From 6f4a074bab39dc640eea3c2920767a400ca413fb Mon Sep 17 00:00:00 2001 From: Anionix Date: Tue, 14 Jul 2026 11:58:41 +0900 Subject: [PATCH] docs: summarize v0.1 benchmark evidence --- README.ja.md | 2 + README.md | 2 + reports/v0.1.0/README.md | 106 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 110 insertions(+) create mode 100644 reports/v0.1.0/README.md diff --git a/README.ja.md b/README.ja.md index 6aec40f..c49884f 100644 --- a/README.ja.md +++ b/README.ja.md @@ -51,6 +51,8 @@ format-bench report --run-dir runs/fair-local 測定値そのものはCIの合否に使いません。公開測定はmacOS ARMとLinux x86_64を別runにし、入力hash、commit、flake lock、依存版、seed、writer設定、失敗理由とともに保存します。 +結果はまず[`v0.1.0`実測総括](reports/v0.1.0/README.md)を参照し、詳細値は機種別レポートとRelease assetで確認してください。 + データはApple系リポジトリに偏り、分類は正解ラベルではありません。過去snapshotの生API応答と分類生成コードが残っていない制約もData Cardに明記しています。 ## 参加方法 diff --git a/README.md b/README.md index ade1723..706b850 100644 --- a/README.md +++ b/README.md @@ -63,6 +63,8 @@ The default fair protocol uses 10 fresh processes per format and operation. Each Release assets are the distribution boundary for the frozen dataset, raw results, checksums, and binary artifacts. Git contains only schemas, manifests, a small fixture, generated Markdown summaries, and code. +Start with the [`v0.1.0` evidence summary](reports/v0.1.0/README.md), then open the platform report or raw Release asset for the measurement details. + macOS ARM and Linux x86_64 are separate runs. Their values must not be combined into a machine ranking. Direct corpus token counts apply only to text; binary formats report `N/A`. Lance data, index, metadata, logical directory size, and external zstd transport size are recorded separately. The Stars Data Card documents the Apple-heavy source bias, the fact that classifications are not ground truth, and missing historical raw API/classifier provenance. See [`datasets/github-stars-2026-07-03/DATA_CARD.md`](datasets/github-stars-2026-07-03/DATA_CARD.md). diff --git a/reports/v0.1.0/README.md b/reports/v0.1.0/README.md new file mode 100644 index 0000000..dafa2e3 --- /dev/null +++ b/reports/v0.1.0/README.md @@ -0,0 +1,106 @@ +# v0.1.0 Evidence Summary + +This release tests the frozen `github-stars-2026-07-03` dataset: 2,331 rows, +13 typed columns, source SHA-256 +`39cc70109d9dddf947257584e15f2f9a6bc97dcdf0a7bf939c26cccbcda0e22e`, +and canonical hash +`1bf35022cce6d752f7959907b6a60d4024123e045a270f4aa286acbefbe4ca39`. +It does not name a universal winner. + +Raw `manifest.json`, `results.json`, generated reports, and checksums are in the +[`v0.1.0` Release](https://github.com/Anionix/data-format-lab/releases/tag/v0.1.0). +Generated platform reports are under [`macos-arm64`](macos-arm64/) and +[`linux-x86_64`](linux-x86_64/). Never rank values across those platforms. + +## Fair storage + +All ranked formats returned 2,331 rows, 119 AI/LLM rows, 15 repositories above +100,000 stars, one exact match, and the canonical hash. + +### macOS ARM run 1 + +| Rank | Format | Native bytes | Warm read-all p50 ms | +| ---: | --- | ---: | ---: | +| 1 | Parquet zstd-19 | 176,713 | 0.838 | +| 2 | Vortex compact | 183,576 | 0.955 | +| 3 | Parquet default | 200,031 | 0.786 | +| 4 | Vortex default | 286,328 | 0.602 | +| 5 | Lance base | 314,428 | 1.429 | +| 6 | CSV | 658,439 | 1.648 | +| 7 | object JSONL | 1,049,957 | 3.459 | + +### Linux x86_64 run 1 + +| Rank | Format | Native bytes | Warm read-all p50 ms | +| ---: | --- | ---: | ---: | +| 1 | Parquet zstd-19 | 176,713 | 1.165 | +| 2 | Vortex compact | 183,576 | 1.363 | +| 3 | Parquet default | 200,031 | 1.139 | +| 4 | Vortex default | 286,416 | 0.963 | +| 5 | Lance base | 314,300 | 2.072 | +| 6 | CSV | 658,439 | 1.260 | +| 7 | object JSONL | 1,049,957 | 2.996 | + +Native size and read latency answer different questions. zstd-19 produced the +smallest artifact here but had a substantially higher write cost than default +Parquet. Vortex default had the lowest warm read-all p50 in both platform runs. + +## Claim workloads + +These results are comparable only inside each workload and platform. + +| Platform | Claim | Result from run 1 | +| --- | --- | --- | +| macOS ARM | Lance FTS | 944,129 logical bytes; 418,696 index bytes; `agent` p50 0.579 ms | +| macOS ARM | Vortex sorted full projection | Parquet 5.637 ms; Vortex 2.357 ms | +| macOS ARM | Vortex unsorted random 1,000 | Parquet 20.886 ms; Vortex 2.148 ms | +| macOS ARM | TsFile time range | Parquet 2.324 ms; TsFile 2.034 ms; 1,000 rows each | +| Linux x86_64 | Lance FTS | 944,578 logical bytes; 418,696 index bytes; `agent` p50 1.566 ms | +| Linux x86_64 | Vortex sorted full projection | Parquet 14.072 ms; Vortex 4.283 ms | +| Linux x86_64 | Vortex unsorted random 1,000 | Parquet 25.169 ms; Vortex 3.973 ms | +| Linux x86_64 | TsFile time range | Parquet 4.129 ms; TsFile 2.767 ms; 1,000 rows each | + +The TsFile claim used 1,000,000 time-series rows. TsFile was 313,744 bytes and +Parquet was 5,777,023 bytes in every run, but TsFile writes were much slower: +5.75 s versus 0.48 s on macOS run 1 and 10.10 s versus 0.74 s on Linux run 1. + +The Vortex stress artifacts were deterministic. Sorted Parquet/Vortex sizes +were 2,761,465/2,026,488 bytes; unsorted sizes were +24,925,948/6,343,112 bytes. This supports the tested scan claim, not a general +database-performance claim. + +## Prompt tokens + +The taxonomy dictionary is included once in every total. + +| Representation | Total bytes | `o200k_base` | `cl100k_base` | +| --- | ---: | ---: | ---: | +| Compact TSV | 345,115 | 93,377 | 92,756 | +| array JSONL | 390,697 | 105,866 | 103,745 | +| object JSONL | 598,156 | 148,736 | 145,070 | + +Both runs on both platforms produced identical prompt metrics. Binary formats +remain `N/A` for direct corpus token counts. Retrieval outputs are normalized +to the same Compact TSV before token measurement. + +## Reproducibility + +- Both platforms produced identical result counts and normalized hashes for all + 42 fair operations. Non-Lance fair artifact sizes were identical between runs. +- macOS fair warm p50 changed by 1.43% at the median and 5.76% at the maximum. + Linux changed by 25.15% at the median and 64.35% at the maximum, so timings + are observations rather than stable constants. +- Lance base changed from 314,428 to 314,683 bytes on macOS and from 314,300 to + 315,516 bytes on Linux. Indexed logical size also changed, while the FTS index + stayed exactly 418,696 bytes. +- An earlier exploratory run observed a 704-byte Lance delta. Its raw artifact + was not retained, so it is a research-log concern rather than release evidence. +- Vortex stress sizes, TsFile/Parquet time-series sizes and result counts, and + all prompt metrics were identical between run 1 and run 2. + +## Incomplete formats + +FastLanes is `PARTIAL/FAILED`, Nimble is `UNAVAILABLE/UNSUPPORTED`, and AnyBlox +is `PARTIAL/FAILED` at the pinned attempts. Their exact commits, failures, and +retry conditions remain in [`research/formats`](../../research/formats/); they +do not enter rankings.