Add a large pure-text corpus (enwik8, the first 100 MB of English Wikipedia, from the Large Text Compression Benchmark) so we can evaluate L9 against the zopfli ceiling on zopfli's home turf.
Why
Our L9 now sits right at zopfli's operating point (slightly faster, slightly worse ratio). Zopfli ships no corpus of its own; its published evaluation used standard corpora including enwik8, and its design target is large text / web content. Big pure-text is exactly the regime where near-optimal parsing pays off most, and where our current ratio gap to zopfli is widest. Canterbury is tiny and Silesia, while large and modern, is heterogeneous; a single large prose file is the missing data point for judging L9-vs-zopfli.
What to do
- Add
bench/corpora/enwik8/ wired through bench/fetch_corpora.sh (SHA-256-pinned, gitignored cache, fetched on demand like Silesia). The harness discovers corpora by directory, so it slots into the per-level Pareto/summary/heatmap charts with no plot.py change.
- Generate the zopfli ceiling for it (
bench-report --zopfli-ceiling) so the L9-vs-zopfli overlay appears on the enwik8 ratio graph. Note: zopfli on 100 MB at default iterations is very slow (~100x zlib); if the one-time run is impractical, use a fixed slice (e.g. enwik8 first 10-20 MB, recorded in the corpus name) — deterministic and SHA-pinned either way.
- Run the reduced (single-pass timing) matrix like Silesia, since it is large.
Acceptance
enwik8 (or the chosen slice) appears in bench/graphs/enwik8_* with the zopfli ceiling overlaid on the ratio graph, and bench/results/latest.json carries its rows. This becomes the reference for the "is L9 strictly better ratio than libdeflate, and how close to zopfli" question (ties to #2638's ratio floor).
Context: requested to evaluate L9 against zopfli on a large-text corpus; sibling of the bench-regeneration chore #2699.
Add a large pure-text corpus (enwik8, the first 100 MB of English Wikipedia, from the Large Text Compression Benchmark) so we can evaluate L9 against the zopfli ceiling on zopfli's home turf.
Why
Our L9 now sits right at zopfli's operating point (slightly faster, slightly worse ratio). Zopfli ships no corpus of its own; its published evaluation used standard corpora including enwik8, and its design target is large text / web content. Big pure-text is exactly the regime where near-optimal parsing pays off most, and where our current ratio gap to zopfli is widest. Canterbury is tiny and Silesia, while large and modern, is heterogeneous; a single large prose file is the missing data point for judging L9-vs-zopfli.
What to do
bench/corpora/enwik8/wired throughbench/fetch_corpora.sh(SHA-256-pinned, gitignored cache, fetched on demand like Silesia). The harness discovers corpora by directory, so it slots into the per-level Pareto/summary/heatmap charts with noplot.pychange.bench-report --zopfli-ceiling) so the L9-vs-zopfli overlay appears on the enwik8 ratio graph. Note: zopfli on 100 MB at default iterations is very slow (~100x zlib); if the one-time run is impractical, use a fixed slice (e.g. enwik8 first 10-20 MB, recorded in the corpus name) — deterministic and SHA-pinned either way.Acceptance
enwik8 (or the chosen slice) appears in
bench/graphs/enwik8_*with the zopfli ceiling overlaid on the ratio graph, andbench/results/latest.jsoncarries its rows. This becomes the reference for the "is L9 strictly better ratio than libdeflate, and how close to zopfli" question (ties to #2638's ratio floor).Context: requested to evaluate L9 against zopfli on a large-text corpus; sibling of the bench-regeneration chore #2699.