Data-derived order-0 literal costs for the cheap-optimal cost model
Small ratio lever for the L9-fast parser (#2638) and any cheap-L8 optimal (see the sibling optimal-floor issue). Today computeChoicesFast prices literals from pure static tables; the exact DP does a full 2-round refit (fit a Huffman histogram from round-1 tokens, re-price). There is a cheap middle: one quick order-0 byte histogram of the region → literal costs from that entropy estimate, keeping the static length/distance costs.
Better cost model than static, far cheaper than a token-collection + Huffman refit round. Could improve L9-fast ratio at ~no speed cost, and makes a cheap-L8 optimal more competitive.
Plan
- Add an order-0 literal-cost seeding pass to the single-round fast DP (one linear histogram +
costOfLen-style conversion).
- Measure ratio + speed vs pure-static on both corpora.
- Adopt if it is a free-or-cheap ratio gain.
Heuristic — consumed by the re-verifying emitter, so proof-invisible.
🤖 Prepared with Claude Code
Data-derived order-0 literal costs for the cheap-optimal cost model
Small ratio lever for the L9-fast parser (#2638) and any cheap-L8 optimal (see the sibling optimal-floor issue). Today
computeChoicesFastprices literals from pure static tables; the exact DP does a full 2-round refit (fit a Huffman histogram from round-1 tokens, re-price). There is a cheap middle: one quick order-0 byte histogram of the region → literal costs from that entropy estimate, keeping the static length/distance costs.Better cost model than static, far cheaper than a token-collection + Huffman refit round. Could improve L9-fast ratio at ~no speed cost, and makes a cheap-L8 optimal more competitive.
Plan
costOfLen-style conversion).Heuristic — consumed by the re-verifying emitter, so proof-invisible.
🤖 Prepared with Claude Code