v3

[Tabaie]

Introduced version 3, with:

• Little-Endian encoding: reading bits is more natural for the decompressor on ARM, RISC-V and AMD64 machines. This improves performance significantly when run as a guest program in a zkVM.
• Huffman coding of back-reference lengths, leading about a 6% improvement in compression of blob data, nearing 94% of zstd's performance on the same.

---

Note

High Risk
Breaking wire format and public API; any consumer of v1 output or old NewCompressor(dict) signatures must migrate and ship matching Huffman tables.

Overview
Breaking format and API change to compressor version 3 (0x0003): the payload is no longer byte-oriented 0xFE/0xFD phrases but an LSB-first bit stream coded with a shared 512-symbol canonical Huffman table (literals plus back-reference lengths). Compressors and decompressors must use the same external table; NewCompressor and Decompress now require a *HuffmanTable, and the CLI adds a required -table flag.

Adds in-repo bit I/O (bitWriter/bitReader), Huffman table build/load (NewHuffmanTable, NewHuffmanTableFromFrequencies), and cmd/hufftable to derive a table from a file corpus. Dictionary special-symbol augmentation and reserved-byte handling are dropped because every byte is a literal symbol. Go 1.26, golangci-lint v2, and README/spec updates document the new on-wire layout and table workflow.

^{Reviewed by Cursor Bugbot for commit b96a53e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	golang/​github.com/​stretchr/​testify@​v1.8.4 ⏵ v1.11.1	+3
	github/​golangci/​golangci-lint-action@​3a919529898de77ec3da873e3063ca4b10e7f5cc ⏵ ba0d7d2ec06a0ea1cb5fa41b2e4a3ab91d21278a	+1

View full report

[Copilot]

Pull request overview

This PR introduces LZSS format version 3, switching the payload to an LSB-first canonical Huffman bitstream and requiring a shared 512-byte external Huffman table for both compression and decompression, along with a new utility to generate such tables from corpus statistics.

Changes:

• Replace the byte-oriented phrase stream with a canonical Huffman-coded bitstream (literals + backref lengths), packed LSB-first.
• Update public APIs and CLI to require a Huffman table (NewHuffmanTable, NewCompressor(dict, table), Decompress(data, dict, table); linzip -table).
• Add a cmd/hufftable generator and update tests/regression baselines for the new format.

Reviewed changes

Copilot reviewed 16 out of 17 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
README.md	Updates usage docs and format specification for v3 + external Huffman table workflow.
main.go	Adds required -table flag and wires table loading into compress/decompress paths.
lzss/testdata/huffman_table	Adds a fixed 512-byte canonical code-length table used by tests.
lzss/regress_test.go	Updates regression ratio baselines and adapts tests to pass the Huffman table.
lzss/huffman.go	Implements canonical Huffman table parsing/generation + symbol encode/decode helpers.
lzss/huffman_test.go	Adds unit tests for table validity, round-trips, and edge cases.
lzss/header.go	Bumps on-disk format version to 3 (header 0x0003).
lzss/decompress.go	Reworks decompression + stream introspection to use Huffman symbols and LSB bit reader.
lzss/compress.go	Reworks compressor to emit Huffman symbols + new bit writer; updates backref heuristics.
lzss/bitstream.go	Adds in-package LSB-first bitWriter/bitReader implementation.
lzss/bitstream_test.go	Tests little-endian bit packing/unpacking correctness.
lzss/backref.go	Adjusts backref encoding to new length-symbol + type-bit + offset scheme.
go.sum	Removes icza/bitio and updates testify dependency checksums.
go.mod	Removes icza/bitio, updates testify, and bumps the Go version directive.
cmd/hufftable/main.go	Adds tool to build deterministic Huffman tables from LZ parse statistics over a corpus.

Suppressed comments (1)

lzss/compress.go:54

• AugmentDict's comment refers to "Version 2", but the current format is Version 3 (and this helper is retained for v1 compatibility). This is a small but concrete documentation mismatch.

// AugmentDict is retained for callers working with version-1 dictionaries.
// Version 2 can encode every byte as a literal and does not call this helper.
func AugmentDict(dict []byte) []byte {

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 6346df6. Configure here.}