[Blocked on .NET 11 GA] Migrate Zstd compression to native System.IO.Compression

[smiggleworth]

⚠️ Not actionable yet

This issue documents a decision and a future migration path — do not schedule or start this work. Half of it is blocked on a .NET runtime release that hasn't shipped. It exists now so the decision and its rationale aren't lost before that release lands. See "Why this is blocked" below for the specific unblock condition.

Context

Midge's SST block compression (src/sst/compression/mod.rs) defines a fixed, on-disk algorithm discriminant, stored as a u8 in the block trailer — this is a wire-format detail Pants must match exactly, not an internal implementation choice:

CompressionAlgo::None = 0
CompressionAlgo::Lz4  = 1   (lz4_flex crate)
CompressionAlgo::Zstd3 = 2  (zstd crate, level 3)
CompressionAlgo::Zstd9 = 3  (zstd crate, level 9+)

Default policy is Adaptive: try LZ4 then Zstd3 then Zstd9 per block (in that order), keep whichever result saves at least min_savings_bytes = 256 bytes and beats a min_ratio = 0.95 compressed/original threshold; blocks under MIN_COMPRESSION_INPUT_BYTES = 256 are stored raw regardless, since framing/codec overhead dominates at that size. Pants already implements this discriminant (MidgeCompressionAlgorithm.cs) and the adaptive selection logic (MidgeSstCodec.CompressBlock/CompressAdaptive), currently via two third-party NuGet packages: K4os.Compression.LZ4 and ZstdSharp.Port (both pure-managed C#, no native binary/P-Invoke — worth noting, since "third-party" here is a vendor-trust question, not a native-deployment-risk one).

The ask: reduce third-party dependency where an out-of-box (BCL/Microsoft-first-party) implementation is available, without breaking the wire-format contract above or the cross-engine SST-readability goal (any conformant Zstd/LZ4 decoder reads any conformant encoder's output — the constraint is bitstream-format compliance, not "same library on both sides").

What changed the calculus

.NET 11 adds native Zstandard support to System.IO.Compression: ZstandardStream, ZstandardEncoder, ZstandardDecoder, ZstandardDictionary, ZstandardCompressionOptions — mirroring the existing BrotliEncoder/Decoder API shape, with quality levels 1–22 and streaming + span-based one-shot operations. It produces standard Zstd-frame bytes, so it's drop-in interoperable with Midge's Rust-side zstd crate output — no shared-spec change required, this is purely an implementation-library swap on the Pants side.

LZ4 has no equivalent. There is no indication of native LZ4 support landing in System.IO.Compression for .NET 10 or .NET 11. Options if/when this is picked up:

1. Keep K4os.Compression.LZ4 (pure-managed, single-purpose, widely used) — the pragmatic default.
2. Hand-write an in-house LZ4 block-format codec. The format is small and well-specified enough to be feasible, but this trades a dependency-count win for taking on a hand-rolled decompressor parsing untrusted-shaped byte streams (length/offset fields, overlapping-copy edge cases) — exactly the class of code where a narrow, widely-used, single-purpose library is usually the lower-risk choice, not the higher one. Do not default to "just write it" without weighing that trade-off explicitly when this is picked up.
3. Drop LZ4 support — not viable; a Midge-written store using LZ4-compressed blocks would become unreadable, breaking the interop goal.

Why this is blocked

• System.IO.Compression's native Zstandard types ship in .NET 11, which is currently Preview 7 (final GA expected November 2026). Pants intentionally targets net10.0 in Directory.Build.props; CI selects the supported 10.0.x SDK line. A prior .NET 11 preview trial was deliberately reverted rather than weakening the analyzer baseline, and this repository intentionally has no global.json.
• There is no supported way to consume the new ZstandardEncoder/Decoder types from a net10.0 target — this is not a "reference a preview package" situation, it's the BCL of a not-yet-released runtime.
• Unblock condition: .NET 11 reaches general availability and Pants makes a deliberate decision to retarget (fully or via multi-targeting) to net11.0. That retarget decision is bigger than this issue and shouldn't be made just to get native Zstd — it should be evaluated on its own merits when the time comes.

Requirements (once unblocked)

• Retarget (or multi-target) Pants to net11.0 as a separate, deliberate decision — this issue does not itself justify that move on its own.
• Replace ZstdSharp.Port usage in MidgeSstCodec with System.IO.Compression.ZstandardEncoder/ZstandardDecoder, preserving the existing Zstd3/Zstd9 discriminant mapping (quality level 3 and level 9, respectively) and the existing adaptive-selection logic (min_savings_bytes, min_ratio, try-order).
• Add a round-trip interop regression test: compress with the new native encoder, decompress with the existing/legacy path (and vice versa if a transition period exists), and confirm byte-for-byte payload equality against a Midge-produced fixture (extend MidgeCompatibilityFixtureTests.cs's pattern).
• Re-evaluate LZ4 (keep K4os.Compression.LZ4 vs. in-house vs. any newer .NET native option that may have shipped by then) as an explicit, separate decision at that time — don't let it ride along silently on the Zstd migration.
• Update docs/compatibility/ to record the library swap and confirm it required no changes to the persisted CompressionAlgo discriminant values or block trailer format.

Acceptance Criteria (once unblocked)

• Pants targets net11.0 (or multi-targets net10.0/net11.0) as a tracked, separate decision referenced from this issue, not decided implicitly here.
• Zstd3/Zstd9 blocks are produced and consumed via System.IO.Compression's native types; ZstdSharp.Port package reference is removed.
• A Midge-produced Zstd-compressed SST fixture round-trips correctly through the new implementation.
• The LZ4 dependency decision is explicitly re-confirmed (not silently carried over) and documented.
• dotnet test --configuration Release passes on the newly targeted SDK.
• No change to the persisted CompressionAlgo discriminant values (None=0, Lz4=1, Zstd3=2, Zstd9=3) or block trailer format.

Evidence / References

• Midge src/sst/compression/mod.rs (algorithm discriminants, CompressionPolicy::Adaptive defaults, MIN_COMPRESSION_INPUT_BYTES, MAX_BLOCK_SIZE)
• Midge Cargo.toml (lz4_flex = "0.14", zstd = "0.13")
• src/Pants/Internal/Storage/Sst/MidgeCompressionAlgorithm.cs, MidgeSstCodec.cs (current adaptive compression implementation)
• Directory.Packages.props (K4os.Compression.LZ4, ZstdSharp.Port)
• Directory.Build.props (net10.0 target) and .github/workflows/*.yml (10.0.x SDK selection); no global.json by design
• .NET 11 native Zstandard: .NET 11 Preview 7 announcement, System.IO.Compression move
• test/Pants.Tests/MidgeCompatibilityFixtureTests.cs (existing pattern for cross-engine fixture round-trip tests)

[smiggleworth]

Revalidated on 2026-08-23: .NET 11 Preview 7 is still a preview, so this issue remains intentionally blocked. Pants remains on net10.0 through Directory.Build.props, CI selects 10.0.x, and there is no global.json by design. The issue body now reflects that repository decision and uses official .NET sources.