encode/zstd: support compressing with a trained dictionary

[sorena-paydar]

Implements the dictionary option discussed in #6204.

@mholt wrote there:

An option to load a dictionary file should be pretty simple, as mentioned above. I'd welcome a PR to review!

This is that option, scoped to a single dictionary loaded from disk — not the full Compression Dictionary Transport negotiation the issue opens with. See Limitations below for why that part can't be done from an encoder module today.

What it does

Adds a dictionary option to the zstd encoder, taking a path to a dictionary produced by zstd --train:

encode {
	zstd {
		dictionary /etc/caddy/api.dict
	}
}

JSON:

{"encodings": {"zstd": {"dictionary": "/etc/caddy/api.dict"}}, "handler": "encode"}

Why it helps

Small, structurally similar payloads are the case that plain zstd cannot do much with, because frame overhead swamps the content. Measured on a 108-byte JSON body against a dictionary trained on 60 similar responses:

	bytes
uncompressed	108
encode zstd	121 (larger than the input)
encode zstd + dictionary	39

That is the shape of a lot of API traffic behind a reverse proxy.

Limitations, and a question

A dictionary frame is only decodable by a client holding that dictionary. The dictionary is not carried in the frame; a decoder without it fails with unknown dictionary rather than degrading. So this option is for traffic between parties that share a dictionary out of band — service-to-service, internal API clients — and is not safe to turn on for browser traffic.

The web-facing answer to that is Compression Dictionary Transport: the client advertises Available-Dictionary: :<sha-256>: and the server replies with Content-Encoding: dcz only when it holds the matching dictionary. That cannot be implemented from an encoder module as things stand, because the interface is:

type Encoding interface {
	AcceptEncoding() string
	NewEncoder() Encoder
}

NewEncoder() receives no request, so the encoder cannot vary per client or inspect Available-Dictionary without changing a shared interface. That felt like the wrong thing to fold into this PR unprompted.

So my question: is a plain dictionary option the right surface, or would you rather this were gated behind a distinct encoding name so it can never be selected by a client that merely sent Accept-Encoding: zstd? I'm happy to rework it either way, including taking on the request-aware interface change if you want the negotiation properly.

The doc comment on the field states the constraint so it is visible from the JSON docs.

Implementation notes

The dictionary is read and validated in Provision, not per request, for two reasons: NewEncoder() has no way to report an error, and passing raw sample data where a trained dictionary belongs is an easy mistake — zstd.WithEncoderDict rejects it with magic number mismatch, which is not obvious. The error names zstd --train instead:

invalid zstd dictionary /etc/caddy/api.dict (expected the output of `zstd --train`): ...

Testing

New modules/caddyhttp/encode/zstd/zstd_test.go covers:

• round trip through a dictionary-aware decoder, and an assertion that the frame header carries a non-zero DictionaryID
• a decoder without the dictionary failing, which is the property that makes this unsafe for browsers
• the no-dictionary path still producing plain, universally decodable zstd
• Provision rejecting a missing file and a file that is not a trained dictionary, with the error naming zstd --train
• Caddyfile parsing: valid, missing argument, repeated
• the dictionary actually compressing better than plain zstd on a small payload

Plus a Caddyfile adapter test at caddytest/integration/caddyfile_adapt/encode_zstd_dictionary.caddyfiletest.

I checked the tests fail without the change, which caught a weak one: the round-trip test originally passed even with the dictionary wiring disabled, because a decoder holding dictionaries also reads plain frames. The DictionaryID assertion is what fixed it. With the wiring disabled three tests now fail; with it, all pass.

go test ./modules/caddyhttp/encode/... -count=1     ok
go test ./caddytest/integration/ -run TestCaddyfileAdaptToJSON  ok
go vet ./modules/caddyhttp/encode/zstd/             ok

testdata/sample.dict is a 2.8 KB dictionary trained on generated JSON samples; testdata/invalid.dict is plain text, for the rejection test.

Assistance Disclosure

AI-assisted. I directed the design and the scoping decisions, and verified the behaviour rather than taking it on trust: the compression figures above are measured, the round-trip and frame-header assertions are in the test file, and I confirmed the tests fail when the implementation is disabled. The Available-Dictionary limitation above came from reading the Encoding interface and finding it takes no request.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[steadytao]

You (your AI) do not understand compression and dictionaries enough to make this change.

[steadytao]

Fails to comply with RFC 8878 and 9842

[sorena-paydar]

Very helpful comment, Thanks a lot !!

[steadytao]

As stated. RFC 8878 and 9842.

[steadytao]

I am more than happy to review a change but do not submit AI-written code that you do not understand the implications of to me and then expect me to explain to you why it is wrong. Be respectful in the future. For such a large change, one would generally wish to discuss further in the issue rather then submit a PR that does not work.