bun tests/golden.ts fails all 54 goldens on a clean checkout of main (272b1a9) while the
rendered pixels are byte-identical to the committed goldens. The assertion is taken on the
encoded PNG, so it also depends on the deflate implementation, which is not part of what the
test means to pin.
release.yml gates a release on this step, so the failure mode is a red release gate with no
visual change.
What I measured
Environment: macOS 26.5.1 arm64, Bun 1.3.11, rustc 1.97.1 (rustup stable), main at 272b1a9,
bun tools/wasm.ts built fresh.
bun tests/golden.ts → 0 passed, 54 failed
Decoding each <label>.png / <label>.actual.png pair and comparing pixels:
54 pairs
by PNG bytes 0 pass / 54 fail
by pixels 54 pass / 0 fail
For cards-main.12 specifically:
480x272 differing pixels 0 / 130560 max channel delta 0
PNG size golden 14,452 B actual 13,080 B
An amplified difference image (per-pixel max|Δchannel| scaled 50x) is uniformly black. As a
control, injecting a single 5/255 difference at one pixel makes that pixel clearly visible under
the same amplification, so the comparison is not simply insensitive.
Where it comes from
tests/golden.ts:182 reports a mismatch when the encoded PNG buffers differ:
console.log("FAIL ", label, "- PNG bytes differ (see " + label + ".actual.png)");
tests/png.ts builds the IDAT with Bun.deflateSync, noted there as deterministic. That holds
within a Bun version — I confirmed encoding the same raw buffer twice under 1.3.11 is
byte-identical — but the compressed output is not a stable contract across Bun versions, and the
assertion inherits that dependency.
To be explicit about what is measured versus inferred: the identical pixels and the differing
PNG bytes are measured. Since the raw input to deflateSync is provably identical and encoding
is stable within a version, the remaining variable is the deflate implementation itself; I have
not bisected which Bun version changed it.
The all-54 failure pattern is itself a hint — a genuine raster regression would usually move a
subset of frames.
Suggested change
Assert on pixels and keep PNG as the human-readable artifact:
- compare the rasterizer's RGBA output directly (or a hash of it) against a stored raw/hashed
golden — golden.ts already holds that buffer, so no decoder is needed
- keep writing
<label>.actual.png on failure for inspection
A side benefit: the failure line can then report how many pixels changed instead of only that
the bytes differ, which is the more useful diagnostic.
I verified this direction on the 54 existing pairs — comparing pixels passes all of them, and a
one-bit change in a single channel of a single pixel still fails, so the check does not get
weaker.
Happy to send a PR if you want it done this way; equally happy to leave the approach to you,
since it touches the golden format and would mean regenerating the stored artifacts.
bun tests/golden.tsfails all 54 goldens on a clean checkout ofmain(272b1a9) while therendered pixels are byte-identical to the committed goldens. The assertion is taken on the
encoded PNG, so it also depends on the deflate implementation, which is not part of what the
test means to pin.
release.ymlgates a release on this step, so the failure mode is a red release gate with novisual change.
What I measured
Environment: macOS 26.5.1 arm64, Bun 1.3.11,
rustc1.97.1 (rustup stable),mainat 272b1a9,bun tools/wasm.tsbuilt fresh.Decoding each
<label>.png/<label>.actual.pngpair and comparing pixels:For
cards-main.12specifically:An amplified difference image (per-pixel
max|Δchannel|scaled 50x) is uniformly black. As acontrol, injecting a single 5/255 difference at one pixel makes that pixel clearly visible under
the same amplification, so the comparison is not simply insensitive.
Where it comes from
tests/golden.ts:182reports a mismatch when the encoded PNG buffers differ:tests/png.tsbuilds the IDAT withBun.deflateSync, noted there as deterministic. That holdswithin a Bun version — I confirmed encoding the same raw buffer twice under 1.3.11 is
byte-identical — but the compressed output is not a stable contract across Bun versions, and the
assertion inherits that dependency.
To be explicit about what is measured versus inferred: the identical pixels and the differing
PNG bytes are measured. Since the raw input to
deflateSyncis provably identical and encodingis stable within a version, the remaining variable is the deflate implementation itself; I have
not bisected which Bun version changed it.
The all-54 failure pattern is itself a hint — a genuine raster regression would usually move a
subset of frames.
Suggested change
Assert on pixels and keep PNG as the human-readable artifact:
golden —
golden.tsalready holds that buffer, so no decoder is needed<label>.actual.pngon failure for inspectionA side benefit: the failure line can then report how many pixels changed instead of only that
the bytes differ, which is the more useful diagnostic.
I verified this direction on the 54 existing pairs — comparing pixels passes all of them, and a
one-bit change in a single channel of a single pixel still fails, so the check does not get
weaker.
Happy to send a PR if you want it done this way; equally happy to leave the approach to you,
since it touches the golden format and would mean regenerating the stored artifacts.