Skip to content

Renderer goldens compare compressed PNG bytes rather than pixels #263

Description

@qianiaoo

bun tests/golden.ts fails all 54 goldens on a clean checkout of main (272b1a9) while the
rendered pixels are byte-identical to the committed goldens. The assertion is taken on the
encoded PNG, so it also depends on the deflate implementation, which is not part of what the
test means to pin.

release.yml gates a release on this step, so the failure mode is a red release gate with no
visual change.

What I measured

Environment: macOS 26.5.1 arm64, Bun 1.3.11, rustc 1.97.1 (rustup stable), main at 272b1a9,
bun tools/wasm.ts built fresh.

bun tests/golden.ts        →  0 passed, 54 failed

Decoding each <label>.png / <label>.actual.png pair and comparing pixels:

54 pairs
  by PNG bytes   0 pass / 54 fail
  by pixels     54 pass /  0 fail

For cards-main.12 specifically:

480x272   differing pixels 0 / 130560   max channel delta 0
PNG size  golden 14,452 B   actual 13,080 B

An amplified difference image (per-pixel max|Δchannel| scaled 50x) is uniformly black. As a
control, injecting a single 5/255 difference at one pixel makes that pixel clearly visible under
the same amplification, so the comparison is not simply insensitive.

Where it comes from

tests/golden.ts:182 reports a mismatch when the encoded PNG buffers differ:

console.log("FAIL ", label, "- PNG bytes differ (see " + label + ".actual.png)");

tests/png.ts builds the IDAT with Bun.deflateSync, noted there as deterministic. That holds
within a Bun version — I confirmed encoding the same raw buffer twice under 1.3.11 is
byte-identical — but the compressed output is not a stable contract across Bun versions, and the
assertion inherits that dependency.

To be explicit about what is measured versus inferred: the identical pixels and the differing
PNG bytes are measured. Since the raw input to deflateSync is provably identical and encoding
is stable within a version, the remaining variable is the deflate implementation itself; I have
not bisected which Bun version changed it.

The all-54 failure pattern is itself a hint — a genuine raster regression would usually move a
subset of frames.

Suggested change

Assert on pixels and keep PNG as the human-readable artifact:

  • compare the rasterizer's RGBA output directly (or a hash of it) against a stored raw/hashed
    golden — golden.ts already holds that buffer, so no decoder is needed
  • keep writing <label>.actual.png on failure for inspection

A side benefit: the failure line can then report how many pixels changed instead of only that
the bytes differ, which is the more useful diagnostic.

I verified this direction on the 54 existing pairs — comparing pixels passes all of them, and a
one-bit change in a single channel of a single pixel still fails, so the check does not get
weaker.

Happy to send a PR if you want it done this way; equally happy to leave the approach to you,
since it touches the golden format and would mean regenerating the stored artifacts.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions