Skip to content

Bind each snapshot's profiles page to its own freeze - #154

Merged
DavidBakerEffendi merged 1 commit into
mainfrom
dave/nifty-joliot-a6fabe
Sep 4, 2026
Merged

DavidBakerEffendi merged 1 commit into
mainfrom
dave/nifty-joliot-a6fabe

Conversation

@DavidBakerEffendi

Copy link
Copy Markdown
Collaborator

analyzerProfiles() read currentSnapshot from module scope and AnalyzerProfiles.astro took no snapshot prop, so every snapshots/<slug>/profiles/ page rendered the current freeze's figures rather than its own.

After #152 this became visible in the built site: dist/snapshots/v0-6-0/profiles/ and dist/snapshots/v0-6-1/profiles/ carried v0.7.0's numbers — Bifrost 225 of 369 positive assertions on all three pages, where v0.6.1 published 223 and v0.6.0 published 219. That contradicts the promise snapshots/index.mdx makes, that "no page below has been rewritten since its freeze".

The fix

  • docs/src/data/profiles.tsanalyzerProfiles(snapshot: Snapshot = currentSnapshot) takes the snapshot as a parameter and derives both coreKernelPopulations(snapshot.results) and kernelCorpusDistributions(snapshot) from it. The default keeps the unversioned call site working.
  • docs/src/components/AnalyzerProfiles.astro — accepts a snapshotVersion prop resolved through snapshotByVersion, in the shape LatencyTier.astro already carries.
  • The three profiles.mdx pages name their own version.
  • The component's "every latency caveat on the latency page" link now resolves to the page's own freeze rather than the newest one — the same leak in link form, which sent a reader of the archived v0.6.0 page to v0.7.0's caveats.

The latency axis is deliberately untouched

profiles.ts reaches latency through kernelCorpusDistributions, which selects its corpus by snapshot.latencyEvidenceRelease. v0.6.0, v0.6.1 and v0.7.0 all bind the v0.6.0 corpus, so passing each page its own snapshot leaves all eight speed marks identical across the three — which is exactly the mixed-run caveat v0.6.1's and v0.7.0's pages already carry, now derived rather than coincidental.

Verification

npm test 7 pass · astro check 0 errors, 0 warnings, 0 hints · 52 pages built. Read back out of the built HTML, the three pages now differ exactly where they should and nowhere else:

v0.6.0 v0.6.1 v0.7.0
Bifrost recall 59.3% (219/369) 60.4% (223/369) 61.0% (225/369)
Bifrost precision 98.2% 100.0% 100.0%
other seven analyzers' axes identical identical identical
eight speed medians identical identical identical

Bifrost alone moving is what both caution blocks assert — v0.6.1's "the other seven analyzers did not move at all between these freezes", and v0.7.0's "three of the four re-pins moved nothing at all on the correctness axes".

Not an amendment, on my reading — flagging for the maintainer's call

No frozen evidence, results model, manifest, or scored cell changes here. The correct figures were already in each snapshot's own results model and the site was reading the wrong one, so this is a rendering defect rather than an evidentiary correction under the docs/new-analyzer.md taxonomy — no A<n> number taken. It did, however, put wrong figures on two published pages for the life of #152, so if that reads as warranting a dated amendment, say so and I will add one.

No regression test

docs/src/data/*.test.ts runs under bare node --test; profiles.tssnapshots.ts pulls in extensionless specifiers and attribute-less JSON imports that only Vite resolves, so the module cannot be loaded there without new machinery. The built-HTML comparison above is the check.

🤖 Generated with Claude Code

`analyzerProfiles()` read `currentSnapshot` from module scope and
`AnalyzerProfiles.astro` took no snapshot prop, so every
`snapshots/<slug>/profiles/` page rendered the *current* freeze's
figures. After #152 this became visible: `v0-6-0/profiles/` and
`v0-6-1/profiles/` published v0.7.0's numbers — Bifrost 225 of 369
positive assertions on all three pages, where v0.6.1 published 223 and
v0.6.0 219. That contradicts the promise on `/snapshots/`, that "no page
below has been rewritten since its freeze".

- `analyzerProfiles(snapshot = currentSnapshot)` takes the snapshot as a
parameter and derives both `coreKernelPopulations` and
`kernelCorpusDistributions` from it.
- `AnalyzerProfiles.astro` accepts `snapshotVersion`, defaulting to the
current freeze, in the shape `LatencyTier.astro` already carries.
- The three `profiles.mdx` pages name their own version.
- The component's "every latency caveat on the latency page" link now
resolves to the page's own freeze rather than the newest one — the same
leak in link form, which sent a reader of the archived v0.6.0 page to
v0.7.0's caveats.

No frozen evidence, results model, or scored cell changes: the correct
figures were already in each snapshot's own results model, and the site
was reading the wrong one.

## Verification

`npm test` 7 pass · `astro check` 0 errors · 52 pages built. In the
built HTML the three pages now differ exactly where they should:

    Bifrost recall     v0.6.0 59.3% (219/369)
                       v0.6.1 60.4% (223/369)
                       v0.7.0 61.0% (225/369)
    Bifrost precision  98.2% · 100.0% · 100.0%

The other seven analyzers' correctness axes are identical across the
three, and the eight speed medians are byte-identical on all three
pages — which is what v0.6.1's and v0.7.0's own caution blocks assert,
and what the shared v0.6.0 latency corpus requires.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@DavidBakerEffendi
DavidBakerEffendi merged commit a95ffc8 into main Sep 4, 2026
3 checks passed
@DavidBakerEffendi
DavidBakerEffendi deleted the dave/nifty-joliot-a6fabe branch September 4, 2026 10:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant