ci(deck-fit): measure every deck slide on every presentation change (closes #2425) - #2478
Merged
Conversation
scripts/deck-fit.mjs is the only thing that can answer "does this deck still fit?", and it ran nowhere -- so a slide could start clipping again with nothing to notice, which is the same shape of bug the script was written for (#2373). A measurement nobody takes is the same as no measurement. It gets its own workflow rather than a job in docs-guards.yml, which the issue suggested: that workflow triggers on `**/*.rs`, so a job there would launch a browser on nearly every pull request in the repository. wire-schema.yml already exists for the same disjoint-paths reason in the other direction (#1439). The job loops over every deck in the directory rather than naming investor-deck.html, so a second deck is covered the day it lands. Verified both directions locally, against Chrome 151.0.7922.77, by running the workflow's own step script verbatim: - the deck passes 21/21 slides on all five passes, exit 0; - twelve filler sentences appended to slide 06's footnote fail it on all five passes, exit 1, naming "slide 06 +25px tall". Worth recording because the issue's suggested repro does not work: three sentences on slide 06 change the measurement by nothing at all. The frame is a flex column whose .body child is `flex: 1 1 auto`, so it shrinks by exactly what the footnote grows and the frame's height never moves. Slide 06 has about nine filler sentences of slack; the tenth is the first that overflows. Closes #2425
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Sorry @macanderson, you have reached your weekly rate limit of 500000 diff characters.
Please try again later or upgrade to continue using Sourcery
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Contributor
Reviewer's GuideAdds a dedicated GitHub Actions workflow that runs the deck-fit measurement script on every change to presentation decks (and related assets), and updates documentation and the script header to describe this CI wiring and behavior. File-Level Changes
Assessment against linked issues
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
4 tasks
macanderson
added a commit
that referenced
this pull request
Aug 9, 2026
…ow (closes #2426) (#2490) ## What & why tbench.ai rank 17 lists **Claude Code on GLM-5.1 at 58.7% ± 1.2%** — nine points above the **44/89 = 49.44%** our preregistered paired run `tb21-hh10-20260731` measured for the same comparator, and the deck offered nothing to reconcile it. An investor or a diligence reviewer who opens the leaderboard sees our comparator scoring nine points higher in public than we measured it. That is the class of gap #2370 was opened to close. This takes the issue's **option 1** (reconcile and say so), because the reconciliation turns out to be decisive rather than a list of excuses. Closes #2426 ### The reconciliation It was already latent in the committed artifacts: - **44/89 carries a 95% exact-binomial interval of [38.67%, 60.25%] — which contains 58.7%.** The two numbers are not in conflict. - **±1.2 is unattainable from one 89-task pass@1 run.** Ours is roughly ±11 points, and *every* row on that board reports ±1.2 to ±1.6, so those figures aggregate more trials than we ran. The page publishes neither its harness configuration nor the metric behind the percentage. - On top of that, the ordinary difference: the row is **GLM-5.1**, both our arms ran **`glm-5.2`**, because the design's binding rule was that the model is held constant across arms and the endpoint pair that makes a paired run possible served `glm-5.2`. So they are different estimators at different precisions. A single paired run cannot resolve a nine-point difference in either direction, and this one does not claim to. ### Ruling out the third outcome rather than assuming it The issue's outcome 3 — "if the comparator arm was genuinely under-configured, that invalidates the headline" — is the one that had to be checked, not asserted. Evidence, all committed: - Every axis but the endpoint was matched, and the endpoint difference is forced (OpenRouter serves no `/v1/messages`), not chosen. - Neither arm carried a per-trial budget or token cap (`budget_usd_per_trial: "unbounded"`); Claude Code ran `reasoning_effort: max` with thinking enabled, the same posture as stella's worker. - **Operational exceptions do not single out the comparator: 45 of 89 Arm A trials and 48 of 89 Arm B trials recorded one** — a near-identical rate. This was the number most likely to hide a handicap, so it is stated explicitly, and the section says an exception does not imply a failed trial. The doc invites the disconfirming read: if someone reads `comparator/trials.jsonl` and finds an asymmetry we missed, the headline is wrong and we want to know. ### What the deck now says it is *not* Added plainly, because it is the honest limit of the design: our comparator figure is the arm that ran beside stella's, on the same box, on the same day — **not an estimate of Claude Code's ceiling**, and no claim in the deck should be read as one. The headline quantity is the paired within-run difference of +15.73 points, which is the only thing a matched design licenses. ### Why not option 2 (re-measure) Re-running the comparator alone on GLM-5.1 produces a **third, unpaired** number with no contemporaneous stella arm beside it — the same unpaired comparison this section exists to refuse. A GLM-5.1 result worth reporting is a new paired run of *both* arms: a measurement to schedule, not a footnote to write. Said so in the doc rather than left implicit. ## The witness - [x] No witness needed (pure refactor / docs / CI) — because: it is prose and one deck table row; no behavior changes. How it was verified instead: - **The leaderboard row was re-checked against the primary source on 2026-08-09** rather than taken from the issue: tbench.ai Terminal-Bench 2.1 confirms rank 17, Claude Code, GLM-5.1, 58.7% ± 1.2%. - Every figure quoted traces to a committed artifact — `score.json` (44 passes, both CIs, `trials_with_an_exception`), `comparator/score.json`, `run-manifest.json` (`budget_usd_per_trial`, posture). Nothing is asserted from the deck's own prose. - **The deck still fits**: `scripts/deck-fit.mjs` passes 21/21 slides on all five passes with the new appendix A1 row (Chrome 151.0.7922.77). Appendix A1 sits at its frame edge, so this was a real risk, not a formality — which is also why the full reconciliation went into `BENCHMARK_METHODOLOGY.md` and only one row into the slide. ## The gate - [x] No Rust changed; `cargo fmt/clippy/test` unaffected by this diff - [x] `BENCHMARK_METHODOLOGY.md` is the authoritative doc for every benchmark figure in the deck, and carries the full reconciliation; the deck's A1 table carries one sourced sentence pointing at it ## Nothing left behind - [x] There is nothing new: everything I noticed in this issue is fixed here. Related work from the same session, filed separately: #2475 (deck-fit reports one overflow as three different numbers), #2486 (`verify_done`'s cost is the shadow worktree lifecycle). Neither is in scope here. ## Anything reviewers should know? **The strongest sentence in the diff is checkable and I would like it checked**: 58.7% falling inside [38.67%, 60.25%] is what turns this from a list of excuses into a reconciliation. Both bounds are in `comparator/score.json` (`ci95_clopper_pearson`) and recompute from `comparator/trials.jsonl` via `bench/evidence/score_dev_baseline.py`. The one thing I did **not** establish is what the leaderboard's ± actually denotes — the page does not say, so the doc claims only that ±1.2 is unattainable from a single 89-task run and that the figures therefore aggregate more trials than ours. That is deliberately weaker than "they ran it five times", which I cannot source. Depends on nothing; stacks cleanly with #2478 (which adds the CI job that would have caught an A1 overflow here). ## Summary by Sourcery Clarify and reconcile the discrepancy between the public tbench.ai GLM-5.1 leaderboard row for Claude Code and the preregistered comparator measurement, and surface that reconciliation in both the benchmark methodology doc and the investor deck. Enhancements: - Extend the investor deck appendix table with a succinct row that summarizes the reconciliation between the public GLM-5.1 leaderboard result and our comparator’s confidence interval, and points to the benchmark methodology for full details. Documentation: - Add a detailed methodology section explaining why the tbench.ai Claude Code GLM-5.1 leaderboard result and our 49.4% comparator measurement are different estimators that nonetheless agree within statistical confidence, including model, protocol, and precision differences. - Document that the comparator arm was configured to match stella across all axes except the forced endpoint, and invite readers to inspect the committed trials for any hidden asymmetry. - Explain why re-running the comparator alone on GLM-5.1 would produce an unacceptable unpaired measurement and that any new GLM-5.1 figure must come from a fresh paired run of both arms.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
scripts/deck-fit.mjsmeasures every slide of a deck against the fixed 1600x900canvas it is authored in, at four viewports plus a webfont-blocked pass. It is
the only thing that can answer "does this deck still fit?" — and it ran nowhere.
Anyone editing a slide's copy could push a clipping deck with both the gate and
CI green, which is the same shape of bug the script was written for (#2373): a
measurement nobody takes is the same as no measurement.
This adds
.github/workflows/deck-fit.yml, triggered on changes underwebsite/public/presentations/**, the bundled fonts, the script, and itself.Closes #2425
Why its own workflow, not a job in
docs-guards.ymlThe issue suggested
docs-guards.yml. That workflow triggers on**/*.rs, so ajob there would launch a browser on nearly every pull request in the repository.
wire-schema.ymlalready exists for the same reason in the other direction — acheck whose trigger paths are disjoint from an existing workflow's gets its own
file rather than widening someone else's path set (#1439). Exemplar named
because it is a structural choice, not taste.
Two smaller calls, both in the workflow's header comment:
and no browser build to bump by hand. The resolved binary's
--versionislogged, because a layout result that changes with no deck change is a browser
bump and the log is where that gets diagnosed instead of guessed at. If a
runner image ever drops Chrome the job fails with the candidate list and the
exact
npx playwright-core@1.62.1 install chromiumfallback.investor-deck.htmlbyname, so a second deck is covered the day it lands rather than the day someone
remembers this file exists.
playwright-coreis pinned to1.62.1(--no-save, so it enters no manifest —it is a measurement dependency, not something the repository ships). There is a
separate step asserting the import resolves, because
deck-fit.mjsexits 2 withthe word "skipping" when it does not; that is a red job either way, but the word
invites a reader to shrug at it.
The witness
there is no Rust behavior to witness.
Verified both directions locally against Chrome 151.0.7922.77 by extracting the
workflow's own
run:block and executing it verbatim (not a paraphrase of it):and with twelve filler sentences appended to slide 06's footnote — exit 1, all
five passes red, slide named:
Every
run:block was also syntax-checked withbash -n, and the workflowparses as YAML with all
uses:SHA-pinned (check-action-pins: 66/66).The issue's suggested repro does not work, and that is worth knowing. Adding
three sentences to slide 06's footnote changes the measurement by nothing —
byte-identical output on the pristine and edited decks. The frame is a flex
column whose
.bodychild isflex: 1 1 auto, so it shrinks by exactly what thefootnote grows (431 -> 392 layout px) and the frame's height never moves. Slide
06 carries about nine filler sentences of slack; the tenth is the first that
overflows. I nearly concluded the guard was blind on the strength of the issue's
own repro passing.
The gate
make guards-fastgreen (the rung the pre-push hook picks for a push thatreaches no crate), including
check-action-pins,check-gate-parity,check-doc-links,check-brand-case,cargo fmt --checksplit and now name the fourth;
scripts/deck-fit.mjs's header no longersays "run it by hand when you touch a deck"
Closes #2425appears both here and as a commit trailerNo Rust changed, so clippy/test/rustdoc are unaffected by this diff.
Nothing left behind
deck-fitreports one overflow as three different numbers.overHtakesMath.maxof a layout-px reading (scrollHeight - clientHeight) and a visual-px one (thegetBoundingClientRectsweep,scaled by the stage's
--k), so the same 25-layout-px overflow prints as+25/+30/+40across viewports. The verdict is sound — bothreadings are exactly 0 when nothing overflows and the layout reading is
never scaled down, so nothing can round away to green — only the reported
magnitude is unit-inconsistent. Deliberately not fixed here: this PR wires
up a guard, and changing what the guard measures in the same breath would
mean the verification above no longer describes the shipped script.
mainis over its own file-size ceilings on three bench Python files — baseline needs regenerating #2472 (mainover its own file-size ceilings): it is sixfiles now, not three, with current numbers — plus the trap that from a
stale base the same check exits non-zero and blames your branch.
Ground-rule check
stella-core; no new deps without justification belowplaywright-core@1.62.1is installed inside the CI job only,--no-save, andappears in no manifest or lockfile the repository ships.
Anything reviewers should know?
It is deliberately not a required check — the issue asks for it to run green
for a while first, and nothing here touches branch protection.
The one thing I could not verify locally is the runner-side Chrome resolution:
the candidate list is written against GitHub's
ubuntu-latestimage, and if itis wrong the job fails loudly with the list and the fallback command rather than
skipping. That is the failure mode I chose on purpose — for a guard whose entire
history is "it silently did not run", a silent skip is the one outcome worth
engineering against.
Summary by Sourcery
Add a dedicated CI workflow to automatically verify that all presentation deck slides fit within the authored canvas on each relevant change.
CI:
Documentation: