A corpus of deliberately awkward apps in CI — every blocking defect in the last field export would have been caught by one

[divshekhar]

This is the meta-finding of the 2026-09-08 feedback export, and the reporter who spent four hours driving a real CAD/CFD app named it as the single highest-leverage change we could make:

The single highest-leverage change would be a corpus of deliberately awkward real apps in CI — file upload, >20s endpoint, WebGL canvas, 401-refresh-retry, noisy localStorage, unnamed icon buttons. Every blocking defect I hit would have been caught by one of those before release, and the pattern of failures suggests the current test surface is simpler than the apps you are aiming at.

Their overall verdict, which is worth quoting because it is the map: "the observation layer is genuinely very good; the automation and replay layer is not production-ready. A good instrument bolted to a weak robot." Reading state, reading network bodies, and grading honestly are the hard, valuable parts and they are strong. Driving an app and replaying journeys need breadth of real-world exposure, and they read like they have been exercised against simple demo apps.

The split was not random. Every single thing that broke was a property of a real application:

property	what it broke
a file upload	recorder wrote {path}, replayer only took {name, content, type} — a recorded upload could never replay
a 22-second model-backed endpoint	replay's fixed 4s wait called a pending request an absent one
a WebGL canvas	invisible to every observation channel; with default sourceMapping it crashed the app outright
a 401-refresh-retry	two false-red contradictions on a standard auth path
an app writing localStorage in a loop	starved the buffer; the verdict then reported net.total: 0 as a fact
an icon-only button with no accessible name	undriveable

None of those is exotic. The five fixes above shipped, and none of them would have needed a field report if a corpus like this existed.

Why nothing we have covers it

Every app under apps/ is one we wrote, and each was written to exercise a feature. They are honest fixtures and they are all cooperative. apps/atlas is the closest thing to an adversarial one and it is still ours. The sibling reticle-fixtures repo installs into pristine upstream apps, which covers install COMPLEXITY — a different question from runtime hostility.

What would close this

Start with one app carrying two or three of these properties, wired into pnpm test:e2e. A corpus is the goal; a corpus is not the first PR. Good candidates, cheapest first:

• an endpoint that takes >20s (a setTimeout in apps/api — nearly free, and it covers the replay-timeout class)
• a localStorage write loop (a few lines; covers the buffer-starvation class)
• an icon-only button with no accessible name
• a 401-then-refresh-then-retry path in apps/api
• a file upload
• a WebGL canvas subtree

Please read apps/README.md first — it says what belongs under apps/ and why — and docs/gates.md for which gate a new fixture should join. pnpm test:e2e is ~8 minutes.

Related

#130 (a standing false-green gate against third-party apps we did not write) is the same instinct aimed at apps we do not control. This one is about apps we DO control that are deliberately hostile, which is cheaper to start and easier to keep green.

[divshekhar]

Filed from the same export that produced this issue: the six critical fixes are #902 and the react-three-fiber crash is #901.

Worth recording here because it is the argument for this issue. Five of the six fixes in #902 were reproduced from the reporters' traces, not from a running app with the property that broke. There was no file-upload flow to replay, no >20s endpoint, no localStorage write storm and no WebGL subtree in this repo to test against — so the unit tests pin the logic and the end-to-end proof that the reported symptom is gone does not exist.

That is exactly the gap this issue describes, seen from the fixing side rather than the reporting side. The corpus would not only have caught these before release; it would let the fixes be proved rather than argued.

[divshekhar]

First slice landed on fix/field-export-2026-09-08 (PR #902), commit 0da5b7d6 — one axis, driven end to end, in apps/atlas rather than a new app (its README already claims "where truncation, buffer eviction and rate caps actually bite", and its spec is already wired into the battery).

The property: an opt-in control that rewrites one localStorage key with byte-identical content, 200 writes every 50ms. The exact field shape that filled the ring buffer (held: 2000, dropped: 70482) and made a verdict report net.total: 0 as a fact while the request carrying the root cause was on the wire.

Two things worth reporting, because they are the argument for this issue:

1. My first guard was decoration. It asserted the capture stayed clean during the storm — and it passed with the observer fix REVERTED. The assert window is short and the transport rate-cap already sheds low-value events, so cleanliness is not sensitive to the defect. Replaced with a direct measurement of what the fix changes: 1 storage event with the guard, 496 without.
2. Running it caught an off-by-one in my own assertion. I asserted zero events; it is one. The first write genuinely changes the key, from absent to its value, and must be reported — only the rewrites are no-ops. Asserting zero would have demanded that Reticle drop a real change, which is the opposite defect and a worse one.

Neither of those was visible from unit tests, and both took ten minutes once a real app was doing the real thing. That is the case for the rest of the corpus better than anything I could argue.

Remaining axes, cheapest first: a >20s endpoint (apps/api already takes a delay param, so this is nearly free and covers the replay-timeout class), a 401-refresh-retry, a file upload, an icon-only button with no accessible name, and a WebGL subtree.

[divshekhar]

Second axis landed: a slow endpoint (7f2fb558), and it proves another fix that had shipped against unit tests only.

apps/api already honoured ?delay= — only the client never asked for it. A ?serverDelay=<ms> knob on the saved-items fixture threads it through, opt-in and defaulting to 0, so every existing spec on that fixture is untouched. Deliberately a different knob from renderDelay: a fast response rendered late and a request that is genuinely slow are different fixtures, and only the second is one a fixed wait cannot survive.

Both polarities are asserted, which is what makes it a guard rather than a demonstration. The same flow against the same 6s endpoint drifts without timeoutMs (reproducing the field's exact signal_not_observed) and passes with it. If that field ever stops being threaded through, the first assertion goes green and the pair stops meaning anything — so the red case is pinned as deliberately as the green one. That is the lesson from the first axis, where my initial check passed with the fix reverted.

6s rather than the field's 22s: the boundary that matters is the 4000ms default, not their exact numbers, so it reproduces the failure at the smallest wall-clock cost that clears it.

Two of six axes done, both of which found something:

• storage churn → caught that my first guard was decoration, and an off-by-one in my own assertion
• slow endpoint → proved the replay-timeout fix end to end for the first time

Remaining: a 401-refresh-retry, a file upload, an icon-only button with no accessible name, and a WebGL subtree. The first three are ordinary app fixtures; the WebGL one is the largest and overlaps #880.

[divshekhar]

Third axis landed: 401 → refresh → retry-once (8b449b04), the first covering the FALSE-RED class.

A field reticle_assert returned pass: true and Reticle overrode it to verified: "no", verifiedReason: "contradicted" with two findings that both described the app working correctly — ui-advanced-request-failed and duplicate-request. The fixture keeps the honest shape rather than simulating it: ?expireFirst=1 makes the first call with a given token 401 exactly once, rejected before any write; the client refreshes and retries once — not a loop, because a second 401 is a real failure and must surface. One click, three real requests.

Four assertions, and the fourth keeps the other three honest: the spec reads reticle_network and requires that a 401 AND a 200 were really on the wire (statuses=[401,200]). Without it, a fixture that quietly stopped expiring would make the three "no contradiction" checks pass for free. That is the decoration failure the storage axis caught after the fact, guarded against here from the start.

Three of six axes done, and the pattern is consistent — each one has either proved a fix that had never been demonstrated, or caught a flaw in my own check:

axis	what it produced
storage churn	caught that my first guard was decoration; caught an off-by-one in my own assertion
slow endpoint	first end-to-end proof of the replay-timeout fix, both polarities
401 retry	first coverage of the false-red class, with an anti-decoration guard built in

Full battery green with all three wired in.

Remaining: a file upload, an icon-only button with no accessible name, and a WebGL subtree. The first two are ordinary fixtures; the WebGL one is the largest and overlaps #880.