feat(ppd): a second verification arm that needs no live upstream - #64
Merged
Conversation
Stage 1's live arm has been unavailable twice: 429 on 2026-09-02, then a 503
after 271.8s on 2026-09-04. Its central criterion is the false empty --
"snapshot_false_empty": snap.count == 0 and live.count > 0
-- which one arm cannot detect, because a wrongly-empty answer is
indistinguishable from a correctly-empty one. The 2026-08-28 snapshot-only
rehearsal passed 85 assertions across 13 cases and still could not close that
gap, and no number of further snapshot-only assertions would.
This adds a second arm that is not the live source: the artifact compared
against itself, read a different way. No network, no HMLR availability.
It can find something because the two geography derivations are genuinely
independent. The build (build.py _DERIVE) uses DuckDB string surgery --
split_part(trim(postcode), ' ', 1) -- which validates nothing and never
upper-cases. The reference here uses property_core.postcode_rules, a validated
regex grammar with exact GIR handling that upper-cases and rejects malformed
input. A postcode they disagree about is a row the adapter cannot reach by
geography, which is a false empty discoverable offline.
That matters specifically because the schema stores outcode/sector as
MATERIALIZED columns. That removes the STRSTARTS("B5")-matches-"B50" bug class
entirely, but it relocates the risk rather than deleting it: if the build wrote
those columns wrongly, every geography query is silently short and querying the
same column can never reveal it.
Three checks: geography derivation; query semantics, with each predicate, the
ordering and the limit reimplemented in plain Python over decoded Parquet, and
filtering on geography RECOMPUTED from postcode rather than the stored column,
so a wrong derivation cannot hide by being wrong identically in both arms; and
partition completeness, since a missing year returns fewer rows rather than
failing.
Memory safety is load-bearing, not incidental. The artifact holds 10,394,935
rows and the deployed Machine has ~1.6 GB available while serving production
traffic. A first draft used fetchall() and accumulated one id per disagreement;
both were fixed before the tool was pointed at the real artifact. Rows now
stream in 50k chunks and findings carry counters plus a capped sample, so peak
memory is independent of artifact size -- and the tool does not fail hardest
exactly when the artifact is most broken. Tests pin both, including a cursor
stand-in whose fetchall() raises.
What it does not establish is stated in the module docstring and in the report
itself, which sets not_stage_1_evidence: true. It shares the snapshot's own
data, so it cannot detect a row that never reached the artifact; only an
independent publication -- the live source, or the 5.5 GB bulk CSV whose
download the build runbook records as unauthorised -- can close that. It
measures no latency and says nothing about agreement with HMLR today. What it
does establish is that the artifact is internally consistent and that the query
path can reach every row the artifact contains.
Out-of-band operator tooling on the same contract as boot_only_verify.py: never
imported by the application, never wired into CMD, inert unless invoked.
./scripts/validate.sh: 2042 passed, 27 skipped.
First run of the second arm against the real artifact, on the deployed Machine.
rows_examined 10,394,935 (every row, not a sample)
passed true
findings []
elapsed 90s
artifact v20260828T194003Z, bundle_sha256 50f802b2...
Geography derivation agrees with the validated grammar for every row, and all
eleven year partitions covering 2016-01-01..2026-06-30 are present. The primary
false-empty vector -- a materialized outcode/sector the build wrote wrongly, so
that equality queries silently miss rows -- is now checked across the whole
artifact and is clean.
Ran alongside production traffic without disturbing it: MemAvailable was
1,599,652 kB after the run against 1,596,868 kB before, health returned 200 in
49-97 ms throughout, and the Fly check stayed passing. That is the streaming
design working; the same run with the first draft's fetchall() would have tried
to materialize 10.4M rows as Python tuples against 1.6 GB.
Scope, as the report itself records: cases_compared is 0. The CLI runs the
geography-derivation and partition-completeness checks; the per-corpus-case
query-semantics comparison is implemented and unit-tested (python_select) but
not yet wired to the adapter and the frozen corpus. The report states 0 rather
than implying coverage it does not have.
Still not Stage 1 evidence, and the report sets not_stage_1_evidence: true. It
shares the snapshot's own data, so a row that never reached the artifact remains
undetectable here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Stage 1's live arm has failed twice — 429 on 2026-09-02, then a 503 after
271.8s on 2026-09-04. Its central criterion is the false empty:
One arm cannot detect that, because a wrongly-empty answer is indistinguishable
from a correctly-empty one. The 2026-08-28 snapshot-only rehearsal passed 85
assertions across 13 cases and still couldn't close the gap — and no number of
further snapshot-only assertions would.
This adds a second arm that is not the live source: the artifact compared
against itself, read a different way. No network, no HMLR dependency.
Why it can find anything
The schema stores
outcode/sectoras materialized columns. That removesthe
STRSTARTS("B5")-matches-B50bug class entirely — but relocates therisk: if the build wrote those columns wrongly, every geography query is
silently short, and querying the same column can never reveal it.
The two derivations are genuinely independent:
build.py_DERIVE)split_part(trim(postcode), ' ', 1)— validates nothing, never uppercasespostcode_rules)A postcode they disagree on is a row the adapter cannot reach by geography.
That's a false empty, discoverable offline.
The checks
postcode, compare to what thebuild stored. Three outcomes kept apart because their causes differ:
unreachable,case_mismatch(build never uppercases, query side does),unvalidated.reimplemented in plain Python, filtering on geography recomputed rather
than the stored column, so a wrong derivation can't hide by being wrong
identically in both arms.
failing.
Result against the real artifact
Ran alongside production traffic without disturbing it:
MemAvailable1,599,652 kB after vs 1,596,868 kB before, health 200 in 49–97ms
throughout, Fly check passing.
Memory safety is load-bearing here
My first draft used
fetchall()and accumulated one id per disagreement.Against 10.4M rows with ~1.6 GB free on a Machine serving production traffic,
that would have OOM-killed the live app — and the id-accumulation would have
failed hardest exactly when the artifact was most broken. Both were fixed
before the tool was pointed at the real artifact. Rows now stream in 50k
chunks; findings carry counters plus a capped sample. Pinned by tests, including
a cursor stand-in whose
fetchall()raises.Tests
31 tests. Every check is driven from a row built to carry its specific
defect and asserted to be found, then a clean row asserted to pass — a
verifier that only ever passes is worthless.
./scripts/validate.sh→ 2042 passed, 27 skipped.Scope limits, stated
not_stage_1_evidence: true. It shares thesnapshot's own data, so a row that never reached the artifact stays
undetectable. Only an independent publication can close that — the live
source, or the 5.5 GB bulk CSV whose download
ppd-snapshot-build.mdrecordsas not authorised.
cases_compared: 0. The CLI runs checks 1 and 3; the per-corpus-casecomparison is implemented and unit-tested (
python_select) but not yet wiredto the adapter and frozen corpus. The report states 0 rather than implying
coverage it lacks.
What it does establish: the artifact is internally consistent, and the query
path can reach every row it contains.
Out-of-band operator tooling on the same contract as
boot_only_verify.py—never imported by the app, never wired into
CMD, inert unless invoked. Notadded to the image in this PR; it was run by copying the single file in.