Skip to content

feat(ppd): a second verification arm that needs no live upstream - #64

Merged
paulieb89 merged 2 commits into
mainfrom
feat/ppd-nonlive-verification
Sep 4, 2026
Merged

feat(ppd): a second verification arm that needs no live upstream#64
paulieb89 merged 2 commits into
mainfrom
feat/ppd-nonlive-verification

Conversation

@paulieb89

Copy link
Copy Markdown
Owner

Why

Stage 1's live arm has failed twice — 429 on 2026-09-02, then a 503 after
271.8s
on 2026-09-04. Its central criterion is the false empty:

"snapshot_false_empty": snap.count == 0 and live.count > 0

One arm cannot detect that, because a wrongly-empty answer is indistinguishable
from a correctly-empty one. The 2026-08-28 snapshot-only rehearsal passed 85
assertions across 13 cases and still couldn't close the gap — and no number of
further snapshot-only assertions would.

This adds a second arm that is not the live source: the artifact compared
against itself, read a different way. No network, no HMLR dependency.

Why it can find anything

The schema stores outcode/sector as materialized columns. That removes
the STRSTARTS("B5")-matches-B50 bug class entirely — but relocates the
risk: if the build wrote those columns wrongly, every geography query is
silently short, and querying the same column can never reveal it.

The two derivations are genuinely independent:

build (build.py _DERIVE) split_part(trim(postcode), ' ', 1) — validates nothing, never uppercases
reference (postcode_rules) validated regex grammar, GIR-exact, uppercases, rejects malformed

A postcode they disagree on is a row the adapter cannot reach by geography.
That's a false empty, discoverable offline.

The checks

  1. Geography derivation — recompute from postcode, compare to what the
    build stored. Three outcomes kept apart because their causes differ:
    unreachable, case_mismatch (build never uppercases, query side does),
    unvalidated.
  2. Query semantics — every predicate, the ordering and the limit
    reimplemented in plain Python, filtering on geography recomputed rather
    than the stored column, so a wrong derivation can't hide by being wrong
    identically in both arms.
  3. Partition completeness — a missing year returns fewer rows rather than
    failing.

Result against the real artifact

rows_examined  10,394,935   (every row, not a sample)
passed         true
findings       []
elapsed        90s

Ran alongside production traffic without disturbing it: MemAvailable
1,599,652 kB after vs 1,596,868 kB before, health 200 in 49–97ms
throughout, Fly check passing.

Memory safety is load-bearing here

My first draft used fetchall() and accumulated one id per disagreement.
Against 10.4M rows with ~1.6 GB free on a Machine serving production traffic,
that would have OOM-killed the live app — and the id-accumulation would have
failed hardest exactly when the artifact was most broken. Both were fixed
before the tool was pointed at the real artifact. Rows now stream in 50k
chunks; findings carry counters plus a capped sample. Pinned by tests, including
a cursor stand-in whose fetchall() raises.

Tests

31 tests. Every check is driven from a row built to carry its specific
defect
and asserted to be found, then a clean row asserted to pass — a
verifier that only ever passes is worthless.

./scripts/validate.sh2042 passed, 27 skipped.

Scope limits, stated

  • Not Stage 1. The report sets not_stage_1_evidence: true. It shares the
    snapshot's own data, so a row that never reached the artifact stays
    undetectable. Only an independent publication can close that — the live
    source, or the 5.5 GB bulk CSV whose download ppd-snapshot-build.md records
    as not authorised.
  • cases_compared: 0. The CLI runs checks 1 and 3; the per-corpus-case
    comparison is implemented and unit-tested (python_select) but not yet wired
    to the adapter and frozen corpus. The report states 0 rather than implying
    coverage it lacks.
  • No latency measured; nothing about agreement with HMLR today.

What it does establish: the artifact is internally consistent, and the query
path can reach every row it contains.

Out-of-band operator tooling on the same contract as boot_only_verify.py
never imported by the app, never wired into CMD, inert unless invoked. Not
added to the image in this PR; it was run by copying the single file in.

Stage 1's live arm has been unavailable twice: 429 on 2026-09-02, then a 503
after 271.8s on 2026-09-04. Its central criterion is the false empty --

    "snapshot_false_empty": snap.count == 0 and live.count > 0

-- which one arm cannot detect, because a wrongly-empty answer is
indistinguishable from a correctly-empty one. The 2026-08-28 snapshot-only
rehearsal passed 85 assertions across 13 cases and still could not close that
gap, and no number of further snapshot-only assertions would.

This adds a second arm that is not the live source: the artifact compared
against itself, read a different way. No network, no HMLR availability.

It can find something because the two geography derivations are genuinely
independent. The build (build.py _DERIVE) uses DuckDB string surgery --
split_part(trim(postcode), ' ', 1) -- which validates nothing and never
upper-cases. The reference here uses property_core.postcode_rules, a validated
regex grammar with exact GIR handling that upper-cases and rejects malformed
input. A postcode they disagree about is a row the adapter cannot reach by
geography, which is a false empty discoverable offline.

That matters specifically because the schema stores outcode/sector as
MATERIALIZED columns. That removes the STRSTARTS("B5")-matches-"B50" bug class
entirely, but it relocates the risk rather than deleting it: if the build wrote
those columns wrongly, every geography query is silently short and querying the
same column can never reveal it.

Three checks: geography derivation; query semantics, with each predicate, the
ordering and the limit reimplemented in plain Python over decoded Parquet, and
filtering on geography RECOMPUTED from postcode rather than the stored column,
so a wrong derivation cannot hide by being wrong identically in both arms; and
partition completeness, since a missing year returns fewer rows rather than
failing.

Memory safety is load-bearing, not incidental. The artifact holds 10,394,935
rows and the deployed Machine has ~1.6 GB available while serving production
traffic. A first draft used fetchall() and accumulated one id per disagreement;
both were fixed before the tool was pointed at the real artifact. Rows now
stream in 50k chunks and findings carry counters plus a capped sample, so peak
memory is independent of artifact size -- and the tool does not fail hardest
exactly when the artifact is most broken. Tests pin both, including a cursor
stand-in whose fetchall() raises.

What it does not establish is stated in the module docstring and in the report
itself, which sets not_stage_1_evidence: true. It shares the snapshot's own
data, so it cannot detect a row that never reached the artifact; only an
independent publication -- the live source, or the 5.5 GB bulk CSV whose
download the build runbook records as unauthorised -- can close that. It
measures no latency and says nothing about agreement with HMLR today. What it
does establish is that the artifact is internally consistent and that the query
path can reach every row the artifact contains.

Out-of-band operator tooling on the same contract as boot_only_verify.py: never
imported by the application, never wired into CMD, inert unless invoked.

./scripts/validate.sh: 2042 passed, 27 skipped.
First run of the second arm against the real artifact, on the deployed Machine.

    rows_examined  10,394,935   (every row, not a sample)
    passed         true
    findings       []
    elapsed        90s
    artifact       v20260828T194003Z, bundle_sha256 50f802b2...

Geography derivation agrees with the validated grammar for every row, and all
eleven year partitions covering 2016-01-01..2026-06-30 are present. The primary
false-empty vector -- a materialized outcode/sector the build wrote wrongly, so
that equality queries silently miss rows -- is now checked across the whole
artifact and is clean.

Ran alongside production traffic without disturbing it: MemAvailable was
1,599,652 kB after the run against 1,596,868 kB before, health returned 200 in
49-97 ms throughout, and the Fly check stayed passing. That is the streaming
design working; the same run with the first draft's fetchall() would have tried
to materialize 10.4M rows as Python tuples against 1.6 GB.

Scope, as the report itself records: cases_compared is 0. The CLI runs the
geography-derivation and partition-completeness checks; the per-corpus-case
query-semantics comparison is implemented and unit-tested (python_select) but
not yet wired to the adapter and the frozen corpus. The report states 0 rather
than implying coverage it does not have.

Still not Stage 1 evidence, and the report sets not_stage_1_evidence: true. It
shares the snapshot's own data, so a row that never reached the artifact remains
undetectable here.
@paulieb89
paulieb89 merged commit 6090db7 into main Sep 4, 2026
1 check passed
@paulieb89
paulieb89 deleted the feat/ppd-nonlive-verification branch September 4, 2026 16:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant