feat(ppd): extend snapshot coverage to the full PPD history, 1995 onward - #67
Merged
Conversation
PARTITION_YEARS 11 -> 32. With a coverage end in 2026 that opens the window on
1995-01-01, the first year of Price Paid Data, across 32 year partitions.
The old value was sized to the `months le=120` API ceiling, on the reasoning
recorded in its comment: "twelve buys coverage nothing asks for". That held only
while every surface was bounded. The subject-property history lookup is not --
it asks for one property's whole sale record, with no date bound -- which is
precisely why it was hardwired to the live SPARQL source, and why it stayed
broken when that source degraded to 503s. Full history is what lets it route to
the snapshot like everything else.
One constant is the whole coverage change. `expected_years()` and
`coverage_start()` both derive from it, and everything downstream reads coverage
from the manifest or derives it -- validate.py's partition and coverage gates,
the runtime's parquet-file count, the adapter, provenance. No hardcoded 2016
exists anywhere in serving code.
The tests needed more care than the assertions suggested. Several used dates in
2011 and 2015 as their "outside the declared window" fixtures; under a 1995
window those rows fall INSIDE and the tests would have kept passing while
quietly testing nothing. They are moved to 1994, so each still exercises the
exclusion it is named for. Changed:
test_coverage_starts_at_january_of_the_eleventh_calendar_year_back
-> test_coverage_starts_at_the_first_year_of_ppd
test_expected_years_are_eleven_consecutive_calendar_years
-> test_expected_years_span_ppd_history
Both keep pinning the arithmetic rather than the number, because 32 is derived
(2026 - 32 + 1 == 1995) and must be incremented when the release year rolls
over; the renamed coverage test is what will say so.
The guarantee gate needed real thought rather than a new number. It fires when
the window cannot reach back 120 months, and at 32 years that is trivially
satisfied for any plausible `today`, so the old failing case (today=2025-01-01)
now passes and tested nothing. 1995-01-01 plus 3600 days is 2004-11-13, so the
pair is re-bracketed to 2004-01-01 failing and 2005-01-01 passing -- tight
around the actual boundary.
Mechanical elsewhere: partition lists 2016..2026 -> 1995..2026, parquet_files
11 -> 32, declared coverage_from 2016-01-01 -> 1995-01-01.
No artifact is built or published here. ./scripts/validate.sh: 2121 passed.
…ery figure that derives from the measurement The full-history build produced a 1,189,365,783 B bundle and the boot check refused it: "manifest declares 1189365783 bytes, above the 1073741824-byte maximum". That is the fail-closed design working -- it built, validated and packaged, then declined to publish an artifact every Machine would have rejected at boot, before a byte transferred. Three limits, raised together because they constrain each other: fetch.DEFAULT_MAX_BUNDLE_BYTES 1 GiB -> 2 GiB (~1.8x over 1.108 GiB) archive.ExtractionLimits.max_total 2 GiB -> 4 GiB boot_only_verify.EXPECTED_BUNDLE 279,109,872 -> 1,189,365,783 max_total_bytes is deliberately kept ABOVE what the fetch ceiling can deliver. A bundle is compressed, so one at the fetch cap unpacks to more than the cap; leaving them equal would admit an archive that extraction then refused -- a failure discovered after the whole transfer rather than before it. Preflight checked rather than assumed: bundle * 2.5 is 2835.7 MiB at the real bundle and 5.37 GiB at the ceiling, against 8.32 GB free on the Machine. The documentation was not optional here. Three tests read the SPECIFICATION and fail when code and doc disagree, which is what stops them drifting. So every figure derived from the measurement is republished in the section that relies on it: bundle size 266.2 -> 1134.3 MiB extracted size 267.9 -> 1139.3 MiB transfer @100 Mbit/s 22.3 -> 95.1 s simultaneous payload 534.1 -> 2273.6 MiB preflight (x2.5) 665.4 -> 2835.7 MiB MAX_BUNDLE_BYTES margin 3.8x -> 1.8x Spec 1.1 is rewritten rather than renumbered. It argued for eleven partitions from the 120-month REST ceiling, and that argument was sound for every BOUNDED surface -- but the subject-property history lookup is unbounded, which is exactly why 2.6 routed it live, and that exception became the system's last hard dependency on live SPARQL. The section now says so, and records that "twelve buys coverage nothing asks for" was true of the bounded surfaces and false of the unbounded one. It also records a consequence rather than burying it: 95.1 s does not fit the 30 s readiness target and no longer needs to. Since rev 9 the boot is non-blocking, so the transfer bounds how long AFTER readiness the snapshot starts answering, not readiness itself -- which Phase E measured at 9.89 s and this does not touch. The Group B honesty guards are generalised rather than re-pointed. They policed one superseded figure (214 MiB); there are now two, and a third arrives at the next rebuild, so SUPERSEDED_BUNDLE_FIGURES is a list and each entry must appear only in text marking it superseded. The sizing-table guards likewise assert the general property -- every row declares measured-or-estimated, every transfer time declares that it is calculated, every non-current row says superseded in the row itself -- instead of naming the rows that happened to exist. Two tests whose subject no longer exists (the 10/12-partition estimate rows) are removed; their principle is carried by test_every_sizing_row_declares_measured_or_estimated. ./scripts/validate.sh: 2125 passed, 28 skipped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The snapshot held 11 years, sized to the largest bounded request (
months le=120).That reasoning was correct for every bounded surface — and one is not bounded.
Subject-property history asks for a single property's whole sale record with no
date bound, which is why it was hardwired to live SPARQL, and that exception
became the system's last hard dependency on a source now returning 503 after
271.8 s.
Full history removes the exception rather than working around it.
The build
in the source. Nothing lost.
d0897c94…e874e.geography, 32/32 partitions present. One benign finding: a single row whose
source postcode is the literal string
UNKNOWN.current artifact), so
coverage_tois 2026-07-31 and freshness improves from66 days to ~35.
One constant, and careful tests
PARTITION_YEARS = 11 → 32is the entire coverage change — everythingdownstream reads coverage from the manifest or derives it.
The tests needed more than renumbering. Several used 2011 and 2015 dates as
their "outside the window" fixtures; under a 1995 window those fall inside,
so the tests would have kept passing while testing nothing. Moved to 1994.
The guarantee gate needed real thought: it fires when the window can't reach
back 120 months, which at 32 years is trivially satisfied, so the old failing
case now passed and tested nothing. Re-bracketed around the actual boundary
(1995-01-01 + 3600 days = 2004-11-13): 2004-01-01 fails, 2005-01-01 passes.
Limits raised, and the docs that had to move with them
The first build was refused at the boot check —
BundleVerificationError: 1189365783 bytes, above the 1073741824-byte maximum— and not promoted. That'sthe fail-closed design working: it built, validated and packaged, then declined
to publish an artifact every Machine would reject at boot.
DEFAULT_MAX_BUNDLE_BYTESExtractionLimits.max_total_bytesEXPECTED_BUNDLE_BYTESmax_total_bytesis deliberately kept above what the fetch ceiling admits —a compressed bundle at the cap unpacks to more than the cap, and equal values
would admit an archive extraction then refused, discovered after the whole
transfer.
The documentation was not optional. Three tests read the specification and
fail when code and doc disagree. Every derived figure is republished in the
section that relies on it: bundle 266.2 → 1134.3 MiB, transfer 22.3 → 95.1 s,
preflight 665.4 → 2835.7 MiB, margin 3.8× → 1.8×.
§1.1 is rewritten rather than renumbered, because it argued for eleven
partitions and that argument no longer holds. It also records a consequence
rather than burying it: 95.1 s does not fit the 30 s readiness target and no
longer needs to — since rev 9 the boot is non-blocking, so the transfer bounds
when the snapshot starts answering, not readiness (9.89 s, unchanged).
The Group B honesty guards are generalised, not re-pointed: they policed one
superseded figure, there are now two and a third arrives next rebuild, so it's a
list — and the sizing-table guards assert the general property (every row
declares measured-or-estimated; every non-current row says superseded in the row
itself).
Deploy ordering — important
The image carrying the raised
MAX_BUNDLE_BYTESmust be deployed BEFOREcurrent.jsonflips to the new artifact. Otherwise every Machine refuses the1.1 GiB bundle at boot and falls back to live.
Publishing to Tigris is owner-held by design and not automatable from here.
./scripts/validate.sh→ 2125 passed, 28 skipped.