Seo geo prerender#20
Merged
Merged
Conversation
…uards Externalize the inline JS data out of each locale homepage so the HTML shell is small and the data caches across pages — the prerequisite for the per-appellation pre-rendered pages in later phases. - map_template.render(): the ~12.3 MB AOCS + ~0.6 MB grape-tooltip blobs are no longer inlined; they ship in a hashed, render-blocking /data/aocs.<lang>.<hash>.js that assigns window.__OWM_DATA before the app runs (read synchronously, no boot re-thread). render() now returns (html, data_filename, data_bytes). Homepage HTML 13.3 MB -> 449 KB (-96.6%). - grapes_info is serialized with sort_keys so the cache-busting hash is deterministic across runs (its dict order is set-derived); aocs keeps its build order, which drives the sidebar appellation list. Reproducible-build / no-op-rerun contract holds. - 04_build_maps.py: write the per-locale bundle to wiki/data/ and prune stale hashed files. - deploy.py: set Content-Type by file extension on upload (html/json/js/xml/ webmanifest/...; pmtiles and others keep octet-stream), and a non-fatal post-deploy check that warns if the apex host isn't 301-redirecting to www (guards the Search Console "Duplicate without user-selected canonical" issue; the redirect itself is a Bunny edge rule, not in this repo). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add scripts/_lib/content_block.py — a faithful Python port of the STABLE subset of the JS renderAocCard pipeline, so a future per-appellation page can ship a crawlable, server-rendered <article> built from the same `aocs` record the client panel uses. Covers: name (+native/Latin), meta line (country chip(s) incl. cross-border aliases, region, kind, commune count), DGC / approx / stub lines, style chips, grape pills with the cahier-vs-canonical bracket logic, the PT role disclaimer, terroir facts (bullet + verbatim modes), summary (shown only when there are no facts), the cross-border note, and the sources block. Each <article> carries a data-ssr-sig (hash of its normalised text) for a runtime JS cross-check in a later phase. Deliberately narrower than the JS panel: the volatile HU dűlők / IT menzioni chip sections are LEFT to the client panel (omitted server-side). Purely additive — not yet imported or emitted by stage 04; Phase 3 injects it into the per-appellation pages. Verified: renders all 2908 records with 0 crashes / 0 leftover placeholders; tests/test_content_block.py (15 cases) pins canonical-bracket show/suppress, sub-denomination line, stub message, cross-border chips, verbatim facts, bullet-facts grouping + summary suppression, native+Latin name, dulok/menzioni omission, escaping, and deterministic output. Mirrors one pre-existing JS quirk faithfully (communes_matched == -1 renders "-1 commune(s)" in both renderers; a `> 0` guard is deferred to a shared fix). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pre-render a crawlable HTML page per appellation for a gated pilot set (23
well-known parents x 4 locales = 92 pages), reusing each locale's shell + the
shared Phase-1 data bundle so the pages cost almost nothing on the CDN.
map_template.render():
- Parameterised the <head> (title / description / hreflang / og / json-ld) and
added a {ssr_content} body slot; split the format call into a shared
shell_kwargs (constant across a locale's pages) + per-page slots. The homepage
output is unchanged except the new SSR-removal JS.
- entity_slugs path emits {slug: html} per pilot slug, each with a
self-referential canonical, a per-slug hreflang cluster (x-default -> EN),
a templated (NON-verbatim) title + meta description, Place/AdministrativeArea
+ BreadcrumbList JSON-LD (GeoShape box derived from bbox, lat/lng order), and
the Phase-2 content_block injected as <article id="ssr-content"> (carrying
data-ssr-sig). EN entities live at /en/<slug>; fr/es/nl at /<locale>/<slug>.
- The app JS removes #ssr-content on boot — the interactive panel supersedes the
no-JS / crawler fallback article.
04_build_maps.py:
- gate_classify folds sub-denominations + stubs + no-geometry records; an
indexable record needs resolved geometry plus its own grapes/summary/terroir.
A hand-picked pilot allowlist (intersected with the gate) + a dry-run counter;
writes wiki/<locale>/<slug>/index.html; sitemap expanded to per-slug 4-locale
hreflang clusters (96 URLs: 4 home + 92 entity).
Determinism: the inline VIVC-derived blobs (VIVC_SIBLINGS / SLUG_TO_CANONICAL /
GRAPE_SYNONYMS) and GRAPE_SEARCH_INDEX were non-deterministic across builds
(set-iteration order) — fixed by sorting vivc_groups member lists, sort_keys on
the three dicts, and a slug tiebreaker on the index sort. The homepage is now
byte-identical across builds (no-op rerun / reproducibility holds).
Verified: 92 entity pages; self-canonical + reciprocal hreflang; valid
Place/BreadcrumbList JSON-LD; SSR article present pre-JS; node --check on every
generated script; FR/ES/NL locale chrome correct; full test suite + dup-keys
green.
Staging gates (NOT verifiable locally, must confirm before relying on
indexation): (1) Bunny must serve the origin /<locale>/<slug>/index.html for the
no-trailing-slash URL (origin-file-wins over the SPA rewrite) — the canonical
uses the no-slash form matching the shipped client deep-link; (2) facet "open"
buttons are still buttons, so pilot pages are sitemap-discoverable but not yet
internally linked (Phase 3.5 follow-up); (3) the communes_matched == -1 display
quirk is shared with the JS panel (a > 0 guard is a deferred shared fix).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-appellation entity pages set a server-rendered <title>, but the SPA
never updated document.title as the user navigated, so the tab title got stuck
on whichever appellation the user landed on or first opened (e.g. stayed
"Priorat" after closing it or opening another).
Add docTitleFor() — mirrors the server _build_entity_meta title format
("<Name> — <kind> · <region>, <country> · Open Wine Map") — and set
document.title on every panel open / focus-change (renderPanelStack), resetting
to the locale homepage title (DEFAULT_TITLE = LABELS.page_title) when the panel
closes (closePanel). So the tab title now tracks the current view: identical to
the server <title> when landing on a pre-rendered /<locale>/<slug> page, updated
when cycling overlapping appellations, and reset on close.
Document-level <title> tags are unchanged (homepage generic, entity per-slug);
this is a client-side UX fix only. node --check + test suite + dup-keys green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the 23-slug pilot with a full-corpus gate: every record gets a prerendered /<locale>/<slug>/index.html so the CDN can serve any deep-link. 1638 indexable records get a full page (self-canonical, Place/BreadcrumbList JSON-LD, server-rendered card); 1270 folded records (sub-denominations, stubs, no-geometry, thin) get a lightweight noindex,follow + self-canonical shell with empty SSR — on the map for the deep-link, kept out of the index as near-duplicates of the parent. - render() streams each page to disk instead of returning a dict (the full corpus would be multi-GB in memory); returns (html, data_filename, data_bytes, n_index, n_fold). - folded pages: noindex + SELF-canonical, not canonical->parent (Google treats noindex + cross-canonical as contradictory; 53 folded parents were themselves noindex). og:url follows canonical_url. - sitemap lists indexable slugs only (4 home + 1638x4 = 6556 URLs). Verified: 11,632 pages + 4 homepages; every deep-linkable slug has a file in all 4 locales; locale-parity, slug-safety, valid sitemap. Pre-deploy adversarial audit (HCU/duplicate, JSON-LD, head/meta, routing) clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Split the monolithic page template at module load into (page, css, app):
the <style> block ships as one shared content-hashed /assets/style.<hash>.css
for the whole corpus, and the ~2,000-line app <script> as a per-locale
/assets/app.<locale>.<hash>.js. Each of the ~11.6k pages is now a ~13-22 KB
shell referencing three cached bundles (data + app + style) instead of
inlining ~450 KB.
- _split_template() lifts the two blocks by unique delimiters with a
lossless-reconstruction assert (fires at import on drift).
- render() splits shell_kwargs into script_kwargs (fill the app JS) and
page_shell (fill the page; the {labels[...]} item-access fields keep labels
as a page field); CSS has no fields (replace-unescape). Both assets are
content-hashed; render returns an assets dict {data, style, app}.
- stage 04 writes + prunes all three bundles per locale.
Verified behavior-preserving: re-inlining the external assets reproduces the
pre-externalization pages byte-for-byte (homepage + index + fold; en/fr/es).
Deploy footprint 5.1 GB -> ~196 MB. Gate/sitemap/canonical invariants
unchanged; 16 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The externalized app.<lang>.js (~407KB cached fetch) takes ~190ms to boot,
during which the server-rendered #ssr-content card was visible before the map
replaced it — a white-card flash. Gate it: the pre-paint inline script (which
already sets view-mode/theme classes before paint) now also adds `js` to
<html>, and `html.js #ssr-content{display:none}` hides the card from first
paint for JS visitors (the app removes it on boot anyway). Non-JS visitors and
non-rendering crawlers still receive the card in the HTML, so SEO +
accessibility are unchanged.
Verified locally: with JS the card is never visible during boot (no flash);
the #ssr-content article remains in the raw HTML source.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The <urlset> namespace used a hyphen (.../schemas/sitemap-0.9) instead of the official sitemaps.org slash form (.../schemas/sitemap/0.9). Google tolerates it, but strict XSD validators reject the sitemap. One-char fix; the deployed sitemap.xml was patched + re-uploaded in place (no full rebuild needed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…y headers - list_remote fans out one-GET-per-directory across BUNNY_LIST_WORKERS (32) threads, eliminating the ~30min sequential walk over ~11k leaf dirs - HTTPAdapter with backoff retry on 429/5xx, pool sized to widest concurrency - ensure_security_headers: idempotent Bunny Edge Rules (HSTS, nosniff, X-Frame-Options, Referrer-Policy, Permissions-Policy) - gitignore logs/ and .hermes/
…ity pages - 02i_fetch_wikidata_qids.py: cached incremental stage resolving each record to a Wikidata QID via P9854 (eAmbrosia ID) join + Wikipedia-title fallback; writes raw/wikidata/qids-by-slug.json - _lib/wikidata.py: QID->URL helper + resolver - entity JSON-LD restructured to a single schema.org @graph (WebSite -> WebPage -> Place/AdministrativeArea -> BreadcrumbList) with stable fragment @ids; sameAs identity cluster (Wikidata/Wikipedia/regulator), isBasedOn source docs, additionalType wine-region; drops invalid country-only breadcrumb ListItem - SRI integrity + crossorigin on unpkg maplibre/pmtiles tags (verified) - h1/h2 heading-hierarchy fix (brand_tag demotes to <p> on SSR card pages) - content_block: suppress 'N communes' when count is 0/None - add unidecode dep; tests for jsonld, headings, wikidata resolver
…g pins - VIVC ambiguous-slug queue closed 2026-06-03 (9 ES/AT/IT slugs pinned with vivc_id + colour + provenance via grape-colour-researcher) - note SSR content-block gap: HU dulok / IT menzioni not yet server-rendered
Diffs two build snapshots, normalising content-hash filenames (foo.<hash>.js -> foo.js) so a renamed-but-identical asset is not a false diff. Stage 04 is deterministic by design, so a move-only refactor must report 'identical'. The golden baseline lives at tmp/golden-v2 (gitignored).
All 24 were identical-value duplicates within their dict literal (zero data conflicts — Python kept the last value, which equalled the first, so the resolved GRAPE_ALIAS/DEFAULT_COLOUR mappings are unchanged). Deleted the later occurrence of each; lifted 5 informative VIVC-number/identity comments onto the surviving line. Guarded by tests/test_no_duplicate_keys.py.
All 30 were the single-char 'l': language/locale loop vars -> lang, candidate provider keys in benchmark_providers -> cand, lost-slug list in audit_extraction_diff -> removed. Loop-body references updated too (F821 guard caught two I'd missed in the 02b translate stages).
select the set actually driven clean; E501 excluded (900 long lines, many in template strings). per-file-ignore E402 for scripts/ — they bootstrap sys.path before importing _lib, so those imports are intentionally not top-of-file (previously held by inline noqa, which isort-splitting could drop).
Runs uv sync --group dev, then the pinned ruff check and pytest. Only the raw/-independent test suite runs in CI (all 43 current tests qualify); a test that needs raw/ should be marked and excluded here.
Move the 1255-char 'Last reconciled' paragraph into docs/reconciliation-log.md as a dated entry; CURATOR_TODO.md keeps a one-line pointer. No open items (❌/🟡/⏳) touched.
10 tests over scripts/02_extract_cahiers.py: pure helpers (slug, normalize_name, candidate_keys alias-splitting, parse_communes parenthetical/soft-hyphen handling) + extract_sections routing. Two are regression guards for the f1ad98c section-X drops (case-insensitive CHAPITRE_RE truncation; \x0c page-break before a Roman heading) — verified they fail on the pre-fix regexes. Adds tests/conftest.py (fixture_text loader) + tests/fixtures/ conventions.
10 tests over scripts/_lib/es/national_pliego.py covering the regional-format variation documented inline: strong/weak section-6 headers, penedes' dropped 'DE' linker, TOC dot-leader rejection, mentrida 6->7 stop, los-cerrillos same-prefix-stop-by-title, and the _normalise_token rejection rules (length, digits, word-count, _DROP noise) that fire before the rapidfuzz vocab matcher. find_variety_section verified against real cached pliegos for penedes/la-mancha/ castilla.
Vendor maplibre-gl@4.7.1 + pmtiles@3.2.0 as tracked git inputs under scripts/_lib/vendor/; stage 04 copy_brand_assets() mirrors them to /assets/vendor/ and map_template.py references the same-origin paths (SRI/crossorigin dropped — first-party now). The fetched bytes' sha384 match the previously-shipped SRI hashes exactly, so behaviour is unchanged. Golden comparator vs tmp/golden-v2: the only output diffs are the 3-tag swap in every HTML page + the 3 new vendor assets; data bundles, app.js, style.css, sitemap, robots are byte-identical. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…parser-fixture-writer) These were authored by the improvements-plan run but left untracked; .claude/ is a tracked dir here (national-spec-layer, research-gaps, grape-colour-researcher are all committed), so they belong in git for the plan's tooling to be reproducible. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-locale aocs bundle carried both map-critical fields and the heavy panel payload (summary, terroir facts, sources, grape display-names, …), ~13.2MB raw / 1.9MB gz per locale. Now the render-blocking startup bundle ships only STARTUP_AOCS_FIELDS (sidebar list, search, facets, fly-to, active-filter chips, map-click geom_source dedup) — 3.2MB raw / 0.42MB gz — and the panel payload emits per slug as /data/d/<locale>/<slug>.json (the field complement). The JS fetches it on first panel open (hydratePanel -> Object.assign into AOCS[slug], deduped + graceful-degrade), with a skeleton placeholder; repeat opens are instant. grapes_info stays inline (a startup dependency via the active-filter chips). Server-rendered entity cards use the full record unchanged, so crawlers see identical content. Golden comparator vs a post-Phase-2 baseline: the only output changes are data/aocs.*.js (shrank), assets/app.*.js (lazy-fetch), assets/style.css (skeleton CSS); every entity HTML, sitemap and robots are byte-identical (asset-hash refs aside). node --check passes on all 4 app.js; per-slug JSON serves 200 with the panel fields; startup blob carries only the 14 startup keys. 63 tests pass, ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… source)
The ~96 KB map application JS lived as an escaped string inside _TEMPLATE
(doubled braces, 23 {slot} format fields). It now lives in a real .js file;
_render_app_js fills __OWM_<slot>__ tokens (sitting exactly where the format
fields were) per locale. Output is byte-for-byte unchanged: the golden
comparator reports 'identical' vs the post-Phase-3 baseline, and all four
emitted app.<locale>.<hash>.js match the prior bytes (same hashes). node
--check passes; eslint.config.js (flat, advisory, token globals self-derived)
reports 0 errors; a non-blocking eslint CI job runs it. _split_template now
lifts only the CSS. 63 tests + ruff clean.
The byte-identity was machine-verified during extraction (a sentinel fill of
the tokenised source must equal _APP_JS.format() with the same sentinel), so
hand-transcription error across 2123 lines is excluded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…/ (no-op)
Phase 6 foundation + first slice. The 13 slug-keyed provenance caches that the
national-spec augmenters populate and _sources_for()/the panel-blob phase read
back move to _lib/augment/_shared.py, so both writer and readers reference the
SAME dict objects across the module split (the augmenters mutate in place —
.clear()/[slug]=, never reassign — so the imported names stay bound). The
ES/SI/HR augmenters + their sidecar-dir constants move to _lib/augment/{es,si,
hr}.py verbatim. 10 augmenters (+ IT extras, CZ helper, the 02f lexicon/geom
blocks) remain in 04 for follow-up.
Verified move-only: golden comparator vs the post-Phase-3/4 baseline reports
'identical' after a full stage-04 rebuild; 63 tests + ruff clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes Phase 6.1: bg/cy/gr/sk/ro/hu/de + it (4 fns: masaf, _backfill, regional-registers, synthesize-sottozone) + cz (with its colour-block map + in-function grape-lexicon import) move verbatim to _lib/augment/<cc>.py. _IT_REGISTER_BY_SLUG joins the shared caches; _IT_MENZIONI_BY_SLUG stays in 04 (only 04's menzioni-harvest touches it). HU/CZ sidecar dirs stay importable from _shared (04 reads them outside the augmenter). synthesize_it_sottozone's call order in main() is preserved. 04_build_maps.py is now ~1400 lines lighter. Verified move-only: golden comparator vs the (yesterday) baseline differs ONLY in sitemap.xml, and ONLY in <lastmod> (the build date rolled 06-10→06-11; grep -vc lastmod = 0). Every HTML page, app.js, aocs.js, data and robots are byte-identical; write_seo_files was untouched. 63 tests + ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
….py (no-op) Phase 6.2 part 1. The self-contained lexicon cluster (load/merge grape + style lexicons, VIVC loaders, build_grapes_info + its private corpus/translation helpers, _truncate_extract, _latin_form_or_empty) moves verbatim to _lib/lexicon_loading.py along with its 5 path constants + _DISAMBIG_SUFFIX. No stage-04 function dependencies; 04 imports back the 6 externally-called names (build_grapes_info, load_style_lexicon, merge_style_lexicon, _load_vivc_by_slug, _load_vivc_colour_by_slug, _latin_form_or_empty). The geometry constant DEPT_NAME_TO_CODE, which physically sat between the lexicon and geometry blocks, stays in 04 with the geometry code (ruff F821 caught the greedy slice). Verified move-only: golden comparator vs today's baseline reports 'identical' after a full stage-04 rebuild. 63 tests + ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…chain.py (no-op) Completes Phase 6.2. The geometry resolution cluster — normalize_commune, load_commune_index, cahier_insee, union_for_appellation, union_from_insee, _find_sibling_umbrella, the DGCGeomResult dataclass, resolve_dgc_geometry, and the ES fallbacks (_resolve_es_igp_fallback, _resolve_es_sigpac) + DEPT_NAME_TO _CODE — moves verbatim to _lib/geom_chain.py. The members are scattered non-contiguously in 04 and interleaved with non-geometry helpers (_join_set, _geojson_bounds, communes_containing), so each is extracted individually by name; the interleaved helpers stay in 04 (the cluster has no 04-function deps). The 11 names the main build loop calls are imported back. Verified move-only: golden comparator vs today's baseline reports 'identical' after a full stage-04 rebuild. 63 tests + ruff clean. Phase 6 done: 04_build_maps.py is ~2300 lines lighter (augmenters + lexicon + geometry now in _lib/). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Flags appellations whose geometry is the Bétard 2022 figshare-pdo fallback but whose GI was registered/amended after the dataset's Nov-2021 data snapshot — the polygon may not reflect a post-snapshot boundary change (or a brand-new GI is on the map only via an incidental file-number match). Cross-references the compact aocs startup blob (geom_source) against each raw/<cc>/eambrosia index's eu_protection_date/modification_date; buckets FLAGGED/REVIEWED/OK/NO-DATE like the sibling geometry audits, with a betard_delta_overrides.json whitelist and --strict / --cutoff flags. Read-only detector. v1 run: 473 Bétard-tier records → 345 OK, 128 FLAGGED, 0 NO-DATE. Phase 7.1; documented in CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stages a ready-to-send email asking JKI whether Open Wine Map may display VIVC synonym strings verbatim under attribution (today it ships only IDs + prime names). Human sends; reply recorded in the draft + folded into CLAUDE.md's VIVC section. CLAUDE.md points at the draft. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds fixture-based regression tests for the RO (document_unic + ONVPV caiet +
commune) and IT (sottozona + menzione + MASAF article carving) parsers, written
by the parser-fixture-writer agent against redacted, licence-clear excerpts of
cached regulator docs. Suite 63 → 103 (+19 RO, +21 IT); ruff clean. A
.gitignore negation lets the redacted OJ-page .html fixtures commit past the
blanket *.html ignore.
The tests pin CURRENT parser behaviour (not idealised), so two latent data bugs
they surfaced are now locked in + will flip-flag when fixed:
- RO: the iana caiet '- soiuri roşii/roze:' colour header glues 'roze:' onto
the first red variety, dropping Cabernet Sauvignon.
- IT: menzione's _NAME_TOKEN_RE rejects a lowercase connector inside a name,
so Chianti Classico's 11 UGAs parse as 10 (San Donato in Poggio dropped).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bunny rejects an empty Triggers list ("At least one condition is required"),
so each SetResponseHeader edge rule now carries a catch-all Url trigger
(Type 0, pattern "*") to apply unconditionally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two latent parser bugs pinned by Phase-5 regression tests; both tests flip to assert the corrected behaviour. - RO `_lib/ro/caiet.py`: the colour header "- soiuri roşii/roze:" left "roze:" glued onto the first red variety, so its name never resolved. `_COLOUR_HEADER_RE` now consumes the "/roze" (and "/rose") second-colour suffix; bucket colour stays the first captured colour (roşii → noir). Recovers Cabernet Sauvignon in iana (11→12) and dealurile-transilvaniei (16→17); no wine loses a grape. - IT `_lib/it/menzione.py`: `_NAME_TOKEN_RE` rejected a lowercase Italian connector inside a name, dropping "San Donato in Poggio" from the numbered-list parser. The token regex now allows a connector (in/di/del/della/…) glued to a following capitalised word. Recovers Chianti Classico's 11th UGA, plus "Corte del Durlo" / "Monte di Colognola" in the Soave family (29→31 each); no name removed or corrupted (connector only accepted before a capitalised word). Verified end-to-end by regenerating the RO 02f sidecars + IT 02 extractions over the real cached regulator documents. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…se 7.2 Adds the 6th official Italian regional production-zone layer, moving 20 Umbrian appellations off the Bétard whole-municipality fallback onto the Regione Umbria delimited zones (CC-BY 4.0). - zone_sources.py: promote `umbria` from todo → active with a new `fetch_type: "ckan_shapefiles"` shape — the region publishes one per-appellation shapefile per CKAN dataset (19 of them, a mix of .zip and .7z) instead of a single WFS/zip layer. Carries the CKAN catalog endpoint, the assigned CRS (EPSG:3004 — the shapefiles ship no .prj), dotted tier-prefix stripping, an "X o Y" alternate-name split, and a curated extra_names map (the combined DOC+DOCG datasets cover the DOCG sub-name on the shared delimited area). - stage 00 `fetch_ckan_shapefiles()`: enumerate the catalog, download + extract each archive into raw/it/regional-zones/umbria/<dataset>/. py7zr (the `bootstrap` group, used by MASAF 02f) is imported lazily and gated like the Playwright WAF bootstrap — absent → the .7z datasets are skipped with a loud warning, the .zip ones still harvest. - ITZoneIndex: split the loader into `_load_layers` (the existing WFS/zip path, unchanged) + `_load_ckan` (glob extracted .shp, assign the declared CRS, strip the dotted ZONE tier prefix, index every name variant). The `ZONE` field is per-feature, so the existing connector- stripped normalisation matching still applies. Geometry verified: an isolated with-vs-without-Umbria rebuild differs on exactly 20 slugs — all Umbrian (14 figshare-pdo + 5 gisco-* + Orvieto lazio→lazio+umbria), zero collateral. Sagrantino, Torgiano Rosso Riserva and Rosso Orvietano all resolve; only Narni (no published shapefile) stays on its commune-union fallback. audit_it_regions green; overlap audit shows only thin border slivers (Italian zones overlap by design). tests/test_it_zones.py locks in the tier-strip / alt-name / DOCG-alias matching. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…(Phase 5)
Completes Phase 5 — FR/ES/RO/IT were already done; this adds redacted-
fixture regression tests for every remaining country parser, written by
the parser-fixture-writer agent (one country each) and mirroring the
existing tests/test_{ro,it}_parser.py style: STRUCTURE-only assertions,
the shared fixture_text conftest fixture, and inline pinning of ACTUAL
parser behaviour (so a test flips and flags the day a quirk is fixed).
+196 tests (109 → 305), all green, ruff clean. Coverage:
- AT (19): einziges_dokument section guard + dash-synonym variety split.
- DE (28): einziges_dokument + all four BLE Produktspezifikation
templates (A/B/C/D) incl. the §3.2-principal vs §8-flat role split.
- HU (16): the monotonic-number + nested-subsection guard (section-4
"Bor – …" decoys must not shadow sections 5–9).
- GR (16): greek_norm final-sigma fold (ς→σ) + OIV colour-letter codes.
- PT (20): caderno_sections 3 variants + subregião patterns A & B +
commune_list concelho shapes.
- BG (24): edinen_dokument nested-subsection guard + IAVV specifikacija
colour split + Cyrillic-preserving commune casefold.
- HR (18): jedinstveni_dokument + the lettered-section specifikacija
(form-feed fold, forward-only letter guard, colour-adjective fallback).
- SI (19): enotni_dokument em-dash split + all three specifikacija
templates incl. the 2007 priporočene/dovoljene role split.
- CZ (22): national_spec variety table + rowspan commune-tree walker +
SZPI CHZO terroir/style parser.
- SK (14): jednotny_dokument old+new title variants + the ÚPV §f
two-column left-Odroda-only extraction (synonym never leaks).
Fixtures are short redacted excerpts (≤103 lines) with source-file
provenance comments; nothing from raw/ is committed. The agents also
pinned several pre-existing latent bugs as current behaviour (GR 2024
country decoy routes as geo_area; BG section-9 drop; PT distrito leak)
— each test flips when the parser is fixed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…less CI Two independent CI failures (31 total), both reproduced locally by reducing raw/ to its tracked files (CI's actual state): 1. test_brand_heading_tag_is_parameterized — the search-rail refactor moved the brand wordmark out of _PAGE_TEMPLATE into _SIDEBAR (composed via the %%SIDEBAR%% placeholder). Point the assertions at _SIDEBAR; keep the no-hard-coded-<h1> invariant checked against both constants. 2. 30 parser-fixture tests (AT/BG/DE/GR/HR/HU/IT/RO/SI/SK) — the grape matcher builds its vocabulary from gitignored raw/vivc/by-slug + raw/*-extracted, absent on CI, so VIVC synonym surfaces vanish and grapes fuzzy-mismatch (Weißer Riesling -> raeuschling, dropped feteasca-alba, KeyError on empty colour buckets). Fix test-side only (production matcher untouched): a committed vocab snapshot fed to _load_vocabulary via an autouse conftest fixture, but only when raw/vivc/by-slug is absent. Verified: CI-state full suite 312 passed (was 31 failed); normal local (raw present) 312 passed; ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.