Add a source currency watch that reports content drift against the corpus

[ChelseaKR]

What

Every rule cites an LCI document, and LCI republishes its checklist and common-mistakes PDFs each year under new URLs. make audit-sources checks that links resolve; nothing checks that the content behind a link still says what the corpus retained. scripts/watch_sources.py (maintainer-run, networked, never in make verify or check) re-fetches each corpus/manifest.json document, compares normalized text against the retained text, and classifies each as unchanged, changed, or unverifiable by kind (transport failure versus 404/410). For a changed source it reports which retained passages, and therefore which rules' source and guidelines bindings, no longer occur verbatim, and writes a dated review record. It adopts nothing.

Why it matters

A rule explained from a superseded checklist is the confident staleness this portfolio is built to catch. permit-bearings runs exactly this watch weekly for HCD and statute sources, with converging issues and the transport versus not_found distinction of its ADR 0005; this proposes the same pattern here so the corpus that grounds ai explain cannot silently outlive its source. The default path is untouched: the watch is a maintainer tool whose output is a dated file, not a change to any rule.

Scope

• scripts/watch_sources.py with --offline-cache for replay; output docs/audits/source-watch-YYYY-MM-DD.json with per-document status, hash, failure kind, and surviving and lost passage ids.
• rules list --source-status FILE shows each rule's source status from a watch file.
• Optional scheduled workflow uploading the watch file as an artifact and commenting on one stable issue (never a new issue per run), modeled on permit-bearings' currency.yml.
• corpus/README.md procedure for adopting a changed source: re-retrieve, rebuild passages, re-run the grounding eval.

Out of scope

• Automatic corpus rebuilds or rule edits.
• Any effect on check output or exit codes.

Done when

• Against a recorded cache where one document changed, the watch names the document, the passages lost, and the rules bound to them; unchanged documents are listed as unchanged.
• A 404 is recorded as not_found, a timeout as transport, and neither marks a passage lost.
• The classifier runs offline over fixtures in the test suite; no test opens a socket.

Pointers

• scripts/check_rule_sources.py, scripts/build_corpus.py, corpus/manifest.json, src/ceqa_preflight/ai/corpus.py
• permit-bearings: src/permit_pathways/harness/watch.py, .github/workflows/currency.yml, docs/adr/0005-separate-a-withdrawn-citation-from-an-unreachable-source.md
• FILE-004 and FILE-005 cite a URL that returns 404, and the HTML report renders it as a "Source" link #38 (the dead-URL incident this would have surfaced earlier)

Proposed with AI assistance.

[ChelseaKR]

Triage, 2026-09-06. Checked against origin/main @ 6d948f7: open PRs (only #105, a draft), merged PRs through #104, all 9 remote branches, no integration/* branch exists, and the file contents on origin.

Verdict: buildable, with a working implementation of the same pattern in a sibling repository to copy from.

Measured on origin: no scripts/watch_sources.py. What exists is scripts/check_rule_sources.py (behind make audit-sources), which checks that links resolve — not that the content behind them still says what the corpus retained. scripts/build_corpus.py and corpus/manifest.json are present, so the retained text and passage ids the watch would compare against are already there. Unbuilt, and the inputs exist.

The risk is real and named precisely: LCI republishes its checklist and common-mistakes PDFs each year under new URLs, so a rule explained from a superseded checklist is "the confident staleness this portfolio is built to catch". A link that still resolves to a rewritten document is the worst case, because the current check passes.

The cross-repo precedent is genuine and worth using. permit-bearings runs this weekly for HCD and statute sources, with the transport-versus-not_found distinction of its ADR 0005 — and a builder should read src/permit_pathways/harness/watch.py before writing this one. The unverifiable kind is the load-bearing part: a fetch that fails for transport reasons is not "unchanged", and collapsing those two would make the watch structurally incapable of reporting the thing it exists for.

Two constraints that keep it safe, both stated in the issue:

• It adopts nothing — output is a dated review record under docs/audits/, and a person decides.
• It never runs in make verify or check, so the default path stays offline and exit codes are untouched.

The detail about commenting on one stable issue, never a new issue per run, is worth keeping — that is a lesson other repos in this portfolio learned the expensive way.

Prioritisation note: the pilot-track roadmap issues here (#80, #81, #83) are blocked on people who are not named, and #85/#86/#103 are owner-only. The feature issues are most of the work actually available right now.

[ChelseaKR]

Unbuilt on origin/main at 6d948f7 — no scripts/watch_sources.py, and make audit-sources still only checks that URLs resolve.

Automatable. The classifier is offline over fixtures, permit-bearings has the pattern including the transport versus not_found distinction, and nothing here touches check output.

The high-value half is the transport-versus-not_found split. A watch that reports "changed" on a timeout would train a reader to ignore it, and a watch that reports "unchanged" because it never reached the document is the same defect as a check reporting a document clean that it never read — the thing #61 fixed in the engine. Keep the two apart from the first commit rather than adding the distinction later.

Triaged with AI assistance during the 2026-09-06 drain; every claim above was checked against origin/main, not against a roadmap.

[ChelseaKR]

Partly delivered in #109 (c8dcd66). What landed:

• scripts/watch_sources.py with --offline-cache, --document and --output, plus a make watch-sources target. It re-fetches every document in corpus/manifest.json, re-derives its text through the same extraction build_corpus.py used, and writes docs/audits/source-watch-YYYY-MM-DD.json. It adopts nothing.
• All three "Done when" clauses: a changed document names the document, the passages lost and the rules bound to them, with unchanged documents listed as unchanged; a 404 is not_found and a timeout is transport, and neither marks a passage lost; and the whole classifier is tested over fixtures with no test opening a socket.
• Two failure kinds beyond the two the issue named: not_checked for a document absent from an offline cache, and extraction for one that was fetched but from which no passage could be re-derived — because an extractor that no longer understands a page is far more common than a source that deleted every passage, and the two must not render the same.
• corpus/README.md gains "When a source changes": the verdicts, the failure kinds, and a five-step adoption procedure that ends by saying a changed verdict is not by itself a reason to change a rule.

What is left, and why this issue stays open:

1. rules list --source-status FILE. It needs new user-facing strings in src/, and make i18n holds EN and ES at key parity, so it would carry Spanish a qualified reviewer has not seen (docs/I18N.md item 3 is still open).
2. The scheduled workflow. It would put real network requests into CI on a schedule, which this repository deliberately does not do, and "comment on one stable issue" needs an issue you choose. Both are your calls.

No watch record is committed: a real run is ~291 live requests, and a record fabricated from anything less would be exactly the asserted-not-measured figure #103 is open about.

Prepared with AI assistance; reviewed before submission.

[ChelseaKR]

rules list --source-status FILE landed on main in PR #110 (commit 6978d4a), so the second of the two items #109 deliberately left is now delivered.

Status of this issue's scope, read against origin/main:

Scope item	State
scripts/watch_sources.py with --offline-cache, dated record under docs/audits/	done (#109)
rules list --source-status FILE	done (#110)
corpus/README.md procedure for adopting a changed source	done (#109), with a per-rule readback section added in #110
Optional scheduled workflow	outstanding, and an owner call

The workflow is the one item left, and it is left on purpose rather than for want of effort. Two parts of it are decisions rather than automation:

1. It would put real network requests into CI on a schedule — roughly 291 fetches per run against LCI and the CCR. This repository deliberately makes no network requests in make verify, in check, or in CI, and README.md says so as a property of the tool. Adding a scheduled job that does is a change to what the project claims about itself.
2. "Comment on one stable issue, never a new issue per run" needs an issue chosen by you. permit-bearings' currency.yml is the model, and it comments on a specific issue number that a person picked.

No watch record is committed either, for the reason recorded on #109: running the real watch means those ~291 live requests, and a record assembled from anything less would be exactly the asserted-not-measured figure #103 is open about.

What --source-status does with a record, for the note: it keeps every distinction the watch makes rather than flattening it. passages_lost, source_changed, not_examined (a bound document could not be read), not_watched (no document in the record names this rule) and unchanged. A rule bound to several documents takes the worst status among them. And because the listing reads only the file it is handed — the installed package ships no corpus manifest to compare against — its preamble names how many documents the record covers, so a run narrowed with --document cannot read as a checked corpus.

One finding from that work, unrelated to this issue but worth recording somewhere: make i18n stays green on a Spanish msgstr that is verbatim English. Its "English identity" row asserts an English msgstr matches its msgid; nothing asserts a Spanish one differs from one. Measured by replacing one translation with its source string, recompiling, and running the gate. It is not trivially fixable — a proper noun, a command name or an acronym can legitimately be identical — so closing it needs an allowlist, which is your call.

Prepared with AI assistance; reviewed before submission.

[ChelseaKR]

Triage 2026-09-07: everything in Scope except the last bullet has landed.

Scope item	Where
scripts/watch_sources.py with --offline-cache, dated docs/audits/source-watch-YYYY-MM-DD.json, per-document status / hash / failure kind / surviving and lost passage ids	#109 (cca38df… c8dcd66)
rules list --source-status FILE	#110 (6978d4a), src/ceqa_preflight/source_status.py
corpus/README.md procedure for adopting a changed source	#109, "When a source changes"
classifier runs offline over fixtures; no test opens a socket	tests/test_watch_sources.py, and the suite runs with --disable-socket in addopts
not_found vs transport vs not_checked vs extraction, none marking a passage lost	#109, failure_kind

Against the Done-when list, all three conditions are met by the committed tests.

Not built: the optional scheduled workflow — "uploading the watch file as an artifact
and commenting on one stable issue (never a new issue per run)". Left alone deliberately
rather than forgotten. It needs a stable issue number that does not exist yet, and
inventing one would be a guess; it puts a weekly networked job on an Actions budget with a
billing-block history in this account; and this portfolio has already measured how a
scheduled live-URL check degrades into a calendar bomb that reports staleness it cannot
see. That is a call for you, not a mechanical follow-on.

So: the substance of this issue is delivered. Whether it stays open for the scheduled
workflow, or that becomes its own issue, is yours.

Prepared with AI assistance.