Skip to content

fix(verify): write the verification the predicate actually accepts - #23

Open
ChelseaKR wants to merge 1 commit into
mainfrom
fix/verify-writes-what-eligibility-requires
Open

fix(verify): write the verification the predicate actually accepts#23
ChelseaKR wants to merge 1 commit into
mainfrom
fix/verify-writes-what-eligibility-requires

Conversation

@ChelseaKR

Copy link
Copy Markdown
Owner

Closes #18.

verify.confirm() was the only writer of a verification in the codebase, and it wrote status, verifier, at and note. core/eligibility.py also requires an evidence reference and an in-date recheck expiry. Nothing in src/ could write a fetch-policy decision at all — eleven fetch_policy hits: parsed, read, published, never written. So the queue the README calls "the most valuable three and a half hours anyone could spend on this repo" could be worked to completion with no effect on what the tool watches, and nothing said so.

Measured, one source, scratch registry

BEFORE
  confirm(path, "ks-kdhe-vital-statistics", verifier="A Named Human", at="2026-08-15")
  on disk:  {'status': 'verified', 'verifier': 'A Named Human', 'at': '2026-08-15'}
  reasons:  ('verification-evidence-missing', 'verification-expiry-missing',
             'fetch-policy-unreviewed')

AFTER
  on disk:  {'status': 'verified', 'verifier': 'A Named Human', 'at': '2026-08-15',
             'evidence': 'var/evidence/verification/ks-2026-08-15.json',
             'expires_at': '2027-02-11'}
  reasons:  ('fetch-policy-unreviewed',)

The one reason left is the one that is genuinely a different person's judgment about somebody else's terms — and it now has a writer.

Two of the four fields are produced, not demanded

The evidence reference points at a receipt written at the moment of the decision: URL, fetch time, HTTP status, the page's own <title>, the excerpt the verifier read, and the hash of the bytes behind it. That is what evidence means here. Asking a volunteer to type a path to a file they have not made produces a plausible string and no receipt.

A source our crawler cannot fetch gets a receipt saying exactly that, claiming no title and no text: the honest evidence for ssa.gov is "we could not see it; a person opened it themselves", and an empty title under a confident statement would read as a page we saw and found blank.

Receipts are written to var/evidence/verification/ — untracked. docs/05-DATA-AND-EVIDENCE.md says raw evidence is never automatically public, and a receipt carries an excerpt of whatever the page happened to be serving. This is a judgment call and it is yours to overturn: --evidence-dir sources/evidence makes them committed and auditable by anyone, at the cost of publishing page excerpts by default.

The expiry is dated 180 days forward from the decision. That number is a starting guess and is commented as one, in the same voice as REMOVAL_THRESHOLD: too long and a verification is a permanent claim about a page that can be reorganised at any time; too short and the queue outruns the people working it. Also yours to change — it is a module constant and a --expires flag.

The second writer

sentinel sources policy records the dated robots/terms decision SRC-03 specifies — outcome, reviewer, date, evidence, reason, expiry — and refuses blanks, and refuses unreviewed (the absence of the decision, not one of its outcomes).

It does not make the decision and cannot. Whether a host's terms permit a weekly watch is a reading of somebody else's document; HttpFetcher obeying robots at request time is one rule, mechanically applied, at one moment, and is not that reading. So --evidence here is required from the reviewer rather than generated: unlike a page excerpt, nothing in this tool has read these terms.

What now says the thing nobody was told

Surface Says
verify --list attempt-eligible today, and every blocking reason — before someone decides to spend the afternoon
end of a verify session the same, derived from the registry on disk through the shared predicate, not from the session's own counters (which can only say how much work was done)
sources policy what that one decision did and did not unlock
docs/VERIFYING.md first screen: both decisions are required, and what a confirmation actually writes
the site a zero denominator is no longer rendered as attempted 0 of 0 eligible sources; 0 successful retrievals — arithmetically true, and reads as a run that had nothing to do and did it perfectly. Completeness over an empty denominator is not a number, so it is reported as not measurable.
the site headline a registry where every source is verified and none is attempt-eligible no longer leads with "All 152 sources are human-verified"

Not changed

The committed registry. Nothing here verifies a source or grants a fetch policy — those are 152 human decisions and they remain undone. 0 of 152 sources are human-verified still holds, and the standing test that asserts it is untouched.

Verification

make verify — 7/7 green. 431 tests, mypy --strict clean, coverage ≥ 90%, coverage --check-docs green across all six gated documents.

Both new site tests fail on the parent commit, on the exact false sentence:

FAILED tests/test_site.py::test_a_run_with_no_eligible_sources_does_not_render_as_a_completed_run
FAILED tests/test_site.py::test_a_fully_verified_registry_that_watches_nothing_does_not_headline_the_verification
   'Read this first: All 1 sources are human-verified' is contained here: …

The eleven new tests/test_verify.py tests cannot even import against the parent (record_fetch_policy does not exist), which is the defect stated as a stack trace.

Note for merge order

This branch and #22 (issue #19) both touch src/id_churn_sentinel/core/site.py#22 adds an unmeasured-retrievals clause to _run_status_section, this one adds the empty-denominator branch to the same function. Whichever merges second wants a rebase; the two changes are independent in intent.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ENsaeU8RwLEBEwTN2Nevzx

`verify.confirm()` was the only writer of a verification in the codebase and
wrote `status`, `verifier`, `at` and `note`. `core/eligibility.py` also requires
an evidence reference and an in-date recheck expiry, and nothing in `src/` could
write a fetch-policy decision at all -- eleven `fetch_policy` hits, parsed,
read, published, never written. So the queue the README calls "the most valuable
three and a half hours anyone could spend on this repo" could be worked to
completion, and the attempt denominator stayed at zero.

Measured on a scratch registry, one source, before this change:

  confirm(path, "ks-kdhe-vital-statistics", verifier="A Named Human")
    -> {'status': 'verified', 'verifier': 'A Named Human', 'at': '2026-08-15'}
  evaluate_source(...)  reasons = ('verification-evidence-missing',
                                   'verification-expiry-missing',
                                   'fetch-policy-unreviewed')

after:

  -> {'status': 'verified', 'verifier': 'A Named Human', 'at': '2026-08-15',
      'evidence': 'var/evidence/verification/ks-2026-08-15.json',
      'expires_at': '2027-02-11'}
  evaluate_source(...)  reasons = ('fetch-policy-unreviewed',)

The one reason left is the one that is genuinely a different person's judgment
about somebody else's terms, and it now has a writer.

Two of the four fields are produced rather than demanded. The evidence reference
points at a receipt written at the moment of the decision -- URL, fetch time,
HTTP status, the page's own title, the excerpt the verifier read, the hash of
the bytes behind it. That is what evidence means here; asking a volunteer to
type a path to a file they have not made produces a plausible string and no
receipt. A source our crawler cannot fetch gets a receipt that says exactly
that, claiming no title and no text, because the honest evidence for ssa.gov is
"we could not see it; a person opened it themselves" -- and an empty title under
a confident statement would read as a page we saw and found blank. Receipts go
to var/evidence/verification/ (untracked): docs/05-DATA-AND-EVIDENCE.md says raw
evidence is never automatically public, and a receipt carries an excerpt of
whatever the page was serving. `--evidence-dir` moves it.

The expiry is dated 180 days forward from the decision. That number is a
starting guess and is commented as one, in the same voice as REMOVAL_THRESHOLD:
too long and a verification becomes a permanent claim about a page that can be
reorganised at any time, too short and the queue outruns the people working it.

`sentinel sources policy` writes the dated robots/terms decision SRC-03
specifies -- outcome, reviewer, date, evidence, reason, expiry -- and refuses
blanks and refuses `unreviewed`, which is the absence of the decision rather
than one of its outcomes. It does not make the decision and cannot: whether a
host's terms permit a weekly watch is a reading of somebody else's document, and
HttpFetcher obeying robots at request time is one rule mechanically applied at
one moment, not that reading.

And the reporting, which is what turns a trap into a queue:

  verify --list            attempt-eligible today, and every blocking reason,
                           BEFORE someone decides to spend the afternoon
  end of a verify session  the same, derived from the registry on disk through
                           the shared predicate -- not from the session's own
                           counters, which can only say how much work was done
  sources policy           what that one decision did and did not unlock
  docs/VERIFYING.md        first screen: both decisions are required
  the site                 a zero attempt denominator is no longer rendered as
                           "attempted 0 of 0 eligible sources; 0 successful
                           retrievals", which is arithmetically true and reads
                           as a run that had nothing to do and did it perfectly.
                           Completeness over an empty denominator is not a
                           number and is reported as not measurable.
  the site headline        a registry where every source is verified and none is
                           attempt-eligible no longer leads with "All 152
                           sources are human-verified" -- the strongest
                           verification claim the page can make, over a feed
                           that will never fill

Not changed: the committed registry. Nothing here verifies a source or grants a
fetch policy; those are 152 human decisions and they remain undone.

make verify: 7/7 green. 431 tests pass, mypy --strict clean, coverage >= 90%.

Closes #18

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ENsaeU8RwLEBEwTN2Nevzx
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Working the whole verification queue still leaves the attempt denominator at zero

1 participant