fix(verify): write the verification the predicate actually accepts - #23
Open
ChelseaKR wants to merge 1 commit into
Open
fix(verify): write the verification the predicate actually accepts#23ChelseaKR wants to merge 1 commit into
ChelseaKR wants to merge 1 commit into
Conversation
`verify.confirm()` was the only writer of a verification in the codebase and
wrote `status`, `verifier`, `at` and `note`. `core/eligibility.py` also requires
an evidence reference and an in-date recheck expiry, and nothing in `src/` could
write a fetch-policy decision at all -- eleven `fetch_policy` hits, parsed,
read, published, never written. So the queue the README calls "the most valuable
three and a half hours anyone could spend on this repo" could be worked to
completion, and the attempt denominator stayed at zero.
Measured on a scratch registry, one source, before this change:
confirm(path, "ks-kdhe-vital-statistics", verifier="A Named Human")
-> {'status': 'verified', 'verifier': 'A Named Human', 'at': '2026-08-15'}
evaluate_source(...) reasons = ('verification-evidence-missing',
'verification-expiry-missing',
'fetch-policy-unreviewed')
after:
-> {'status': 'verified', 'verifier': 'A Named Human', 'at': '2026-08-15',
'evidence': 'var/evidence/verification/ks-2026-08-15.json',
'expires_at': '2027-02-11'}
evaluate_source(...) reasons = ('fetch-policy-unreviewed',)
The one reason left is the one that is genuinely a different person's judgment
about somebody else's terms, and it now has a writer.
Two of the four fields are produced rather than demanded. The evidence reference
points at a receipt written at the moment of the decision -- URL, fetch time,
HTTP status, the page's own title, the excerpt the verifier read, the hash of
the bytes behind it. That is what evidence means here; asking a volunteer to
type a path to a file they have not made produces a plausible string and no
receipt. A source our crawler cannot fetch gets a receipt that says exactly
that, claiming no title and no text, because the honest evidence for ssa.gov is
"we could not see it; a person opened it themselves" -- and an empty title under
a confident statement would read as a page we saw and found blank. Receipts go
to var/evidence/verification/ (untracked): docs/05-DATA-AND-EVIDENCE.md says raw
evidence is never automatically public, and a receipt carries an excerpt of
whatever the page was serving. `--evidence-dir` moves it.
The expiry is dated 180 days forward from the decision. That number is a
starting guess and is commented as one, in the same voice as REMOVAL_THRESHOLD:
too long and a verification becomes a permanent claim about a page that can be
reorganised at any time, too short and the queue outruns the people working it.
`sentinel sources policy` writes the dated robots/terms decision SRC-03
specifies -- outcome, reviewer, date, evidence, reason, expiry -- and refuses
blanks and refuses `unreviewed`, which is the absence of the decision rather
than one of its outcomes. It does not make the decision and cannot: whether a
host's terms permit a weekly watch is a reading of somebody else's document, and
HttpFetcher obeying robots at request time is one rule mechanically applied at
one moment, not that reading.
And the reporting, which is what turns a trap into a queue:
verify --list attempt-eligible today, and every blocking reason,
BEFORE someone decides to spend the afternoon
end of a verify session the same, derived from the registry on disk through
the shared predicate -- not from the session's own
counters, which can only say how much work was done
sources policy what that one decision did and did not unlock
docs/VERIFYING.md first screen: both decisions are required
the site a zero attempt denominator is no longer rendered as
"attempted 0 of 0 eligible sources; 0 successful
retrievals", which is arithmetically true and reads
as a run that had nothing to do and did it perfectly.
Completeness over an empty denominator is not a
number and is reported as not measurable.
the site headline a registry where every source is verified and none is
attempt-eligible no longer leads with "All 152
sources are human-verified" -- the strongest
verification claim the page can make, over a feed
that will never fill
Not changed: the committed registry. Nothing here verifies a source or grants a
fetch policy; those are 152 human decisions and they remain undone.
make verify: 7/7 green. 431 tests pass, mypy --strict clean, coverage >= 90%.
Closes #18
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ENsaeU8RwLEBEwTN2Nevzx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #18.
verify.confirm()was the only writer of a verification in the codebase, and it wrotestatus,verifier,atandnote.core/eligibility.pyalso requires an evidence reference and an in-date recheck expiry. Nothing insrc/could write a fetch-policy decision at all — elevenfetch_policyhits: parsed, read, published, never written. So the queue the README calls "the most valuable three and a half hours anyone could spend on this repo" could be worked to completion with no effect on what the tool watches, and nothing said so.Measured, one source, scratch registry
The one reason left is the one that is genuinely a different person's judgment about somebody else's terms — and it now has a writer.
Two of the four fields are produced, not demanded
The evidence reference points at a receipt written at the moment of the decision: URL, fetch time, HTTP status, the page's own
<title>, the excerpt the verifier read, and the hash of the bytes behind it. That is what evidence means here. Asking a volunteer to type a path to a file they have not made produces a plausible string and no receipt.A source our crawler cannot fetch gets a receipt saying exactly that, claiming no title and no text: the honest evidence for
ssa.govis "we could not see it; a person opened it themselves", and an empty title under a confident statement would read as a page we saw and found blank.Receipts are written to
var/evidence/verification/— untracked.docs/05-DATA-AND-EVIDENCE.mdsays raw evidence is never automatically public, and a receipt carries an excerpt of whatever the page happened to be serving. This is a judgment call and it is yours to overturn:--evidence-dir sources/evidencemakes them committed and auditable by anyone, at the cost of publishing page excerpts by default.The expiry is dated 180 days forward from the decision. That number is a starting guess and is commented as one, in the same voice as
REMOVAL_THRESHOLD: too long and a verification is a permanent claim about a page that can be reorganised at any time; too short and the queue outruns the people working it. Also yours to change — it is a module constant and a--expiresflag.The second writer
sentinel sources policyrecords the dated robots/terms decisionSRC-03specifies — outcome, reviewer, date, evidence, reason, expiry — and refuses blanks, and refusesunreviewed(the absence of the decision, not one of its outcomes).It does not make the decision and cannot. Whether a host's terms permit a weekly watch is a reading of somebody else's document;
HttpFetcherobeying robots at request time is one rule, mechanically applied, at one moment, and is not that reading. So--evidencehere is required from the reviewer rather than generated: unlike a page excerpt, nothing in this tool has read these terms.What now says the thing nobody was told
verify --listverifysessionsources policydocs/VERIFYING.mdNot changed
The committed registry. Nothing here verifies a source or grants a fetch policy — those are 152 human decisions and they remain undone.
0 of 152 sources are human-verifiedstill holds, and the standing test that asserts it is untouched.Verification
make verify— 7/7 green. 431 tests, mypy--strictclean, coverage ≥ 90%,coverage --check-docsgreen across all six gated documents.Both new site tests fail on the parent commit, on the exact false sentence:
The eleven new
tests/test_verify.pytests cannot even import against the parent (record_fetch_policydoes not exist), which is the defect stated as a stack trace.Note for merge order
This branch and #22 (issue #19) both touch
src/id_churn_sentinel/core/site.py— #22 adds an unmeasured-retrievals clause to_run_status_section, this one adds the empty-denominator branch to the same function. Whichever merges second wants a rebase; the two changes are independent in intent.🤖 Generated with Claude Code
https://claude.ai/code/session_01ENsaeU8RwLEBEwTN2Nevzx