claim-to-source

Open the live page

Health journalism cites studies. Often the article's claim is not what the study found.

Point this at an article. It pulls out the health claims, resolves the studies they cite through Europe PMC, and reports, per claim, what the cited paper does and does not support, with the sentence from the source that each verdict rests on.

$ python3 -m claim_to_source fixtures/articles/relative-risk.html

[NOT SUPPORTED] c2  RISK_MAGNITUDE_OVERSTATED  confidence 0.85
    claim: "Pushing systolic blood pressure below 120 halves your risk of a heart attack or
    stroke, a landmark trial found (SPRINT, NEJM 2015)."
    source: A Randomized Trial of Intensive versus Standard Blood-Pressure Control
            The New England journal of medicine 2015 - rct - https://pubmed.ncbi.nlm.nih.gov/26551272/
    why: the article says 50% lower ('halves'); the largest same-direction estimate in the
    source is hazard ratio 0.73, which is 27%. That is 1.85x smaller than the article's figure
    source says: "All-cause mortality was also significantly lower in the intensive-treatment
    group (hazard ratio, 0.73; 95% CI, 0.60 to 0.90; P=0.003)."
    also: MISSING_ABSOLUTE_RISK: the article gives only the relative change (50% lower). The
    source reports a baseline of 2.19% per year (1 in 46). Applied to that baseline the claim
    means 1.095% per year, a change of 1.095 percentage points, or about 110 fewer people per
    10,000, one per 91 people. The source's own figures are 2.19% and 1.65% per year, a
    difference of 0.54 percentage points or one event per 185 people.
    confidence basis: arithmetic on two numbers, both quoted above
    not checked here: contradiction (the abstract states no negative result to contradict)

The distinction the whole thing is built around

A tool that prints "unsupported" because a fetch timed out is accusing a journalist of something it never checked. So there are three outcomes, not two:

	meaning	can it be an accusation?
SUPPORTED	the cited paper says this, and no named failure mode fired	no
NOT SUPPORTED	the tool read the source and found a specific, named mismatch	yes, and it always quotes the source
NOT CHECKED	the tool did not read the source	never

NOT CHECKED splits further, because an unreachable server, a paywalled paper, a DOI that resolves to nothing, and a sentence with no citation at all are four different situations: NO_CITATION, SOURCE_NOT_FOUND, FETCH_FAILED, NO_ABSTRACT, plus CLAIM_TOO_VAGUE and three model-specific reasons.

The type system enforces the separation rather than relying on care:

• Verdict.__bool__ raises TypeError. if verdict: is a bug, and it fails loudly.
• Reason codes for the three statuses are three disjoint sets, checked in __post_init__. Verdict(Status.NOT_SUPPORTED, "FETCH_FAILED", ...) raises.
• A NOT CHECKED verdict may not carry a misrepresentation finding, and must report confidence exactly 0.0.
• Every misrepresentation must carry a verbatim quote from the source. A Finding without one raises at construction, so the report can always show its evidence.

Exit codes follow the same logic. 1 means something is not supported. 2 means nothing is unsupported but something could not be checked. They are not interchangeable.

What it detects

Named failure modes, not a score. A score cannot be argued with.

code	what it means
SPECIES_MISMATCH	the study was in mice, rats, flies or yeast; the article does not say so
IN_VITRO_AS_HUMAN	the study treated cells in culture; the article does not say so
CAUSAL_FROM_OBSERVATIONAL	the source reports an association, the article states cause, unhedged
RISK_MAGNITUDE_OVERSTATED	the article's relative effect is larger than any the source reports
DIRECTION_CONTRADICTED	the source says it found no effect, the article says it found one
NOT_IN_SOURCE	the cited paper does not discuss the claim at all
MISSING_ABSOLUTE_RISK	a relative change with no absolute risk anywhere in the article
UNDISCLOSED_SAMPLE_SIZE	a study of 100 people or fewer, and the article never says how few
UNDISCLOSED_DURATION	a study of 90 days or less, and the article never says how short

The last three are omissions, not misrepresentations. They are reported but they never make a claim NOT SUPPORTED, because "you did not mention the sample size" is not "you made it up".

Absolute against relative, done properly

This is the highest-value single check and it is exactly computable whenever the source states the baseline. From a real run over relative-risk.html, which cites SPRINT:

what the article says	50% lower risk ("halves")
baseline in the source	2.19% per year (1 in 46)
what the article's number implies	1.095% per year
change implied, in points	1.095
people affected per 10,000	110
one event per	91 people
what the study itself found	2.19% to 1.65% per year, a difference of 0.54 points, one event per 185 people

The article's phrasing implies a benefit twice the size of the one the trial measured, and the number a reader actually needs, 0.54 percentage points a year, appears nowhere in it.

The corpus, and this run

Eight articles, written for this repository, each built around one named failure mode. The studies they cite are real papers, resolved from a committed Europe PMC cache: SPRINT (PMID 26551272), the NIH-AARP coffee cohort (22591295), Baur's resveratrol mouse study (17086191), a sulforaphane cell-line study (41819438), a null beetroot-juice trial (42148901), and a record with no abstract (31124288).

well-sourced.html is the control. Its four claims report their sources faithfully, so a tool that reaches good numbers by calling everything unsupported fails on it.

Regenerate this block with python3 scripts/build_page.py; scripts/verify.sh fails if it is stale.

causal-from-cohort.html       3 claims   1 supported   2 not supported   0 not checked   0 omissions  exit 1
contradicted.html             2 claims   0 supported   2 not supported   0 not checked   4 omissions  exit 1
fabricated-attribution.html   1 claims   0 supported   1 not supported   0 not checked   0 omissions  exit 1
in-vitro-as-human.html        2 claims   0 supported   2 not supported   0 not checked   0 omissions  exit 1
mouse-as-human.html           2 claims   0 supported   2 not supported   0 not checked   0 omissions  exit 1
relative-risk.html            2 claims   0 supported   2 not supported   0 not checked   2 omissions  exit 1
unchecked.html                4 claims   0 supported   0 not supported   4 not checked   0 omissions  exit 2
well-sourced.html             4 claims   4 supported   0 not supported   0 not checked   0 omissions  exit 0

TOTAL                        20 claims   5 supported  11 not supported   4 not checked   6 omissions

What a real article looks like

The eight fixture articles are written, and a written fixture only ever tests what its author already thought of. So the tool was pointed at six live NIH news releases, which are US government works in the public domain. One of them is committed at fixtures/real/nih-prediabetes-multimorbidity.html because of what it found.

$ python3 -m claim_to_source fixtures/real/nih-prediabetes-multimorbidity.html
10 claims: 2 supported, 0 not supported, 8 not checked, 0 omission(s) noted

Two things came out of that run, and neither would have come out of the written corpus.

A false accusation, at confidence 0.80. The release faithfully reports the lifestyle arm of the DPPOS follow-up. The abstract of that paper also contains the sentence "There was no difference between participants in the metformin and placebo groups", about an arm the release makes no claim about. detect_contradiction fired on any null sentence anywhere in an abstract, so it told a reader that an accurate press release contradicted its own source. The detector now requires the null sentence to share a distinctive term with the claim, and tests/test_real_article.py holds that fix in place with a control that proves the detector still fires on a claim that really is about the metformin arm.

Eight of ten claims came back NOT CHECKED, and that is the honest answer. A press release puts its reference in a line at the foot of the page, not beside each sentence, and the proximity rule will not reach that far. A tool willing to guess would have attributed ten claims to one paper and judged them all. The distinction this project is built around is what stops that: eight sentences the tool could not check are reported as eight sentences the tool could not check.

The run also extracted institutional boilerplate ("About the National Institute on Aging (NIA): NIA seeks to understand the nature of aging...") as a health claim. There is now a narrow boilerplate filter, tested alongside a control that a real sentence starting with "About half of participants" is still extracted.

Using it

python3 -m claim_to_source article.html               # offline, cached sources only
python3 -m claim_to_source --network --write-cache article.html
python3 -m claim_to_source --url https://example.org/story --network
python3 -m claim_to_source --json article.html        # machine-readable, same data

Source lookups are off by default. With no --network, a source that is not in the cache is reported FETCH_FAILED, which is the honest outcome: the tool did not read it.

The model backend

The support check has two backends. lexical is the default and is deterministic: after stemming and a small synonym table, does the source share any distinctive vocabulary with the claim? model asks a local Ollama model the same question.

python3 -m claim_to_source --backend model --model-network article.html
python3 scripts/record_model_answers.py --model gemma4:26b-a4b-it-q4_K_M

The model is fenced in on both sides:

• its supporting quote must appear verbatim in the abstract, or the result becomes NOT CHECKED, so it cannot invent evidence;
• it may not return NOT_IN_SOURCE unless the lexical check independently agrees. A model may confirm an accusation or downgrade a result. It may never originate an accusation.

An empty response is never a label. Reasoning models can spend the entire token budget thinking and return message.content == "" with done_reason: "length". Measured here on /api/chat with gemma4:26b-a4b-it-q4_K_M and num_predict 120:

request	content	eval_count	done_reason
"think": false	32 chars of JSON	15	stop
"think": "low"	0 chars, 398 chars of thinking	120	length
think omitted	0 chars, 398 chars of thinking	120	length

So the retry ladder starts with thinking off, and keeps a "low" rung at the end where it exercises the gate rather than the answer. If every rung comes back empty, judge raises ModelEmptyResponse, the claim becomes NOT CHECKED / MODEL_EMPTY_RESPONSE, and the CLI prints a warning on stderr. It is never scored.

Over the 16 checkable fixture claims, gemma4 recorded 11 SUPPORTED, 3 NOT_IN_SOURCE and 2 UNCLEAR, with 0 empty responses and 0 retries. All three of its NOT_IN_SOURCE answers were wrong (the beetroot trial plainly does discuss sprint performance); the lexical corroboration rule caught two of them and turned them into NOT CHECKED instead of false accusations. The article-level verdicts are identical under both backends, because the named detectors, which use no model at all, decide those cases first.

What it cannot check

Stated plainly, because a checker that hides its limits is worse than no checker.

• It reads abstracts, not full texts. A trial whose abstract omits its sample size is reported as "the abstract does not state a sample size", and that check is listed as one that could not run. It is never reported as a study with no sample size.
• Sources it cannot reach are not evidence. Paywalled, unresolvable and unreachable all come back NOT CHECKED.
• Claim extraction is a rule engine. A health claim phrased in a way the rules do not recognise is never judged, and its absence from the report is not a pass. The independent checker recounts claims with different rules, which is how a large extraction gap would surface.
• Attribution is by proximity, plus two narrow inheritance rules: a headline inherits the body's citation only when the body cites exactly one source, and a sentence that points back explicitly ("the same trial", "the authors") inherits the previous citation. A claim whose citation sits further away is reported uncited, not misattributed.
• NOT_IN_SOURCE is deliberately conservative. It fires only when no distinctive term from the claim appears anywhere in the source. Real fabrications that reuse the source's vocabulary will be missed. Missing one is a smaller failure than telling a reader an honest sentence was invented.
• Synonyms are a hand-written table. Journalism says "heart attack", abstracts say "myocardial infarction"; a pair not in entail.SYNONYM_ROWS can produce a spurious NOT_IN_SOURCE.
• A multi-group sample size is reported as the largest single group. The coffee cohort abstract says "229,119 men and 173,141 women"; the tool reports 229,119 rather than summing two numbers it has no grounds to add. So the sample size it prints is a floor, not a total.
• Ratio extraction is regex over prose. The magnitude check only ever accuses when the article's figure exceeds the source's largest same-direction estimate, so a missed ratio makes the check silent rather than wrong. A ratio picked up from the wrong subgroup would still weaken it.
• It judges the article against the study, not the study. A faithfully reported bad study comes out SUPPORTED.
• It is not medical advice, and a SUPPORTED verdict is not an endorsement of the underlying science.

Verification

bash scripts/verify.sh is the whole thing. It exits nonzero on any failure and does not skip sections quietly.

• Unit tests, hermetic. The source store is built with RefusingFetcher, which raises on every call, so a cache miss inside the suite is a hard error rather than a slow success. A green suite can never be the network's doing.
• Every fixture checked against fixtures/expected.json, which was written from the design of each article. Every positive assertion is paired with a negative control naming a verdict that must not be reached.
• scripts/independent_check.py recounts claims, study designs, sample sizes, ratio estimates and the whole absolute-risk arithmetic from the raw HTML and the raw cache JSON, importing nothing from claim_to_source. On its first run it disagreed about two hazard ratios; the bug was in the checker, whose naive <[^>]+> had swallowed the text between a bare P<0.001 and the next real tag. The tool was right. That is what an independent check is for.
• scripts/sabotage.py breaks the implementation in specific ways, on a throwaway copy, and requires the suite to notice each one. It prints the changed output first, so a sabotage that turned out to be a no-op cannot be mistaken for a gap in the checks.
• A live Europe PMC lookup and a live Ollama call, both of which must run. If either is unreachable, verify fails and names the environment variable that says so deliberately.

Layout

claim_to_source/
  verdict.py    Status, Finding, Verdict, Summary; the anti-collapse invariants
  sources.py    Europe PMC lookup, the on-disk cache, the four resolution outcomes
  evidence.py   reading an abstract: species, design, sample size, duration, effect sizes
  numbers.py    risk arithmetic; the only place a percentage is computed
  article.py    HTML to paragraphs, sentences, citations
  claims.py     which sentences are claims, and which source each one points at
  detect.py     the named failure modes
  entail.py     lexical and model support checks
  llm.py        Ollama client, the empty-response gate
  check.py      the pipeline
  report.py     text and JSON output
scripts/
  verify.sh              the verify command
  independent_check.py   a recount that shares no code with the above
  sabotage.py            proves the checks have teeth
  build_page.py          regenerates docs/index.html and this README's corpus block
  check_page.mjs         loads the page in a real browser
  record_model_answers.py
  splice_status.py       pastes a real verify run into the README

Status

$ bash scripts/verify.sh
1. unit tests, hermetic
Ran 104 tests in 0.362s

OK
  ok    unit tests
  ok    104 tests ran, which is the number this script expects

2. end to end exit codes, replayed from the committed Europe PMC cache
  ok    a mouse study reported as a human result (exit 1)
  ok    cells in a dish reported as a human result (exit 1)
  ok    an association reported as cause (exit 1)
  ok    an overstated relative risk (exit 1)
  ok    a null trial reported as positive (exit 1)
  ok    a claim attributed to a study that never mentions it (exit 1)
  ok    four claims that cannot be checked are exit 2, not exit 1 (exit 2)
  ok    the faithful control article passes (exit 0)
  ok    a missing article file is a tool error (exit 3)

2b. a real published article, offline, from the committed snapshot
  ok    the NIH release is exit 2 (nothing unsupported, most claims uncited) (exit 2)
  ok    no false DIRECTION_CONTRADICTED on the real article

3. the corpus is not one-sided
  {'SUPPORTED': 5, 'NOT SUPPORTED': 11, 'NOT CHECKED': 4}
  ok    all three statuses appear, including supported claims

4. the model backend, replayed from fixtures/llm-cache with the model unreachable
  ok    model backend: the mouse article (exit 1)
  ok    model backend: the control article (exit 0)
  ok    model backend: the unchecked article (exit 2)

5. the independent recount (shares no code with claim_to_source)
  ok    the independent checker imports nothing from claim_to_source
88 independent checks, 0 disagreement(s)
  ok    independent recount agrees

6. the published page and the README regenerate
docs/index.html and the README corpus block match a fresh run
  ok    docs/index.html and the README corpus block match a fresh run
  ok    README has a Status section
  ok    the README's Status section carries this script's own success line
  ok    no placeholder text in the README's prose

7. no secrets, no absolute home paths, no unreadable files
  ok    no tracked file carries a NUL or other control byte, so the text scans below can read every one
  ok    no absolute home paths in tracked files
  ok    no credential-shaped strings in tracked files
  ok    the credential scan catches all three bait patterns and does not fire on a URL slug
  ok    no tracked file over 1 MB

8. the page in a real browser
  ok    the port serves this project's page, verified by content and not by status code
  ok    doctype is first
  ok    charset declared
  ok    viewport declared
  ok    the inline script ran and rendered: 8 articles, 20 claims: 5 supported, 11 not supported, 4 not checked, 6 omissions noted
  ok    #sources rendered: 5 cited studies resolved
  ok    8 article blocks rendered
  ok    20 claim blocks rendered
  ok    status tags on the page match the run: {"SUPPORTED":5,"NOT SUPPORTED":11,"NOT CHECKED":4}
  ok    the caveats and the three-state legend are on the page
  ok    every claim states its status as a word, not only a colour
  ok    border style also varies by status: solid, double, dashed
  ok    nothing hides horizontal overflow, so this probe is not vacuous
  ok    no horizontal body scroll at 390px (390 <= 390)
  ok    no element escapes the viewport at 390px
  ok    no element's content overflows its own box outside a scroll container
  ok    all 1 table(s) sit in an overflow-x container (1 actually scroll at 390px)
  ok    prefers-color-scheme: light -> toggle changes background rgb(251, 250, 247) to rgb(20, 22, 26)
  ok    prefers-color-scheme: light -> toggling back restores it
  ok    prefers-color-scheme: light -> data-theme="dark" overrides the media query
  ok    prefers-color-scheme: dark -> toggle changes background rgb(20, 22, 26) to rgb(251, 250, 247)
  ok    prefers-color-scheme: dark -> toggling back restores it
  ok    prefers-color-scheme: dark -> data-theme="light" overrides the media query
  ok    no requests to any remote host
  ok    no console errors through load and theme toggle
BROWSER PASS
  ok    browser check

9. live Europe PMC
  live: SPRINT resolved from the network with n=9361, HR 0.75, 2.19%/yr baseline
  ok    live Europe PMC lookup, plus a negative control that resolves to nothing

10. live ollama, including the empty-response gate against a real model
  live: gemma4:26b-a4b-it-q4_K_M answered 'SUPPORTED' in 1 attempt(s)
  live: starved of tokens the model returned [0, 0, 0] chars with done_reason ['length', 'length', 'length'], and the gate raised instead of scoring it
  ok    live ollama: a real answer, and the empty-response gate proved on a real model

11. sabotage: the checks above must be able to fail
--- attack: species detector always returns nothing
  PASS  the sabotage was caught
--- attack: NOT CHECKED counted as NOT SUPPORTED
  PASS  the sabotage was caught
--- attack: absolute risk arithmetic drops the multiplication
  PASS  the sabotage was caught
--- attack: support check accuses every claim of not being in its source
  PASS  the sabotage was caught
--- attack: a failed fetch is reported as a source that does not exist
  PASS  the sabotage was caught
--- attack: Verdict gets a truth value again
  PASS  the sabotage was caught
--- attack: the contradiction detector stops requiring topical overlap
  PASS  the sabotage was caught
--- attack: the page stops regenerating from a real run
  PASS  the sabotage was caught
8 attacks, 0 problem(s)
  ok    every sabotage changed real output and every one was caught

VERIFY OK

Licence

MIT. Source metadata comes from Europe PMC. Not medical advice.

Part of 722 things to build.