From b498b9e0094f7f9ba6d82e19e08b27c9f85e3f10 Mon Sep 17 00:00:00 2001 From: Ken Okamoto Date: Thu, 13 Aug 2026 14:54:39 +0900 Subject: [PATCH] Make fact-check verification stage-gated with four loops. Add a claim ledger, engine/semantic/adversarial/correction loops (cap 3), and a mandatory red-team (KILL/MAJOR/MINOR/PASS) so the skill cannot score A while fabricated citations or unsafe advice remain. --- skills/medical-fact-check/SKILL.md | 148 ++++++++++++------ .../references/adversarial-review.md | 80 ++++++++++ .../references/checklist.md | 4 +- .../references/verification-workflow.md | 136 ++++++++++++++++ .../templates/claim-ledger.md | 24 +++ .../templates/report-template.md | 85 ++++++++-- 6 files changed, 414 insertions(+), 63 deletions(-) create mode 100644 skills/medical-fact-check/references/adversarial-review.md create mode 100644 skills/medical-fact-check/references/verification-workflow.md create mode 100644 skills/medical-fact-check/templates/claim-ledger.md diff --git a/skills/medical-fact-check/SKILL.md b/skills/medical-fact-check/SKILL.md index 86a5576..37b8b4b 100644 --- a/skills/medical-fact-check/SKILL.md +++ b/skills/medical-fact-check/SKILL.md @@ -7,21 +7,11 @@ description: "Comprehensive medical fact-checking and critical appraisal skill. Comprehensive critical appraisal and fact-checking for medical information, producing a structured report. -> **Scope.** This is a pre-publication aid for writers, editors, and researchers — **not clinical decision support.** It evaluates how medical *content* is written and sourced; it does not diagnose, treat, or replace professional medical judgment. - -> **Reference files.** This skill bundles three reference files. Read them with paths -> relative to this skill's directory: `references/checklist.md`, `references/evidence-levels.md`, -> and `templates/report-template.md`. Depending on how the skill was installed, that -> directory is `~/.claude/skills/medical-fact-check/` (manual copy) or -> `${CLAUDE_PLUGIN_ROOT}/skills/medical-fact-check/` (installed as the Evidentia plugin). -> -> **Deterministic citation engine.** When checking citations (Step 4), prefer the local -> `evidentia` engine over verifying each identifier by hand. Use the `evidentia` -> binary first; fall back to `npx -y evidentia` only if a local install is unavailable. -> Evidentia resolves DOI/PMID/arXiv/NCT identifiers against CrossRef, PubMed, -> OpenAlex, arXiv, and ClinicalTrials.gov, and emits the 4-tier classification plus -> `lookupVerified` and `resolverOutcomes` lookup traces. Use a cache path when possible -> so repeat checks are stable and fast. See **Step 4** for how to call it. +> **Scope.** This is a pre-publication aid for writers, editors, and researchers — **not clinical decision support.** It evaluates how medical *content* is written and sourced; it does not diagnose, treat, or replace professional medical judgment. Do not recommend treatments. + +> **Must-read before scoring.** Use the `Read` tool on `references/verification-workflow.md` (operating model, loops, hard rules) and `references/adversarial-review.md` (red-team, KILL/MAJOR/MINOR/PASS) before you emit a letter grade. Also load `references/checklist.md`, `references/evidence-levels.md`, `templates/claim-ledger.md`, and `templates/report-template.md`. Depending on install, the skill directory is `~/.claude/skills/medical-fact-check/` (manual copy) or `${CLAUDE_PLUGIN_ROOT}/skills/medical-fact-check/` (Evidentia plugin). + +> **Deterministic citation engine.** When checking citations (Step 4), prefer the local `evidentia` engine over verifying each identifier by hand. Use the `evidentia` binary first; fall back to `npx -y evidentia` only if a local install is unavailable. Evidentia resolves DOI/PMID/arXiv/NCT identifiers against CrossRef, PubMed, OpenAlex, arXiv, and ClinicalTrials.gov, and emits the 4-tier classification plus `lookupVerified` and `resolverOutcomes` lookup traces. Use a cache path when possible so repeat checks are stable and fast. See **Step 4** for how to call it. ## Overview @@ -76,8 +66,32 @@ Each item is rated **Excellent / Good / Fair / Poor**. The overall score: | **D** | 3+ Poor | | **F** | 5+ Poor, or critical ethical issues | +**Adversarial gates (mandatory).** A **KILL** forces overall score ≤ D (use **F** if ethics, harm, or fabrication). A **MAJOR** cannot be an A. Do not emit a clean A as if the piece is publishable when the verdict is KILL or MAJOR. + +## Two layers + +- **Engine (deterministic):** DOI/PMID/arXiv/NCT vs CrossRef/PubMed/OpenAlex/arXiv/ClinicalTrials.gov. Tiers 1/3/4 with certainty. ISBN/guideline/title-only → Tier 2 (**Content review needed**), never Hallucination. Never invent a tier if the engine was not run. Never override Tier 4 to "probably real." +- **Skill (judgment):** claim ledger, semantic honesty, 15-criteria appraisal, adversarial red-team. The engine cannot tell you whether a real paper is used honestly. + +Named loops (detail in `references/verification-workflow.md`): **engine** (retry once → `unresolved`, not Hallucination), **semantic** (one extra abstract lookup per T1 cite), **adversarial** (max 3; KILL/MAJOR → content must change; re-enter from the engine), **correction** (cap 3; then stop and report remaining issues). + +```mermaid +flowchart TD + A[Acquire content] --> B[Extract claim ledger] + B --> C[Run evidentia engine] + C --> D[Semantic honesty check] + D --> E[15-criteria appraisal] + E --> F[Adversarial red-team] + F --> G[Score plus report] + G --> H{User revises?} + H -->|yes: max 3| C + H -->|no or cap| I[Stop] +``` + ## Workflow +Stage-gated. Do not skip the ledger, the engine, or adversarial review. + ### Step 1: Acquire & Analyze Receive the target medical content from the user. Depending on the input format: @@ -125,6 +139,12 @@ Adjust the evaluation lens based on detected media type: - Check for "confident but wrong" patterns typical of LLM output - Verify all specific numbers, dates, and named entities +### Step 1.5: Claim ledger (mandatory) + +Read `templates/claim-ledger.md` and extract **every testable claim** plus its attached citation (or `none`) into the table: `#`, claim (verbatim), citation/id, engine tier, semantic (`supports` / `cherry-pick` / `mismatch` / `n/a`), adversarial note. + +Do **not** skip this even for short social posts (1–3 claims). Headlines count. Leave engine tier / semantic blank until Steps 4 and 4b fill them. Do not score from vibes. + ### Step 2: Load Evaluation Checklist Read `references/checklist.md` (in this skill's directory — see the note at the top) with the `Read` tool to load the detailed 15-item evaluation checklist. @@ -138,40 +158,44 @@ If the content references research studies, read `references/evidence-levels.md` - GRADE assessment for overall quality - Domain-specific considerations (pediatrics, oncology, etc.) -### Step 4: Verify Citations +### Step 4: Verify Citations (engine loop) If the content cites papers or sources, verify them. **Prefer the deterministic engine** for existence and bibliographic checks, then use `WebSearch` for the semantic context check that the engine cannot do. +Engine output is **ground truth for existence**. The LLM must not override Tier 4 to "probably real." Never invent a tier if the engine was not run. + #### 4a. Run the deterministic engine (existence + bibliographic accuracy) -If `evidentia` (or the `verify_citations` MCP tool) is available, run it on the content first. It resolves DOI/PMID/arXiv/NCT identifiers against CrossRef, PubMed, OpenAlex, arXiv, and ClinicalTrials.gov and returns Tiers 1, 3, and 4 with certainty - no model guesswork. Books (ISBN), guidelines, title-only citations, and other non-indexed sources are returned as Tier 2 (**Content review needed**), never as fabrications: +If `evidentia` (or the `verify_citations` MCP tool) is available, run it on the content first. It resolves DOI/PMID/arXiv/NCT identifiers against CrossRef, PubMed, OpenAlex, arXiv, and ClinicalTrials.gov and returns Tiers 1, 3, and 4 with certainty — no model guesswork. Books (ISBN), guidelines, title-only citations, and other non-indexed sources are returned as Tier 2 (**Content review needed**), never as fabrications. -```bash -evidentia check --format json --cache "$HOME/.cache/evidentia/verification-cache.json" --mailto -``` +Preferred command (local binary): -If the local binary is unavailable, fall back to the npm package: + evidentia check --format json --cache "$HOME/.cache/evidentia/verification-cache.json" --mailto -```bash -npx -y evidentia check --format json --cache "$HOME/.cache/evidentia/verification-cache.json" --mailto -``` +Fallback if the local binary is unavailable: -Use its output as the ground truth for citation *existence*. Inspect `lookupVerified` and `resolverOutcomes` when explaining why a citation was classified: + npx -y evidentia check --format json --cache "$HOME/.cache/evidentia/verification-cache.json" --mailto + +**Engine loop:** if unreachable, retry **once**. If still down, mark those citations `unresolved` (not Hallucination) and continue. Never guess a Hallucination without a failed identifier lookup. + +Use its output as the ground truth for citation *existence*. Inspect `lookupVerified` and `resolverOutcomes` when explaining why a citation was classified. Write the engine tier into the claim ledger. - **Tier 1 (Verified)** — the paper, preprint, or trial exists and metadata matches. Proceed to the context check in 4b. - **Tier 2 (Content review needed)** — the source may be real, but the engine cannot deterministically verify it in registries, or semantic use still needs review. - **Tier 3 (Bibliographic mismatch)** — a real record exists, but the DOI/PMID/arXiv/NCT identifier or metadata is wrong. Record the discrepancy. - **Tier 4 (Hallucination)** — the identifier resolves to nothing or to a different paper. Flag as a fabricated citation immediately; this is the highest-severity finding. -If the engine is not available, fall back to verifying each identifier manually with `WebSearch` (steps below). +If the engine is not available, fall back to verifying each identifier manually with `WebSearch` (steps below). Still do not invent a Hallucination for ISBN/guideline/title-only sources. + +#### 4b. Semantic loop (honesty — the engine cannot do this) -#### 4b. Semantic context check (Tier 2 — the engine cannot do this) +For every citation the engine marked **Verified** (Tier 1), still confirm it is used honestly: -For every citation the engine marked **Verified**, still confirm it is used honestly: +1. Fetch the abstract (WebSearch / WebFetch) and cross-check it against the cited claim — primary outcome, population, direction of effect. +2. Evaluate context — is the citation cherry-picked or accurately represented? +3. Downgrade to **Tier 2 (Content review needed)** if a real paper is being misrepresented or cited out of context. Set semantic to `mismatch` or `cherry-pick`. -1. **Cross-check** the abstract or full text against the cited claim -2. **Evaluate context** — is the citation cherry-picked or accurately represented? -3. Downgrade to **Tier 2 (Content review needed)** if a real paper is being misrepresented or cited out of context. +**Semantic loop cap:** one extra lookup per citation, then stop. Paywalled with no abstract → semantic `n/a`, note "abstract unavailable," leave the engine tier in place. Do not use this loop to upgrade a Tier 3 or 4. #### Manual fallback (if the engine is unavailable) @@ -187,7 +211,7 @@ AI-generated text (ChatGPT, Claude, Gemini, etc.) frequently contains plausible - Does the author actually exist and publish in this field? - Do the journal name, volume, and page numbers match a real publication? -**Do NOT stop at "could not verify."** Actively determine whether the citation is unverifiable or provably fabricated. +**Do NOT stop at "could not verify."** Actively determine whether the citation is unverifiable or provably fabricated. Engine-down is `unresolved`, not Hallucination. Classify each citation into one of 4 tiers: @@ -207,6 +231,8 @@ Rate each of the 15 items using these dimensions: - **Suggestions**: concrete, actionable improvements (if issues exist) - **Rating**: Excellent / Good / Fair / Poor +Do not recommend treatments. Suggestions are about how the *content* should be rewritten (cite the primary paper, add ARR, hedge the causal verb) — not about what a patient or clinician should take. + #### Media-Specific Evaluation Adjustments | Criterion | Social Media | Marketing | Guidelines | Patient Materials | @@ -219,9 +245,20 @@ Rate each of the 15 items using these dimensions: | #10 Ethics | Check stigma/fear | Check manipulation | Check COI panel | Check dignity/autonomy | | #12 Images | Memes, infographics | Selective visuals | Evidence figures | Clear illustrations | +### Step 5.5: Adversarial review (mandatory) + +Read `references/adversarial-review.md` with the `Read` tool. Run the five lenses (citation integrity, claim support, statistics and language, harm, steelman-then-attack). Answer the 10-line attack checklist (yes/no + evidence). Emit **KILL / MAJOR / MINOR / PASS**. + +- **KILL** — any Tier 4 presented as real, or advice that could cause harm if followed. Content must not be published as-is. Overall score ≤ D (F if ethics/harm/fabrication). This is the system working. +- **MAJOR** — real sources, dishonest use, causal overclaim, missing fair balance. Must fix before publish. Cannot be an A. +- **MINOR** — hedging, currency, readability. Should fix. +- **PASS** — ship with stated caveats. Human still owns publish. + +KILL or MAJOR: do **not** emit a clean A-score as if publishable; tell the user the content must change. If they revise, re-enter from the **engine** (Step 4), not from scoring. Max **3** adversarial passes per document. + ### Step 6: Determine Overall Score -Aggregate the 15 item ratings into an A–F score using the criteria table in the Overview section. +Aggregate the 15 item ratings into an A–F score using the criteria table in the Overview section, then apply the adversarial gates above. Additionally, flag a **Public Health Risk Assessment**: @@ -235,14 +272,16 @@ Read the report template from `templates/report-template.md` (in this skill's di **Required sections:** 1. Content Overview — title, source, audience, date, media type -2. Overall Assessment — score, key issues summary, risk level, recommended actions +2. Overall Assessment — score, **adversarial verdict**, key issues summary, risk level, recommended actions 3. Detailed Evaluation — all 15 items with ratings, issues, and suggestions -4. Citation Verification Results — tier classification for each citation (if applicable) -5. Critical Concerns — flagged high-severity issues -6. Strengths — positive aspects worth noting -7. Suggested Corrections — before/after comparison text (if issues found) -8. References — sources used during evaluation -9. Evaluator Notes — overall commentary and caveats +4. Citation Verification Results — engine vs semantic columns; paste or summarize engine JSON; use **Content review needed** (the previous mismatch wording is retired) +5. Adversarial review — lenses, steelman, attack, checklist, verdict +6. Loop log — engine runs, semantic lookups, adversarial pass #, remaining issues +7. Critical Concerns — flagged high-severity issues +8. Strengths — positive aspects worth noting +9. Suggested Corrections — before/after comparison text (if issues found) +10. References — sources used during evaluation +11. Evaluator Notes — overall commentary and caveats ### Step 8: Deliver Report @@ -253,19 +292,23 @@ Save the completed report as a Markdown file using `Write`: - Provide the user with: - The file path - A concise summary of findings (3–5 sentences) - - The overall score and risk level + - The overall score, adversarial verdict, and risk level - Top 3 most important issues to address -### Step 9: Post-Correction Re-Verification (Optional) +### Step 9: Correction loop (up to 3) + +Not optional. If the user revises the content based on the report: -If the user revises the content based on the report and requests re-evaluation: +1. Re-read the revised content. +2. Re-run the **engine** on the document (at least every changed or new citation). +3. Re-run the **semantic** loop on changed Tier 1 citations. +4. Re-run **adversarial** review (counts toward the 3-pass cap). Re-enter from the engine, not from scoring. +5. Update the claim ledger, loop log, and recommended-actions checklist. +6. Check that corrections have not introduced new problems (shifted reference numbers, new causal verbs). +7. List remaining unresolved issues. +8. Save the updated report with a `-rev2` (or `-rev3`) suffix. -1. Re-read the revised content -2. Check that flagged issues have been properly addressed -3. Update the recommended-actions checklist (mark resolved items) -4. Verify that corrections haven't introduced new problems (e.g., shifted reference numbers) -5. List any remaining unresolved issues -6. Save the updated report with a `-rev2` (or `-rev3`, etc.) suffix +**Cap: 3.** Then stop and report what is still open. Do not raise the letter grade while a KILL/MAJOR or a new Tier 4 remains. ## Media-Specific Handling @@ -343,6 +386,7 @@ Patient materials prioritize accessibility and safety: 5. **Consider the audience** — evaluation standards differ for professional vs. public content 6. **Stay practical** — improvement suggestions should be realistic and actionable 7. **Disclose limitations** — acknowledge what this AI-based review can and cannot verify +8. **Record the loops** — engine JSON, semantic lookups, adversarial pass number, remaining issues belong in the report, not only in your head ## Caveats @@ -351,12 +395,16 @@ Patient materials prioritize accessibility and safety: 3. **Image evaluation is limited** — cannot deeply analyze embedded figures or video content 4. **Rapidly evolving fields** — the most current evidence may not yet be indexed 5. **Final medical decisions** should always be made by qualified healthcare professionals +6. **Not CDS** — do not recommend treatments; the human publishes ## Reference Files +- `references/verification-workflow.md` — operating model, two layers, named loops, hard rules (must-read before scoring) +- `references/adversarial-review.md` — five lenses, attack checklist, KILL/MAJOR/MINOR/PASS (must-read before scoring) - `references/checklist.md` — detailed 15-item evaluation checklist - `references/evidence-levels.md` — evidence hierarchy & quality assessment tools -- `templates/report-template.md` — structured report template +- `templates/claim-ledger.md` — claim table filled before the engine call +- `templates/report-template.md` — structured report template (verdict, engine JSON, loop log) ## External References diff --git a/skills/medical-fact-check/references/adversarial-review.md b/skills/medical-fact-check/references/adversarial-review.md new file mode 100644 index 0000000..c45876b --- /dev/null +++ b/skills/medical-fact-check/references/adversarial-review.md @@ -0,0 +1,80 @@ +# Adversarial review + +Red-team for *content*, not a manuscript. Run this after the 15-criteria pass and before you emit a letter grade that implies the piece is publishable. + +**Purpose:** try to make the content fail before a reader, editor, or regulator does. + +This is not clinical decision support. Do not invent treatments. The AI recommends; the human publishes. Record the verdict in the report. + +## Five lenses + +Short and operational. Answer each with evidence from the claim ledger and the engine JSON, not with a vibe. + +### 1. Citation integrity + +Trust the engine. Look for swapped DOIs, invented PMIDs, "the author exists but this paper doesn't," title–identifier mismatches, and NCT IDs that resolve to a different trial. + +- Engine Tier 4 presented as a real source → this lens fails hard (KILL). +- Engine Tier 3 (bibliographic mismatch) presented as clean → at least MAJOR unless already disclosed in the draft. +- Do not "rescue" a Tier 4 because a similarly titled paper exists. + +### 2. Claim support + +A real paper can still be misquoted. Compare the sentence that cites it with the paper's **primary outcome**, population, and direction of effect. + +- Secondary endpoint or subgroup sold as the main finding → cherry-pick (semantic), usually MAJOR. +- Numeric claim with no abstract check on a Tier 1 cite → you have not finished this lens. Go back to the semantic loop (one extra lookup, then stop). + +### 3. Statistics and language + +Flag RRR without ARR, "causes" from observational data, "breakthrough" / "miracle" / "proven," and surrogate endpoints treated as clinical outcomes. p < 0.05 is not a license for causal verbs. + +### 4. Harm + +Could a patient or clinician act on this unsafely? Unapproved treatments framed as ready, a dose, an adult result applied to a child, a contraindicated population, or omitted safety-critical caveats in actionable copy. + +If following the text as written could cause harm, this is KILL — even when every citation is Tier 1. + +### 5. Steelman, then attack + +First write **one paragraph** stating the author's strongest case in their own terms (no sarcasm). Then write the strongest case that the piece is wrong or misleading. If the steelman cannot survive that attack, the verdict **cannot be PASS**. + +## Verdicts + +| Verdict | Meaning | Score consequence | +|---------|---------|-------------------| +| **KILL** | Fabricated citations (any Tier 4 presented as real) **or** advice that could cause harm if followed. Content must not be published as-is. This is the system working. | Overall ≤ D. Use **F** if ethics, harm, or fabrication. | +| **MAJOR** | Real sources, dishonest use: causal overclaim, missing fair balance, cherry-picked primary vs cited sentence. Must fix before publish. | Cannot be an **A**. | +| **MINOR** | Hedging, currency, readability, formatting. Should fix. | Grade as earned; list the fixes. | +| **PASS** | Ship with stated caveats. Human still owns publish. | Grade as earned. | + +KILL or MAJOR: do **not** emit a clean A-score as if publishable. Tell the user the content must change. If they revise, re-enter from the **engine**, not from scoring. Max **3** adversarial passes per document (see `references/verification-workflow.md`). + +## Attack checklist + +The agent must actually answer each line: **yes / no** plus a one-line evidence note. "n/a" is allowed only where a line cannot apply (e.g. no citations). Blank is a fail. + +1. Did I run the engine (`evidentia check --format json` or MCP `verify_citations`)? If not, did I mark citations `unresolved` instead of guessing a tier? +2. Did I refuse to override any engine Tier 4 to "probably real"? +3. Did I treat ISBN / guideline / title-only cites as Tier 2 (Content review needed), never Hallucination? +4. Did I read an abstract for each Tier 1 citation used to support a numeric or causal claim? (One extra lookup, then stop.) +5. Does every cited numeric claim match the paper's actual primary finding, population, and effect direction? +6. Is every causal verb earned (not observational data dressed as "causes")? +7. Is relative risk accompanied by absolute risk / NNT where a magnitude is claimed? +8. Could a patient or clinician act on this unsafely (unapproved treatment, dose, adult→child, omitted critical caveat)? +9. Did the steelman survive the attack, or am I grading a piece I already know is misleading? +10. Would I be embarrassed if this DOI 404'd in peer review — or if a regulator quoted this paragraph back at the author? + +All ten answered. Then pick KILL / MAJOR / MINOR / PASS and write it in the report. + +## How to record it + +In the report's **Adversarial review** section: + +- One short note per lens (pass / fail + evidence). +- Steelman paragraph. +- Attack paragraph. +- Checklist answers (the ten lines). +- Verdict, and whether this is adversarial pass 1, 2, or 3. + +Do not hide a KILL in "Evaluator notes." diff --git a/skills/medical-fact-check/references/checklist.md b/skills/medical-fact-check/references/checklist.md index a035cab..017ee73 100644 --- a/skills/medical-fact-check/references/checklist.md +++ b/skills/medical-fact-check/references/checklist.md @@ -48,12 +48,12 @@ AI-generated content (ChatGPT, Claude, Gemini, etc.) frequently contains plausib ### Citation Classification (4 Tiers) -Classify each citation verification result: +Classify each citation verification result. Run the engine first — see `references/verification-workflow.md` for the engine loop (retry once if unreachable; mark `unresolved`, not Hallucination, if it stays down; never override a returned Tier 4). | Tier | Classification | Description | |------|---------------|-------------| | 1 | **Verified** | Paper exists and content matches the citation | -| 2 | **Content mismatch** | Paper exists but is cited out of context or misrepresented | +| 2 | **Content review needed** | Covers both semantic misuse (real paper, wrong or cherry-picked claim) AND unindexed sources the engine cannot resolve in registries (ISBN, guidelines, title-only). Never treat those as Hallucination. | | 3 | **Bibliographic mismatch** | Paper exists but DOI, author, or journal info is incorrect | | 4 | **Hallucination** | DOI points to an unrelated paper, or the paper does not exist at all | diff --git a/skills/medical-fact-check/references/verification-workflow.md b/skills/medical-fact-check/references/verification-workflow.md new file mode 100644 index 0000000..5d3cb74 --- /dev/null +++ b/skills/medical-fact-check/references/verification-workflow.md @@ -0,0 +1,136 @@ +# Verification workflow + +Operating model for the medical-fact-check skill. Read this before scoring. This is a **content** fact-check before publish — not clinical decision support, and not a manuscript pipeline. Do not recommend treatments. + +## Pipeline + +```mermaid +flowchart TD + A[Acquire content] --> B[Extract claim ledger] + B --> C[Run evidentia engine] + C --> D{Engine reachable?} + D -->|no: retry once| C + D -->|still down| E[Mark citations unresolved] + D -->|yes| F[Tiers 1 / 3 / 4 plus T2 unindexed] + E --> G[Semantic honesty check] + F --> G + G --> H[15-criteria appraisal] + H --> I[Adversarial red-team] + I --> J{Verdict} + J -->|PASS or MINOR| K[Score plus report] + J -->|KILL or MAJOR| L[Do not ship as-is] + L --> M{User revises?} + K --> M + M -->|yes: max 3| C + M -->|no or cap| N[Stop and report remaining issues] +``` + +Acquire content → Extract claim ledger → Run evidentia engine (existence) → Semantic honesty check (abstract vs claim) → 15-criteria appraisal → Adversarial red-team → Score + report → Correction loop (if the user revises). + +## Two layers + +### Engine (deterministic) + +The `evidentia` CLI or MCP `verify_citations` resolves DOI / PMID / arXiv / NCT against CrossRef, PubMed, OpenAlex, arXiv, and ClinicalTrials.gov. It emits Tiers **1 / 3 / 4** with certainty, plus Tier **2** for sources it cannot index. + +- Engine output is **ground truth for existence**. +- Never invent a tier if the engine was not run. +- Never override Tier 4 to "probably real." +- ISBN, guidelines, title-only, and other unindexed sources are **Tier 2 (Content review needed)**, never Hallucination. +- If the engine is down after one retry, mark those citations `unresolved` — not Hallucination. A Hallucination label requires a failed identifier lookup, not a missing tool. + +### Skill (judgment) + +Context, cherry-picking, statistics, ethics, language, and adversarial review. The engine cannot tell you whether a real paper is being used honestly. That is the skill's job. + +This skill evaluates how medical *content* is written and sourced. It does not diagnose, treat, or replace professional medical judgment. + +## Claim ledger + +Fill `templates/claim-ledger.md` **before** the engine call, even for a 1–3 claim social post. Do not score from vibes. + +| Column | What goes in it | +|--------|-----------------| +| # | Sequential claim number | +| claim | Verbatim sentence or tight paraphrase of the testable assertion | +| citation / id | DOI, PMID, arXiv, NCT, or "none" | +| engine tier | 1 / 2 / 3 / 4 / unresolved / n/a (no identifier) | +| semantic | supports / cherry-pick / mismatch / n/a | +| adversarial note | Short flag from red-team, or blank | + +Every numeric, causal, or safety-relevant sentence is a claim. Headlines count. + +## Loops + +Four named loops. Each has a stop condition. "FAIL" means the gate did not clear — not that you invent a worse tier. + +### 1. Engine loop + +**Do:** run `evidentia check --format json` (or MCP `verify_citations`) with a cache path and `--mailto` when using the CLI. + +**FAIL:** engine unreachable or non-zero with no JSON. + +**Retry:** once. + +**Then:** if still down, mark affected citations `unresolved`. Continue the rest of the appraisal. Do **not** guess Hallucination without a failed identifier lookup. + +**Do not:** override a returned Tier 4 because the title "sounds real." Do not invent Tier 1 from WebSearch when the engine already said 3 or 4. Manual WebSearch is the fallback only when the engine is unavailable, and even then a 404/empty registry hit is Hallucination — a book, guideline, or title-only cite is still Tier 2. + +### 2. Semantic loop + +**Scope:** every Tier 1 citation that supports a claim in the ledger (especially numeric or causal claims). + +**Do:** fetch the abstract (WebSearch / WebFetch). Test: does the cited claim match the paper's actual finding — primary outcome, population, direction of effect? + +**FAIL:** abstract contradicts the claim, or the claim uses a secondary/subgroup finding as if it were the main result → downgrade that citation to **Tier 2 (Content review needed)** and set semantic to `mismatch` or `cherry-pick`. + +**Retry:** one extra lookup per citation, then stop. Paywalled with no abstract → semantic `n/a`, note "abstract unavailable," leave the engine tier in place. + +**Do not:** use the semantic loop to "upgrade" a Tier 3 or 4. Existence is the engine's job. + +### 3. Adversarial loop + +**When:** after the 15-criteria pass, before emitting a clean score. + +**Do:** Read `references/adversarial-review.md`. Run the five lenses and the attack checklist. Verdict: **KILL / MAJOR / MINOR / PASS**. + +**FAIL:** +- **KILL** or **MAJOR** — do not emit a clean A as if the piece is publishable. Tell the user the content must change. A KILL forces overall score ≤ D (F if ethics, harm, or fabrication). A MAJOR cannot be an A. +- **MINOR** — ship only with stated caveats; still list the fixes. +- **PASS** — score as earned; human still owns publish. + +**Retry:** if the user revises, re-enter from the **engine** (not from scoring). Max **3** adversarial passes per document. After 3, stop and report remaining issues. + +### 4. Correction loop + +Step 9 is a protocol, not an optional courtesy. + +**When:** the user revises the content after a report. + +**Do:** +1. Re-read the revised text. +2. Re-run the engine on the document (or at least on every changed / new citation). +3. Re-run the semantic loop on changed Tier 1 citations. +4. Re-run adversarial review (counts toward the 3-pass cap). +5. Update the ledger, loop log, and recommended-actions checklist. +6. Check that fixes did not introduce new problems (shifted reference numbers, new causal verbs). + +**Cap:** 3. Then stop and report what is still open. + +**FAIL:** a new Tier 4, a new KILL/MAJOR, or an unfixed previous KILL/MAJOR. Do not raise the letter grade while those remain. + +## Hard rules + +1. Engine output is ground truth for existence. The LLM must not override Tier 4 to "probably real." +2. Never invent a tier if the engine was not run. If it was not run, say so and use `unresolved`. +3. ISBN / guideline / title-only → Tier 2 (Content review needed), never Hallucination. +4. Semantic mismatch of a real paper is Tier 2, not Tier 4. +5. Not CDS. Do not recommend treatments, doses, or "what the patient should do." +6. The AI recommends; the human publishes. + +## What FAIL does *not* mean + +- Engine down ≠ Hallucination. +- Unindexed book ≠ Hallucination. +- Abstract unavailable ≠ mismatch. +- MINOR ≠ a hidden KILL. Record the actual verdict. diff --git a/skills/medical-fact-check/templates/claim-ledger.md b/skills/medical-fact-check/templates/claim-ledger.md new file mode 100644 index 0000000..2523811 --- /dev/null +++ b/skills/medical-fact-check/templates/claim-ledger.md @@ -0,0 +1,24 @@ +# Claim ledger + +Fill **before** the engine call and **before** scoring. Copy this table into the working notes (and into the report if useful). Do not skip it for short social posts — those still have 1–3 testable claims. + +**Content:** [title or one-line identifier] +**Date:** [YYYY-MM-DD] + +| # | claim (verbatim) | citation / id | engine tier | semantic | adversarial note | +|---|------------------|---------------|-------------|----------|------------------| +| 1 | | | | | | +| 2 | | | | | | +| 3 | | | | | | + +Add rows as needed. + +**Column values** + +- **claim (verbatim):** the testable assertion as written (or a tight quote). Headlines count. +- **citation / id:** DOI, PMID, arXiv id, NCT number, or `none`. +- **engine tier:** `1` / `2` / `3` / `4` / `unresolved` / `n/a`. Never invent a tier if the engine was not run. ISBN / guideline / title-only → `2`, never `4`. +- **semantic:** `supports` / `cherry-pick` / `mismatch` / `n/a` (no abstract, no identifier, or not a literature claim). +- **adversarial note:** short flag from red-team (`T4 presented as real`, `RRR only`, `adult→child`, or blank). + +Engine tier is existence. Semantic is honesty. Do not collapse them into one column. diff --git a/skills/medical-fact-check/templates/report-template.md b/skills/medical-fact-check/templates/report-template.md index 067aeee..5692a74 100644 --- a/skills/medical-fact-check/templates/report-template.md +++ b/skills/medical-fact-check/templates/report-template.md @@ -15,6 +15,8 @@ **Overall score**: [A / B / C / D / F] +**Adversarial verdict**: [KILL / MAJOR / MINOR / PASS] + **Public health risk level**: [LOW / MEDIUM / HIGH] **Key issues summary**: @@ -27,6 +29,8 @@ - [ ] Minor improvements suggested - [ ] No issues found +Scoring gates: a **KILL** forces overall score ≤ D (F if ethics, harm, or fabrication). A **MAJOR** cannot be an A. Do not emit a clean A if the verdict is KILL or MAJOR. + --- ## 3. Detailed Evaluation @@ -258,29 +262,88 @@ ## 4. Citation Verification Results -[Include this section when the content contains citations. Omit if no citations are present.] +[Include this section when the content contains citations. Omit if no citations are present. Engine tier is existence; semantic is honesty — do not collapse them.] + +| # | Citation | Engine tier | Semantic | Classification | Notes | +|---|----------|-------------|----------|----------------|-------| +| 1 | [Author, Year, Journal; DOI/PMID/NCT] | [1–4 / unresolved / n/a] | [supports / cherry-pick / mismatch / n/a] | [Verified / Content review needed / Bibliographic mismatch / Hallucination] | [Details; include lookupVerified / resolverOutcomes when useful] | + +**Engine JSON** (paste or summarize `evidentia check --format json`; never invent a tier if the engine was not run): + +```json +[paste engine output or write "engine not run — citations marked unresolved"] +``` + +**Summary**: [X] of [Y] citations verified. [Z] hallucinations detected. [N] marked Content review needed (semantic misuse and/or unindexed sources). [U] unresolved (engine unreachable). + +--- + +## 5. Adversarial review + +**Pass number**: [1 / 2 / 3 of 3] + +### Lenses + +1. **Citation integrity**: [pass / fail] — [evidence] +2. **Claim support**: [pass / fail] — [evidence] +3. **Statistics and language**: [pass / fail] — [evidence] +4. **Harm**: [pass / fail] — [evidence] +5. **Steelman then attack**: [steelman survived / did not] — [evidence] + +### Steelman + +[One paragraph: the author's strongest case, in their own terms, no sarcasm.] + +### Attack + +[The strongest case that the piece is wrong or misleading.] + +### Attack checklist + +| # | Question | Yes / No / n/a | Evidence | +|---|----------|----------------|----------| +| 1 | Did I run the engine? If not, marked unresolved rather than guessed a tier? | | | +| 2 | Refused to override any engine Tier 4 to "probably real"? | | | +| 3 | ISBN / guideline / title-only treated as Content review needed, never Hallucination? | | | +| 4 | Read an abstract for each T1 citation used to support a numeric or causal claim? | | | +| 5 | Every cited numeric claim matches the paper's primary finding, population, and effect direction? | | | +| 6 | Every causal verb earned? | | | +| 7 | Relative risk accompanied by absolute risk / NNT where a magnitude is claimed? | | | +| 8 | Could a patient or clinician act on this unsafely? | | | +| 9 | Did the steelman survive the attack? | | | +| 10 | Would I be embarrassed if this DOI 404'd in peer review? | | | + +**Verdict**: [KILL / MAJOR / MINOR / PASS] + +--- + +## 6. Loop log -| # | Citation | Tier | Classification | Notes | -|---|----------|------|---------------|-------| -| 1 | [Author, Year, Journal] | [1–4] | [Verified / Content mismatch / Bibliographic mismatch / Hallucination] | [Details] | +| Loop | Runs | Result | Remaining issues | +|------|------|--------|------------------| +| Engine | [1 or 2; command used] | [JSON received / unreachable → unresolved] | | +| Semantic | [N abstracts fetched; 1 extra lookup max per cite] | [supports / cherry-pick / mismatch / n/a counts] | | +| Adversarial | [pass # of 3] | [KILL / MAJOR / MINOR / PASS] | | +| Correction | [0–3] | [re-entered from engine / cap reached] | | -**Summary**: [X] of [Y] citations verified. [Z] hallucinations detected. +**Remaining issues after this pass**: +- [List, or "None"] --- -## 5. Critical Concerns +## 7. Critical Concerns [Flag any high-severity issues that require immediate attention. If none, state "No critical concerns identified."] --- -## 6. Strengths +## 8. Strengths [List positive aspects of the content that are worth noting.] --- -## 7. Suggested Corrections +## 9. Suggested Corrections [Provide before/after examples for the most important issues.] @@ -294,14 +357,14 @@ --- -## 8. References +## 10. References [List the guidelines, papers, and trusted sources used during this evaluation.] --- -## 9. Evaluator Notes +## 11. Evaluator Notes [Overall commentary, caveats about the evaluation, and any limitations of this AI-based review.] -**Disclaimer**: This report was generated by an AI-based evaluation tool and does not replace expert medical judgment. Final medical decisions should always be made by qualified healthcare professionals. +**Disclaimer**: This report was generated by an AI-based evaluation tool and does not replace expert medical judgment. Final medical decisions should always be made by qualified healthcare professionals. This is not clinical decision support.