Skill: claim ledger, four loops, mandatory adversarial review

skills/medical-fact-check/SKILL.md

@@ -7,21 +7,11 @@ description: "Comprehensive medical fact-checking and critical appraisal skill.
 
 Comprehensive critical appraisal and fact-checking for medical information, producing a structured report.
 
-> **Scope.** This is a pre-publication aid for writers, editors, and researchers — **not clinical decision support.** It evaluates how medical *content* is written and sourced; it does not diagnose, treat, or replace professional medical judgment.
-
-> **Reference files.** This skill bundles three reference files. Read them with paths
-> relative to this skill's directory: `references/checklist.md`, `references/evidence-levels.md`,
-> and `templates/report-template.md`. Depending on how the skill was installed, that
-> directory is `~/.claude/skills/medical-fact-check/` (manual copy) or
-> `${CLAUDE_PLUGIN_ROOT}/skills/medical-fact-check/` (installed as the Evidentia plugin).
->
-> **Deterministic citation engine.** When checking citations (Step 4), prefer the local
-> `evidentia` engine over verifying each identifier by hand. Use the `evidentia`
-> binary first; fall back to `npx -y evidentia` only if a local install is unavailable.
-> Evidentia resolves DOI/PMID/arXiv/NCT identifiers against CrossRef, PubMed,
-> OpenAlex, arXiv, and ClinicalTrials.gov, and emits the 4-tier classification plus
-> `lookupVerified` and `resolverOutcomes` lookup traces. Use a cache path when possible
-> so repeat checks are stable and fast. See **Step 4** for how to call it.
+> **Scope.** This is a pre-publication aid for writers, editors, and researchers — **not clinical decision support.** It evaluates how medical *content* is written and sourced; it does not diagnose, treat, or replace professional medical judgment. Do not recommend treatments.
+
+> **Must-read before scoring.** Use the `Read` tool on `references/verification-workflow.md` (operating model, loops, hard rules) and `references/adversarial-review.md` (red-team, KILL/MAJOR/MINOR/PASS) before you emit a letter grade. Also load `references/checklist.md`, `references/evidence-levels.md`, `templates/claim-ledger.md`, and `templates/report-template.md`. Depending on install, the skill directory is `~/.claude/skills/medical-fact-check/` (manual copy) or `${CLAUDE_PLUGIN_ROOT}/skills/medical-fact-check/` (Evidentia plugin).
+
+> **Deterministic citation engine.** When checking citations (Step 4), prefer the local `evidentia` engine over verifying each identifier by hand. Use the `evidentia` binary first; fall back to `npx -y evidentia` only if a local install is unavailable. Evidentia resolves DOI/PMID/arXiv/NCT identifiers against CrossRef, PubMed, OpenAlex, arXiv, and ClinicalTrials.gov, and emits the 4-tier classification plus `lookupVerified` and `resolverOutcomes` lookup traces. Use a cache path when possible so repeat checks are stable and fast. See **Step 4** for how to call it.
 
 ## Overview
 
@@ -76,8 +66,32 @@ Each item is rated **Excellent / Good / Fair / Poor**. The overall score:
 | **D** | 3+ Poor |
 | **F** | 5+ Poor, or critical ethical issues |
 
+**Adversarial gates (mandatory).** A **KILL** forces overall score ≤ D (use **F** if ethics, harm, or fabrication). A **MAJOR** cannot be an A. Do not emit a clean A as if the piece is publishable when the verdict is KILL or MAJOR.
+
+## Two layers
+
+- **Engine (deterministic):** DOI/PMID/arXiv/NCT vs CrossRef/PubMed/OpenAlex/arXiv/ClinicalTrials.gov. Tiers 1/3/4 with certainty. ISBN/guideline/title-only → Tier 2 (**Content review needed**), never Hallucination. Never invent a tier if the engine was not run. Never override Tier 4 to "probably real."
+- **Skill (judgment):** claim ledger, semantic honesty, 15-criteria appraisal, adversarial red-team. The engine cannot tell you whether a real paper is used honestly.
+
+Named loops (detail in `references/verification-workflow.md`): **engine** (retry once → `unresolved`, not Hallucination), **semantic** (one extra abstract lookup per T1 cite), **adversarial** (max 3; KILL/MAJOR → content must change; re-enter from the engine), **correction** (cap 3; then stop and report remaining issues).
+
+```mermaid
+flowchart TD
+  A[Acquire content] --> B[Extract claim ledger]
+  B --> C[Run evidentia engine]
+  C --> D[Semantic honesty check]
+  D --> E[15-criteria appraisal]
+  E --> F[Adversarial red-team]
+  F --> G[Score plus report]
+  G --> H{User revises?}
+  H -->|yes: max 3| C
+  H -->|no or cap| I[Stop]
+```
+
 ## Workflow
 
+Stage-gated. Do not skip the ledger, the engine, or adversarial review.
+
 ### Step 1: Acquire & Analyze
 
 Receive the target medical content from the user. Depending on the input format:
@@ -125,6 +139,12 @@ Adjust the evaluation lens based on detected media type:
 - Check for "confident but wrong" patterns typical of LLM output
 - Verify all specific numbers, dates, and named entities
 
+### Step 1.5: Claim ledger (mandatory)
+
+Read `templates/claim-ledger.md` and extract **every testable claim** plus its attached citation (or `none`) into the table: `#`, claim (verbatim), citation/id, engine tier, semantic (`supports` / `cherry-pick` / `mismatch` / `n/a`), adversarial note.
+
+Do **not** skip this even for short social posts (1–3 claims). Headlines count. Leave engine tier / semantic blank until Steps 4 and 4b fill them. Do not score from vibes.
+
 ### Step 2: Load Evaluation Checklist
 
 Read `references/checklist.md` (in this skill's directory — see the note at the top) with the `Read` tool to load the detailed 15-item evaluation checklist.
@@ -138,40 +158,44 @@ If the content references research studies, read `references/evidence-levels.md`
 - GRADE assessment for overall quality
 - Domain-specific considerations (pediatrics, oncology, etc.)
 
-### Step 4: Verify Citations
+### Step 4: Verify Citations (engine loop)
 
 If the content cites papers or sources, verify them. **Prefer the deterministic engine** for existence and bibliographic checks, then use `WebSearch` for the semantic context check that the engine cannot do.
 
+Engine output is **ground truth for existence**. The LLM must not override Tier 4 to "probably real." Never invent a tier if the engine was not run.
+
 #### 4a. Run the deterministic engine (existence + bibliographic accuracy)
 
-If `evidentia` (or the `verify_citations` MCP tool) is available, run it on the content first. It resolves DOI/PMID/arXiv/NCT identifiers against CrossRef, PubMed, OpenAlex, arXiv, and ClinicalTrials.gov and returns Tiers 1, 3, and 4 with certainty - no model guesswork. Books (ISBN), guidelines, title-only citations, and other non-indexed sources are returned as Tier 2 (**Content review needed**), never as fabrications:
+If `evidentia` (or the `verify_citations` MCP tool) is available, run it on the content first. It resolves DOI/PMID/arXiv/NCT identifiers against CrossRef, PubMed, OpenAlex, arXiv, and ClinicalTrials.gov and returns Tiers 1, 3, and 4 with certainty — no model guesswork. Books (ISBN), guidelines, title-only citations, and other non-indexed sources are returned as Tier 2 (**Content review needed**), never as fabrications.
 
-```bash
-evidentia check <file-or-url> --format json --cache "$HOME/.cache/evidentia/verification-cache.json" --mailto <your-email>
-```
+Preferred command (local binary):
 
-If the local binary is unavailable, fall back to the npm package:
+    evidentia check <file-or-url> --format json --cache "$HOME/.cache/evidentia/verification-cache.json" --mailto <your-email>
 
-```bash
-npx -y evidentia check <file-or-url> --format json --cache "$HOME/.cache/evidentia/verification-cache.json" --mailto <your-email>
-```
+Fallback if the local binary is unavailable:
 
-Use its output as the ground truth for citation *existence*. Inspect `lookupVerified` and `resolverOutcomes` when explaining why a citation was classified:
+    npx -y evidentia check <file-or-url> --format json --cache "$HOME/.cache/evidentia/verification-cache.json" --mailto <your-email>
+
+**Engine loop:** if unreachable, retry **once**. If still down, mark those citations `unresolved` (not Hallucination) and continue. Never guess a Hallucination without a failed identifier lookup.
+
+Use its output as the ground truth for citation *existence*. Inspect `lookupVerified` and `resolverOutcomes` when explaining why a citation was classified. Write the engine tier into the claim ledger.
 
 - **Tier 1 (Verified)** — the paper, preprint, or trial exists and metadata matches. Proceed to the context check in 4b.
 - **Tier 2 (Content review needed)** — the source may be real, but the engine cannot deterministically verify it in registries, or semantic use still needs review.
 - **Tier 3 (Bibliographic mismatch)** — a real record exists, but the DOI/PMID/arXiv/NCT identifier or metadata is wrong. Record the discrepancy.
 - **Tier 4 (Hallucination)** — the identifier resolves to nothing or to a different paper. Flag as a fabricated citation immediately; this is the highest-severity finding.
 
-If the engine is not available, fall back to verifying each identifier manually with `WebSearch` (steps below).
+If the engine is not available, fall back to verifying each identifier manually with `WebSearch` (steps below). Still do not invent a Hallucination for ISBN/guideline/title-only sources.
+
+#### 4b. Semantic loop (honesty — the engine cannot do this)
 
-#### 4b. Semantic context check (Tier 2 — the engine cannot do this)
+For every citation the engine marked **Verified** (Tier 1), still confirm it is used honestly:
 
-For every citation the engine marked **Verified**, still confirm it is used honestly:
+1. Fetch the abstract (WebSearch / WebFetch) and cross-check it against the cited claim — primary outcome, population, direction of effect.
+2. Evaluate context — is the citation cherry-picked or accurately represented?
+3. Downgrade to **Tier 2 (Content review needed)** if a real paper is being misrepresented or cited out of context. Set semantic to `mismatch` or `cherry-pick`.
 
-1. **Cross-check** the abstract or full text against the cited claim
-2. **Evaluate context** — is the citation cherry-picked or accurately represented?
-3. Downgrade to **Tier 2 (Content review needed)** if a real paper is being misrepresented or cited out of context.
+**Semantic loop cap:** one extra lookup per citation, then stop. Paywalled with no abstract → semantic `n/a`, note "abstract unavailable," leave the engine tier in place. Do not use this loop to upgrade a Tier 3 or 4.
 
 #### Manual fallback (if the engine is unavailable)
 
@@ -187,7 +211,7 @@ AI-generated text (ChatGPT, Claude, Gemini, etc.) frequently contains plausible
 - Does the author actually exist and publish in this field?
 - Do the journal name, volume, and page numbers match a real publication?
 
-**Do NOT stop at "could not verify."** Actively determine whether the citation is unverifiable or provably fabricated.
+**Do NOT stop at "could not verify."** Actively determine whether the citation is unverifiable or provably fabricated. Engine-down is `unresolved`, not Hallucination.
 
 Classify each citation into one of 4 tiers:
 
@@ -207,6 +231,8 @@ Rate each of the 15 items using these dimensions:
 - **Suggestions**: concrete, actionable improvements (if issues exist)
 - **Rating**: Excellent / Good / Fair / Poor
 
+Do not recommend treatments. Suggestions are about how the *content* should be rewritten (cite the primary paper, add ARR, hedge the causal verb) — not about what a patient or clinician should take.
+
 #### Media-Specific Evaluation Adjustments
 
 | Criterion | Social Media | Marketing | Guidelines | Patient Materials |
@@ -219,9 +245,20 @@ Rate each of the 15 items using these dimensions:
 | #10 Ethics | Check stigma/fear | Check manipulation | Check COI panel | Check dignity/autonomy |
 | #12 Images | Memes, infographics | Selective visuals | Evidence figures | Clear illustrations |
 
+### Step 5.5: Adversarial review (mandatory)
+
+Read `references/adversarial-review.md` with the `Read` tool. Run the five lenses (citation integrity, claim support, statistics and language, harm, steelman-then-attack). Answer the 10-line attack checklist (yes/no + evidence). Emit **KILL / MAJOR / MINOR / PASS**.
+
+- **KILL** — any Tier 4 presented as real, or advice that could cause harm if followed. Content must not be published as-is. Overall score ≤ D (F if ethics/harm/fabrication). This is the system working.
+- **MAJOR** — real sources, dishonest use, causal overclaim, missing fair balance. Must fix before publish. Cannot be an A.
+- **MINOR** — hedging, currency, readability. Should fix.
+- **PASS** — ship with stated caveats. Human still owns publish.
+
+KILL or MAJOR: do **not** emit a clean A-score as if publishable; tell the user the content must change. If they revise, re-enter from the **engine** (Step 4), not from scoring. Max **3** adversarial passes per document.
+
 ### Step 6: Determine Overall Score
 
-Aggregate the 15 item ratings into an A–F score using the criteria table in the Overview section.
+Aggregate the 15 item ratings into an A–F score using the criteria table in the Overview section, then apply the adversarial gates above.
 
 Additionally, flag a **Public Health Risk Assessment**:
 
@@ -235,14 +272,16 @@ Read the report template from `templates/report-template.md` (in this skill's di
 
 **Required sections:**
 1. Content Overview — title, source, audience, date, media type
-2. Overall Assessment — score, key issues summary, risk level, recommended actions
+2. Overall Assessment — score, **adversarial verdict**, key issues summary, risk level, recommended actions
 3. Detailed Evaluation — all 15 items with ratings, issues, and suggestions
-4. Citation Verification Results — tier classification for each citation (if applicable)
-5. Critical Concerns — flagged high-severity issues
-6. Strengths — positive aspects worth noting
-7. Suggested Corrections — before/after comparison text (if issues found)
-8. References — sources used during evaluation
-9. Evaluator Notes — overall commentary and caveats
+4. Citation Verification Results — engine vs semantic columns; paste or summarize engine JSON; use **Content review needed** (the previous mismatch wording is retired)
+5. Adversarial review — lenses, steelman, attack, checklist, verdict
+6. Loop log — engine runs, semantic lookups, adversarial pass #, remaining issues
+7. Critical Concerns — flagged high-severity issues
+8. Strengths — positive aspects worth noting
+9. Suggested Corrections — before/after comparison text (if issues found)
+10. References — sources used during evaluation
+11. Evaluator Notes — overall commentary and caveats
 
 ### Step 8: Deliver Report
 
@@ -253,19 +292,23 @@ Save the completed report as a Markdown file using `Write`:
 - Provide the user with:
   - The file path
   - A concise summary of findings (3–5 sentences)
-  - The overall score and risk level
+  - The overall score, adversarial verdict, and risk level
   - Top 3 most important issues to address
 
-### Step 9: Post-Correction Re-Verification (Optional)
+### Step 9: Correction loop (up to 3)
+
+Not optional. If the user revises the content based on the report:
 
-If the user revises the content based on the report and requests re-evaluation:
+1. Re-read the revised content.
+2. Re-run the **engine** on the document (at least every changed or new citation).
+3. Re-run the **semantic** loop on changed Tier 1 citations.
+4. Re-run **adversarial** review (counts toward the 3-pass cap). Re-enter from the engine, not from scoring.
+5. Update the claim ledger, loop log, and recommended-actions checklist.
+6. Check that corrections have not introduced new problems (shifted reference numbers, new causal verbs).
+7. List remaining unresolved issues.
+8. Save the updated report with a `-rev2` (or `-rev3`) suffix.
 
-1. Re-read the revised content
-2. Check that flagged issues have been properly addressed
-3. Update the recommended-actions checklist (mark resolved items)
-4. Verify that corrections haven't introduced new problems (e.g., shifted reference numbers)
-5. List any remaining unresolved issues
-6. Save the updated report with a `-rev2` (or `-rev3`, etc.) suffix
+**Cap: 3.** Then stop and report what is still open. Do not raise the letter grade while a KILL/MAJOR or a new Tier 4 remains.
 
 ## Media-Specific Handling
 
@@ -343,6 +386,7 @@ Patient materials prioritize accessibility and safety:
 5. **Consider the audience** — evaluation standards differ for professional vs. public content
 6. **Stay practical** — improvement suggestions should be realistic and actionable
 7. **Disclose limitations** — acknowledge what this AI-based review can and cannot verify
+8. **Record the loops** — engine JSON, semantic lookups, adversarial pass number, remaining issues belong in the report, not only in your head
 
 ## Caveats
 
@@ -351,12 +395,16 @@ Patient materials prioritize accessibility and safety:
 3. **Image evaluation is limited** — cannot deeply analyze embedded figures or video content
 4. **Rapidly evolving fields** — the most current evidence may not yet be indexed
 5. **Final medical decisions** should always be made by qualified healthcare professionals
+6. **Not CDS** — do not recommend treatments; the human publishes
 
 ## Reference Files
 
+- `references/verification-workflow.md` — operating model, two layers, named loops, hard rules (must-read before scoring)
+- `references/adversarial-review.md` — five lenses, attack checklist, KILL/MAJOR/MINOR/PASS (must-read before scoring)
 - `references/checklist.md` — detailed 15-item evaluation checklist
 - `references/evidence-levels.md` — evidence hierarchy & quality assessment tools
-- `templates/report-template.md` — structured report template
+- `templates/claim-ledger.md` — claim table filled before the engine call
+- `templates/report-template.md` — structured report template (verdict, engine JSON, loop log)
 
 ## External References