diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index 14029b8..4fbfaba 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -3,7 +3,7 @@ "name": "evidentia", "displayName": "Evidentia — Medical Evidence & Citation Audit", "version": "1.1.0", - "description": "Pre-publication evidence audit for medical and health content. Catches AI-fabricated DOIs, PMIDs, arXiv IDs, and trial registrations, then grades any health claim with a 15-criteria, physician-built rubric (A–F report).", + "description": "Pre-publication evidence audit for medical and health content. Catches AI-fabricated DOIs, PMIDs, arXiv IDs, and trial registrations; fills a claim ledger; then grades with a 15-criteria rubric and an adversarial KILL gate (A–F report).", "author": { "name": "Ken Okamoto, MD" }, diff --git a/README.ja.md b/README.ja.md index 8a6fa3d..f0c43ed 100644 --- a/README.ja.md +++ b/README.ja.md @@ -4,7 +4,7 @@ ### AIが捏造した医学引用を、公開前に捕まえる。 -Evidentia は医学文章中のすべての引用を **CrossRef・PubMed・OpenAlex・arXiv・ClinicalTrials.gov** に照合し、4段階で分類します。さらにエージェントスキルでは、15項目のエビデンス評価を重ねて A〜F のレポートを生成します。作者は小児科専門医。 +Evidentia は医学文章中のすべての引用を **CrossRef・PubMed・OpenAlex・arXiv・ClinicalTrials.gov** に照合し、4段階で分類します。エージェントスキルはまず **主張台帳** を作り、エンジン / 意味 / 敵対 / 修正の4つのループを回してから A〜F を出します。作者は小児科専門医。 **サイト:** [https://kgraph57.github.io/evidentia/](https://kgraph57.github.io/evidentia/) @@ -36,7 +36,7 @@ Evidentia は医学文章中のすべての引用を **CrossRef・PubMed・OpenA npx evidentia check your-article.md ``` -**エージェントスキルとして**(15項目の完全評価。`SKILL.md` が `skills/medical-fact-check/SKILL.md` にあるので動きます): +**エージェントスキルとして**(主張台帳 + 4つの検証ループ。`SKILL.md` が `skills/medical-fact-check/SKILL.md` にあるので動きます): ```bash npx skills add kgraph57/evidentia @@ -72,7 +72,7 @@ Evidentia: 4 citations — 1 verified, 1 mismatch, 2 hallucinated (75.0% fabrica [MIS] doi:10.1002/14651858.cd012734 — 論文は実在するが、メタデータ(年)が不一致 ``` -実在は1件。1件はDOIをでっち上げ。1件のPMIDは無関係な論文を指し、1件は年が誤り。**人間のレビュアーなら4件すべてを手で確認する必要があります。** Evidentia は数秒で終えました。 +実在は1件。1件はDOIをでっち上げ。1件のPMIDは無関係な論文を指し、1件は年が誤り。**人間のレビュアーなら4件すべてを手で確認する必要があります。** Evidentia は数秒で終えました。[エンジンの全文レポート](examples/reports/ai-generated-answer.report.md)。同じファイルをスキルのパイプライン(主張台帳 → 意味チェック → 敵対レビューで **KILL**)で歩いたものが [examples/case-studies/vitamin-d-adversarial.md](examples/case-studies/vitamin-d-adversarial.md) です。 ## 4段階分類 @@ -91,36 +91,42 @@ Evidentia は「機械が完璧にできる部分」と「判断を要する部 **1. エンジン(CLI + MCPサーバー)** — 純粋に決定論的な引用検証。APIキー不要、LLM不要、それ自身がハルシネーションを起こすこともありません。「この引用論文は本当に実在し、識別子はそれを指しているか?」という1点を確実に答えます。ターミナル、CI、または任意のエージェントの MCP ツールとして使えます。 -**2. スキル(Claude Code)** — エンジンを **15項目の批判的評価ルーブリック** で包みます。エビデンスレベル、統計解釈(相対リスク vs 絶対リスク、NNT)、因果と相関、利益相反、誇張、対象集団の適合、倫理など — **A〜F のレポート** と具体的な修正案を生成。これはエンジン単独ではできない Tier 2「正しく使われているか」の層です。 +**2. スキル(Claude Code)** — エンジンを **主張台帳** と4つの名前付きループで包みます。エンジン、意味の誠実さ、敵対レビュー、修正(上限3回)。15項目のルーブリックはそのパイプラインの *中の一工程* であって、商品そのものではありません。判定は **KILL / MAJOR / MINOR / PASS**。KILL または MAJOR は A では出せません。エンジンだけではできない「正しく使われているか」の層です。 どちらも単独で使えます。組み合わせれば、引用の *存在*(決定論的)と引用の *誠実さ*(評価)の両方をカバーします。 -## MCPツールとして使う +## スキルはどう検証するか -任意のエージェントに引用検証能力を与えます: +エンジンは「実在するか」に答える。スキルは「誠実に使っているか」に答える。最初に15個の箱を埋める作業ではない。 -```bash -claude mcp add evidentia -- npx -y evidentia-mcp -``` - -JSON出力には、人間向けの4段階判定に加えて `lookupVerified` と `resolverOutcomes` が含まれます。どのレジストリに何を照合し、`matched / unmatched / unreachable / skipped` のどれだったかを機械的に追跡できます。 - -## CIで使う +**まず主張台帳。** 照合の前に、検証できる主張とその引用を全部抜き出す。見出しも主張のうち。書式は [`templates/claim-ledger.md`](skills/medical-fact-check/templates/claim-ledger.md)。 -捏造引用を混入させたプルリクエストをブロックします: +**名前の付いたループは4つ**(それぞれ打ち切り条件あり。詳細は [`skills/medical-fact-check/references/verification-workflow.md`](skills/medical-fact-check/references/verification-workflow.md)): -```yaml -- run: npx evidentia check content/**/*.md --fail-on-fabrication -``` +1. **エンジン** — `evidentia check`。届かなければ1回だけ再試行。それでもダメなら `unresolved`。推測のハルシネーションは付けない。存在についてのエンジン出力が正本。T4 を「たぶん実在」に書き換えない。 +2. **意味** — **Tier 1** の引用だけ、抄録を1回余分に取る。主張は主要アウトカム・対象集団・効果の向きと合うか。T3 や T4 をこのループで格上げしない。 +3. **敵対** — 5つのレンズと10行のチェックリスト。判定は **KILL / MAJOR / MINOR / PASS**。最大3パス。直したら採点からではなくエンジンから入り直す。 +4. **修正** — 著者が直したら、エンジン → 意味 → 敵対を再実行。上限3回。その後は止まって、残件を書く。 -`--fail-on-fabrication` は、いずれかの引用が不一致・ハルシネーションなら非ゼロ終了します。 +**KILL または MAJOR は A では出せない。** **KILL**(実在として出した T4、または従うと害が出る助言)は総合 ≤ D。捏造なら **F**。それが故障ではなく、仕組みが動いているということ。 -## 15項目の評価スキル(Claude Code) +実例: [ビタミンD — 主張台帳 → エンジン → 意味 → KILL](examples/case-studies/vitamin-d-adversarial.md)。収録してある引用4件のデモを、スキルのパイプラインで歩いたもの。 -スキルとして呼び出すと、Evidentia は医学コンテンツを15の観点で評価し、メディアの種類(研究論文・ニュース記事・SNS投稿・患者向けリーフレット・学会スライド・診療ガイドライン・製薬マーケティング・AI生成テキスト)に応じて評価軸を調整します。 +```mermaid +flowchart TD + A[原稿を取る] --> B[主張台帳を作る] + B --> C[エンジンを回す] + C --> D[意味の誠実さ] + D --> E[15項目評価] + E --> F[敵対レビュー] + F --> G[スコアとレポート] + G --> H{直す?} + H -->|はい・最大3回| C + H -->|いいえ / 上限| I[止める] +```
-15の評価項目 +15の評価項目(パイプラインの中の一工程であって、商品ではない) 1. エビデンスレベルと研究デザイン 2. 引用・出典の正確さ *(上記エンジンが担当)* @@ -138,9 +144,29 @@ JSON出力には、人間向けの4段階判定に加えて `lookupVerified` と 14. 臨床的妥当性 15. 情報の完全性 -各項目を **Excellent / Good / Fair / Poor** で評価し、総合 **A〜F** スコアと **公衆衛生リスクレベル**(LOW / MEDIUM / HIGH)に集約します。詳細は [`skills/medical-fact-check/SKILL.md`](skills/medical-fact-check/SKILL.md)。 +各項目を **Excellent / Good / Fair / Poor** で評価し、総合 **A〜F** と **公衆衛生リスク**(LOW / MEDIUM / HIGH)にまとめたうえで、敵対判定でゲートする。詳細は [`skills/medical-fact-check/SKILL.md`](skills/medical-fact-check/SKILL.md)。
+## MCPツールとして使う + +任意のエージェントに引用検証能力を与えます: + +```bash +claude mcp add evidentia -- npx -y evidentia-mcp +``` + +JSON出力には、人間向けの4段階判定に加えて `lookupVerified` と `resolverOutcomes` が含まれます。どのレジストリに何を照合し、`matched / unmatched / unreachable / skipped` のどれだったかを機械的に追跡できます。 + +## CIで使う + +捏造引用を混入させたプルリクエストをブロックします: + +```yaml +- run: npx evidentia check content/**/*.md --fail-on-fabrication +``` + +`--fail-on-fabrication` は、いずれかの引用が不一致・ハルシネーションなら非ゼロ終了します。 + ## エージェントから使う スキルはオープンな [Agent Skills](https://code.claude.com/docs/en/skills) の `SKILL.md` 標準に準拠しているため、Claude Code で今すぐ読み込めるほか、この形式を採用する任意のエージェントで動きます。エンジンは素のnpmパッケージかつMCPサーバーなので、Claude Code・Codex CLI・Cursor・自作スクリプトから利用できます。 @@ -212,6 +238,7 @@ cp -r evidentia/skills/medical-fact-check ~/.claude/skills/ | 入力 | 結果 | |------|------| | [AI生成回答](examples/inputs/ai-generated-answer.md)(実在+捏造の混在) | [捏造率75%](examples/reports/ai-generated-answer.report.md) | +| [ビタミンDの敵対レビュー実例](examples/case-studies/vitamin-d-adversarial.md)(同じファイルをスキルのパイプラインで) | **KILL**、スコア **F** — 実在として出した T4 が2件 | | [クリーンな参考文献リスト](examples/inputs/clean-references.md)(全て実在) | [0%(全件検証済み)](examples/reports/clean-references.report.md) | ## ロードマップ diff --git a/README.md b/README.md index a5e9165..19711c0 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ **Site:** [https://kgraph57.github.io/evidentia/](https://kgraph57.github.io/evidentia/) -The `evidentia` command verifies every citation in a piece of medical writing against **CrossRef, PubMed, OpenAlex, arXiv, and ClinicalTrials.gov** and grades each one in a 4-tier classification. The companion agent skill adds a full 15-criteria evidence appraisal on top. Built by a board-certified pediatrician. +The `evidentia` command verifies every citation in a piece of medical writing against **CrossRef, PubMed, OpenAlex, arXiv, and ClinicalTrials.gov** and grades each one in a 4-tier classification. The companion agent skill starts with a **claim ledger**, then runs four named loops (engine, semantic, adversarial, correction) before any A–F score. Built by a board-certified pediatrician. Evidentia flagging 3 of 4 citations in an AI-generated medical answer as fabricated or mismatched @@ -42,7 +42,7 @@ Homepage: [https://kgraph57.github.io/evidentia/](https://kgraph57.github.io/evi npx evidentia check your-article.md ``` -**As an agent skill** (full 15-criteria appraisal; works because `SKILL.md` is at `skills/medical-fact-check/SKILL.md`): +**As an agent skill** (claim ledger + four verification loops; works because `SKILL.md` is at `skills/medical-fact-check/SKILL.md`): ```bash npx skills add kgraph57/evidentia @@ -79,7 +79,7 @@ Evidentia: 4 citations — 1 verified, 1 mismatch, 2 hallucinated (75.0% fabrica [MIS] doi:10.1002/14651858.cd012734 — Paper exists, but cited metadata disagrees with the record (year). ``` -One citation was real. One DOI was invented. One PMID pointed to an unrelated paper. One had the wrong year. **A human reviewer would have to check all four by hand.** Evidentia did it in seconds. See the [full report](examples/reports/ai-generated-answer.report.md). +One citation was real. One DOI was invented. One PMID pointed to an unrelated paper. One had the wrong year. **A human reviewer would have to check all four by hand.** Evidentia did it in seconds. See the [full engine report](examples/reports/ai-generated-answer.report.md). The skill pipeline on this same file — claim ledger → semantic → adversarial **KILL** — is walked in [examples/case-studies/vitamin-d-adversarial.md](examples/case-studies/vitamin-d-adversarial.md). > This is a deliberately tough example. Most carefully written articles score far lower — Evidentia's value is catching the handful that slip through, every time, without fatigue. @@ -100,36 +100,42 @@ Evidentia is deliberately split into a part a computer can do perfectly and a pa **1. The engine (CLI + MCP server)** — pure, deterministic citation verification. No API key, no LLM, no hallucination of its own. It answers one question with certainty: *does this cited paper actually exist, and does the identifier point to it?* Use it in a terminal, in CI, or as an MCP tool inside any agent. -**2. The skill (Claude Code)** — wraps the engine in a full **15-criteria critical-appraisal rubric**: evidence level, statistical interpretation (relative vs. absolute risk, NNT), causation vs. correlation, conflicts of interest, exaggeration, population fit, ethics, and more — producing an **A–F report** with concrete fixes. This is the Tier-2 "is it used correctly?" layer the engine can't do alone. +**2. The skill (Claude Code)** — wraps the engine in a **claim ledger** and four named loops: engine, semantic honesty, adversarial red-team, correction (cap 3). The 15-criteria rubric is one pass *inside* that pipeline, not the product. Verdicts are **KILL / MAJOR / MINOR / PASS**. KILL or MAJOR cannot ship as A. This is the "is it used honestly?" layer the engine can't do alone. You can use either on its own. Together they cover citation *existence* (deterministic) and citation *honesty* (appraisal). -## Use it as an MCP tool +## How the skill verifies -Give any agent the ability to verify citations: +The engine answers existence. The skill answers honesty. It does not start by scoring 15 boxes. -```bash -claude mcp add evidentia -- npx -y evidentia-mcp -``` - -The server exposes one tool, `verify_citations(text)`, returning the tiered report as Markdown or JSON. JSON responses include both the public 4-tier verdict and a machine-readable `lookupVerified` / `resolverOutcomes` trace for agents and CI. - -## Use it in CI +**Claim ledger first.** Extract every testable claim and its attached citation *before* any lookup. Headlines count. See [`templates/claim-ledger.md`](skills/medical-fact-check/templates/claim-ledger.md). -Block a pull request that introduces a fabricated citation. Drop [`.github/workflows/evidentia.yml`](examples/ci/evidentia.yml) into any medical-content repo: +**Four named loops** (each has a stop condition — full rules in [`skills/medical-fact-check/references/verification-workflow.md`](skills/medical-fact-check/references/verification-workflow.md)): -```yaml -- run: npx evidentia check content/**/*.md --fail-on-fabrication -``` +1. **Engine** — `evidentia check`. Retry once if unreachable; then mark `unresolved`, never a guessed Hallucination. Engine output is ground truth for existence. Never override T4 to "probably real." +2. **Semantic** — one extra abstract lookup per **Tier 1** cite. Does the claim match primary outcome, population, direction of effect? Do **not** use this loop to upgrade a T3 or T4. +3. **Adversarial** — five lenses + a 10-line checklist. Verdict: **KILL / MAJOR / MINOR / PASS**. Max 3 passes. Re-enter from the engine, not from scoring. +4. **Correction** — if the author revises, re-run engine → semantic → adversarial. Cap 3, then stop and report what is still open. -`--fail-on-fabrication` exits non-zero if any citation is a mismatch or hallucination. +**KILL or MAJOR cannot ship as A.** A **KILL** (any T4 presented as real, or advice that could cause harm if followed) forces overall score ≤ D — **F** if fabrication. That is the system working, not a failure of the tool. -## The 15-criteria skill (Claude Code) +Worked example: [vitamin D — claim ledger → engine → semantic → KILL](examples/case-studies/vitamin-d-adversarial.md) on the committed four-citation demo. -When invoked as a skill, Evidentia evaluates medical content across 15 dimensions and adapts to the media type — research paper, news article, social post, patient leaflet, conference slide, guideline, pharma marketing, or AI-generated text. +```mermaid +flowchart TD + A[Acquire content] --> B[Extract claim ledger] + B --> C[Run evidentia engine] + C --> D[Semantic honesty check] + D --> E[15-criteria appraisal] + E --> F[Adversarial red-team] + F --> G[Score plus report] + G --> H{User revises?} + H -->|yes: max 3| C + H -->|no or cap| I[Stop] +```
-The 15 criteria +The 15 criteria (one pass inside the pipeline, not the product) 1. Evidence level & study design 2. Citation & source accuracy *(powered by the engine above)* @@ -147,9 +153,29 @@ When invoked as a skill, Evidentia evaluates medical content across 15 dimension 14. Clinical relevance 15. Information completeness -Each item is rated **Excellent / Good / Fair / Poor**, aggregated into an overall **A–F** score with a **public-health risk level** (LOW / MEDIUM / HIGH). See [`skills/medical-fact-check/SKILL.md`](skills/medical-fact-check/SKILL.md). +Each item is rated **Excellent / Good / Fair / Poor**, aggregated into an overall **A–F** score with a **public-health risk level** (LOW / MEDIUM / HIGH) — then gated by the adversarial verdict. See [`skills/medical-fact-check/SKILL.md`](skills/medical-fact-check/SKILL.md).
+## Use it as an MCP tool + +Give any agent the ability to verify citations: + +```bash +claude mcp add evidentia -- npx -y evidentia-mcp +``` + +The server exposes one tool, `verify_citations(text)`, returning the tiered report as Markdown or JSON. JSON responses include both the public 4-tier verdict and a machine-readable `lookupVerified` / `resolverOutcomes` trace for agents and CI. + +## Use it in CI + +Block a pull request that introduces a fabricated citation. Drop [`.github/workflows/evidentia.yml`](examples/ci/evidentia.yml) into any medical-content repo: + +```yaml +- run: npx evidentia check content/**/*.md --fail-on-fabrication +``` + +`--fail-on-fabrication` exits non-zero if any citation is a mismatch or hallucination. + ## Works with your agent The skill follows the open [Agent Skills](https://code.claude.com/docs/en/skills) `SKILL.md` standard, so it loads in Claude Code today and in any agent that adopts the format. The engine is a plain npm package and an MCP server — usable from Claude Code, Codex CLI, Cursor, or your own scripts. @@ -223,6 +249,7 @@ All registries are free and keyless. Pass `--mailto` to join the faster "polite | Input | Result | |-------|--------| | [AI-generated answer](examples/inputs/ai-generated-answer.md) (real + fabricated mix) | [75% fabrication rate](examples/reports/ai-generated-answer.report.md) | +| [Vitamin D adversarial walkthrough](examples/case-studies/vitamin-d-adversarial.md) (skill pipeline on the same file) | **KILL**, score **F** — two T4 presented as real | | [Clean reference list](examples/inputs/clean-references.md) (all real) | [0% — all verified](examples/reports/clean-references.report.md) | ## Limitations diff --git a/docs/index.html b/docs/index.html index 7b183c5..a9d9b1a 100644 --- a/docs/index.html +++ b/docs/index.html @@ -18,6 +18,7 @@ ★ Star on GitHub @@ -50,10 +51,26 @@

A computer that does not guess, plus a skill that judges

Engine · CLI + MCP

Does this paper exist?

Pure registry lookup. DOI, PMID, arXiv, NCT vs CrossRef, PubMed, OpenAlex, arXiv, ClinicalTrials.gov. No API key, no LLM, no hallucination of its own.

-

npx evidentia check · claude mcp add evidentia -- npx -y evidentia-mcp

Skill · 15 criteria +

npx evidentia check · claude mcp add evidentia -- npx -y evidentia-mcp

Skill · four loops

Is it used honestly?

-

Wraps the engine in a 15-criteria appraisal: evidence level, statistics, causation vs correlation, conflicts, exaggeration, population fit, ethics — an A–F report with concrete fixes. This is the Tier 2 layer the engine cannot do alone.

+

Claim ledger, then four named loops: engine, semantic, adversarial, correction. The 15-criteria rubric is one pass inside that pipeline. KILL or MAJOR cannot ship as A.

npx skills add kgraph57/evidentia

+
How the skill actually verifies +

Claim ledger first. Four loops. KILL is a feature.

+

The 15-criteria rubric is one pass inside the pipeline, not the product. Existence is the engine. Honesty is the skill. A fabricated DOI presented as real is a KILL — overall ≤ D, F if fabrication. That is the system working.

+
01 · Engine +

Does it exist?

+

Registry lookup. Retry once if unreachable; then unresolved, never a guessed hallucination.

+
02 · Semantic +

Is the real paper used honestly?

+

One extra abstract lookup per Tier 1 cite. Primary outcome, population, direction of effect. Does not upgrade a T3 or T4.

+
03 · Adversarial +

KILL is a feature

+

Five lenses. KILL / MAJOR / MINOR / PASS. KILL or MAJOR cannot ship as A. Max 3 passes.

+
04 · Correction +

Revise, then re-enter from the engine

+

Cap 3. Then stop and report what is still open. Do not raise the letter while a T4 remains.

+

Worked example: vitamin D — claim ledger → engine → semantic → KILL.

60-second start

Three ways in. CLI first.

No API key. MIT. Pick the path that matches how you write.

@@ -70,5 +87,5 @@

Three ways in. CLI first.


MIT License · Evidentia v1.1.0 · Copyright (c) 2026 Ken Okamoto, MD

Not clinical decision support. Does not diagnose, treat, or replace professional medical judgment.

-
+ diff --git a/docs/ja/index.html b/docs/ja/index.html index b44e266..d1cbddb 100644 --- a/docs/ja/index.html +++ b/docs/ja/index.html @@ -18,6 +18,7 @@ ★ GitHubでStar @@ -50,10 +51,26 @@

推測しないエンジンと、判断するスキル

エンジン · CLI + MCP

この論文は実在するか

純粋なレジストリ照合。DOI / PMID / arXiv / NCT を CrossRef、PubMed、OpenAlex、arXiv、ClinicalTrials.gov へ。APIキー不要、LLMなし、エンジン自身はハルシネーションしない。

-

npx evidentia check · claude mcp add evidentia -- npx -y evidentia-mcp

スキル · 15項目 +

npx evidentia check · claude mcp add evidentia -- npx -y evidentia-mcp

スキル · 4ループ

正しく使われているか

-

エンジンを15項目の評価ルーブリックで包む。エビデンスレベル、統計、因果と相関、利益相反、誇張、対象集団、倫理 — A–F のレポートと具体的な直し方。エンジンだけではできない Tier 2 の層。

+

まず主張台帳。つづいてエンジン、意味、敵対、修正の4ループ。15項目はその中の一工程。KILL または MAJOR は A では出せない。

npx skills add kgraph57/evidentia

+
スキルの検証の仕方 +

まず主張台帳。ループは4つ。KILL は仕様。

+

15項目のルーブリックはパイプラインの中の一工程であって、商品そのものではない。存在はエンジン、誠実さはスキル。実在として出した捏造DOIは KILL — 総合 ≤ D、捏造なら F。それが故障ではなく、仕組みが動いているということ。

+
01 · エンジン +

実在するか

+

レジストリ照合。届かなければ1回だけ再試行。それでもダメなら unresolved。推測のハルシネーションは付けない。

+
02 · 意味 +

実在論文を誠実に使っているか

+

Tier 1 だけ、抄録を1回余分に取る。主要アウトカム、対象集団、効果の向き。T3 や T4 は格上げしない。

+
03 · 敵対 +

KILL は仕様

+

レンズは5つ。KILL / MAJOR / MINOR / PASS。KILL または MAJOR は A では出せない。最大3パス。

+
04 · 修正 +

直したら、エンジンから入り直す

+

上限3回。その後は止まって残件を書く。T4 が残っているあいだは評点を上げない。

+

実例: ビタミンD — 主張台帳 → エンジン → 意味 → KILL

60秒スタート

入り方は3つ。まず CLI。

APIキー不要。MIT。書き方に合う経路を選ぶ。

@@ -70,5 +87,5 @@

入り方は3つ。まず CLI。


MIT License · Evidentia v1.1.0 · Copyright (c) 2026 Ken Okamoto, MD

臨床判断支援ではありません。診断・治療・医療専門家の判断を代替しません。

-
+ diff --git a/docs/site/css/style.css b/docs/site/css/style.css index f5c7624..1a541a9 100644 --- a/docs/site/css/style.css +++ b/docs/site/css/style.css @@ -425,3 +425,35 @@ h2 { @media (prefers-reduced-motion: reduce) { .js .cli[data-cli] .cli-line { opacity: 1; transform: none; transition: none; } } + + +/* ---------- verify loops ---------- */ + +.loop-grid { + display: grid; + grid-template-columns: 1fr 1fr; + gap: 16px; +} +@media (max-width: 720px) { .loop-grid { grid-template-columns: 1fr; } } +.loop-card { + background: var(--paper-raised); + border: 1px solid var(--line); + border-top: 3px solid var(--navy); + border-radius: var(--radius); + padding: 20px 22px; + box-shadow: var(--shadow-card); +} +.loop-card h3 { margin: 0 0 8px; font-size: 17px; } +.loop-card p { margin: 0; font-size: 14.5px; color: var(--ink-soft); } +.loop-kicker { + display: block; font-family: var(--mono); font-size: 11px; letter-spacing: 0.1em; + text-transform: uppercase; color: var(--navy); font-weight: 700; margin-bottom: 8px; +} +.loop-card.loop-kill { border-top-color: var(--tier-hal); } +.loop-card.loop-kill .loop-kicker { color: var(--tier-hal); } +.verify-case { + margin: 22px 0 0; + font-size: 15px; + color: var(--ink-soft); + max-width: 48em; +} diff --git a/examples/case-studies/vitamin-d-adversarial.md b/examples/case-studies/vitamin-d-adversarial.md new file mode 100644 index 0000000..60a0c6f --- /dev/null +++ b/examples/case-studies/vitamin-d-adversarial.md @@ -0,0 +1,237 @@ +# Vitamin D adversarial walkthrough + +Worked example of the **new** skill pipeline on a committed file. Claim ledger → engine → semantic → 15-criteria skim → adversarial **KILL**. This is not a 15-box recitation. It is not clinical decision support. + +| | | +|---|---| +| **Input** | [`examples/inputs/ai-generated-answer.md`](../inputs/ai-generated-answer.md) | +| **Engine report** | [`examples/reports/ai-generated-answer.report.md`](../reports/ai-generated-answer.report.md) (committed) | +| **Media type** | AI-generated medical answer | +| **Adversarial verdict** | **KILL** | +| **Score** | **F** | +| **Public-health risk** | **HIGH** | + +KILL is forced by two Tier 4 citations presented as real. Score cannot be A; KILL forces ≤ D. F because fabrication. + +--- + +## Input + +Four citations, all formatted as if they were papers. Opening frame: vitamin D for **pediatric** ARTI prevention. + +Source: [`examples/inputs/ai-generated-answer.md`](../inputs/ai-generated-answer.md). + +What the piece actually asserts, in order: + +1. Vitamin D for ARTI has been studied extensively in children. +2. Martineau 2017 IPD meta-analysis: vitamin D reduced at least one ARTI, strongest in the deficient. +3. Yamamoto/Tanaka NEJM 2021 RCT: 1,200 schoolchildren, 47% reduction in influenza A. +4. Hansdottir 2008: vitamin D → cathelicidin in airway epithelium. +5. Bergman 2019 Cochrane: routine supplementation in replete children does not meaningfully reduce ARTI. + +Numbers used below are only those written in the input. The 47% figure is a claim in the draft, not a finding we treat as real. + +--- + +## Claim ledger + +Filled before the engine call. Engine tier and semantic filled after. Headlines count; the pediatric frame is claim 1. + +| # | claim (verbatim) | citation / id | engine tier | semantic | adversarial note | +|---|------------------|---------------|-------------|----------|------------------| +| 1 | "Vitamin D supplementation has been studied extensively for the prevention of acute respiratory tract infections (ARTIs) in pediatric populations." | none (frame) | n/a | n/a | Adult/mixed IPD sold as a pediatric stack | +| 2 | "A large individual participant data meta-analysis found that vitamin D supplementation reduced the risk of at least one ARTI, with the strongest effect in those who were deficient at baseline" | doi:10.1136/bmj.i6583 | **1** | supports (ARTI, deficient); cherry-pick (population) | Real paper, stretched toward children | +| 3 | "A subsequent randomized controlled trial in 1,200 schoolchildren demonstrated a 47% reduction in influenza A incidence with daily vitamin D3 supplementation" | doi:10.1056/nejmoa2105512 | **4** | n/a | **T4 presented as real → KILL** | +| 4 | "Mechanistic work has linked vitamin D to cathelicidin-mediated antimicrobial activity in the airway epithelium" | pmid:18768876 | **4** | n/a | **T4 presented as real → KILL** | +| 5 | "A 2019 Cochrane review concluded that routine supplementation in vitamin-D-replete children does not meaningfully reduce ARTI frequency" | doi:10.1002/14651858.cd012734 | **3** | n/a | DOI is a 2017 cirrhosis Cochrane, not vitamin D. Keep T3. Dishonest use → at least MAJOR | + +Engine tier is existence. Semantic is honesty. They stay in separate columns. T3/T4 do not get a semantic upgrade. + +--- + +## Engine loop + +Engine was reachable. No retry. No `unresolved`. Output is ground truth for existence. Do not override any tier. + +Committed CLI summary: + +```text +$ npx evidentia check examples/inputs/ai-generated-answer.md + +Evidentia: 4 citations — 1 verified, 1 mismatch, 2 hallucinated (75.0% fabrication rate) + [OK ] doi:10.1136/bmj.i6583 — Paper exists and the cited metadata matches the registry record. + [HAL] doi:10.1056/nejmoa2105512 — DOI does not resolve in CrossRef or OpenAlex, and no matching paper was found. + [HAL] pmid:18768876 — Identifier resolves to a different paper ("Trafficking of antigen-specific + CD8+ T lymphocytes…") than the one cited. + [MIS] doi:10.1002/14651858.cd012734 — Paper exists, but cited metadata disagrees with the record (year). +``` + +Full report: [`examples/reports/ai-generated-answer.report.md`](../reports/ai-generated-answer.report.md). + +| Identifier | Engine | Why (from the committed report) | +|------------|--------|----------------------------------| +| doi:10.1136/bmj.i6583 | T1 Verified | CrossRef record matches cited metadata. Martineau et al., *BMJ* 2017, IPD meta-analysis of vitamin D for ARTI. **REAL.** | +| doi:10.1056/nejmoa2105512 | T4 Hallucination | DOI does not resolve. Yamamoto/Tanaka *NEJM* 2021 “1,200 schoolchildren, 47% influenza A” is fabricated. | +| pmid:18768876 | T4 Hallucination | Resolves to a *different* paper: “Trafficking of antigen-specific CD8+ T lymphocytes to mucosal surfaces following intramuscular vaccination.” Not Hansdottir on vitamin D/cathelicidin. | +| doi:10.1002/14651858.cd012734 | T3 Bibliographic mismatch | Year cited 2019 vs record 2017. Registry record: “Aminoglycosides and metronidazole for people with cirrhosis and hepatic encephalopathy” — **not** a vitamin D Cochrane. | + +**Hard rule on the Cochrane DOI:** the engine said T3. This walkthrough does **not** promote it to T4. A real Cochrane record exists at that identifier; the metadata (year) disagrees, and the record title is a different review. Existence = T3. Honesty of use is adversarial, below. + +--- + +## Semantic loop + +Scope: Tier 1 only. One extra abstract lookup per T1 cite, then stop. Do not use this loop to upgrade a T3 or T4. Existence already failed for those. + +### doi:10.1136/bmj.i6583 — Martineau et al., *BMJ* 2017 (T1) + +- **Supports** the ARTI finding as written: an IPD meta-analysis reported that vitamin D supplementation reduced the risk of at least one ARTI, with the strongest effect in people who were deficient at baseline. That sentence is in the right family. +- **Cherry-pick / stretch on population.** The example frames the whole stack as pediatric. The IPD included mixed ages. Strongest signal in the deficient, not “in children.” The draft never says the IPD was mixed-age. That is a population stretch, not a fabricated paper. +- No numbers beyond the input are used here. We do not invent an odds ratio, a percentage, or an NNT for Martineau. + +### T4 and T3 — semantic `n/a` + +- **doi:10.1056/nejmoa2105512** — no paper. There is no abstract to fetch. Semantic n/a. The 47% is unverifiable because the trial does not exist. Do not treat 47% as a real statistic. +- **pmid:18768876** — the identifier’s paper is about CD8+ T-cell trafficking after intramuscular vaccination, not vitamin D or cathelicidin. Semantic n/a; existence already failed (T4). Do not “rescue” it by finding some other Hansdottir paper. +- **doi:10.1002/14651858.cd012734** — engine T3. Semantic loop does not run to upgrade or re-tier. The registry title is a cirrhosis/hepatic-encephalopathy Cochrane. Using that DOI as a 2019 vitamin D Cochrane is an honesty problem for adversarial review, not a new engine tier. + +--- + +## 15-criteria skim + +Not a fake A–F of all 15. The items that actually move this draft: + +| # | Criterion | Rating | Why it moves | +|---|-----------|--------|--------------| +| 2 | Citation & source accuracy | **Poor** | 2× T4 presented as real; 1× T3 (cirrhosis Cochrane sold as vitamin D, 2019 vs 2017). Fabrication rate 75% on the committed engine run. | +| 3 | Statistical interpretation | **Poor** | “47% reduction in influenza A” has no paper behind it. Unverifiable. Do not interpret it as RRR vs ARR; there is no trial. | +| 4 | Causation vs. correlation | **Poor** | “Demonstrated a 47% reduction” is causal language hanging on a fabricated RCT. Martineau’s “reduced the risk” is earned *for that IPD*; the stack then launders causation through two fakes and a wrong Cochrane. | +| 6 | Exaggeration & overclaiming | **Poor** | One real mixed-age IPD is padded with a fake pediatric RCT, a fake mechanism paper, and a misidentified Cochrane so the piece looks like a complete evidence pyramid. | +| 7 | Target population fit | **Poor** | Opens as pediatric ARTI. The only real paper is not a pediatric-only IPD. The schoolchild RCT that would have closed the gap does not exist. | +| 10 | Ethical considerations | **Poor** | Fabricated pediatric evidence, presented as NEJM/Cochrane-grade fact. That is an ethics failure of the *content*, not a treatment recommendation. | + +Items 1, 5, 8, 9, 11–15: **not scored in depth for this demo.** Honest reason: the draft dies on citation integrity before a full rubric pass would change the letter. A complete 15-box report would still be gated by KILL. + +Overall letter is not computed from a 15-item average. See verdict. + +--- + +## Adversarial + +Five lenses, then steelman, then attack, then the 10-line checklist. Evidence from the ledger and the committed engine report — not vibes. + +### 1. Citation integrity — FAIL (KILL) + +Two engine T4s presented as real sources: + +- `doi:10.1056/nejmoa2105512` does not resolve. Yamamoto/Tanaka *NEJM* 2021 is fabricated. +- `pmid:18768876` resolves to an unrelated CD8+ trafficking paper, cited as Hansdottir vitamin D/cathelicidin. + +T4 presented as real is a hard KILL on this lens. Do not rescue either cite because “a similarly titled paper might exist.” + +The Cochrane DOI stays **T3**. Real record, wrong year, wrong review (cirrhosis, not vitamin D). Presented as a clean 2019 vitamin D Cochrane → at least MAJOR on this cite. It does not need to be promoted to T4 to kill the piece; the two T4s already do. + +### 2. Claim support — FAIL + +- Martineau: ARTI reduction in the deficient is supported; pediatric-only framing is a stretch (cherry-pick on population). +- 1,200 schoolchildren / 47% influenza A: no paper, so the claim has zero support. +- Cathelicidin mechanism: identifier points at a different paper. +- “2019 Cochrane… replete children”: the DOI’s record is a 2017 aminoglycosides/metronidazole review in cirrhosis. + +### 3. Statistics and language — FAIL + +The only specific magnitude in the draft is **47%**. It is attached to a T4. Unverifiable. No ARR, no NNT, and no trial from which those could be derived. Causal verb (“demonstrated”) on a ghost RCT. + +### 4. Harm — FAIL (fabrication; not a dosing instruction) + +The copy does not give a dose and does not tell a clinician to give or withhold vitamin D. This walkthrough does not either. + +What it does: publish a fake pediatric influenza RCT and a fake mechanism paper as if they were evidence. A parent or a non-expert reader could treat “47% fewer influenza A cases in 1,200 schoolchildren, *NEJM*” as a reason to act. That is why public-health risk is **HIGH**. The KILL trigger remains **fabrication** (two T4s presented as real), not a CDS harm rule. + +### 5. Steelman, then attack + +**Steelman (honest):** Vitamin D for pediatric ARTI has a real IPD signal in deficient people (Martineau 2017); the piece tries to add an RCT, a mechanism paper, and a Cochrane. + +**Attack:** Two of four citations are fabrications; the “Cochrane” DOI is a cirrhosis review; the only real paper is stretched toward children. + +The steelman does not survive the attack. Verdict cannot be PASS. + +### Attack checklist + +Each line: yes/no + evidence. Blank would be a fail. + +| # | Question | Answer | +|---|----------|--------| +| 1 | Did I run the engine (`evidentia check --format json` or MCP `verify_citations`)? If not, did I mark citations `unresolved` instead of guessing a tier? | **Yes.** Committed run on `examples/inputs/ai-generated-answer.md`. Engine reachable. Report: `examples/reports/ai-generated-answer.report.md`. No `unresolved`. | +| 2 | Did I refuse to override any engine Tier 4 to “probably real”? | **Yes.** Both T4s stay T4. No “Yamamoto might be real under another DOI.” | +| 3 | Did I treat ISBN / guideline / title-only cites as Tier 2, never Hallucination? | **n/a.** All four cites have DOI or PMID. | +| 4 | Did I read an abstract for each Tier 1 citation used to support a numeric or causal claim? (One extra lookup, then stop.) | **Yes.** One T1: Martineau. Semantic loop run once. T3/T4 not upgraded via this loop. | +| 5 | Does every cited numeric claim match the paper’s actual primary finding, population, and effect direction? | **No.** 47% has no paper. Cochrane “conclusion” is attached to a cirrhosis review. | +| 6 | Is every causal verb earned (not observational data dressed as “causes”)? | **No.** “Demonstrated a 47% reduction” is unearned. Martineau’s “reduced the risk” is earned for that IPD only. | +| 7 | Is relative risk accompanied by absolute risk / NNT where a magnitude is claimed? | **No.** The 47% stands alone — and is unverifiable. | +| 8 | Could a patient or clinician act on this unsafely (unapproved treatment, dose, adult→child, omitted critical caveat)? | **No dose, no give/withhold instruction.** The copy is still unsafe *as evidence*: a fake schoolchild RCT plus mixed-age IPD framed as pediatric. KILL is fabrication, not a treatment order. This report does not recommend giving or withholding vitamin D. | +| 9 | Did the steelman survive the attack, or am I grading a piece I already know is misleading? | **No.** Steelman dies. Two T4s + a cirrhosis Cochrane + a population stretch. | +| 10 | Would I be embarrassed if this DOI 404’d in peer review — or if a regulator quoted this paragraph back at the author? | **Yes.** Two of four identifiers fail in public registries. The “Cochrane” DOI 200s to the wrong review. | + +All ten answered. Adversarial pass **1**. + +--- + +## Verdict + +| | | +|---|---| +| **Adversarial** | **KILL** | +| **Score** | **F** | +| **Public-health risk** | **HIGH** | +| **Publish as-is?** | **No.** | + +**Why KILL:** two Tier 4 citations presented as real (Yamamoto/Tanaka *NEJM* DOI that does not resolve; PMID 18768876 pointing at a CD8+ trafficking paper). That is the hard rule. + +**Why F, not D:** KILL already forces ≤ D. Fabrication takes it to F. + +**Why not A:** KILL or MAJOR cannot ship as A. Irrelevant here; the piece is not close. + +**Why HIGH:** fabricated pediatric RCT evidence, stacked on a real IPD that was stretched toward children, plus a Cochrane DOI that belongs to a cirrhosis review. Vulnerable population (children), respiratory-infection claims, confident scholarly voice. + +The T3 Cochrane is **MAJOR** on honesty (wrong review, wrong year, presented as vitamin D 2019). It is not re-tiered to T4. + +--- + +## Loop log + +| Loop | Runs | Result | +|------|------|--------| +| Engine | **1** | Reachable. 1× T1, 1× T3, 2× T4. 0 unresolved. No retry. | +| Semantic | **1** (T1 Martineau only) | supports ARTI/deficient; cherry-pick on pediatric framing. T3/T4 = n/a. | +| Adversarial | **pass 1** | **KILL** | +| Correction | **0** | Not run. No revised article in this demo. | + +Remaining issues if anyone tries to publish this file: both T4s, the T3 Cochrane identity, the pediatric stretch on Martineau. + +--- + +## What a correction loop would demand + +Do not write a fake revised article. If the author actually revises, re-enter from the **engine**, not from scoring. Cap 3. + +A publishable rewrite would have to: + +1. **Remove** `doi:10.1056/nejmoa2105512` and the Yamamoto/Tanaka 1,200-schoolchildren / 47% influenza A sentence. There is no paper to swap in from this identifier. +2. **Remove** `pmid:18768876` as a Hansdottir/cathelicidin cite. The PMID is a different paper. Either drop the mechanism sentence or cite a real paper whose identifier actually points at it — then re-run the engine on the new identifier. +3. **Replace or drop** `doi:10.1002/14651858.cd012734`. It is not a 2019 vitamin D Cochrane. Keeping the DOI and changing only the year is not a fix; the record is a cirrhosis review. Engine tier stays T3 until the identifier changes. +4. **Hedge Martineau on population.** Say what the IPD is: mixed ages, strongest in deficient people. Do not open as if the evidence base were pediatric-only. + +After those edits: engine again → semantic on remaining T1 → adversarial pass 2. Do not raise the letter while a T4 or a KILL remains. + +This walkthrough stops here. No invented replacement citations. No “corrected” vitamin D article. + +--- + +## Not CDS + +Evidentia is a pre-publication aid for writers, editors, and researchers — **not clinical decision support.** This case study evaluates how the *content* is sourced. It does not diagnose, treat, or replace professional medical judgment. + +**Do not read this file as a reason to give vitamin D, withhold vitamin D, or change a pediatric ARTI protocol.** The only operational output is: this draft must not ship. + +Operating model: [`skills/medical-fact-check/references/verification-workflow.md`](../../skills/medical-fact-check/references/verification-workflow.md). Adversarial rules: [`skills/medical-fact-check/references/adversarial-review.md`](../../skills/medical-fact-check/references/adversarial-review.md). diff --git a/skills/medical-fact-check/SKILL.md b/skills/medical-fact-check/SKILL.md index 37b8b4b..6297311 100644 --- a/skills/medical-fact-check/SKILL.md +++ b/skills/medical-fact-check/SKILL.md @@ -88,6 +88,8 @@ flowchart TD H -->|no or cap| I[Stop] ``` +Worked example: `examples/case-studies/vitamin-d-adversarial.md`. + ## Workflow Stage-gated. Do not skip the ledger, the engine, or adversarial review.