snomed-coder

▶ Live demo: snomed-coder.kareemghazal.com

Paste a clinical note and get suggested SNOMED codes with the supporting phrase — or an abstention when nothing fits. The model can only pick from retrieved candidates, so it can't invent a code. (First run ~10–20s.)

How it works — input → pipeline → output, with the eval harness that measures it:

Demo / educational — not for clinical use. Uses a small illustrative SNOMED subset (not the full release). Output is coding suggestions for a human to confirm.

Suggests SNOMED CT codes for a free-text clinical note — built so the LLM can't hallucinate a code. Concept ids are the classic failure mode for LLMs (they'll happily emit a plausible, wrong id), so this tool never lets the model produce one from memory.

How it stays safe:

1. Retrieval first. The note is embedded and matched against the concept subset — this produces a shortlist of candidate concepts (real ids + terms).
2. The LLM only chooses. It's shown the candidates and picks the ones the note documents, with a confidence and the evidence phrase from the note. It's told it may only use the given ids.
3. Validation enforces it. After selection, any id that wasn't in the retrieved candidates is dropped — so even if the model invents one, it can't reach the output. The term is normalised back to the canonical subset term.
4. Evidence is grounded. Each code carries an evidence_span — any span that is not a verbatim substring of the note (whitespace/case-normalised) drops its candidate. If the model can't quote it, it can't code it.
5. Context is checked (NegEx-style, deterministic). A negated mention ("no diabetes", "denies chest pain") or a family-history mention ("family history of MI") is not a current condition of the patient — the candidate is excluded, with scope terminators handled so "no chest pain but has type 2 diabetes" still codes the diabetes. These are the classic primary-care coding traps.
6. The hierarchy is modelled. SNOMED is an is-a hierarchy: when a concept and its parent are both selected (Type 2 diabetes mellitus + Diabetes mellitus), the parent is subsumed by the more specific child and dropped.
7. Abstains when nothing survives, rather than forcing a weak match — and every exclusion is surfaced on the result with its reason, so the safety layer is visible, not silent.

Architecture

clinical note ─▶ Voyage embed ─▶ cosine search over SNOMED subset ─▶ candidate concepts
                                                                          │
                                                                          ▼
                                                        LLM selects from candidates only
                                                                          │
       validate: ids ⊆ candidates · evidence verbatim · negation/FH context · subsumption
                                                                          │
                                                                          ▼
                                              CodingResult (+ exclusions with reasons)

Quickstart

pip install -e .
cp .env.example .env   # add ANTHROPIC_API_KEY + VOYAGE_API_KEY

snomed-coder code "68yo with type 2 diabetes and high blood pressure, on methotrexate for RA"

Evals

python evals/run_evals.py            # deterministic gates
python evals/run_evals.py --judge    # + opus LLM-as-judge on every case

• Top-1 accuracy — the highest-confidence pick is a correct code.
• Recall@k — expected codes appear among the selections.
• Abstention — non-codable notes (e.g. "attended for a flu jab") are refused.
• No-hallucinated-id — every selected id exists in the bundled subset (the core guarantee).
• Evidence-grounded — every evidence span is a verbatim substring of the note (re-verified independently of the coder's own filter).
• No-forbidden-id — the trap cases carry forbidden_ids: the negated diabetes, the father's MI, the subsumed parent concept, the symptom-as-diagnosis and over-specification traps must NOT appear in the output.
• LLM-as-judge (--judge) — a separate, stronger model (claude-opus-4-8) grades coding appropriateness, evidence support, and over-claiming per case.

Every run writes a reproducible artifact to evals/results/latest.json — per-case outcomes (including what was excluded and why), metrics, models, timestamp. The numbers below come from that file.

Latest run (claude-sonnet-4-6 coder, voyage-3 embeddings, claude-opus-4-8 judge): all gates pass — Top-1 7/7, recall@k 1.00, correct abstention, no hallucinated id, every evidence span verbatim, every forbidden id avoided (negation, family history, subsumption), and the opus judge passes 8/8 cases.

The judge taught the coder clinical-coding discipline. On an earlier run the judge failed three cases where every deterministic gate passed: "low mood" coded as Depressive disorder (a symptom coded as a diagnosis), "raised BMI" coded as the obesity (BMI 30+) concept, and "high blood pressure" coded as Essential hypertension (specificity the note never documented). That is precisely the discipline real coders call "code to the documentation, not beyond it" — the coder prompt now encodes it, the three cases became labelled traps with forbidden_ids, and all 8/8 pass. Set-membership gates can't see over-coding; that's what the judge layer is for.

Tests

pytest -q   # offline: concept index + the anti-hallucination guarantee (fake embedder + client)

The key test proves that when the model returns an id that wasn't retrieved, the coder drops it.

Web

web/ — a Next.js UI: paste a note, get suggested codes with the evidence phrase, or an honest abstention.

Run it locally in two terminals:

# terminal 1 — the API
pip install -e .
cp .env.example .env                  # add ANTHROPIC_API_KEY and VOYAGE_API_KEY
python -m uvicorn snomed_coder.api:app --port 8000

# terminal 2 — the UI
cd web
npm install
echo "NEXT_PUBLIC_API_URL=http://localhost:8000" > .env.local
npm run dev                           # open http://localhost:3000

See DEPLOY.md.

License

MIT — see LICENSE.