Clinical ASR Evaluation Copilot

An agentic evaluation harness for clinical speech recognition, built on NVIDIA NIMs and NVIDIA's digital-health-clinical-asr-eval skill.

I built this to show an agentic evaluation workflow on NVIDIA's stack, using a problem where the stakes are obvious: clinical speech recognition. If a transcription system mishears a drug name like "cefazolin," that's a patient-safety failure, not a typo. So the interesting question isn't just "how accurate is the model" but "does it get the words that matter right, and if not, why?"

This agent runs that evaluation end to end. It takes a manifest of synthetic clinical audio, transcribes it with an NVIDIA ASR NIM, scores four metrics with keyword error rate (KER) as the clinical headline, produces a five-section leaderboard, and reads the result to recommend whether to fine-tune, expand the test set, or stop.

User: Score my synthetic clinical-ASR manifest and tell me whether we should fine-tune.

Copilot:
  KER by ipa_source:
    - magpie_g2p:      KER 0.667 (n=24)
    - merriam-webster: KER 0.033 (n=30)
    - override:        KER 0.000 (n=4)

  Recommendation [coverage_gap]: route to /digital-health-clinical-asr-build (Stage 2d)
    Curated pronunciations are fine (KER 0.03); the grapheme-to-phoneme long-tail
    terms are not (KER 0.67). This is a pronunciation-coverage gap, not a model
    gap. Do NOT fine-tune. Fix the pronunciations and re-run.

The part I most wanted to get right is that final recommendation. A naive tool sees a high error rate and says "fine-tune." But if the curated pronunciations score well and only the auto-generated ones fail, the model is fine. The pronunciation hints are the problem, and fine-tuning would be the wrong fix. Making the agent draw that distinction is the whole point, and it's the judgment the NVIDIA skill is built around.

A note on data (and why this suits Sovereign AI)

Clinical audio is among the most sensitive data there is. Two properties matter:

• Synthetic only, never PHI. The manifest is synthetic clinical speech (Stage-2 TTS over a curated term list). The agent surfaces a data-handling disclosure before any audio is transmitted, and the design never accepts real patient recordings.
• Keeps audio on-shore. Transcription is the only step that sends audio off-box. Point ASR_ENDPOINT at a self-hosted Riva/Parakeet NIM and the audio never leaves the operator's own infrastructure — the requirement for regulated, in-country healthcare deployments. Scoring is pure-Python and transmits nothing.

What it maps to

Layer	What it is	Source
Reasoning	Drives the eval flow and reads the leaderboard	NVIDIA Nemotron NIM, build.nvidia.com
ASR	Transcribes each clip	NVIDIA Parakeet TDT NIM (NVCF or self-hosted)
Scoring + routing	WER/CER/KER/SER, five-section leaderboard, decision tree	NVIDIA digital-health-clinical-asr-eval skill

Quickstart (runs offline, zero installs)

git clone https://github.com/na2802/clinical-asr-eval.git
cd clinical-asr-eval
python -m clinical_asr_eval demo

Standard-library only — no pip install, no key. Without credentials it uses a deterministic mock ASR (modelling the curated-vs-g2p error pattern the skill describes) so the full leaderboard and decision tree run and reproduce every run.

Go live with NVIDIA

cp .env.example .env      # set NVIDIA_API_KEY for Nemotron reasoning + Parakeet ASR,
                          # or ASR_ENDPOINT for a self-hosted (on-shore) ASR NIM
python -m clinical_asr_eval ask "Score the manifest; is the drug KER acceptable?"

Offline demo vs. live NVIDIA services

Capability	Offline demo	With NVIDIA_API_KEY / ASR_ENDPOINT
Reasoning	deterministic planner runs the flow	Nemotron NIM decides the tool sequence
Transcription	deterministic mock ASR	Parakeet TDT NIM (NVCF gRPC or self-hosted Riva)
Scoring + leaderboard + decision tree	full, identical either way	full, identical either way

Scoring is backend-independent, so the leaderboard is real in both modes; only the hypotheses differ (mock vs. a real ASR NIM).

The five metrics and sections

Scoring (scoring.py) is the skill's recipe verbatim: WER, CER, KER (strict contiguous keyword match — cefazolin → cefa zolin is a miss because a pharmacy lookup would fail on the split token), and SER. The leaderboard order is fixed: headline → KER by entity_category → KER by ipa_source → KER by noise_level → per-term worst-first. The by-ipa_source split is the deployment-story number.

Architecture

   User request
        |
        v
   HARNESS (harness.py) — plan -> call tools -> observe -> repeat -> answer
        |                                   |
        | tool schemas                      | chat()
        v                                   v
   TOOLS                              REASONING (nim.py)
   load_manifest                      Nemotron NIM  — or OfflinePlanner
   disclose_data_handling
   transcribe_manifest --------> ASR (asr.py): Parakeet NIM — or mock
   score_and_leaderboard ------> scoring.py + leaderboard.py (skill recipes)
   recommend_next  ------------> decision tree (incl. coverage-gap special case)

Why this harness

I chose to write the agent loop myself rather than reach for a framework. It's about 90 lines, and I can read every step in one file. For a clinical workflow I cared more about that transparency than about the conveniences a framework adds, and this way I can point the agent at any NVIDIA NIM by changing one environment variable. The cost is that I don't get orchestration or tracing for free — a fair trade at this size, and the reason I name NeMo Agent Toolkit as the next step below.

Scaling

If this grew past a prototype, the next step would be the NVIDIA NeMo Agent Toolkit for orchestration, tracing, and serving. Because the tools are already plain functions with OpenAI-compatible schemas, moving them over wouldn't mean rewriting the agent logic.

Project layout

clinical_asr_eval/
  harness.py          the agent loop
  nim.py              Nemotron NIM client + offline planner
  asr.py              Parakeet NIM hook + deterministic offline mock
  scoring.py          WER/CER/KER/SER (NVIDIA skill recipes, attributed)
  leaderboard.py      five-section leaderboard + post-eval decision tree
  manifest.py         NeMo manifest + clinical-extension fields
  state.py            per-run state
  tools/__init__.py   agent tools + schemas
  data/               synthetic clinical manifest (no PHI)
skills/               vendored NVIDIA digital-health-clinical-asr-eval skill (+ attribution)
tests/                scoring, decision tree, agent flow

Development

pip install -e ".[dev]"
pytest -q

Attribution

Follows and vendors NVIDIA's digital-health-clinical-asr-eval skill (© NVIDIA CORPORATION & AFFILIATES; CC-BY-4.0 docs / Apache-2.0 code). The scoring recipes in scoring.py derive from that skill. See skills/README.md. Licensed Apache-2.0. Not affiliated with or endorsed by NVIDIA. Uses synthetic data only; not a medical device and not for clinical use.