Part of #51.
Summary
Add an optional, additive pipeline step that cross-checks the Vision-LLM's
structured findings against the independent signals MedCheck already computes
(ML anomaly scores, selected/“top” slices, detected anatomy), and annotates the
report with a Validation section: agreement, contradictions, and confidence
flags. This catches the classic failure mode where a Vision-LLM confidently
reports a finding that the quantitative analysis does not support (or misses one
the anomaly detector flagged).
This is the lowest-risk child of #51: no new heavy dependencies, no model
downloads, no network calls — it only reasons over data already present in the
PipelineContext.
Why (vision)
For a research tool whose outputs "must be reviewed by a qualified radiologist,"
the most valuable thing we can add is not another model — it's making
disagreement visible. A finding that all signals agree on is more trustworthy;
a finding the LLM asserts with high confidence but no anomaly-score support
deserves a flag. This is defense-in-depth, not a new diagnosis.
Where
- New file:
src/medcheck/pipeline/validate.py → ValidationStep(PipelineStep)
- Register in
src/medcheck/main.py::_build_registry() as "validate"
- Runs after
vision_analysis, before report
- Extend
PipelineContext (src/medcheck/core/context.py) with a
validation: list[ValidationFinding] (or dict) field, default empty
- Surface results in
src/medcheck/pipeline/report.py (JSON + PDF + HTML) under a
new "Validation / Quality checks" section
- Optionally expose a
validate entry in the default workflows/*.yml
Proposed checks (all heuristic, no ML)
- Anomaly corroboration — for each
StructureFinding with a non-"normal"
status, check whether the series it maps to has an elevated anomaly_scores
entry / appears in top_slices. Flag "asserted abnormality with no
quantitative support" and vice-versa ("high anomaly score in a region the LLM
called normal").
- Confidence floor — flag findings with
confidence below a configurable
threshold (default 0.5) as "low confidence — verify".
- Anatomy consistency — flag structures that don't belong to the
detected_anatomy region (e.g. an "ACL" finding on a shoulder study) as a
possible mislabel/hallucination.
- Schema/sanity — confidence outside [0,1], empty findings text, duplicate
structures.
- Overall quality score — a simple aggregate (e.g. fraction of findings that
are corroborated) shown in the report header.
Each produces a ValidationFinding(level: info|warn|flag, message, related_structure).
Configuration
- Step config (via workflow YAML /
step_config): confidence_threshold,
anomaly_percentile, toggles per check.
- Must be a no-op if the relevant inputs are absent (e.g. no anomaly scores,
no findings) — fail-soft, never raises.
Acceptance criteria
Effort
~3–4 h. No new dependencies.
Part of #51.
Summary
Add an optional, additive pipeline step that cross-checks the Vision-LLM's
structured findings against the independent signals MedCheck already computes
(ML anomaly scores, selected/“top” slices, detected anatomy), and annotates the
report with a Validation section: agreement, contradictions, and confidence
flags. This catches the classic failure mode where a Vision-LLM confidently
reports a finding that the quantitative analysis does not support (or misses one
the anomaly detector flagged).
This is the lowest-risk child of #51: no new heavy dependencies, no model
downloads, no network calls — it only reasons over data already present in the
PipelineContext.Why (vision)
For a research tool whose outputs "must be reviewed by a qualified radiologist,"
the most valuable thing we can add is not another model — it's making
disagreement visible. A finding that all signals agree on is more trustworthy;
a finding the LLM asserts with high confidence but no anomaly-score support
deserves a flag. This is defense-in-depth, not a new diagnosis.
Where
src/medcheck/pipeline/validate.py→ValidationStep(PipelineStep)src/medcheck/main.py::_build_registry()as"validate"vision_analysis, beforereportPipelineContext(src/medcheck/core/context.py) with avalidation: list[ValidationFinding](ordict) field, default emptysrc/medcheck/pipeline/report.py(JSON + PDF + HTML) under anew "Validation / Quality checks" section
validateentry in the defaultworkflows/*.ymlProposed checks (all heuristic, no ML)
StructureFindingwith a non-"normal"status, check whether the series it maps to has an elevated
anomaly_scoresentry / appears in
top_slices. Flag "asserted abnormality with noquantitative support" and vice-versa ("high anomaly score in a region the LLM
called normal").
confidencebelow a configurablethreshold (default 0.5) as "low confidence — verify".
detected_anatomyregion (e.g. an "ACL" finding on a shoulder study) as apossible mislabel/hallucination.
structures.
are corroborated) shown in the report header.
Each produces a
ValidationFinding(level: info|warn|flag, message, related_structure).Configuration
step_config):confidence_threshold,anomaly_percentile, toggles per check.no findings) — fail-soft, never raises.
Acceptance criteria
ValidationStepregistered as"validate"; not in the default step listunless explicitly added (additive).
context.validation; never raises on missing/partial inputs.low-confidence flag, anatomy-mismatch flag, empty-context no-op.
Effort
~3–4 h. No new dependencies.