Skip to content

Add local VLM second-opinion / consensus stage (cross-check cloud LLM against offline model) #55

Description

@Liohtml

Part of #51. Depends on #18 (local VLM provider).

Summary

Once a local, offline vision-language model provider exists (#18), add an
optional stage that runs it in parallel with the primary cloud provider
(Claude/GPT/Gemini) and produces a consensus / second-opinion comparison:
where both agree, confidence is reinforced; where they diverge, the report flags
it for the radiologist. This complements the heuristic cross-validation in #52
with a second independent model.

Why (vision)

Two independent models disagreeing on a finding is exactly the signal a reviewing
radiologist most wants surfaced. A cloud frontier model and a medical-domain
local model have different failure modes; comparing them is cheap defense in
depth. It also gives users who cannot send data to the cloud (consent gate /
offline, see #27) a fully local path that still benefits from cross-checking.

Recommended model (license-verified — corrects #18)

  • Lingshu-7BMIT license (code + weights), explicitly supports MRI
    among 12+ modalities. This is the recommended default for Add local LLaVA-Med vision model provider #18 instead of
    LLaVA-Med, whose weights are research-only ("any deployed use case —
    commercial or otherwise — is out of scope") and therefore unsuitable as a
    default in an Apache-2.0 tool. Alternatives: HuatuoGPT-Vision-7B-Qwen2.5VL
    (Apache-2.0). See comment on Add local LLaVA-Med vision model provider #18.
  • Runs behind the optional local-models extra (torch + transformers).

Where

  • Builds on the LocalLLMProvider from Add local LLaVA-Med vision model provider #18 (currently a stub in
    src/medcheck/llm/local.py).
  • New optional step src/medcheck/pipeline/consensus.py
    ConsensusStep(PipelineStep), OR a flag on vision_analysis to run a second
    registered provider and diff the results.
  • Extend PipelineContext with second_opinion: AnalysisResult | None and a
    consensus: list[...] comparison.
  • Report renderers show an "Independent model comparison" section (agreement per
    structure, divergences highlighted).

Design constraints

Acceptance criteria

  • Depends on Add local LLaVA-Med vision model provider #18 landing a working local provider (Lingshu-7B recommended).
  • Optional ConsensusStep (or vision_analysis flag) compares two providers'
    structured findings and records agreements/divergences.
  • Report includes an independent-comparison section when a second opinion exists.
  • Honors the consent gate; supports a fully-offline consensus configuration.
  • Tests with two mocked providers (agreeing case, diverging case); no real
    model download in CI.

Effort

~1 day on top of #18.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions