A reproducible evaluation harness for scalable oversight protocols on MedQA-USMLE-4-options, accompanying the master's thesis Interpretability, Safety, and Cross-Environment Evaluation of LLM Agents (Theis Andersen, SDU 2026).
SOP wraps a deliberately weak LLM judge in five structured oversight protocols and tests whether multi agent deliberation helps the judge identify correct answers when one of the expert doctors may be adversarial.
- Direct Answer — single-agent baseline; the judge answers with no doctor input.
- Consultancy — one doctor consults with a patient over multiple turns, then answers.
- Debate — two (or more) doctors with opposing locked stances debate; the judge reads the transcript and chooses.
- Interactive Debate — same as debate, but the judge questions each doctor between rounds.
- Audit — single-doctor analogue of interactive debate: one doctor under judge cross-examination.
- Aligned (A) — defends the gold answer.
- Misaligned assigned (MA) — defends an experimenter-chosen non-gold answer.
- Misaligned free (MF) — picks its non-gold stance from the visible option list.
- Misaligned unconstrained (MU) — invents its own non-gold stance with no list.
- Python 3.10 and conda (Miniconda or Anaconda)
- Ollama installed locally
- An Ollama Cloud account with cloud access enabled
git clone https://github.com/Theis-thesis-project/SOP.git
cd SOPconda env create -f conda_env/mac.yml
conda activate macStart the Ollama daemon (leave it running in the background):
ollama serveIn a separate shell, link the device to your Ollama Cloud account:
ollama signinThis step is required — without a device key linked to your account, every *-cloud model call will fail.
The full experiment matrix (37 runs at N=100 MedQA items each) is launched from main.py:
python main.pyIndividual experiment groups can be enabled selectively by commenting/uncommenting the relevant calls:
experiments = (
run_protocol_comparison()
+ run_stronger_adversary()
+ run_debate_structure_ablations()
+ run_strategy_ablations()
+ run_model_matching()
+ run_direct_answer_baselines()
+ run_judge_size_ablations()
+ run_rounds_ablation()
)Single ad hoc variants can be added with a direct setup_debate(...) call — see main.py for the signature.
After runs complete, generate per-run, per-protocol, and cross-protocol reports with:
python run_analysis.py # run all analysis
python run_analysis.py --list # list discovered runs
python run_analysis.py --force # re-run even if outputs exist
python run_analysis.py --per-run-only
python run_analysis.py --cross-onlyOutputs land under runs/<protocol>/analysis/<judge_model>/ (protocol-group view) and runs/analysis/<judge_model>/ (cross-protocol view). Each analysis folder contains a tier-1 set (comparison table, transition table, summary report) for the article body and a tier-2 set (confusion matrices, confidence calibration, per-case flip log) for the appendix.
All roles run on the Gemma 3 family via ollama-cloud:
| Role | Model |
|---|---|
| Judge | gemma3:4b-cloud |
| Doctor (aligned) | gemma3:12b-cloud |
| Doctor (misaligned / redteam) | gemma3:27b-cloud |
| Evaluator | gemma3:27b-cloud |
| Patient | gemma3:4b-cloud (bumped to 27B in Consultancy) |
The Stronger Adversary batch swaps the doctor to gemma3:27b-cloud and the red-team to gemma4:31b-cloud; all other roles unchanged.
SOP/
├── agents/ # doctor, doctor_red_team, judge, evaluator, patient, shared_instructions
├── protocols/ # protocol (base), audit, consultancy, debate, interactive_debate, direct_answer
├── data/ # MedQA loader
├── metrics/ # analysis, parsing, scoring
├── utils/ # config, LLMBundle, helpers
├── conda_env/ # conda environment file (mac.yml)
├── runs/ # output: per-case JSONs, run summaries, analysis reports
├── main.py # entry point for the experiment matrix
└── run_analysis.py # analysis pipeline