Measure unconditional hedging rate of BOHDI wrapper before score filtering

[sebasmos]

Why this matters for the paper

If the BOHDI wrapper produces hedging language regardless of question difficulty, the training signal is uninformative — the LoRA learns to hedge unconditionally, not calibrated to uncertainty. This is the most basic sanity check for the training data and it has never been run. If the hedge rate is near-uniform across score quartiles, the paper's calibration framing is wrong.

Scope

• Write scripts/analyze_hedge_rate.py that reads data/sft/raw_traces.jsonl (before score filtering) and classifies each response as hedged or not using a lightweight lexical classifier (presence of phrases: "I'm not certain", "consult a doctor", "I cannot diagnose", "you should seek", "I don't know", etc. — list to be reviewed before finalizing).
• Report:
  • (a) overall hedge rate
  • (b) hedge rate stratified by rubric score quartile
  • (c) hedge rate stratified by example_tags
• If hedge rate is above 80% unconditionally, add a paragraph to the paper's limitation section acknowledging the training signal may reflect unconditional hedging rather than calibrated abstention.

Done when

scripts/analyze_hedge_rate.py runs end-to-end and prints a table with overall rate, per-quartile rate, and per-tag rate. Results are saved to eval/hedge_rate.json. The paper's limitations section is updated if the 80% threshold is crossed.

Effort

Small — lexical analysis, no GPU.

Good for

sahilk873.

[sebasmos]

Team allocation: the lexical hedge-classifier construction involves manual list curation (which phrases count as hedging). This is annotation / manual-verification work — assign Max (maximinl), Ashley, Martha, and Sohyeon. Once a candidate phrase list exists, clinical review (Zineb, Ash Doulla, Hillary, Felipe) should validate that 'hedging' as defined captures the clinical sense, not just lexical patterns. Add the new contributors' GitHub handles.