Clinical Concept Extraction from Patient Notes

Token-level span extraction on the NBME Score Clinical Patient Notes dataset: given a patient note and a rubric feature such as Family-history-of-thyroid-disorder, locate the character spans in the note that express that concept.

Three architectures are compared under a single preprocessing, splitting and evaluation protocol, so the only variable is the model itself.

Results

Character-level micro F1 on a patient-grouped validation fold. The decision threshold is tuned per model on validation.

Model	char F1	Precision	Recall	Threshold
TF-IDF + logistic regression	0.1308	0.0716	0.7602	0.50
BERT-base	0.7833	0.8170	0.7522	0.83
DeBERTa-base	0.8276	0.7898	0.8691	0.925

The lexical baseline achieves high recall (0.76) at very low precision (0.072) — it marks large regions of text as positive and cannot discriminate. Contextual encoders close that gap almost entirely.

DeBERTa is trained in two phases: an initial fine-tune, then a resume from the best checkpoint at a quarter of the learning rate. Phase 1 peaked at 0.8262 token-level validation F1; phase 2 reached 0.8335.

Reported figures are read directly from results/*.json, written by the training run. Nothing in this README is transcribed by hand.

Evaluation notes

Patient-grouped splitting. The 14,300 training rows are drawn from only 1,000 unique patient notes — each note is paired with roughly 14 rubric features. A row-level random split places the same note text on both sides, so the model is evaluated on text it has already seen. All splits here group on pn_num, and the split function raises if any note appears on both sides.

Character-level metric. The competition scores character spans, so token predictions are projected back onto character offsets before scoring. Token-level F1 runs slightly optimistic relative to character-level F1 and is not comparable across tokenizers.

Positive-class F1 for the baseline. At a 1.8% positive token rate, accuracy is uninformative — predicting all-negative scores 0.982. The baseline is scored on positive-class F1 with accuracy reported alongside for contrast.

Confidence triage

Predictions are ranked by mean binary entropy across non-padding tokens. Routing the least-confident share to a human reviewer captures disproportionate error:

Review budget	Errors caught (DeBERTa)	Lift vs random
10%	35.0%	3.50x
15%	45.2%	3.02x
20%	53.5%	2.67x
30%	68.0%	2.27x

BERT shows the same pattern (3.89x at a 10% budget), so the ranking generalises across architectures rather than fitting one model.

Negation

Annotated spans are split by whether a negation cue (denies, no, without, negative for) appears within 40 characters before the span. Rows with no annotation are excluded from both slices, since they admit false positives but no true positives.

Model	With negation	Without negation	Δ
BERT-base	0.7975 (n=290)	0.8005 (n=1678)	+0.0030
DeBERTa-base	0.8546 (n=290)	0.8470 (n=1678)	-0.0076

Neither model degrades measurably on negated spans. This is a null result, and it runs against the common assumption that negation is a dominant failure mode for token-classification approaches on clinical text. Both models do become more conservative under negation — precision rises while recall falls — but aggregate F1 is unchanged.

Repository layout

src/
  train.py       training CLI: grouped split, weighted BCE, optional two-phase schedule
  nbme_eval.py   evaluation: character F1, threshold sweep, triage, negation slices
  inference.py   SpanExtractor — probabilities to character spans with review flags
  app.py         FastAPI service
notebooks/
  tfidf_baseline.ipynb    lexical baseline, CPU only
  bert_baseline.ipynb     BERT-base fine-tune
  nbme_deberta.ipynb      EDA and DeBERTa two-phase training
results/                  metrics written by training runs

Usage

Download the competition data into data/, then:

pip install -r requirements.txt

python src/train.py --model microsoft/deberta-base --fold 0 --phases 2
python src/train.py --model bert-base-uncased --fold 0 --batch-size 16 \
                    --lr 3e-5 --dropout 0.4 --mixed-precision

Serving:

PYTHONPATH=src uvicorn src.app:app --port 8000

curl -X POST localhost:8000/extract -H 'Content-Type: application/json' \
  -d '{"note": "17yo M presents with palpitations. FHx: mom with thyroid disease",
       "feature": "Family-history-of-thyroid-disorder"}'

{"spans": [{"start": 46, "end": 66, "text": "mom with thyroid dis",
             "confidence": 0.97}],
 "needs_review": false, "threshold": 0.925}

Docker:

docker build -t nbme-extract .
docker run -v $(pwd)/weights:/app/weights -p 8000:8000 nbme-extract

Implementation notes

Mixed precision is BERT-only. The TensorFlow DeBERTa implementation pins several internal tensors to float32 — the embedding mask, two TensorArray allocations in the disentangled attention, and a scaling division — so an fp16 global policy fails with a dtype mismatch at the embedding multiply. train.py rejects the flag for DeBERTa rather than failing at model construction. TFBert has no such hardcoding and trains fine under mixed_float16.

Padding excluded from the loss. The attention mask is passed as sample_weight with an element-wise loss, so padded positions contribute no gradient. Without this a large share of the training signal is the model learning to predict zero on [PAD].

Sequence length. MAX_LEN is 384 rather than 512, chosen from the empirical token-length distribution — 384 covers over 99% of notes. Attention cost is quadratic in sequence length, so this is a material saving.

Limitations

• Single fold. make folds runs all five for a variance estimate; the numbers above are fold 0 only.
• POS_WEIGHT is fixed at 8.0 across models for protocol parity and was not tuned. Both models select thresholds well above 0.5 to compensate, so a lower weight would likely improve calibration.
• No domain-adapted encoder (BioBERT, ClinicalBERT) in the comparison.
• Binary token classification does not model span boundaries as structured objects, which is a plausible source of the residual boundary errors.

Data

The dataset is released by the National Board of Medical Examiners for the Kaggle competition and is subject to its terms. It is not redistributed here. Notes describe standardized patients, not real individuals, but should still be treated as sensitive clinical text.