Predict-It — Confidence-Aware News Triage

Vayu Hackathon submission — final package (v2)

A supervised news-article classifier reframed as a confidence-aware, human-in-the-loop routing system: it auto-files the articles the model is sure about, escalates the uncertain ones to an editor, and learns from every correction. Built end-to-end on the Vayu ecosystem (Object Storage → AI Studio → MLflow → Model Registry → Model Serving → Streamlit).

---

TL;DR — what changed in v2

	v1	v2 (this package)
Shipped model	TF-IDF + Logistic Regression	TF-IDF + soft-vote ensemble [LogReg + ComplementNB]
Holdout weighted-F1	0.9797 (6 errors / 298)	0.9899 (3 errors / 298)
5-fold CV weighted-F1	0.9748	0.9798
Weakest class	tech F1 0.95 (recall 0.92)	tech F1 0.98
Confidence gate	85%, assumed	0.70, measured — swept on holdout, curve in metrics.json
Auto-route @ gate	56% at 0.85 (measured post-hoc)	87% at 0.70, zero holdout misroutes (88% on validation set)
Gate safety	—	every holdout error < 0.65 confidence → gate caught all of them
Confidence-gate UI	designed only	shipped — app/app.py: gate slider, triage queue, explanations, KPI live stream
Artifact size	~3.2 MB	~6.8 MB (RF baseline: ~20 MB)
Trained with	scikit-learn 1.5.2	scikit-learn 1.5.2 (same — artifact drop-in compatible)

---

Results

Holdout = 20% stratified split of news.csv (1192 train / 298 holdout), random_state=42. Model selection by 5-fold CV weighted-F1 on the training split (holdout used once, for confirmation and the gate sweep).

Model	CV weighted-F1	Holdout weighted-F1	Holdout accuracy
Soft-vote [LogReg(C=5) w=1 + ComplementNB(α=0.2) w=2] (SHIPPED)	0.9798 ± 0.0067	0.9899	0.9899
ComplementNB (α=0.2)	0.9765	0.9899	0.9899
Logistic Regression (v1 shipped)	0.9748	0.9797	0.9799
Random Forest (starter-kit baseline)	—	0.9596	0.9597

Per-class F1 (shipped): business 0.99 · entertainment 0.99 · politics 0.99 · sport 0.99 · tech 0.98.

Confidence-gate sweep (holdout)

Gate	Auto-route coverage	Accuracy on auto-routed	Escalated
0.60	90.9%	99.6%	27 / 298
0.65	88.9%	100%	33 / 298
0.70 (default)	86.9%	100%	39 / 298
0.80	77.5%	100%	67 / 298
0.85	70.1%	100%	89 / 298
0.90	59.1%	100%	122 / 298

Highest confidence among the 3 holdout errors: 0.62 — the 0.70 gate catches every mistake with margin. Full numbers in model/metrics.json.

Honesty note for Q&A: the holdout is 298 articles, so "100% on auto-routed" means zero errors observed there, not a guarantee — which is exactly why the MLflow drift monitor and the shadow-deployment promotion policy exist.

---

Package layout

predict-it-submission/
├── README.md                     ← you are here
├── train_final.py                ← reproduces every artifact below (v2)
├── model/
│   ├── model.joblib              ← DEPLOY THIS (register in Step 4, serve in Step 5)
│   ├── category_labels.json      ← integer code → category-name map
│   ├── validation_predictions.csv← predictions on validation.csv (ArticleId, Text, Category)
│   └── metrics.json              ← CV + holdout metrics, benchmarks, full gate sweep
├── app/
│   └── app.py                    ← drop-in replacement for 06_build_app/app.py:
│                                    confidence gate · triage queue · explanations · KPI stream
└── docs/
    ├── Predict-It_System_Architecture.pdf   ← the supporting document (submit this)
    ├── Predict-It_System_Architecture.md    ← same, editable source
    └── solution_approach.txt                ← copy-paste text for the submission box

To slot into the official repo: copy the four model/ files into 03_starter_kit/, and app/app.py over 06_build_app/app.py.

---

The approach

User. Managing Editor of Digital Operations — accountable for newsroom throughput and quality.

Problem. Manually triaging every incoming article into business / politics / sport / tech / entertainment is slow, repetitive, and inflates Time-to-Publish.

Core idea — uncertainty is a feature, not a bug. Every prediction carries a confidence score (max class probability):

• confidence ≥ 0.70 → auto-route straight to the category desk (zero human touch). Measured: 87% of holdout articles, zero misroutes; 88% of the unlabeled validation set.
• confidence < 0.70 → escalate to the human triage queue (never force a wrong label; out-of-category articles land here too).

The 0.70 default was chosen from the measured sweep above — with margin over the highest-confidence holdout error (0.62). The KPI translation: of ~14.6 addressable manual-sorting hours/week (500 articles/day × ~15 s × 7 days), ~12.7 hours are eliminated at the measured auto-route rate.

Three workflows (full diagrams in the architecture doc):

1. Happy Path (auto-route) — Ingestion → TF-IDF + ensemble → predict_proba → ≥0.70? → category desk.
2. Exception Path (human-in-the-loop) — <0.70 → Streamlit triage queue → word-level explanation → editor confirms/corrects in one click → correction appended to corrections.csv → synced to Object Storage.
3. MLOps Loop — Object Storage → AI Studio retraining (train_final.py) → MLflow (eval + gate curve + drift) → Model Registry → Model Serving shadow deployment → promote only if it beats live.

Explainability without dependencies. Both ensemble members are linear, so app.py computes word-level explanations straight from the model: contribution = article's TF-IDF values × (LogReg coefficients + weighted ComplementNB log-probabilities). The editor sees exactly which words drove the call. SHAP remains an optional deep-dive; nothing in the shipped path needs it.

---

Reproduce / retrain

# expects 01_dataset/news.csv + validation.csv (official repo layout), or:
export PREDICT_IT_DATA=/path/to/01_dataset
python train_final.py          # writes model/ artifacts + prints gate sweep
# optional: export MLFLOW_TRACKING_URI=... first to log the run + gate curve

Trained and exported with scikit-learn 1.5.2 (model/metrics.json records the exact version). If the Vayu serving image resolves a different sklearn version and refuses the pickle, re-run train_final.py in the workspace — it regenerates identical artifacts in under two minutes.

Run the demo UI

cd app && streamlit run app.py

The app auto-locates model.joblib, category_labels.json, and news.csv whether it sits in this package or inside the official repo (06_build_app/). Sidebar: gate slider (default 0.70) + optional Vayu Model Serving host for a remote cross-check. Tabs: Classify (single article, confidence bar, explanation), Triage queue (escalations, one-click confirm/correct, corrections buffer), Live stream (news.csv feed with auto-routed / escalated / editor-seconds-saved counters).

---

Improvement backlog (post-hackathon)

• Wire corrections.csv sync to Object Storage on a schedule (currently a local buffer + documented flow).
• Add an explicit out-of-distribution detector (e.g., max-probability + entropy combo) ahead of the gate.
• Calibrate probabilities (isotonic) on a validation fold if the gate is pushed below 0.65.
• SHAP panel as an optional "deep explain" for contested escalations.