SNT Classifier v0.6

Sweenk News Taxonomy — fine-tuned xlm-roberta-large for multi-label news classification.

Model weights (public): huggingface.co/sweenk/snt-classifier — loads with AutoModel.from_pretrained("sweenk/snt-classifier", trust_remote_code=True); the model card there documents the full taxonomy, per-class metrics, and known limitations.

Trained on ~264K labeled news articles across 12 top-level categories (multi-label) and 71 sub-categories (multi-label). A trade-deal article correctly comes back as world + money_and_business + politics — one label was never enough for real news.

Performance (v0.6, 26,412 held-out test articles)

Metric	raw @0.5	tuned thresholds
L1 macro F1	0.817	0.847
L1 micro F1	0.831	0.857
Primary-L1 accuracy	0.833	—
L2 macro F1	0.647	0.685
L2 micro F1	0.753	0.754

All 12 L1 categories clear the 0.65 per-class F1 floor after threshold tuning (lowest: human_stories 0.67; highest: sports 0.95). v0.6 retrains on the corrected v0.5.1 corpus plus ~23K newly teacher-labeled articles — Sweenk production rows and a daily.dev science/tech ingest aimed at the previously starved science_and_space / tech_and_ai classes, which rose to 0.83 / 0.86 tuned F1 — with majority-class capping (--cap-per-class 25000) on top of per-class pos_weight. Aggregate F1 is not directly comparable across versions because the gold labels and corpus changed between them. Note: gold labels are LLM-teacher-assisted, not human-annotated — see the model card for the full honesty section.

Labels

L1 — 12 top-level categories (multi-label)

sports · politics · world · entertainment_and_pop_culture · money_and_business · crime_and_justice · tech_and_ai · science_and_space · health_and_wellness · lifestyle · weather_and_environment · human_stories

L2 — 71 sub-categories (multi-label)

Nested under the L1s — the full machine-readable tree (l1_keys, l2_keys, l2_parent, tuned thresholds) ships in the HF repo's config.json, in snt_model/taxonomy.py, and is served live at GET /taxonomy.

Quick start

git clone https://github.com/sweenk/snt-model
cd snt-model
uv sync

# Download checkpoint from HuggingFace
python -c "
from huggingface_hub import snapshot_download
snapshot_download('sweenk/snt-classifier', local_dir='checkpoints/v0.6/best')
"

# Start the HTTP API
make serve
# → http://localhost:8090

# Predict from text
curl -s -X POST http://localhost:8090/predict \
  -H "Content-Type: application/json" \
  -d '{"title": "OpenAI releases GPT-5 with improved reasoning", "body": "OpenAI today announced..."}'

# Predict from URL
curl -s -X POST http://localhost:8090/predict_url \
  -H "Content-Type: application/json" \
  -d '{"url": "https://techcrunch.com/..."}'

API response (v0.5+ multi-label)

{
  "l1": [{"key": "world", "p": 0.95}, {"key": "money_and_business", "p": 0.94}, {"key": "politics", "p": 0.85}],
  "primary_l1": "world",
  "primary_l1_confidence": 0.95,
  "l2": [{"key": "markets_and_investing", "p": 0.88}],
  "l2_by_l1": {"money_and_business": ["markets_and_investing"]},
  "model_version": "v0.6"
}

l1 contains every category above its per-class tuned threshold (thresholds.json, tuned on validation). primary_l1 is the best thresholded hit.

Labeling tool

A browser UI for reviewing and correcting labels:

make label-tool DB=../snt-data/data/corpus.sqlite SNT_DATA=../snt-data
# → http://localhost:8091

Architecture

• Encoder: xlm-roberta-large (560M params, 100 languages)
• L1 head: linear → 12-class sigmoid (BCEWithLogitsLoss + per-class pos_weight, capped at 10×)
• L2 head: linear → 71-class sigmoid (BCEWithLogitsLoss, weight 2.0)
• Training: 3 epochs, AdamW lr=2e-5, batch=4, grad_accum=8, warmup=10%, bf16 on CUDA
• Class balance: per-class pos_weight boosts rare classes; --cap-per-class N downsamples the majority L1 classes (v0.6 uses 25K primary-label rows) so they don't swamp the rest
• Thresholds: per-class, F1-optimal on validation (tune_thresholds_v0_5.py)
• Taxonomy: 12 L1 / 71 L2, vendored in snt_model/taxonomy.py so training is fully self-contained; pass --sweenk-core <path> to use Sweenk's internal canonical copy instead. Inference reads the taxonomy baked into the checkpoint — no dependency either way.
• Checkpoint marker: head_config.json records head activations + version — inference auto-detects sigmoid vs legacy softmax checkpoints

Repository layout

snt_model/
  serve.py               FastAPI inference service (port 8090)
  label_tool.py          Human review UI (port 8091)
  predict.py             SNTPredictor class + CLI
  train_v0_5.py          v0.5/v0.6 multi-label training script
  data_v0_5.py           dataset + split utilities (+ cap_per_class)
  tune_thresholds_v0_5.py Per-class threshold tuning
  taxonomy.py            Vendored public taxonomy (12 L1 / 71 L2)
  train_v0_3.py, data.py Legacy v0.3 (single-label L1)
scripts/
  export_hf.py           Push model + card to HuggingFace
  regen_taxonomy.py      Regenerate taxonomy.py from sweenk_core
  mine_boundary_rows.py  Active-learning boundary miner
checkpoints/
  v0.6/best/             Trained weights (not in git — download from HF)

Roadmap

• v0.5.1 — corrective retrain (route-by-cause fixes for disasters/terror/pharma clusters)
• v0.6 — retrain on corrected + augmented corpus, majority-class capping, self-contained (vendored) taxonomy
• HuggingFace public release — sweenk/snt-classifier
• Gradio Space demo
• Automated improvement loop — continuous eval → targeted relabel → retrain
• L3 dynamic topics — entity-level tags feeding the topic registry

Related

• Model weights + card: huggingface.co/sweenk/snt-classifier — the machine-readable taxonomy ships in its config.json (and in snt_model/taxonomy.py; Sweenk's internal canonical copy lives in the sweenk_core.taxonomy package)
• Data pipeline: sweenk/snt-data

License

Code: MIT. Model weights: MIT (base model xlm-roberta-large is MIT). The training corpus contains article text from public news sources and is not redistributed with this repo or the model.