snt-data

Private data pipeline for the Sweenk News Taxonomy (SNT) classifier.

Builds a unified corpus from multiple sources, deduplicates, runs quality filters, and produces gold-quality labels via Sonnet.

Layout

snt_data/
  corpus/           # ingest, dedup, quality filter, labelers
  snt_taxonomy.py   # canonical 17 IPTC top-level + 133 sub-cats + 16 regions
data/
  corpus.sqlite     # canonical labeled corpus (Git LFS)
analyses/           # validation reports

Sources currently ingested

Source	Rows	Notes
Sweenk eval	~12K	production reads
HuffPost	~200K	Kaggle dataset
CC-News	~30K	2017–2018 snapshot

Future sources

• sweenk/archived-databases — older Sweenk corpus snapshots; useful for source discovery (which publishers we've historically scraped) but not for re-import, since current corpus already covers the topical range we need.

Pipeline

1. ingest/* → raw rows into data/staging.sqlite
2. dedupe.py → mark duplicates
3. quality_filter.py → mark quality_passed=1
4. label_classla.py → silver labels via classla classifier
5. label_sonnet_full.py → gold labels via Sonnet (run in rounds)

Labeling rounds

# 1. Pick stratified sample, write NDJSON chunks
uv run python -m snt_data.corpus.label_sonnet_full prepare --per-category 100 --chunk 100

# 2. Dispatch Sonnet subagents to label each chunk → results NDJSON
# 3. Persist labels back to corpus.sqlite
uv run python -m snt_data.corpus.label_sonnet_full writeback

Run on the Mac Mini via launchd (see scripts/); each round commits and pushes.