News Topic Classification with AG News

Multi-class text classification on 120,000 real news articles, comparing classical NLP models on the AG News benchmark.

---

I built this as part of my work in the NLP space during my final year at the School of Artificial Intelligence, Nanjing University of Information Science and Technology (NUIST). The goal was straightforward: given a short news article, can a classical ML pipeline — without any deep learning — reliably categorise it into one of four topics? The answer turns out to be yes, and the more interesting question becomes why certain models work better than others, and where they still fail.

The dataset loads directly from Hugging Face Hub, so there's nothing to download manually.

---

Dataset

AG News (Zhang et al., NIPS 2015) — 127,600 news articles from 2,000+ news sources.

Split	Samples
Train	120,000
Test	7,600

Categories: World · Sports · Business · Sci/Tech (30,000 train / 1,900 test per class — perfectly balanced)

---

What's in the Notebook

The notebook is structured as a complete research workflow, not just a training script:

1. Dataset loading — pulled live from Hugging Face; no local CSV needed
2. EDA — class balance, article length distributions, sample inspection
3. Preprocessing — lightweight pipeline (lower, clean, stopword removal); I deliberately kept it light and explain why aggressive stemming can hurt on short texts
4. Feature extraction — TF-IDF with sublinear scaling and bigrams; the sublinear_tf=True trick is worth understanding
5. Model comparison — four classifiers timed and evaluated, with an accuracy vs. training speed trade-off plot
6. Full evaluation — confusion matrix, per-class accuracy breakdown
7. Feature interpretability — top coefficient weights per category from Logistic Regression
8. Error analysis — identifies which category pairs are most often confused and shows real examples
9. Live prediction — predict_topic() function you can test on any headline

---

Models

Model	Notes
Complement Naive Bayes	Better for multi-class than standard Multinomial NB
Logistic Regression	Strong baseline; also gives interpretable feature weights
Linear SVM	Fast, robust on high-dimensional sparse data
SGD Classifier	Online learning, good when scaling to larger datasets

All models use the same TF-IDF representation for a fair comparison.

---

Results

Run the notebook to reproduce. Typical results on the test set:

Model	Test Accuracy
Complement Naive Bayes	~90%
Logistic Regression	~92–93%
Linear SVM	~92–93%
SGD Classifier	~91%

The hardest pair to separate is World vs. Business — economic and geopolitical news often shares vocabulary, which is visible in both the confusion matrix and the error analysis.

---

Getting Started

git clone https://github.com/mdarifhasanbadsha/news-topic-classification.git
cd news-topic-classification
pip install -r requirements.txt
jupyter notebook news_topic_classification.ipynb

The dataset downloads automatically on first run (~30 seconds).

---

Requirements

datasets>=2.14.0
scikit-learn>=1.2.0
numpy
pandas
matplotlib
seaborn
nltk
jupyter

---

Project Structure

news-topic-classification/
│
├── news_topic_classification.ipynb   # Main notebook
├── requirements.txt
├── README.md
│
└── outputs/
    ├── eda.png                        # Class distribution & length analysis
    ├── model_comparison.png           # Accuracy vs speed trade-off
    ├── best_model_eval.png            # Confusion matrix + per-class accuracy
    └── feature_importance.png         # Top predictive words per category

---

A Note on Approach

This project intentionally uses classical ML rather than a pre-trained transformer. The point is to understand what TF-IDF features actually capture, where the representation breaks down, and why certain model families (linear models vs. tree-based) behave differently on sparse high-dimensional text data. Fine-tuning DistilBERT would get you to 95%+ easily, but you'd learn less about the problem.

---

Citation

@inproceedings{Zhang2015CharacterlevelCN,
  title   = {Character-level Convolutional Networks for Text Classification},
  author  = {Xiang Zhang and Junbo Jake Zhao and Yann LeCun},
  booktitle = {NIPS},
  year    = {2015}
}

---

Md Arif Hasan Badsha
School of Artificial Intelligence, Nanjing University of Information Science and Technology
github.com/mdarifhasanbadsha