Fake News Detection Using Machine Learning

A complete, leakage-controlled fake news classification pipeline built from scratch on the ISOT dataset, comparing seven classifiers under identical conditions. Best model on the held-out test set: Logistic Regression — 98.45% accuracy, F1 0.9831, with only 121 errors on 7,820 held-out articles.

Built as Project 1 of the IICT Summer Internship on AI & Machine Learning (2026), VIT Chennai.

Why this project is a little different

The ISOT dataset is famous for handing out 99%+ accuracies. Before training anything, this project measured why: the subject column identifies the label by itself (its categories never overlap between classes), and the "(Reuters)" dateline appears in 99.2% of real articles vs 0.0% of fake ones. Both leaks are removed before any model sees the text, 5,795 duplicated rows are dropped, and only then are the models compared. The paper's discussion section shows what the winning model actually learned — newsroom style vs aggregator style — which is a more honest reading of near-ceiling scores than "the model detects lies."

Results (Held-Out Test Set)

All seven models trained on the same 5,000-feature CountVectorizer matrix and the same stratified 80/20 split (seed 42). The dataset was split into a training set (31,278 articles) used only for model training, and a held-out test set of 7,820 articles that no model saw during training or model selection. Every number in the table below, every confusion matrix in graphs/, and every model-comparison chart were computed on that held-out test set. Precision / Recall / F1 refer to the fake class.

Model	Accuracy	Precision	Recall	F1
Logistic Regression	0.9845	0.9860	0.9802	0.9831
MLP Neural Network	0.9844	0.9870	0.9788	0.9829
Linear SVM	0.9803	0.9815	0.9754	0.9784
Random Forest	0.9795	0.9891	0.9659	0.9774
Decision Tree	0.9481	0.9426	0.9441	0.9434
Multinomial Naive Bayes	0.9405	0.9283	0.9430	0.9356
KNN (k = 5)	0.7739	0.7314	0.8001	0.7642

Three findings worth clicking through for:

• The three (near-)linear models cluster within 0.005 F1 — the classes are close to linearly separable in count space.
• KNN collapses on the very same features (Euclidean distance on raw counts follows article length, not content) and inverts the cost profile: 0.01 s to "train", 4.9 s to predict.
• CountVectorizer beat TF-IDF in a measured cross-validated probe (0.9798 vs 0.9764 F1) — the opposite of the usual default, which is exactly why the choice was measured instead of assumed.

Evaluation protocol

The training set (31,278 articles) was used only for model training, and the CountVectorizer vs TF-IDF choice was decided by 3-fold cross-validation on the training set only — the test set never participated in that decision. All reported numbers above (Accuracy, Precision, Recall, F1) and every generated figure (confusion matrices, model-comparison charts, feature-importance plot) come from the held-out test set of 7,820 articles that no model saw during training or model selection. No training-set or validation-fold scores are reported as final results. Best-model selection also uses held-out-test F1; the same convention is used in the IEEE paper, the project report, and the presentation.

Screenshots

The class-distribution and top-words figures come from the cleaned corpus; the F1 comparison and confusion matrix are computed on the held-out test set (7,820 articles).

All 17 generated figures (EDA, word clouds, per-model confusion matrices on the held-out test set, four held-out-test metric comparisons) are in graphs/.

Deliverables

• 📄 IEEE paper (4 pages, IEEEtran): ieee_paper/fake_news_paper.pdf (LaTeX source included)
• 📝 Project report (10 pages): report/Fake_News_Detection_Project_Report.pdf (.docx)
• 🎤 Presentation (6 slides + speaker notes): presentation/Fake_News_Detection_Presentation.pdf (.pptx, speaker notes)
• 📊 Evidence files: every number in the documents is written to disk by the pipeline itself — see outputs/ (inspection reports, EDA observations, vectorizer comparison, model comparison, classification reports, best-model analysis).

Dataset (not included — 2 files, ~116 MB)

This repository does not ship the dataset. The project uses the ISOT Fake News Dataset (Ahmed, Traore & Saad, University of Victoria ISOT Lab), most easily downloaded from its Kaggle mirror:

• Kaggle: Fake and Real News Dataset

Setup:

1. Download and extract the two files Fake.csv (23,481 articles) and True.csv (21,417 articles).
2. Place both files in dataset/raw/.
3. Run the pipeline (below). The cleaned dataset (dataset/processed/clean_news.csv, ~160 MB) and the three large model dumps are generated locally and are intentionally git-ignored.

Installation

git clone https://github.com/<your-username>/Fake-News-Detection.git
cd Fake-News-Detection
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Requirements: Python 3.10+ with pandas, numpy, matplotlib, scikit-learn, nltk, wordcloud, joblib (see requirements.txt). The NLTK stop-word list downloads automatically on first run.

How to run

python main.py

One command reproduces everything in roughly 6–10 minutes on a normal laptop: cleaning, preprocessing, all EDA figures, the vectorizer comparison, all seven models, every metric, every chart, and the saved best model. All randomness is fixed at random_state=42.

To classify new text with the shipped trained model:

import joblib
model = joblib.load("models/best_model.joblib")
vectorizer = joblib.load("models/count_vectorizer.joblib")
print(model.predict(vectorizer.transform(["your article text here"])))  # 1 = fake, 0 = real

Project pipeline

Week 1 — Data understanding & cleaning. Merge the two files with a label column; inspect (reports saved to outputs/); remove 211 exact duplicates and 5,584 repeated article contents; strip the (Reuters) dateline and drop the leaking subject/date columns; rescue 631 empty-body articles by combining title + body; normalise text (URLs, HTML, mentions, punctuation, numbers, stop words). Final corpus: 39,098 articles (21,196 real / 17,902 fake).

Week 2 — EDA & feature engineering. Five figures with written, number-backed observations; stratified 80/20 split before any vectorizer is fitted (31,278 / 7,820, 45.8% fake on both sides); CountVectorizer vs TF-IDF compared with a cross-validated Logistic Regression probe on the training set only.

Week 3 — Models & evaluation. Seven classifiers trained on identical matrices; accuracy/precision/recall/F1, confusion matrix and classification report for each computed on the held-out test set (7,820 unseen articles); four comparison charts; best model auto-selected by held-out-test F1 and saved with the fitted vectorizer.

Week 4 — Documentation. IEEE paper, project report and presentation, written exclusively from the pipeline's own outputs.

Repository structure

Fake-News-Detection/
├── src/                  # modular pipeline (config, loading, inspection,
│                         # cleaning, preprocessing, EDA, features, models,
│                         # evaluation)
├── main.py               # runs the whole pipeline end to end
├── dataset/
│   ├── raw/              # put Fake.csv and True.csv here (git-ignored)
│   └── processed/        # clean_news.csv generated here (git-ignored)
├── graphs/               # all 17 generated figures
├── outputs/              # evidence files: reports, tables, analyses
├── models/               # best model + vectorizer + small model dumps
├── ieee_paper/           # LaTeX source, figures, compiled PDF
├── report/               # project report (.docx + PDF)
├── presentation/         # 6-slide deck (.pptx + PDF) + speaker notes
├── requirements.txt
├── LICENSE               # MIT
└── README.md

Reproducibility note

Everything is seeded (random_state=42), so a rerun on the same environment reproduces identical numbers. Across different library versions (pandas/NLTK/scikit-learn), tiny drifts of ≤0.003 in individual metrics are normal — the stop-word list and solver internals evolve — but the ranking, the best model, and every conclusion in the paper are stable. The shipped outputs/ files are the exact run all documents were written from.

Future work

Cross-source evaluation to measure out-of-distribution performance; n-gram and character-level features; a cosine-distance KNN as the controlled fix for the distance-metric failure; calibrated probabilities for human-in-the-loop review; a small fine-tuned transformer under the same leakage-controlled protocol; a Flask/Streamlit demo around the saved model.

Acknowledgments

Developed during the IICT Summer Internship on AI & Machine Learning (2026) under the mentorship of Dr. Ashok. Dataset credit: H. Ahmed, I. Traore and S. Saad — "Detection of Online Fake News Using N-Gram Analysis and Machine Learning Techniques" (ISDDC 2017) and "Detecting opinion spams and fake news using text classification" (Security and Privacy, 2018).

License

Released under the MIT License.