A natural language processing project applying TF, TF-IDF, and Zipf's Law to analyze word frequency patterns and topic evolution across five years of public tweet data.
Identify how public discourse topics shift over time by applying text mining techniques to a large-scale tweet dataset, tracking which themes dominate each year and how language patterns follow statistical distributions.
Text Preprocessing
- Tokenization, stopword removal, and normalization across 5 years of tweet data
- Each year treated as a separate document corpus
Analysis Techniques
- Term Frequency (TF) — normalized word counts to surface dominant topics per year
- TF-IDF — identified statistically significant terms unique to each year, filtering out common noise
- Zipf's Law validation — log-log frequency-rank plots confirming power-law distribution in natural language data
Visualization
- Word frequency histograms per year
- Zipf's Law distribution plots
- Top-N term comparison across years
| Year | Dominant themes | Signal |
|---|---|---|
| 2017 | Automotive, performance | Product-focused discourse |
| 2018 | Hardware, innovation | R&D exploration phase |
| 2019 | Tesla, Starship, energy | Multi-domain expansion |
| 2020 | COVID, technology, self | Crisis + reflection period |
| 2021 | AI, artificial intelligence | Strategic pivot signal |
Topic evolution tracked through TF-IDF reveals clear shifts in strategic focus — from product to R&D to crisis response to AI, demonstrating how text mining can surface narrative patterns in unstructured social data.
Python · NLP · TF-IDF · Zipf's Law · Pandas · Matplotlib · Jupyter Notebook
This project demonstrates core text analysis and NLP skills applicable to social listening and trend detection, content analysis and brand monitoring, behavioral signal extraction from unstructured data, and foundation techniques used in modern LLM and RAG pipelines.
Data source: Kaggle — Public Tweet Archive (2010–2021)