Toxic Comment Topic Modeling with LDA

Project overview

This natural language processing project uses Latent Dirichlet Allocation (LDA) to discover themes in Wikipedia comments and compare those unsupervised topics with six supervised toxicity labels.

The analysis focuses on the relationship between what users discuss and how they communicate. It examines whether certain topics are more strongly associated with toxic behavior and where topic-based groupings disagree with the original toxicity labels.

Research questions

• What thematic topics emerge naturally from Wikipedia comments?
• Which topics are most strongly associated with toxic behavior?
• Where do unsupervised topics disagree with supervised toxicity labels?
• Can topic modeling support exploratory content-moderation analysis?

Dataset

The project uses the Jigsaw Toxic Comment Classification dataset from Kaggle.

Dataset source:

Wikipedia Toxic Comment Dataset on Kaggle

The full dataset contains 159,571 comments and six toxicity labels:

• toxic
• severe_toxic
• obscene
• threat
• insult
• identity_hate

The analysis uses a stratified sample of 19,980 comments.

The raw train.csv file is not included in this repository.

Tools and methods

• Python
• pandas and NumPy
• NLTK
• CountVectorizer
• Latent Dirichlet Allocation
• Unigrams and bigrams
• Word clouds
• Perplexity-based topic selection
• Topic assignment
• Topic-level toxicity analysis
• Agreement and disagreement analysis
• Data visualization

Analytical workflow

1. Load and inspect the labeled comment dataset
2. Create a stratified analytical sample
3. Clean and normalize comment text
4. Tokenize and lemmatize text with NLTK
5. Remove standard and domain-specific stop words
6. Create a document-term matrix with CountVectorizer
7. Compare candidate topic counts using perplexity
8. Train a 12-topic LDA model
9. Assign a dominant topic to each comment
10. Compare topic-level toxicity rates with the dataset baseline
11. Analyze agreement and disagreement between topic groups and labels
12. Interpret the results and limitations

Main findings

• Twelve interpretable topics emerged across technical editing discussions, administrative actions, and interpersonal conflict.
• General discussion represented approximately 48.2% of the analytical sample.
• Personal-attack topics showed toxicity rates of approximately 52%, compared with approximately 4.6% for article-content discussions.
• Topic groupings and toxicity labels agreed approximately 46.5% of the time.
• Many comments assigned to higher-risk topics were still labeled non-toxic, indicating that topic context alone is not sufficient for toxicity classification.
• LDA is more useful here for exploratory analysis and label diagnostics than as a standalone toxicity classifier.

Repository structure

toxic-comment-topic-modeling/
├── README.md
├── toxic_comment_lda_analysis.ipynb
├── requirements.txt
└── toxic_comment_lda_presentation.pdf

Presentation

View the project presentation

How to run the notebook

1. Download train.csv from the Kaggle dataset linked above.
2. Place train.csv in the repository root, next to the notebook.
3. Install the required packages:

pip install -r requirements.txt

4. Open the notebook:

jupyter notebook toxic_comment_lda_analysis.ipynb

5. Run the notebook from top to bottom.

The notebook may download required NLTK resources during execution.

Limitations

• Bag-of-words topic modeling does not preserve word order or contextual meaning.
• LDA cannot reliably capture sarcasm, irony, or context-dependent language.
• Topic labels require manual interpretation.
• The selected number of topics and the toxicity threshold involve judgment.
• Results are specific to Wikipedia comments and may not generalize to other platforms.
• Topic modeling should not be treated as a substitute for a supervised toxicity classifier.

Author

Qihong Huang
Data Science graduate student