Demographic Bias in Automated Toxicity Detection

INFSCI 2185: Fairness, Accountability, Transparency & Ethics in AI

Team: Varun Shelke (VPS27) · Shubham Sarvankar (SHS702)

---

Overview

This project is a systematic fairness audit of Detoxify's unbiased toxicity classifier — a RoBERTa-based model trained with debiasing objectives — applied to the Civil Comments dataset. Despite its name, the model exhibits significant and statistically robust demographic disparities. Comments mentioning Black identity are flagged as toxic at 3.02× the overall rate even when non-toxic, and the LGBTQ+, Muslim, and white identity groups are all elevated above baseline. We characterize the bias across four complementary lenses (subgroup AUC, FPR disparity, counterfactual causal analysis, and intersectionality), compare three model variants to test whether debiasing training actually helps, and demonstrate a threshold-optimization mitigation that reduces maximum FPR disparity from 3.07× to 1.20× at a cost of only * −1.1% F1*.

---

Dataset

Property	Value
Source	Civil Comments (Jigsaw Unintended Bias in Toxicity Classification)
Evaluation subset	21,293 identity-annotated comments
Toxic prevalence	11.1%
Identity dimensions	24 groups across race, religion, gender, sexuality, disability
Identity labeling	Continuous annotations (fraction of annotators who agreed); binarized at ≥ 0.5
Toxicity labeling	Continuous; binarized at ≥ 0.5

The 21,293-comment subset comprises all comments with at least one identity annotation — the population where fairness disparities are most measurable and most consequential.

---

Model

Primary model: Detoxify unbiased — RoBERTa-base fine-tuned on Civil Comments with identity-debiasing objectives ( balanced training over identity groups).

Baseline performance on test set (21,293 comments):

Metric	Value
AUC-ROC	0.9492
PR-AUC	0.7520
F1 Score	0.6611
Precision	0.7054
Recall	0.6220
Overall FPR	3.24%
Accuracy	92.9%

---

Findings

1. False Positive Rate Disparities

The False Positive Rate (FPR) — the fraction of genuinely non-toxic comments incorrectly flagged as toxic — is the primary fairness metric, as it directly measures who gets wrongly accused.

Identity Group	FPR	Disparity Ratio
black	9.78%	3.02×
white	6.10%	1.88×
homosexual / gay / lesbian	4.09%	1.26×
muslim	3.90%	1.20×
transgender	3.77%	1.17×
psychiatric / mental illness	3.32%	1.02×
Overall	3.24%	—

All disparities for the top groups are statistically significant after Benjamini-Hochberg FDR correction at α = 0.05 across 24 simultaneous comparisons.

The FPR disparity is robust across identity annotation thresholds (0.5, 0.6, 0.7) — the ranking of affected groups does not change.

2. Subgroup AUC

Subgroup AUC measures how well the model discriminates toxic from non-toxic content within each identity group. Lower values indicate the model is a worse judge of toxicity specifically when that identity is mentioned.

Identity Group	Subgroup AUC	BPSN AUC	BNSP AUC
heterosexual	0.740	0.884	0.894
black	0.853	0.829	0.961
homosexual / gay / lesbian	0.856	0.890	0.939
white	0.870	0.866	0.955
muslim	0.880	0.906	0.941
psychiatric / mental illness	0.900	0.906	0.947
jewish	0.919	0.932	0.938

BPSN AUC (Background Positive, Subgroup Negative): measures whether the model assigns higher scores to genuinely toxic background comments than to non-toxic comments from the identity subgroup. A low BPSN AUC for a group means the model is confusing neutral identity-mentioning comments with toxic content from other contexts.

The black group's BPSN AUC of 0.829 is the lowest of any group with sufficient data, confirming the FPR finding: the model systematically over-scores neutral comments about Black identity relative to actually toxic content from other contexts.

3. Multi-Model Comparison: Does Debiasing Training Work?

We compared three Detoxify variants on the same 21,293 test comments to test whether the unbiased model's debiasing training actually reduces disparities:

Model	AUC-ROC	F1	Overall FPR	Max FPR Disparity	Worst Group
original (BERT-base)	0.8999	0.4918	2.19%	2.92×	gay/lesbian
unbiased (RoBERTa)	0.9492	0.6611	3.24%	3.02×	black
multilingual (XLM-R)	0.9475	0.6545	4.49%	3.36×	transgender

Key finding: The unbiased model is substantially better at detecting toxicity (AUC +0.05, F1 +0.17) but its FPR disparity for the Black identity group is worse than the original BERT model (3.02× vs. 1.73×). The debiasing training appears to have reduced disparities for LGBTQ+ groups (original: 2.92× → unbiased: 1.26×) while amplifying them for Black-identity comments. This suggests that identity-term debiasing may shift rather than eliminate demographic harms.

4. Counterfactual Analysis: Causal Evidence

Purely correlational FPR analysis cannot determine whether the model reacts to identity terms themselves or to other confounding features of comments that mention those identities. To test causality, we swapped identity terms in false-positive comments (non-toxic comments incorrectly flagged) and measured score changes.

Swap	n FPs	Mean Score Change	% Resolved (< 0.5)
black → white	48	−0.142	41.7%
white → black	53	+0.086	0.0%
muslim → christian	16	−0.081	6.3%
christian → muslim	12	+0.090	0.0%
gay → straight	5	−0.227	60.0%

The asymmetry is striking: replacing "black" with "white" drops the toxicity score by −0.142 on average and resolves 42% of false positives entirely. The reverse swap (white → black) raises scores by +0.086 with zero false positives resolved. This asymmetry provides causal evidence that the model reacts to the identity term itself — not merely to content that happens to mention that identity. The same pattern holds for muslim/christian and gay/straight.

5. Intersectionality Analysis

Single-axis analysis misses harms to individuals belonging to multiple marginalized groups simultaneously. We computed FPR for all identity pairs with at least 20 negative-class examples (n ≥ 20).

Highest-disparity intersections:

Intersection	FPR	Disparity Ratio	Compounding Effect
black × jewish	19.0%	5.88×	+0.093
jewish × male	10.0%	3.09×	+0.069
muslim × white	9.4%	2.90×	+0.033
male × muslim	8.6%	2.66×	+0.043
black × homosexual/lesbian	8.0%	2.47×	−0.018

The black × jewish intersection (5.88× FPR) substantially exceeds either group's individual FPR (black: 3.02×, jewish: 0.96×), a clear compounding effect. 7 of 33 pairs with sufficient data show positive compounding (intersection FPR exceeds the worst individual group's FPR). Single-axis fairness audits miss these compounded harms entirely.

6. AAVE / Dialect Analysis

Prior work (Sap et al. 2019) found that African American Vernacular English (AAVE) text is disproportionately flagged as offensive by NLP classifiers. We tested whether dialect, rather than identity terms, drives the elevated FPR for Black-mentioning comments.

Using an expanded 30-pattern weighted AAVE feature classifier (617 comments detected vs. 10 with the baseline keyword approach), AAVE-classified comments show FPR = 0.89× below the overall rate — not elevated. This result holds across all AAVE detection thresholds tested.

Interpretation: Linguistic features associated with AAVE are not driving the elevated FPR for Black-identity comments in this dataset. The evidence points to identity-term presence (the word "black") as the primary driver, consistent with the counterfactual analysis above. Note that only 617 of 21,293 comments (2.9%) were classified as AAVE, so power is limited.

---

Mitigation: Per-Group Threshold Optimization

Method

Per-subgroup decision thresholds were learned on a 70% development set using 5-fold cross-validation to equalize FPR across identity groups, then evaluated on a held-out 30% test set (6,388 comments, never seen during optimization).

Learned thresholds for high-FPR groups:

Identity	Baseline Threshold	Optimized Threshold	Change
black	0.50	0.716	+0.216
white	0.50	0.668	+0.168
psychiatric / mental illness	0.50	0.592	+0.092
muslim	0.50	0.532	+0.032

For comments mentioning multiple identities, the mean of applicable thresholds is used (a conservative choice that reduces over-raising for low-prevalence identity co-occurrences).

Evaluation (Held-out Test Set)

Metric	Baseline	Mitigated	Change
F1 Score	0.6337	0.6267	−1.10%
Accuracy	0.9234	0.9230	−0.04 pp
Overall FPR	3.59%	3.47%	−0.12 pp
Max FPR Disparity	3.07×	1.20×	−61%
FPR Range (max − min)	9.99 pp	4.17 pp	−58%

The mitigation reduces maximum FPR disparity by 61% at a cost of only 1.1% relative F1. The cost is primarily borne as increased false negatives (toxic comments that escape detection) for the groups whose thresholds are raised — a fundamental accuracy-fairness tradeoff that cannot be eliminated by threshold adjustment alone.

---

Limitations

1. Correlational vs. causal (single-axis): The core FPR and subgroup AUC analyses are correlational. The counterfactual analysis partially addresses this but is limited to a mechanical term-swap approach that can produce semantically altered text (e.g., "black market" → "white market"). Idiom-containing comments were not filtered.

2. Annotator bias: Higher FPR for a group is consistent with three explanations: model bias (spurious association with identity terms), annotator labeling bias (raters more likely to mark identity-mentioning comments as toxic), and genuine content differences. This analysis cannot fully disentangle these.

3. AAVE detection: The keyword classifier, even at 617 comments with 30 features, is an approximation. No gold-standard AAVE labels exist for this dataset.

4. Intersectionality power: Many intersectional groups have very small sample sizes (n < 30 negative examples). Estimates for these groups are unstable.

5. Mitigation generalizability: Per-group thresholds were optimized on the Civil Comments test subset. Performance on other domains (social media, news comments) is unknown. Threshold optimization does not address the underlying representation problem.

6. Single model family: All analysis uses Detoxify. Findings may not generalize to other toxicity classifiers ( Perspective API, HateBERT, etc.).

---

Notebook Pipeline

Notebook	Description
01_data_exploration.ipynb	EDA: label distributions, identity group frequencies
02_progress_report.ipynb	Early-stage progress report
03_model_loading.ipynb	Load Detoxify model, verify setup
04_generate_predictions.ipynb	Run unbiased model on 21,293 test comments
05_classification_metrics.ipynb	Baseline accuracy metrics (AUC, F1, confusion matrix)
06_fairness_metrics.ipynb	FPR/TPR/precision by identity group + bootstrap CIs + BH correction
07_subgroup_auc_metrics.ipynb	Subgroup AUC, BPSN, BNSP metrics
08_error_analysis.ipynb	False positive case analysis
09_aave_analysis.ipynb	AAVE dialect analysis (keyword classifier)
09b_aave_improved.ipynb	Expanded AAVE analysis (30 features, threshold sensitivity)
10_threshold_optimization.ipynb	Per-group threshold optimization (CV on dev, eval on held-out test)
11_calibration.ipynb	Post-hoc calibration (Platt scaling, isotonic regression)
12_accuracy_fairness_tradeoff.ipynb	Pareto frontier analysis; mitigation evaluation on held-out test
13_final_report.ipynb	Summary report with all key findings
14_model_comparison.ipynb	Compare original/unbiased/multilingual Detoxify variants
15_counterfactual_analysis.ipynb	Causal identity-swap analysis on false positive set
16_intersectionality.ipynb	Multi-identity co-occurrence and compounding FPR analysis

---

Setup

Requirements

• Python 3.13+
• GPU recommended for notebooks 04, 14 (model inference)

Installation

git clone https://github.com/class-data-mining-master/2026-spring-fate-project-flagged-unfairly.git
cd 2026-spring-fate-project-flagged-unfairly

python -m venv .venv
# Windows:
.venv\Scripts\activate
# macOS/Linux:
source .venv/bin/activate

pip install -r requirements.txt

Dataset

Download the Civil Comments dataset from Kaggle: https://www.kaggle.com/competitions/jigsaw-unintended-bias-in-toxicity-classification/data

Place train.csv in data/. The test subset (test_public_expanded.csv) used throughout this analysis must also be in data/.

Running the Pipeline

Notebooks are designed to be run in order (01 → 16). Notebooks 03–04 require the dataset and will generate data/predictions_test.csv. All subsequent notebooks load from that file.

Notebooks can be executed headlessly:

jupyter nbconvert --to notebook --execute --inplace notebooks/05_classification_metrics.ipynb

---

References

• Dixon et al. (2018). Measuring and Mitigating Unintended Bias in Text Classification. AIES. — Introduced Civil Comments dataset and identity-term bias framework.
• Borkan et al. (2019). Nuanced Metrics for Measuring Unintended Bias with Real Data. WWW. — Introduced subgroup AUC, BPSN, BNSP metrics.
• Sap et al. (2019). The Risk of Racial Bias in Hate Speech Detection. ACL. — AAVE over-flagging; annotator racial bias.
• Garg et al. (2019). Counterfactual Fairness in Text Classification through Robustness. AAAI. — Counterfactual data augmentation methodology.
• Buolamwini & Gebru (2018). Gender Shades. FAccT. — Intersectional analysis of AI systems.
• Crenshaw (1989). Demarginalizing the Intersection of Race and Sex. — Foundational intersectionality framework.
• Hanu & Unitary Team (2020). Detoxify. GitHub. — Model used throughout this audit.

---

Authors

Varun Shelke (VPS27) · Shubham Sarvankar (SHS702)