AGIR Dreaddit Experiments

Dataset Overview

Dreaddit is a specialized dataset designed for detecting psychological stress in social media text. The dataset contains Reddit posts from users discussing various mental health challenges and stressful life situations.

Dataset Statistics

• Total samples: 3,553 records (excluding headers)
• Training set: 2,837 samples
• Test set: 715 samples
• Label distribution: Binary classification (0: No stress, 1: Stress)

Data Format

The dataset includes comprehensive features:

• Text data: Original Reddit post content
• Metadata: Subreddit, post ID, sentence range, timestamp, karma
• Linguistic features: 100+ LIWC (Linguistic Inquiry and Word Count) features
• Sentiment scores: Emotion and valence measures
• Target variable: Binary stress labels (0/1) with confidence scores

Experiments and Results

This repository contains emotion detection experiments using various language models on the Dreaddit dataset. All experiments evaluate models on 715 test samples.

Experiment Code Links

Experiment	Code Location	Description
AGIR	src/agir/emotion_master.py	Base AGIR model for emotion detection
AGIR-CoT	src/agir-cot/emotion_master.py	AGIR with Chain-of-Thought prompting
AGIR-O3-Mini	src/agir-o3-mini/emotion_master.py	AGIR using O3-Mini model
AGIR-O3-Mini-CoT	src/agir-o3-mini-cot/emotion_master.py	AGIR O3-Mini with Chain-of-Thought
AGIR-After-Cases	src/agir-after-cases/emotion_master.py	AGIR with after-cases prompting
AGIR-After-Cases-CoT	src/agir-after-cases-cot/emotion_master.py	AGIR after-cases with Chain-of-Thought
AGIR-After-Cases-Few-Shot	src/agir-after-cases-few-shot/emotion_master.py	AGIR after-cases with few-shot learning
AGIR-After-Cases-Few-Shot-CoT	src/agir-after-cases-few-shot-cot/emotion_master.py	AGIR after-cases few-shot with CoT
AGIR-O3-Mini-After-Cases-Few-Shot	src/agir-o3-mini-after-cases-few-shot/emotion_master.py	AGIR O3-Mini with after-cases few-shot
AGIR-O3-Mini-After-Cases-Few-Shot-CoT	src/agir-o3-mini-after-cases-few-shot-cot/emotion_master.py	AGIR O3-Mini after-cases few-shot with CoT
O3-Mini	src/o3-mini/o3_mini.py	Direct O3-Mini model implementation
GPT-4-1-Nano	src/gpt/gpt_4_1_nano.py	GPT-4 nano model baseline

Analysis Tools

• Balanced Accuracy Calculator: src/balanced_accuracy.py
• Batch Error Checker: src/o3-mini/check_batch_errors.py
• Completed Batch Processor: src/o3-mini/process_completed_batches.py

Performance Results

Summary Rankings (Balanced Accuracy)

Rank	Model	Balanced Accuracy	Regular Accuracy	Sensitivity	Specificity
🥇 1st	AGIR-O3-Mini-CoT	79.94%	80.28%	90.51%	69.36%
🥈 2nd	AGIR-O3-Mini-After-Cases-Few-Shot-CoT	77.44%	77.90%	91.87%	63.01%
🥉 3rd	AGIR-O3-Mini-After-Cases-Few-Shot	72.89%	73.57%	94.04%	51.73%
4th	AGIR-O3-Mini	71.24%	72.03%	95.66%	46.82%
5th	AGIR-After-Cases	70.77%	71.61%	96.75%	44.80%
6th	AGIR-After-Cases-CoT	70.58%	71.47%	98.10%	43.06%
7th	O3-Mini	68.99%	69.93%	98.10%	39.88%
8th	AGIR	68.81%	69.79%	99.19%	38.44%
9th	AGIR-After-Cases-Few-Shot	68.26%	69.23%	98.37%	38.15%
10th	AGIR-After-Cases-Few-Shot-CoT	68.12%	69.09%	98.37%	37.86%
11th	AGIR-CoT	67.82%	68.81%	98.64%	36.99%
12th	GPT-4-1-Nano	62.43%	63.64%	100.00%	24.86%

Detailed Performance Analysis

🥇 1st Place: AGIR-O3-Mini-CoT

src/agir-o3-mini-cot/emotion_master.py

Confusion Matrix:

• True Positives (TP): 334 | True Negatives (TN): 240
• False Positives (FP): 106 | False Negatives (FN): 35

Key Metrics:

• Balanced Accuracy: 79.94% | Regular Accuracy: 80.28%
• Sensitivity (TPR): 90.51% | Specificity (TNR): 69.36%

---

🥈 2nd Place: AGIR-O3-Mini-After-Cases-Few-Shot-CoT

src/agir-o3-mini-after-cases-few-shot-cot/emotion_master.py

Confusion Matrix:

• True Positives (TP): 339 | True Negatives (TN): 218
• False Positives (FP): 128 | False Negatives (FN): 30

Key Metrics:

• Balanced Accuracy: 77.44% | Regular Accuracy: 77.90%
• Sensitivity (TPR): 91.87% | Specificity (TNR): 63.01%

---

🥉 3rd Place: AGIR-O3-Mini-After-Cases-Few-Shot

src/agir-o3-mini-after-cases-few-shot/emotion_master.py

Confusion Matrix:

• True Positives (TP): 347 | True Negatives (TN): 179
• False Positives (FP): 167 | False Negatives (FN): 22

Key Metrics:

• Balanced Accuracy: 72.89% | Regular Accuracy: 73.57%
• Sensitivity (TPR): 94.04% | Specificity (TNR): 51.73%

---

4th Place: AGIR-O3-Mini

src/agir-o3-mini/emotion_master.py

Confusion Matrix:

• True Positives (TP): 353 | True Negatives (TN): 162
• False Positives (FP): 184 | False Negatives (FN): 16

Key Metrics:

• Balanced Accuracy: 71.24% | Regular Accuracy: 72.03%
• Sensitivity (TPR): 95.66% | Specificity (TNR): 46.82%

---

5th Place: AGIR-After-Cases

src/agir-after-cases/emotion_master.py

Confusion Matrix:

• True Positives (TP): 357 | True Negatives (TN): 155
• False Positives (FP): 191 | False Negatives (FN): 12

Key Metrics:

• Balanced Accuracy: 70.77% | Regular Accuracy: 71.61%
• Sensitivity (TPR): 96.75% | Specificity (TNR): 44.80%

---

6th Place: AGIR-After-Cases-CoT

src/agir-after-cases-cot/emotion_master.py

Confusion Matrix:

• True Positives (TP): 362 | True Negatives (TN): 149
• False Positives (FP): 197 | False Negatives (FN): 7

Key Metrics:

• Balanced Accuracy: 70.58% | Regular Accuracy: 71.47%
• Sensitivity (TPR): 98.10% | Specificity (TNR): 43.06%

---

7th Place: O3-Mini

src/o3-mini/o3_mini.py

Confusion Matrix:

• True Positives (TP): 362 | True Negatives (TN): 138
• False Positives (FP): 208 | False Negatives (FN): 7

Key Metrics:

• Balanced Accuracy: 68.99% | Regular Accuracy: 69.93%
• Sensitivity (TPR): 98.10% | Specificity (TNR): 39.88%

---

8th Place: AGIR

src/agir/emotion_master.py

Confusion Matrix:

• True Positives (TP): 366 | True Negatives (TN): 133
• False Positives (FP): 213 | False Negatives (FN): 3

Key Metrics:

• Balanced Accuracy: 68.81% | Regular Accuracy: 69.79%
• Sensitivity (TPR): 99.19% | Specificity (TNR): 38.44%

---

9th Place: AGIR-After-Cases-Few-Shot

src/agir-after-cases-few-shot/emotion_master.py

Confusion Matrix:

• True Positives (TP): 363 | True Negatives (TN): 132
• False Positives (FP): 214 | False Negatives (FN): 6

Key Metrics:

• Balanced Accuracy: 68.26% | Regular Accuracy: 69.23%
• Sensitivity (TPR): 98.37% | Specificity (TNR): 38.15%

---

10th Place: AGIR-After-Cases-Few-Shot-CoT

src/agir-after-cases-few-shot-cot/emotion_master.py

Confusion Matrix:

• True Positives (TP): 363 | True Negatives (TN): 131
• False Positives (FP): 215 | False Negatives (FN): 6

Key Metrics:

• Balanced Accuracy: 68.12% | Regular Accuracy: 69.09%
• Sensitivity (TPR): 98.37% | Specificity (TNR): 37.86%

---

11th Place: AGIR-CoT

src/agir-cot/emotion_master.py

Confusion Matrix:

• True Positives (TP): 364 | True Negatives (TN): 128
• False Positives (FP): 218 | False Negatives (FN): 5

Key Metrics:

• Balanced Accuracy: 67.82% | Regular Accuracy: 68.81%
• Sensitivity (TPR): 98.64% | Specificity (TNR): 36.99%

---

12th Place: GPT-4-1-Nano

src/gpt/gpt_4_1_nano.py

Confusion Matrix:

• True Positives (TP): 369 | True Negatives (TN): 86
• False Positives (FP): 260 | False Negatives (FN): 0

Key Metrics:

• Balanced Accuracy: 62.43% | Regular Accuracy: 63.64%
• Sensitivity (TPR): 100.00% | Specificity (TNR): 24.86%

---

Model Performance Insights

1. Best Overall Performance: AGIR-O3-Mini-CoT achieves the highest balanced accuracy with good performance on both stress and non-stress detection.

2. High Sensitivity Models: Most models show excellent sensitivity (>90%), indicating strong ability to detect stress cases.

3. Specificity Challenge: Lower specificity scores suggest models tend to over-predict stress, leading to false positives.

4. Chain-of-Thought Impact: CoT prompting generally improves performance, especially when combined with O3-Mini models.

5. After-Cases & Few-Shot Learning: These techniques show promising results, particularly when combined with O3-Mini models.

Repository Structure

├── dataset/                    # Dreaddit dataset files
│   ├── dreaddit-train.csv     # Training data (2,837 samples)
│   ├── dreaddit-test.csv      # Test data (715 samples)
│   └── dreaddit-all.csv       # Combined dataset
├── src/                       # Experiment source code
│   ├── agir/                  # Base AGIR experiments
│   ├── agir-cot/             # AGIR with Chain-of-Thought
│   ├── agir-o3-mini/         # AGIR with O3-Mini model
│   ├── agir-o3-mini-cot/     # AGIR O3-Mini + CoT
│   ├── o3-mini/              # Direct O3-Mini implementation
│   ├── gpt/                  # GPT-4 baseline experiments
│   └── balanced_accuracy.py  # Performance analysis tool
└── experiments/results/       # Experimental results and outputs

Getting Started

1. Data Preparation: Ensure the Dreaddit dataset is available in the dataset/ directory
2. Environment Setup: Install required dependencies for your chosen experiment
3. Run Experiments: Execute the corresponding Python script from the src/ directory
4. Analyze Results: Use balanced_accuracy.py to evaluate model performance

Key Findings

• AGIR-O3-Mini-CoT demonstrates superior performance with balanced accuracy approaching 80%
• Chain-of-Thought prompting consistently improves model performance
• O3-Mini models outperform traditional GPT-4 nano baseline
• All models show strong sensitivity but struggle with specificity, indicating a bias toward stress detection