ModernBERT Bias Classification Training

This project replicates the training of the ModernBERT-large bias type classifier from scratch using the same dataset and hyperparameters.

Overview

The model is designed to detect and classify bias in text across 11 categories:

• Racial
• Religious
• Gender
• Age
• Nationality
• Sexuality
• Socioeconomic
• Educational
• Disability
• Political
• Physical

Project Structure

├── config.py              # Configuration and hyperparameters
├── data_preprocessing.py   # Data loading and preprocessing
├── evaluation.py          # Evaluation metrics and reporting
├── train.py               # Main training script
├── inference.py           # Inference and testing script
├── requirements.txt       # Python dependencies
├── README.md              # This file
└── test.py                # Original exploration script

Installation

1. Clone or download the project files

2. Install dependencies:

   pip install -r requirements.txt

3. Optional: Set up Weights & Biases for experiment tracking

   wandb login

   Then set USE_WANDB = True in config.py

Usage

1. Data Exploration

First, explore the dataset structure:

python data_preprocessing.py

This will:

• Analyze the dataset distribution
• Show label statistics
• Test data loading functionality

2. Training

Run the main training script:

python train.py

This will:

• Load the base ModernBERT-large model
• Prepare the dataset with train/validation split
• Train the model using the original hyperparameters
• Save the trained model and results
• Generate comprehensive evaluation reports

Training Parameters (from original model):

• Learning rate: 3e-5
• Batch size: 16
• Weight decay: 0.01
• Warmup steps: 500
• Epochs: 3
• Optimizer: AdamW

3. Inference

Test your trained model:

python inference.py

This will:

• Load your trained model (or the original for comparison)
• Run inference on example texts
• Provide an interactive mode for testing your own text

4. Configuration

Modify config.py to customize:

• Model and dataset settings
• Training hyperparameters
• Evaluation settings
• Output directories
• Wandb configuration

Key Features

Multi-label Classification

• Handles multiple bias types per text
• Uses sigmoid activation with configurable threshold
• Comprehensive metrics (precision, recall, F1, MCC)

Robust Evaluation

• Macro and micro averages
• Per-label detailed metrics
• Exact match accuracy
• Hamming loss calculation

Best Practices

• Reproducible training with fixed seeds
• Proper train/validation splitting
• Early stopping to prevent overfitting
• Automated model saving and checkpointing
• Comprehensive logging and reporting

Expected Results

The original model achieved these metrics on the test set:

Metric	Value
Accuracy	0.983
Precision	0.930
Recall	0.914
F1	0.921
MCC	0.912

Your replicated model should achieve similar performance.

Hardware Requirements

• GPU: Recommended for training (RTX 3080+ or equivalent)
• RAM: 16GB+ recommended
• Storage: ~10GB for model, dataset, and outputs
• Training time: ~2-3 hours on modern GPU

Troubleshooting

Common Issues

1. CUDA out of memory: Reduce batch size in config.py
2. Slow training: Ensure GPU is being used, check config.USE_CUDA
3. Dataset loading errors: Check internet connection and HuggingFace access

Performance Tips

• Use mixed precision training (enabled by default)
• Adjust DATALOADER_NUM_WORKERS based on your CPU cores
• Monitor GPU memory usage during training

File Descriptions

config.py

Contains all configuration settings including:

• Model and dataset parameters
• Training hyperparameters
• Evaluation settings
• Hardware configuration

data_preprocessing.py

Handles data loading and preprocessing:

• Dataset loading from HuggingFace
• Train/validation splitting
• Multi-label preparation
• Custom PyTorch Dataset class

evaluation.py

Comprehensive evaluation metrics:

• Multi-label classification metrics
• Per-label detailed analysis
• Prediction analysis and reporting
• Custom compute_metrics for Trainer

train.py

Main training script:

• Model and tokenizer setup
• Training loop with HuggingFace Trainer
• Automatic saving and checkpointing
• Final evaluation and reporting

inference.py

Testing and inference:

• Load trained models
• Batch and single text prediction
• Interactive testing mode
• Formatted output display

Dataset Information

• Source: ethical-spectacle/biased-corpus
• Size: 37,507 examples
• Format: Multi-label (binary labels for each bias type)
• Split: 80% training, 20% validation (automatically created)

Model Architecture

• Base Model: ModernBERT-large (396M parameters)
• Classification Head: Linear layer with 11 outputs
• Activation: Sigmoid for multi-label classification
• Problem Type: Multi-label classification

Contributing

To improve this training setup:

1. Test different hyperparameters
2. Experiment with different train/validation splits
3. Add additional evaluation metrics
4. Optimize for different hardware configurations

License

This project follows the same MIT license as the original model.

Citation

If you use this training code, please cite:

@misc{modernbert-bias-replica,
  title={ModernBERT Bias Classification Training Replica},
  author={Your Name},
  year={2024},
  howpublished={\url{https://github.com/yourusername/modernbert-bias-training}},
}

Original model citation:

@inproceedings{JunquedeFortuny2025c,
  title = {Bias Detection with ModernBERT-Large},
  author = {Enric Junqué de Fortuny},
  year = {2025},
  howpublished = {\url{https://huggingface.co/cirimus/modernbert-large-bias-type-classifier}},
}