A comprehensive pipeline benchmarking four transformer architectures (BERT, RoBERTa, DistilBERT, FinBERT) on financial sentiment classification, with trading signal generation and backtesting capabilities.
- Project Overview
- Results
- Models Under Comparison
- Quick Start
- Project Structure
- Technical Details
- Trading Strategies
- Troubleshooting
- License
This project demonstrates:
- Architecture Comparison: Fair benchmarking of multiple transformer models
- Domain Adaptation: Comparing general-purpose vs. finance-specific models
- Production Thinking: Balancing accuracy, speed, and resource constraints
- End-to-End Pipeline: From raw text to trading signals
| Model | Parameters | Accuracy | F1 | Inference | Training |
|---|---|---|---|---|---|
| FinBERT | 109M | 87.2% | 0.873 | 0.7ms | 4.2min |
| RoBERTa | 125M | 84.5% | 0.846 | 0.5ms | 4.3min |
| BERT | 109M | 84.1% | 0.842 | 0.5ms | 4.1min |
| DistilBERT | 66M | 81.4% | 0.816 | 0.2ms | 2.1min |
Key Findings:
- Domain-specific FinBERT outperforms general models by 3-6%
- DistilBERT achieves 93% of FinBERT's accuracy with 40% fewer parameters and 3.5x faster inference
- All models show strong neutral class performance (majority class)
| Model | Parameters | HuggingFace ID | Key Insight |
|---|---|---|---|
| BERT | 110M | bert-base-uncased |
Baseline transformer |
| RoBERTa | 125M | roberta-base |
Better training procedure |
| DistilBERT | 66M | distilbert-base-uncased |
40% smaller, 2x faster |
| FinBERT | 110M | ProsusAI/finbert |
Domain-specific pre-training |
- Multi-Model Benchmarking: Fair comparison of BERT, RoBERTa, DistilBERT, and FinBERT
- End-to-End Pipeline: From raw financial text to trading signals
- News Sentiment Analysis: Real-time sentiment from Yahoo Finance headlines
- Backtesting Framework: 4 trading strategies with performance metrics
- Interactive Dashboard: Streamlit-based visualization and exploration
- Production-Ready: Docker containerization with comprehensive tests
financial_sentiment_analysis/
├── README.md
├── requirements.txt
├── Dockerfile
├── docker-compose.yml
├── .dockerignore
├── .gitignore
├── src/
│ ├── data/
│ │ ├── dataset.py # Data loading, splits, class weights
│ │ └── tokenizer.py # Unified tokenization pipeline
│ ├── models/
│ │ ├── classifier.py # SentimentClassifier (supports all 4 models)
│ │ └── trainer.py # Training loop, scheduler, early stopping
│ ├── analysis/
│ │ └── error_analysis.py # Model comparison and error analysis
│ ├── news/
│ │ ├── fetcher.py # Yahoo Finance news fetching
│ │ └── processor.py # Headline cleaning and processing
│ ├── inference/
│ │ └── predictor.py # Sentiment prediction pipeline
│ ├── signals/
│ │ ├── aggregator.py # Daily sentiment aggregation
│ │ └── prices.py # Price fetching and returns
│ ├── backtesting/
│ │ ├── strategy.py # Trading strategies
│ │ └── metrics.py # Performance metrics
│ └── utils/
├── scripts/
│ ├── train_baseline.py # BERT baseline training script
│ ├── train_all_models.py # Multi-model fine-tuning script
│ ├── run_error_analysis.py # Error analysis script
│ ├── run_news_pipeline.py # News sentiment pipeline
│ ├── run_aggregation.py # Sentiment aggregation
│ └── run_backtest.py # Backtesting strategies
├── tests/
│ ├── test_environment.py # Environment verification
│ ├── test_data_loading.py # Data loading tests
│ ├── test_baseline.py # Model and trainer tests
│ ├── test_finetune.py # Fine-tuning tests
│ ├── test_error_analysis.py # Error analysis tests
│ ├── test_news_pipeline.py # News pipeline tests
│ ├── test_aggregation.py # Aggregation tests
│ └── test_backtest.py # Backtesting tests
├── data/
│ └── raw/ # Financial PhraseBank dataset
├── models/ # Saved model checkpoints
├── outputs/ # Training outputs, metrics
├── notebooks/ # EDA and experimentation
├── app/
│ └── dashboard.py # Streamlit dashboard
├── docs/
│ └── model_selection.md # Model selection recommendations
└── outputs/
├── training/ # Model checkpoints and comparison
├── analysis/ # Error analysis outputs
├── news/ # News sentiment results
├── signals/ # Aggregated signals
└── backtest/ # Backtest results
- Docker and Docker Compose
- ~4GB disk space for models
-
Clone the repository
git clone https://github.com/nimeshk03/financial-sentiment-transformers.git cd financial-sentiment-transformers -
Download the dataset
Download Financial PhraseBank from Kaggle and place
all-data.csvindata/raw/. -
Build and run with Docker
docker compose build docker compose run --rm app pytest tests/test_environment.py -v
docker compose run --rm app python scripts/train_baseline.pyExpected output: ~62% accuracy with frozen BERT encoder.
# Train all 4 models (recommended: use GPU via Google Colab)
docker compose run --rm app python scripts/train_all_models.py
# Train specific models
docker compose run --rm app python scripts/train_all_models.py --models bert distilbertResults saved to outputs/training/.
# All tests
docker compose run --rm app pytest -v
# Specific test file
docker compose run --rm app pytest tests/test_baseline.py -v# Fetch news and predict sentiment for default tickers
docker compose run --rm app python scripts/run_news_pipeline.py
# Specific tickers
docker compose run --rm app python scripts/run_news_pipeline.py --tickers AAPL NVDA TSLA# Aggregate sentiment and run backtest
docker compose run --rm app python scripts/run_aggregation.py
docker compose run --rm app python scripts/run_backtest.pydocker compose run --rm -p 8501:8501 app streamlit run app/dashboard.py --server.address 0.0.0.0Open http://localhost:8501 in your browser.
No API keys are required for basic functionality. The project uses:
- Yahoo Finance: Public API for news and price data (no key needed)
- Hugging Face: Public model downloads (no key needed for inference)
Note: For production deployments or high-volume usage, consider setting up rate limiting or caching.
All models use standardized hyperparameters for fair comparison:
TrainingConfig(
learning_rate=2e-5,
batch_size=16,
epochs=3,
warmup_ratio=0.1, # 10% of steps for warmup
weight_decay=0.01,
max_grad_norm=1.0, # Gradient clipping
early_stopping_patience=2,
)- Financial PhraseBank: ~4,845 financial news sentences
- Classes: Negative (0), Neutral (1), Positive (2)
- Split: 80% train, 10% validation, 10% test (stratified)
- Class weights: Applied to handle imbalance (neutral is majority)
- Wraps any HuggingFace transformer
- Supports frozen/unfrozen encoder
- Unified interface for all 4 models
- Linear warmup + decay learning rate scheduler
- Gradient clipping
- Early stopping
- Weighted cross-entropy loss
- Comprehensive metrics (accuracy, F1, per-class)
- Unified tokenization for all models
- PyTorch Dataset and DataLoader creation
- Configurable max length and batch size
| Task | Environment | Reason |
|---|---|---|
| Development | Local (CPU) | Fast iteration |
| Single model training | Local (CPU) | ~30-60 min acceptable |
| All 4 models | Cloud GPU | Parallel, faster |
| Inference | Local (CPU) | Fast enough |
Four sentiment-based strategies are implemented:
| Strategy | Rule | Return | Sharpe |
|---|---|---|---|
| Sentiment Threshold | Long if > 0.2, Short if < -0.2 | +0.03% | 0.59 |
| Long Only | Long if > 0.1, else flat | +0.14% | 2.75 |
| Momentum | Trade on sentiment momentum | +0.18% | 1.87 |
| Rolling 3d | Long if 3d rolling > 0.15 | +0.32% | 9.07 |
Important Disclaimer: These backtest results are based on a 7-day sample with only 2-5 trades per strategy. The high Sharpe ratios (especially 9.07) are not statistically significant and should not be interpreted as realistic performance expectations. Key limitations:
- Sample size: 7 days is insufficient for reliable statistics
- Trade count: 2-5 trades cannot establish statistical significance
- Annualization artifacts: Sharpe ratios are annualized (
* sqrt(252)), which amplifies small edges from limited data- No transaction costs: Slippage, commissions, and spreads are not included
For production use, run backtests with at least 1 year of data:
--price-period 1y
STOCK_TICKERS = ['AAPL', 'AMZN', 'BAC', 'GLD', 'GOOGL',
'JPM', 'MSFT', 'NVDA', 'SPY', 'TLT']| Issue | Solution |
|---|---|
CUDA out of memory |
Reduce batch_size in training config or use CPU |
Dataset not found |
Ensure all-data.csv is in data/raw/ directory |
Docker build fails |
Check Docker daemon is running; try docker system prune |
Model download slow |
Models are cached after first download in ~/.cache/huggingface |
Yahoo Finance rate limit |
Add delays between requests or reduce ticker count |
- GPU Training: Use Google Colab or cloud GPU for training all models
- Inference: CPU is sufficient for real-time inference (0.2-0.7ms per sample)
- Memory: DistilBERT uses 40% less memory than BERT/FinBERT
This project is licensed under the MIT License - see the LICENSE file for details.
Nimesh Kulatunga
- GitHub: @nimeshk03
- Hugging Face: nimeshk03
- Financial PhraseBank dataset by Malo et al.
- Hugging Face Transformers library
- ProsusAI/finbert for domain-specific pre-training
- Yahoo Finance for news and price data
Built with Transformers and Python