Poker AI Training Pipeline

A machine learning pipeline for training a GPT-2 model to predict poker actions from game states. The project uses transformer-based language modeling to learn poker decision-making from hand history data.

Project Overview

This project trains a neural network to predict poker actions (FOLD, CALL, RAISE) given the current game state. The model learns from real poker hand histories using a masked language modeling approach where the context (game state) is provided and the model must predict the action.

Setup

1. Clone the repository

   git clone https://github.com/YOUR_USERNAME/pokerAI.git
   cd pokerAI

2. Create virtual environment

   python -m venv venv
   venv\Scripts\activate  # On Windows
   source venv/bin/activate  # On Linux/Mac

3. Install dependencies

   pip install -r requirements.txt

4. Prepare your data

   • Place your poker hand history data in data/hands.txt
   • Each line should be a complete hand history in the format: POSITION,STACK,CARDS,PLAYER_INFO,ACTION

Training Pipeline

1. Data Preparation (data/prepare_data.py)

Cleans the raw poker data and creates train/test splits:

• Removes redundant "0BB" from FOLD actions
• Splits data into 95% training / 5% validation
• Saves cleaned data to data/hands_fold_update.txt

python data/prepare_data.py

2. Tokenizer Training (data/train_tokenizer.py)

Trains a custom BPE tokenizer on the poker domain:

• Vocabulary size: 4000 tokens
• Special tokens: <|startoftext|>, <|pad|>
• Saves tokenizer to tokenizer/ directory

python data/train_tokenizer.py

3. Baseline Model (data/train_bigram.py)

Trains a simple bigram model for comparison:

• Provides a baseline for model performance
• Expected validation loss: ~1.53

python data/train_bigram.py

4. GPT-2 Model Training

Option A: Raw PyTorch Loop (data/build_model.py)

• GPT-2 architecture (6 layers, 256 hidden dimensions, 8 heads)
• Context length: 192 tokens
• Masked training: only predicts the action, not the context
• 3 epochs with AdamW optimizer (lr=3e-4)

python data/build_model.py

Option B: TRL Trainer (data/TRL_model.py)

• Uses Hugging Face TRL library for supervised fine-tuning
• Same model architecture as raw loop
• Prompt/completion format for masked training
• Saves model to model_out_trl/

python data/TRL_model.py

Model Architecture

• Model Type: GPT-2 (decoder-only transformer)
• Parameters: ~2.6M
• Layers: 6
• Hidden Size: 256
• Attention Heads: 8
• Context Window: 192 tokens
• Vocabulary Size: 4000 (domain-specific)

Training Strategy

The model uses masked language modeling:

• Input: Full hand history (game state + action)
• Masking: All tokens up to the last comma (the game state) are masked
• Prediction: Model learns to predict only the action tokens
• This ensures the model focuses on decision-making, not memorizing game states

Data Format

Each hand history line should follow this format:

POSITION,STACK,CARDS,PLAYER1_INFO,PLAYER2_INFO,...,ACTION

Example:

BTN,1.5BB,9d 10c,P1:101.0BB/0.0BB,P2:100.23BB/0.0BB,FOLD

File Structure

pokerAI/
├── data/
│   ├── prepare_data.py      # Data cleaning and splitting
│   ├── train_tokenizer.py   # Custom tokenizer training
│   ├── train_bigram.py       # Baseline model training
│   ├── build_model.py       # GPT-2 training (raw loop)
│   ├── TRL_model.py         # GPT-2 training (TRL)
│   ├── hands.txt            # Raw input data (not in repo)
│   └── hands_fold_update.txt # Cleaned data (not in repo)
├── tokenizer/                # Trained tokenizer (not in repo)
├── model_out_trl/           # Trained model (not in repo)
├── requirements.txt         # Python dependencies
└── README.md                # This file

Performance

• Bigram baseline: Validation loss ~1.53
• GPT-2 model: Expected to significantly outperform baseline
• Compare final validation loss against baseline to assess improvement

Notes

• Large files (data, models, tokenizer) are excluded from git via .gitignore
• These can be regenerated by running the training scripts
• Model training requires CUDA GPU for reasonable speed
• Adjust batch size and learning rate based on your hardware

License

[Add your license here]