Skip to content

loki-smip/EvoTransformer

BranchesTags

Repository files navigation

EvoTransformer — Self-Correcting, Self-Evolving AI

A CPU-efficient language model that corrects its own mistakes and evolves its own architecture over time, built from scratch in PyTorch.

you can find the pretraned models on huggingface.co at EvoTransformer-v0.5-0.000614b

Table of Contents

What Is This?

EvoTransformer is a small language model with two unique features that most AI models don't have:

1. Self-Correction Loop

Most AI models give you one answer and that's it. EvoTransformer has a CorrectionHead — a second neural network head that looks at the model's own initial prediction and hidden states, then produces a corrected version. Think of it like proofreading your own work.

Input → Transformer → Initial Prediction
                ↓              ↓
          Hidden States + Initial Prediction
                ↓
         CorrectionHead → Corrected Prediction (better!)

You can run multiple correction passes — each pass feeds the previous correction back through the head, refining the output further.

2. Self-Evolution

During training, an evolutionary algorithm runs alongside normal gradient descent:

  • A population of different model architectures (different sizes, learning rates, dropout) is maintained
  • Every few epochs, each architecture is evaluated on validation data
  • The best ones survive, the worst ones are replaced
  • New architectures are created via mutation (random changes) and crossover (combining two good architectures)
  • If an evolved architecture beats the current model, the model switches to the better one

This means the model doesn't just learn from data — it searches for the best version of itself.

How It Compares to Other Models

Feature EvoTransformer GPT-2 Small Llama 3 (8B) BERT Base
Parameters 614K 124M 8B 110M
Runs on CPU ✅ Yes Slow ❌ No Slow
Self-Correction ✅ Built-in ❌ No ❌ No ❌ No
Self-Evolution ✅ Built-in ❌ No ❌ No ❌ No
Training from scratch ✅ ~2 hours (CPU) Days (GPU) Months (GPU cluster) Days (GPU)
Checkpoint Resume ✅ Full state ✅ ✅ ✅
Graceful Stop ✅ Ctrl+C safe ❌ Manual ❌ Manual ❌ Manual
Text Quality Basic/Shakespeare Good Excellent N/A (not generative)
Memory Usage ~100 MB ~500 MB ~16 GB+ ~500 MB
Dependencies PyTorch + NumPy Many Many Many

When to Use EvoTransformer

  • ✅ Learning how AI language models work from the inside
  • ✅ Experimenting with self-correction and evolutionary AI concepts
  • ✅ Running on a computer with no GPU
  • ✅ Rapid prototyping and experimentation

When NOT to Use It

  • ❌ Production chatbot (too small for real conversations)
  • ❌ When you need human-level text quality (use GPT-4, Llama, etc.)
  • ❌ Large-scale deployment

Architecture Walkthrough

Overview

┌─────────────────────────────────────────────────────┐
│                   EvoTransformer                     │
│                                                      │
│  Input Tokens                                        │
│       ↓                                              │
│  [Embedding Layer] (token → 128-dim vector)          │
│       ↓                                              │
│  [Sinusoidal Positional Encoding] (adds position)    │
│       ↓                                              │
│  ┌──────────────────────────────────┐  × 4 blocks    │
│  │  LayerNorm → Multi-Head Attention (4 heads)  │    │
│  │  LayerNorm → Feed-Forward (GELU, 256-dim)    │    │
│  └──────────────────────────────────┘                │
│       ↓                                              │
│  [Output Head] → Initial Logits (first guess)        │
│       ↓                                              │
│  [CorrectionHead] → Corrected Logits (refined)       │
│                                                      │
│  Total: ~614K parameters                             │
└─────────────────────────────────────────────────────┘

Key Components

Component File What It Does
EvoTransformer model.py The main neural network — embedding, transformer blocks, output head
CorrectionHead model.py Takes hidden states + initial prediction → produces corrected output
EvolutionEngine evolution.py Manages a population of architectures, runs mutation/crossover/selection
SelfCorrectionLoop self_correction.py Runs multiple correction passes with blending during inference
CharTokenizer dataset.py Maps characters ↔ integers (67 tokens for Shakespeare)
CosineWarmupScheduler train.py Learning rate: linear warmup then cosine decay to zero

Training Flow

1. Load English text dataset (Tiny Shakespeare)
2. Tokenize into character-level sequences
3. For each epoch:
   a. Forward pass: input → model → initial logits + corrected logits
   b. Compute combined loss: (0.7 × base_loss) + (0.3 × correction_loss)
   c. Backward pass with gradient clipping
   d. Update learning rate (cosine schedule)
   e. Every 5 epochs: run evolutionary architecture search
   f. Every 2 epochs: save checkpoint
4. On Ctrl+C: save checkpoint immediately, then exit

Setup Guide

Windows

# 1. Install Python 3.10+ from https://python.org or Microsoft Store

# 2. Open PowerShell, navigate to the project
cd "d:\New folder"

# 3. (Optional) Create a virtual environment
python -m venv venv
.\venv\Scripts\Activate

# 4. Install dependencies
pip install -r requirements.txt

# 5. Train the model
python train.py --epochs 20

# 6. Chat with it
python chat.py

Linux (Ubuntu/Debian)

# 1. Install Python
sudo apt update
sudo apt install python3 python3-pip python3-venv

# 2. Navigate to the project folder
cd /path/to/project

# 3. Create virtual environment
python3 -m venv venv
source venv/bin/activate

# 4. Install dependencies
pip install -r requirements.txt

# 5. Train
python train.py --epochs 20

# 6. Chat
python chat.py

macOS

# 1. Install Python (via Homebrew)
brew install python3

# 2. Navigate to the project
cd /path/to/project

# 3. Create virtual environment
python3 -m venv venv
source venv/bin/activate

# 4. Install dependencies
pip install -r requirements.txt

# 5. Train (Apple Silicon Macs will auto-detect MPS if PyTorch supports it)
python train.py --epochs 20

# 6. Chat
python chat.py

Notes for All Platforms

  • Python 3.10+ is required (3.12 recommended)
  • No GPU needed — the model auto-detects CUDA and falls back to CPU
  • The dataset (~1 MB) downloads automatically on first run
  • All paths work cross-platform (using os.path)

Step-by-Step: Training

1. Start Training

python train.py --epochs 20

What happens:

  • Auto-detects your device (CPU or CUDA GPU)
  • Downloads Tiny Shakespeare (~1 MB) on first run
  • Creates ~31K training samples from the text
  • Trains the model with self-correction loss
  • Saves checkpoints every 2 epochs to checkpoints/
  • Runs evolutionary architecture search every 5 epochs

2. Monitor Progress

Watch the console output — key metrics:

  • Train Loss: should decrease over time (started ~4.4, should reach ~1.5-1.6)
  • Val Loss: should track train loss (if it diverges, the model is overfitting)
  • Base Acc → Corrected Acc: shows if the correction head is helping
  • Sample text: generated every 5 epochs so you can see quality improving

3. Stop Training (Gracefully)

Press Ctrl+C at any time:

[!] Ctrl+C detected — will save checkpoint after current batch...
[✓] Checkpoint saved: checkpoints/evo_transformer_ckpt.pt (7.1 MB)

The checkpoint contains: model weights, optimizer state, scheduler state, epoch number, best loss, evolution state, and random number generator states. Everything needed for a perfect resume.

4. Resume Training

# Continue from where you stopped
python train.py --resume --epochs 50

# The output will show:
# [✓] Resumed from epoch 20 (best loss: 1.6661)

5. Training Timeline (CPU, 4 cores)

Epochs Time Loss Text Quality
1-2 ~10 min ~3.5 Random letters
5-10 ~1 hour ~1.8 English words, names
10-20 ~2-3 hours ~1.6 Sentences, Shakespeare style
20-50 ~5-8 hours ~1.4 Coherent paragraphs
50+ ~12+ hours ~1.2 Diminishing returns

Step-by-Step: Chat & Inference

Chat Mode

python chat.py

This opens an interactive chat that shows initial vs corrected output side-by-side:

  You > ROMEO:

  +-- Initial (no correction) --------+
  | And thee to your graces to him     |
  +------------------------------------+

  +-- Corrected (2 passes) -----------+
  | O, what light through yonder       |
  | window breaks! It is the east,     |
  +------------------------------------+

Chat Commands

Command What It Does
/temp 0.5 Lower temperature = more predictable text
/temp 1.2 Higher temperature = more creative/random
/len 300 Generate more text per response
/passes 4 More correction passes (slower but potentially better)
/compare Toggle showing initial vs corrected comparison
/clear Clear conversation context
/settings Show current settings
/quit Exit chat

Tips for Better Chat Output

  1. Start with character names: ROMEO:, KING HENRY:, HAMLET:
  2. Start mid-sentence: The king shall, My lord, I beg
  3. Lower the temperature for more coherent text: /temp 0.5
  4. Train more epochs for fundamentally better quality

Inference Script (Non-Interactive)

# Quick demo
python inference.py

# Interactive mode
python inference.py --interactive

# More correction passes
python inference.py --passes 5

How to Change Parameters

All parameters live in config.py. Here's what each one does and recommended values:

Model Size

@dataclass
class ModelConfig:
    vocab_size: int = 512       # Set automatically from tokenizer (don't change)
    d_model: int = 128          # Embedding dimension (↑ = smarter but slower)
    n_heads: int = 4            # Attention heads (must divide d_model evenly)
    n_layers: int = 4           # Transformer blocks (↑ = deeper understanding)
    d_ff: int = 256             # Feed-forward hidden size (usually 2-4× d_model)
    max_seq_len: int = 64       # Context window (↑ = sees more text at once)
    dropout: float = 0.1        # Regularization (0.1-0.2 is good)
    correction_passes: int = 2  # Self-correction iterations during training

Recommended presets:

Preset d_model n_heads n_layers d_ff Params Speed
Tiny (fast) 64 2 2 128 ~100K Very fast
Small (default) 128 4 4 256 ~614K Fast
Medium 256 8 6 512 ~2.5M Moderate
Large (needs GPU) 512 8 8 1024 ~10M Slow on CPU

Training Speed

@dataclass
class TrainConfig:
    batch_size: int = 32        # ↑ = faster but uses more RAM
    lr: float = 3e-4            # Learning rate (3e-4 is a good default)
    epochs: int = 50            # Total training epochs
    grad_clip: float = 1.0      # Prevents exploding gradients
    warmup_steps: int = 100     # LR warmup (prevents initial instability)
    log_every: int = 10         # Print loss every N batches

Evolution

@dataclass
class EvolutionConfig:
    population_size: int = 6         # Architectures to evaluate (↑ = better search, slower)
    mutation_rate: float = 0.3       # Probability of mutating each gene
    evolve_every_n_epochs: int = 5   # Run evolution every N epochs
    max_param_count: int = 10000000  # Reject architectures larger than this

After Changing Parameters

# Delete old checkpoint (architecture changed, old weights won't fit)
# Windows:
del checkpoints\evo_transformer_ckpt.pt
# Linux/macOS:
rm checkpoints/evo_transformer_ckpt.pt

# Train fresh
python train.py --epochs 20

Important: If you only change training parameters (lr, batch_size, epochs), you CAN resume. If you change model parameters (d_model, n_layers, etc.), you MUST delete the old checkpoint and train fresh.

How to Use Your Own Dataset

Step 1: Replace the Text File

Put your text file at data/english_text.txt. It can be:

  • A novel, book, or collection of books
  • Chat logs or conversations
  • Code, documentation, or technical writing
  • Any plain text in any language
# Example: download a different book
# Windows (PowerShell):
Invoke-WebRequest -Uri "https://www.gutenberg.org/files/1342/1342-0.txt" -OutFile "data/english_text.txt"

# Linux/macOS:
wget -O data/english_text.txt "https://www.gutenberg.org/files/1342/1342-0.txt"

Step 2: Delete Old Tokenizer and Checkpoint

The tokenizer is built from the text, so it needs to be rebuilt:

# Windows:
del data\tokenizer.json
del checkpoints\evo_transformer_ckpt.pt

# Linux/macOS:
rm data/tokenizer.json checkpoints/evo_transformer_ckpt.pt

Step 3: Train on the New Data

python train.py --epochs 20

The tokenizer will automatically rebuild from your new text.

Dataset Size Guidelines

Your Data Size Recommended Model (d_model) Training Time (CPU)
< 1 MB 128 (default) 1-3 hours
1-5 MB 128-256 3-8 hours
5-20 MB 256 8-24 hours
20-100 MB 512 (needs GPU) 1-3 days (GPU)
100+ MB 512+ (needs GPU) Days (GPU)

Using Multiple Files

To combine multiple text files into one:

# Windows (PowerShell):
Get-Content file1.txt, file2.txt, file3.txt | Set-Content data/english_text.txt

# Linux/macOS:
cat file1.txt file2.txt file3.txt > data/english_text.txt

Project Structure

EvoTransformer/
├── config.py            # All hyperparameters in one place
├── model.py             # EvoTransformer neural network + CorrectionHead
├── evolution.py         # Evolutionary algorithm (mutation, crossover, selection)
├── self_correction.py   # Multi-pass self-correction loop
├── dataset.py           # Data download, tokenizer, dataset class
├── train.py             # Training loop with checkpoint save/resume
├── chat.py              # Interactive chat interface
├── inference.py         # Batch inference demo
├── requirements.txt     # Dependencies (torch, numpy)
├── README.md            # This file
├── data/
│   ├── english_text.txt # Training text (downloaded automatically)
│   └── tokenizer.json   # Character-to-ID mapping (built automatically)
└── checkpoints/
    └── evo_transformer_ckpt.pt  # Saved model state (created during training)

FAQ

Q: Can I train on languages other than English? A: Yes! Replace data/english_text.txt with text in any language. The character tokenizer will automatically learn the character set. Works with Chinese, Arabic, Hindi, etc.

Q: Why is the text quality not amazing? A: The model is intentionally tiny (~614K params) so it can run on CPU. For comparison, ChatGPT has ~175 billion parameters — 285,000× larger. The quality is impressive for its size.

Q: Can I use a GPU to train faster? A: Yes! If you have an NVIDIA GPU with CUDA, install torch with CUDA support and the model will automatically use it. Training will be 10-50× faster.

Q: What does the evolution actually change? A: It mutates these hyperparameters: d_model (embedding size), n_heads (attention heads), n_layers (depth), d_ff (feed-forward size), lr (learning rate), and dropout. It keeps the best-performing combination.

Q: How is self-correction different from just making the model bigger? A: Self-correction gives the model a "second chance" to fix mistakes using a dedicated neural network head. A bigger model might make fewer mistakes initially, but it doesn't have a mechanism to review and fix its own output. The correction head adds this ability with minimal extra parameters.

Q: What happens if training is interrupted without Ctrl+C (e.g., power outage)? A: You'll lose progress since the last checkpoint save (every 2 epochs by default). The previous checkpoint will still be valid. Change save_every_n_epochs in config.py to save more frequently.

About

a ai model that self-correction loop and self evolves its self and is super efficient and can run on cpu and can be trained on cpu or gpu

Resources

Readme
MIT license
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages