LoRA Fine-tuning Experiments for SmolLM2

This repository contains a complete, end-to-end reproducible workflow for:

1. Setting up llama.cpp from scratch
2. Converting and quantizing SmolLM2-1.7B base model
3. Training custom LoRA adapters with PEFT
4. Deploying LoRA-enhanced models for inference

Inspired by: The comprehensive llama.cpp guide by steelph0enix

🎯 What's Inside

Complete llama.cpp Setup

• Build Instructions: Compile llama.cpp with GPU support
• Model Conversion: Convert HuggingFace models to GGUF format
• Quantization: Create optimized Q4_K_M quantized models

LoRA Training Pipeline

• Training Script: Train custom LoRA adapters using PEFT + Transformers
• GGUF Conversion: Convert trained adapters to llama.cpp-compatible format
• Validation Tools: Verify your LoRA adapters are correctly formatted
• Test Scripts: Compare base model vs LoRA-enhanced outputs

Production Deployment

• CLI Usage: Run inference with llama-cli
• Server Mode: Deploy as OpenAI-compatible API
• Multiple Adapters: Load and combine multiple LoRAs

📊 Results

This experiment successfully:

• ✅ Trained a LoRA adapter on 100 Wikitext samples (1 epoch)
• ✅ Reduced training loss: 3.432 → 3.094
• ✅ Created a 6.01 MB GGUF adapter file (192 tensors)
• ✅ Validated inference with llama.cpp
• ✅ Achieved 13-14 tokens/second generation speed

Trainable Parameters: 3.1M (0.18% of total model parameters)

🚀 Complete Setup Guide

This guide walks you through the entire process, from setting up llama.cpp to testing your trained LoRA adapter.

Part 1: Setup llama.cpp and Base Model

Following the llama.cpp guide:

1. Install llama.cpp

Option A: Pre-built binaries (Windows - easiest)

winget install ggml.llamacpp

Option B: Build from source

# Clone repository
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

# Build with GPU support (CUDA/Vulkan/Metal)
cmake -B build
cmake --build build --config Release

# Executables will be in build/bin/

2. Get SmolLM2 Base Model

Download from HuggingFace:

# Option 1: Using llama-cli (if installed via winget)
llama-cli -hf HuggingFaceTB/SmolLM2-1.7B-Instruct

# Option 2: Manual download
git lfs install
git clone https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct

3. Convert to GGUF Format

cd llama.cpp

# Convert HuggingFace model to GGUF
python convert_hf_to_gguf.py \
  ../repos/SmolLM2-1.7B-Instruct \
  --outfile ../SmolLM2.gguf \
  --outtype f16

4. Quantize Model (Optional but Recommended)

# Quantize to Q4_K_M (4-bit, good quality/size balance)
./build/bin/llama-quantize \
  ../SmolLM2.gguf \
  ../SmolLM2.Q4_K_M.gguf \
  Q4_K_M

Result: SmolLM2.Q4_K_M.gguf (~1GB) - your base model!

5. Test Base Model

llama-cli -m SmolLM2.Q4_K_M.gguf -p "Once upon a time" -n 50

Part 2: LoRA Fine-tuning

Prerequisites

# Python 3.10+
python --version

# Install dependencies
pip install -r requirements.txt

Step 1: Train LoRA Adapter

python train_lora.py

This will:

• Download SmolLM2-1.7B-Instruct from HuggingFace
• Load 100 Wikitext samples for training
• Train for 1 epoch (~2 hours on CPU)
• Save adapter to ./lora-output/

Customize Training: Edit these variables in train_lora.py:

MODEL_NAME = "HuggingFaceTB/SmolLM2-1.7B-Instruct"
LORA_RANK = 8          # Higher = more capacity
LORA_ALPHA = 16        # Scaling factor
NUM_EPOCHS = 1         # Training epochs
BATCH_SIZE = 1         # Adjust for your RAM

Step 2: Convert to GGUF

cd ../../llama.cpp

python convert_lora_to_gguf.py \
  --base ../repos/SmolLM2-1.7B-Instruct \
  --outtype f16 \
  ../experiments/lora-feasibility/lora-output/

Output: lora-output/Lora-Output-F16-LoRA.gguf

Step 3: Validate

python validate_lora.py

Checks:

• ✅ Valid GGUF format
• ✅ Correct tensor count
• ✅ File integrity

Step 4: Test with Base Model

# Using llama-cli (from lora-feasibility directory)
llama-cli \
  -m ../../SmolLM2.Q4_K_M.gguf \
  --lora lora-output/Lora-Output-F16-LoRA.gguf \
  -p "Once upon a time" \
  -n 100

📁 Complete Project Structure

Lllamacpp/                                # Root directory
├── llama.cpp/                           # llama.cpp repository
│   ├── build/bin/                       # Compiled executables
│   │   ├── llama-cli.exe               # CLI inference tool
│   │   ├── llama-server.exe            # API server
│   │   ├── llama-quantize.exe          # Quantization tool
│   │   └── llama-export-lora.exe       # LoRA merge tool
│   ├── convert_hf_to_gguf.py           # Model converter
│   └── convert_lora_to_gguf.py         # LoRA converter
│
├── repos/                               # HuggingFace models
│   └── SmolLM2-1.7B-Instruct/          # Base model (HF format)
│
├── SmolLM2.gguf                        # Converted base model (F16)
├── SmolLM2.Q4_K_M.gguf                 # Quantized model (~1GB)
│
└── experiments/
    └── lora-feasibility/               # This project!
        ├── train_lora.py               # Main training script
        ├── validate_lora.py            # GGUF validation
        ├── quick_test.py               # Python-based testing
        ├── requirements.txt            # Python dependencies
        ├── lora-output/               # Training outputs
        │   ├── adapter_model.safetensors
        │   ├── adapter_config.json
        │   └── Lora-Output-F16-LoRA.gguf  # Final adapter
        ├── README.md                  # This file
        ├── RESULTS.md                 # Detailed results
        └── HOW_TO_RUN.md              # Troubleshooting

🔧 Configuration Options

LoRA Hyperparameters

Parameter	Default	Description
LORA_RANK	8	Rank of update matrices (4-64)
LORA_ALPHA	16	Scaling factor (typically 2×rank)
LORA_DROPOUT	0.1	Dropout rate for regularization
MAX_SEQ_LENGTH	256	Maximum sequence length
BATCH_SIZE	1	Training batch size
NUM_EPOCHS	1	Number of training epochs
LEARNING_RATE	2e-4	AdamW learning rate

Target Modules

By default, we target attention layers:

target_modules=["q_proj", "v_proj", "k_proj", "o_proj"]

You can also target FFN layers:

target_modules=["q_proj", "v_proj", "k_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]

📚 Using Your Own Dataset

Replace the dataset loading in train_lora.py:

# Option 1: From HuggingFace
dataset = load_dataset("your-username/your-dataset", split="train")

# Option 2: From local file
from datasets import Dataset
with open("your_data.txt") as f:
    texts = [line.strip() for line in f]
dataset = Dataset.from_dict({"text": texts})

# Option 3: JSON/CSV
dataset = load_dataset("json", data_files="your_data.json")

🎮 Advanced Usage

Multiple LoRA Adapters

Load multiple adapters simultaneously:

llama-cli -m base.gguf \
  --lora adapter1.gguf,adapter2.gguf \
  -p "Your prompt"

Custom Scaling

Apply different scaling factors:

llama-cli -m base.gguf \
  --lora-scaled adapter1.gguf:0.8,adapter2.gguf:1.2 \
  -p "Your prompt"

Export Merged Model

Create a single merged model file:

llama-export-lora \
  -m base.gguf \
  --lora adapter.gguf \
  -o merged-model.gguf

Server Mode

Run as an OpenAI-compatible API server:

llama-server \
  -m base.gguf \
  --lora adapter.gguf \
  --port 8080

Test with curl:

curl http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Once upon a time",
    "max_tokens": 50
  }'

🐛 Troubleshooting

Out of Memory During Training

# Reduce batch size
BATCH_SIZE = 1

# Increase gradient accumulation
gradient_accumulation_steps = 8

# Reduce LoRA rank
LORA_RANK = 4

🏗️ Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                    Complete Workflow                             │
└─────────────────────────────────────────────────────────────────┘

1. Base Model Setup (llama.cpp)
   HuggingFace Model → convert_hf_to_gguf.py → GGUF (F16)
                    ↓
   GGUF (F16) → llama-quantize → GGUF (Q4_K_M) ← Base Model Ready

2. LoRA Training (This Project)
   Dataset → train_lora.py → adapter_model.safetensors
                           ↓
   adapter_model.safetensors → convert_lora_to_gguf.py → LoRA.gguf

3. Inference (llama.cpp)
   Base Model (GGUF) + LoRA (GGUF) → llama-cli/server → Output

📖 References & Credits

• llama.cpp - High-performance inference engine by Georgi Gerganov
• The comprehensive llama.cpp guide by steelph0enix - Excellent tutorial that inspired this project
• PEFT - Parameter-Efficient Fine-Tuning library by HuggingFace
• SmolLM2 - Compact language model by HuggingFace
• Reduce dataset size for faster iteration

DLL/Compiler Errors

If llama.cpp executables fail:

1. Download pre-built binaries from llama.cpp releases
2. Or install via package manager: winget install ggml.llamacpp
3. Or use Python bindings: pip install llama-cpp-python

See HOW_TO_RUN.md for detailed solutions.

📖 References

• llama.cpp - Inference engine
• PEFT - Parameter-Efficient Fine-Tuning
• SmolLM2 - Base model
• The comprehensive llama.cpp guide - Inspiration

📝 Citation

If you use this work, consider citing:

@misc{smollm2-lora-experiments,
  title={LoRA Fine-tuning Experiments for SmolLM2},
  author={Your Name},
  year={2026},
  url={https://github.com/yourusername/lora-experiments}
}

📄 License

MIT License - See LICENSE file for details

🙏 Acknowledgments

• Inspired by steelph0enix's llama.cpp guide
• Built with llama.cpp, HuggingFace PEFT, and Transformers
• Trained on the SmolLM2-1.7B-Instruct model by HuggingFace

---

Happy Fine-tuning! 🚀

For questions or issues, please open a GitHub issue.