BitForge: Production-Ready LLM Inference Backend

A robust, scalable FastAPI backend for serving LLM inference via persistent subprocess workers. Designed for CLI-based language models with comprehensive error handling, health monitoring, and automatic recovery.

🎯 What This Infrastructure Provides

This backend serves as a production-grade template for deploying any CLI-based LLM with:

Persistent Worker Pools: Reusable subprocess workers that stay alive between requests (eliminates model reload overhead)
Automatic Health Monitoring: Detects and restarts crashed workers automatically
Request Queuing: Handles concurrent requests with configurable timeouts
Graceful Degradation: Continues serving requests even if individual workers fail
Comprehensive Logging: Structured logging with structlog for debugging
Docker Deployment: Multi-stage build optimized for Hugging Face Spaces / cloud platforms
Metrics & Monitoring: Built-in request tracking and latency metrics

🏗️ Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                        FastAPI App                          │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐     │
│  │   /health    │  │   /metrics   │  │  /api/v1/*   │     │
│  └──────────────┘  └──────────────┘  └──────────────┘     │
└───────────────────────────┬─────────────────────────────────┘
                            │
                            ▼
                ┌───────────────────────┐
                │  Production Worker    │
                │        Pool           │
                │                       │
                │  ┌─────────────────┐ │
                │  │ Health Monitor  │ │  ← Async background task
                │  │  (checks every  │ │
                │  │   5 seconds)    │ │
                │  └─────────────────┘ │
                │                       │
                │  Worker Queue:        │
                │  ┌───────┐ ┌───────┐│
                │  │Worker0│ │Worker1││  ← Multiple workers
                │  │(IDLE) │ │(BUSY) ││     (configurable)
                │  └───┬───┘ └───────┘│
                └──────┼───────────────┘
                       │
                       ▼
            ┌─────────────────────┐
            │  Subprocess         │
            │  (LLM Binary)       │
            │                     │
            │  stdin  ← Prompts   │
            │  stdout → Responses │
            │  stderr → Logs      │
            └─────────────────────┘

Key Components

1. Worker Pool (`app/inference/worker.py`)

Manages multiple persistent subprocess workers
Implements IDLE/BUSY/UNHEALTHY state machine
Automatic restart on failure with exponential backoff
Captures stdout/stderr for debugging

2. BitNet Engine (`app/inference/bitnet.py`)

Abstraction layer between API and workers
Handles request queuing and worker assignment
Implements timeout logic

3. Configuration (`app/core/config.py`)

Environment-based settings (Pydantic)
CRITICAL: Use absolute paths for binaries/models

4. API Routes (`app/api/v1/chat.py`)

RESTful endpoints with Pydantic validation
Request/response schemas
Error handling with appropriate HTTP status codes

🚀 Quick Start

Prerequisites

Docker (for containerized deployment)
Python 3.10+ (for local development)
A compiled LLM binary (e.g., llama.cpp, BitNet, etc.)

Local Development

Install Dependencies:

python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate
pip install -r requirements.txt

Configure Environment:

cp .env.example .env
# Edit .env with your settings

Place Your Binary:

mkdir -p bin
# Copy your compiled LLM binary to bin/your-binary
chmod +x bin/your-binary

Download Model:

mkdir -p models
# Download your GGUF model to models/

Run Server:

uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

Docker Deployment

docker build -t llm-backend .
docker run -p 7860:7860 llm-backend

� Adapting for Your LLM

Step 1: Configure Your Binary

Edit app/core/config.py:

class Settings(BaseSettings):
    # Point to your binary (absolute path in Docker)
    BINARY_PATH: str = "/app/bin/your-llm-binary"
    
    # Point to your model file
    MODEL_PATH: str = "/app/models/your-model.gguf"
    
    # Model download URL (optional)
    MODEL_URL: str = "https://huggingface.co/.../your-model.gguf"

Step 2: Update Worker Command

Edit app/inference/worker.py - modify the start() method:

async def start(self):
    cmd = [
        settings.BINARY_PATH,
        "-m", settings.MODEL_PATH,
        # Add your binary's specific flags:
        "-t", "2",          # threads
        "-c", "512",        # context size
        "-n", "-1",         # max tokens (-1 = unlimited)
        "--temp", "0.7",    # temperature
        "-i",               # interactive mode
        # etc...
    ]

Step 3: Adjust Output Parsing

The generate() method in worker.py reads from stdout. Adapt the parsing logic based on your binary's output format:

async def generate(self, prompt: str, max_tokens: int = 128):
    # Write prompt to stdin
    self.process.stdin.write(f"{prompt}\n".encode())
    await self.process.stdin.drain()
    
    # Read response from stdout
    # CUSTOMIZE THIS based on your binary's output format
    generated_text = ""
    # ... parsing logic ...

Step 4: Update Dockerfile

In Dockerfile, replace the BitNet-specific build steps:

# --- Stage 1: Builder ---
FROM python:3.10-slim AS builder
WORKDIR /build

# Install build dependencies for your LLM
RUN apt-get update && apt-get install -y \
    build-essential \
    cmake \
    git \
    # Add other dependencies

# Clone and build your LLM
RUN git clone https://github.com/your-repo/your-llm.git
WORKDIR /build/your-llm
RUN mkdir build && cd build && cmake .. && make

# --- Stage 2: Runtime ---
FROM python:3.10
# ... copy binary and shared libraries ...
COPY --from=builder /build/your-llm/build/bin/your-binary /app/bin/your-binary

📋 Configuration Reference

Environment Variables

Variable	Description	Default	Required
`BINARY_PATH`	Absolute path to LLM binary	`/app/bin/bitnet`	✅
`MODEL_PATH`	Absolute path to model file	`/app/models/*.gguf`	✅
`MODEL_URL`	URL to download model from	-	❌
`NUM_WORKERS`	Number of worker processes	`1`	❌
`MAX_QUEUE_SIZE`	Max queued requests	`10`	❌
`QUEUE_TIMEOUT`	Queue timeout (seconds)	`30`	❌
`GENERATION_TIMEOUT`	Generation timeout (seconds)	`60`	❌
`LOG_LEVEL`	Logging level	`INFO`	❌

Worker States

IDLE: Worker is ready to accept requests
BUSY: Worker is processing a request
STARTING: Worker subprocess is initializing
UNHEALTHY: Worker crashed or malfunctioning
SHUTTING_DOWN: Worker is being terminated

🔌 API Endpoints

Health Check

GET /health

Response:

{
  "status": "ok",
  "version": "0.2.0"
}

Metrics

GET /metrics

Response:

{
  "total_requests": 42,
  "successful_requests": 40,
  "failed_requests": 2,
  "average_latency_ms": 1234.56
}

Chat Completion

POST /api/v1/chat
Content-Type: application/json

{
  "message": "What is the capital of France?",
  "max_tokens": 128,
  "temperature": 0.7
}

Response:

{
  "response": "The capital of France is Paris.",
  "latency_ms": 1234.56
}

🐛 Debugging

Check Worker Health

Workers log their stderr output with the tag "Worker {id} stderr". Look for these in logs:

docker logs <container-id> | grep "stderr"

Common Issues

Exit Code 127

Problem: Binary or shared libraries not found
Solution:

Verify BINARY_PATH is absolute
Check ldd /app/bin/your-binary to see missing libraries
Copy required .so files to /usr/local/lib/ and run ldconfig

Workers Constantly Restarting

Problem: Binary crashes on startup
Solution:

Check stderr logs for the actual error
Test binary manually: docker exec -it <container> /app/bin/your-binary --help
Verify model file exists and is readable

503 Service Unavailable

Problem: All workers are unhealthy or queue is full
Solution:

Increase NUM_WORKERS
Increase MAX_QUEUE_SIZE
Check what's causing workers to crash

📊 Performance Tuning

Worker Count

NUM_WORKERS = max(1, cpu_count - 1)  # Leave 1 CPU for API

Memory Optimization

Use quantized models (Q4_K_M, Q5_K_M)
Reduce context size (-c flag)
Monitor with docker stats

Latency Optimization

Keep workers persistent (already implemented)
Use streaming responses (requires SSE implementation)
Enable GPU acceleration (add CUDA support in Dockerfile)

🚢 Deployment

Hugging Face Spaces

Push repository to Hugging Face
Set Space SDK to "Docker"
Configure secrets in Settings
Space will auto-build from Dockerfile

Cloud Platforms

AWS: Use ECS/Fargate with ALB
Google Cloud: Deploy to Cloud Run
Azure: Use Container Instances

Scaling

For high-traffic scenarios:

Deploy multiple instances behind a load balancer
Use Redis for request queue (replace in-memory queue)
Add horizontal pod autoscaling (Kubernetes)

📁 Project Structure

BitForge/
├── app/
│   ├── api/
│   │   └── v1/
│   │       └── chat.py          # API endpoints
│   ├── core/
│   │   ├── config.py            # Configuration
│   │   ├── logging.py           # Structured logging
│   │   └── metrics.py           # Basic metrics
│   ├── inference/
│   │   ├── bitnet.py            #Engine abstraction
│   │   └── worker.py            # 🔴 Worker pool (core logic)
│   ├── schemas/
│   │   └── chat.py              # Pydantic models
│   ├── utils/
│   │   └── model.py             # Model download helper
│   └── main.py                  # FastAPI app
├── bin/                         # Place compiled binaries here
├── models/                      # Place model files here
├── examples/                    # Usage examples
├── Dockerfile                   # Multi-stage build
├── requirements.txt
└── README.md

🛡️ Production Checklist

🤝 Contributing

This is a template project. To adapt it:

Fork the repository
Update binary/model paths in config.py
Modify worker command in worker.py
Test locally with your LLM
Update this README with your specific setup

📝 License

MIT License - feel free to use this infrastructure for your own projects.

🔗 Related Projects

llama.cpp - Fast C++ inference
BitNet - 1-bit quantization
FastAPI - Modern Python web framework
Uvicorn - Lightning-fast ASGI server

Built with ❤️ for the LLM community. Contributions welcome!

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

BitForge: Production-Ready LLM Inference Backend

🎯 What This Infrastructure Provides

🏗️ Architecture Overview

Key Components

1. Worker Pool (app/inference/worker.py)

2. BitNet Engine (app/inference/bitnet.py)

3. Configuration (app/core/config.py)

4. API Routes (app/api/v1/chat.py)

🚀 Quick Start

Prerequisites

Local Development

Docker Deployment

� Adapting for Your LLM

Step 1: Configure Your Binary

Step 2: Update Worker Command

Step 3: Adjust Output Parsing

Step 4: Update Dockerfile

📋 Configuration Reference

Environment Variables

Worker States

🔌 API Endpoints

Health Check

Metrics

Chat Completion

🐛 Debugging

Check Worker Health

Common Issues

Exit Code 127

Workers Constantly Restarting

503 Service Unavailable

📊 Performance Tuning

Worker Count

Memory Optimization

Latency Optimization

🚢 Deployment

Hugging Face Spaces

Cloud Platforms

Scaling

📁 Project Structure

🛡️ Production Checklist

🤝 Contributing

📝 License

🔗 Related Projects

1. Worker Pool (`app/inference/worker.py`)

2. BitNet Engine (`app/inference/bitnet.py`)

3. Configuration (`app/core/config.py`)

4. API Routes (`app/api/v1/chat.py`)