English | 中文

vLLM Service Management System

A simple and easy-to-use vLLM service management tool that can run multiple large language models simultaneously and provides a unified API interface.

Core Features

• Multi-model Management: Run multiple different AI models simultaneously
• GPU Resource Allocation: Assign independent GPUs for each model
• Unified API Interface: Access all models through one interface, compatible with OpenAI API
• Security Authentication: Support Token authentication protection
• Real-time Response: Support streaming output
• Complete Monitoring: Provide logging and status monitoring

Project Structure

vllm_serve/
├── config.json              # Configuration file (model list, ports, GPU, etc.)
├── docker-compose.yml       # Docker Compose configuration (auto-generated from config.json)
├── Dockerfile               # Docker image configuration
├── docker.py                # Docker management tool (generate/start/stop/restart)
├── run_docker.sh            # Docker service management script
├── proxy.py                 # FastAPI proxy service
├── vllmer.py                # vLLM process management tool
├── run.sh                   # Local deployment service management script
├── requirements.txt          # Python dependencies
├── log/                     # Log directory
│   ├── proxy.log
│   ├── vllm_{model_name}.log
│   └── ...
└── pid/                     # Process ID file directory
    ├── proxy.pid
    ├── vllm_{model_name}.pid
    └── ...

Requirements

• Python 3.8 or higher
• NVIDIA GPU with CUDA support (for running vLLM)
• Docker and Docker Compose (optional, for containerized deployment)

Installation

1. Install Python Dependencies

pip install -r requirements.txt

2. Install vLLM

vLLM is a high-performance large language model inference engine. Choose the appropriate installation method based on your CUDA version:

pip install vllm==0.11.0

Verify installation:

python -c "import vllm; print(vllm.__version__)"

3. Download Docker Image

If using Docker to deploy model services, you need to download the vLLM Docker image:

docker pull vllm/vllm:0.11.0

Prepare Models

Ensure model files are downloaded to the host:

# Example model paths
/models/bge-m3
/models/qwen3

Configuration

Modify the config.json file to configure model paths, ports, GPU, and other parameters.

Configuration File Example

{
    "proxy": {
        "token":"your_token",
        "listen_port": 20191,
        "host": "0.0.0.0"
    },
    "models": [
        {
            "enable": true,
            "name": "bge-m3",
            "path": "/models/bge-m3",
            "host": "127.0.0.1",
            "port": 8001,
            "vllm_env": {
                "CUDA_VISIBLE_DEVICES": "1"
            },
            "vllm_args": [
                "--tensor-parallel-size", "1",
                "--dtype", "half",
                "--task", "embedding",
                "--gpu-memory-utilization", "0.3"
            ]
        },
        {
            "enable": true,
            "name": "qwen3",
            "path": "/models/qwen3",
            "host": "127.0.0.1",
            "port": 8002,
            "vllm_env": {
                "CUDA_VISIBLE_DEVICES": "0"
            },
            "vllm_args": [
                "--tensor-parallel-size", "1",
                "--gpu-memory-utilization", "0.8",
                "--max-model-len", "32768"
            ]
        }
    ]
}

Configuration Parameters

Proxy Configuration

Field	Type	Description	Default
token	string	API authentication token, empty means no verification	""
listen_port	int	Proxy service listening port	8000
host	string	Proxy service listening address	"127.0.0.1"

Model Configuration

Field	Type	Description	Default
enable	boolean	Whether to enable this model	true
name	string	Model name, used when calling API	-
path	string	Model file path	-
host	string	vLLM service listening address	"127.0.0.1"
port	int	vLLM service listening port	-
vllm_env	object	Environment variables for vLLM process	{}
vllm_args	array	vLLM startup parameters	[]

The CUDA_VISIBLE_DEVICES configuration item is used to specify the GPU devices visible to the vLLM process. In config.json, each model can be configured with different GPU devices to achieve model parallel computing.

Local Deployment

Use the run.sh script to manage local vLLM services and start the proxy service simultaneously.

Using run.sh to Manage Services

# Start all services
./run.sh start

# Stop all services
./run.sh stop

# Restart all services
./run.sh restart

# Check service status
./run.sh status

# Operate proxy separately
./run.sh start proxy
./run.sh stop proxy
./run.sh restart proxy

# Operate vLLM services separately
./run.sh start vllm
./run.sh stop vllm
./run.sh restart vllm

View Logs

• Proxy logs: log/proxy.log
• vLLM model logs: log/vllm_{model_name}.log

Access API

• Proxy service: http://localhost:20191/v1/chat/completions
• Model services: Each model has an independent port, such as http://localhost:8001/v1/chat/completions

Docker Deployment

This project uses Docker Compose to deploy model services with a hybrid architecture:

• Proxy service: Runs locally (managed by run.sh), responsible for request routing
• Model containers: Each model in an independent container, running with vllm/vllm-openai image

Architecture

┌─────────────┐
│   Proxy    │ (Local process, port 20191)
│  (Local)   │
└──────┬──────┘
       │
       ├──────────────┐
       │              │
       ▼              ▼
┌─────────────┐  ┌─────────────┐
│  Model 1   │  │  Model 2   │
│  Container  │  │  Container  │
│ (Port 8001) │  │ (Port 8002) │
└─────────────┘  └─────────────┘

Managing Docker Services

Use the run_docker.sh script to manage Docker model containers and local Proxy service:

./run_docker.sh generate    # Generate docker-compose.yml
./run_docker.sh start       # Start model containers and local Proxy
./run_docker.sh stop        # Stop model containers and local Proxy
./run_docker.sh restart     # Restart model containers and local Proxy
./run_docker.sh status      # Check all service status
./run_docker.sh logs        # View all model container logs
./run_docker.sh logs model-bge-small-zh-v1.5 # View specific model logs

This script will:

• Read config.json configuration
• Generate independent container services for each enabled model
• Automatically configure GPU allocation and port mapping
• Generate complete docker-compose.yml file
• Automatically start local Proxy service after starting model containers
• Provide unified service management commands

Note: After modifying config.json, you need to run ./run_docker.sh generate to update docker-compose.yml.

Start All Services

./run_docker.sh generate

./run_docker.sh start

Check Service Status

./run_docker.sh status

View Logs

# View all model container logs
./run_docker.sh logs

# View specific model logs
./run_docker.sh logs model-bge-small-zh-v1.5
./run_docker.sh logs model-rsguard-v1

# View Proxy logs (local process)
tail -f log/proxy.log

Stop Services

./run_docker.sh stop

Restart Services

./run_docker.sh restart

Regenerate docker-compose.yml

./run_docker.sh generate

Docker Notes

1. NVIDIA Container Toolkit: Ensure the host has NVIDIA Container Toolkit installed
2. Model paths: Model files need to be correctly mounted into the container
3. GPU allocation: Different models are recommended to use different GPUs to avoid resource conflicts
4. Network configuration: All model containers are in the same network, Proxy accesses through host ports
5. Configuration synchronization: Configuration in docker-compose.yml needs to be consistent with config.json
6. Port mapping: Ensure host ports are not occupied
7. Log viewing: Use ./run_docker.sh logs to view model container logs, use tail -f log/proxy.log to view Proxy logs
8. Resource limits: GPU and memory limits can be configured in deploy.resources
9. Proxy service: Proxy service runs locally, managed by ./run.sh
10. Model paths: Model files need to be correctly mounted into the container, path configuration in config.json cannot contain spaces or special characters

API Usage

The proxy service provides an OpenAI-compatible API interface, and all requests need to include the Authorization header.

Health Check

curl http://127.0.0.1:20191/health

Chat Completions

curl -X POST http://127.0.0.1:20191/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your_token" \
  -d '{
    "model": "rsguard:v1",
    "messages": [
      {"role": "user", "content": "Hello!"}
    ],
    "stream": true
  }'

Text Completions

curl -X POST http://127.0.0.1:20191/v1/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your_token" \
  -d '{
    "model": "rsguard:v1",
    "prompt": "How is the weather today?",
    "max_tokens": 100
  }'

Text Embeddings

curl -X POST http://127.0.0.1:20191/v1/embeddings \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your_token" \
  -d '{
    "model": "bge-small-zh-v1.5",
    "input": "This is a text that needs to be embedded"
  }'

Python Code Example

import requests

url = "http://127.0.0.1:20191/v1/chat/completions"
headers = {
    "Content-Type": "application/json",
    "Authorization": "Bearer your_token"
}
data = {
    "model": "rsguard:v1",
    "messages": [
        {"role": "user", "content": "Hello!"}
    ]
}

response = requests.post(url, headers=headers, json=data)
print(response.json())

Log Viewing

Docker Deployment

• Proxy logs (local): tail -f log/proxy.log
• Model container logs: ./run_docker.sh logs
• Specific model logs: ./run_docker.sh logs model-name

Local Deployment

• Proxy logs: tail -f log/proxy.log
• vLLM model logs: tail -f log/vllm_{model_name}.log

Troubleshooting

Docker Deployment

Model Container Cannot Start

1. Check if Docker is running normally: docker ps
2. Check if model path is correct
3. Check if GPU is available: nvidia-smi
4. View container logs: ./run_docker.sh logs model-name
5. Check if port is occupied: lsof -i :{port}

Proxy Cannot Connect to Model Container

1. Check if model container is running: ./run_docker.sh status
2. Check model service host and port configuration (host should be 127.0.0.1)
3. View Proxy logs: tail -f log/proxy.log

Local Deployment

Model Cannot Start

1. Check if model path is correct
2. Check if GPU is available: nvidia-smi
3. View model logs: tail -f log/vllm_{model_name}.log
4. Check if port is occupied: lsof -i :{port}

Proxy Cannot Connect to Model

1. Check if model is running: python3 vllmer.py status
2. Check model service host and port configuration
3. View proxy logs: tail -f log/proxy.log

Token Authentication Failed

1. Check if Authorization header format in request is correct
2. Confirm token configuration in config.json

Notes

Docker Deployment

1. Ensure Docker and Docker Compose are correctly installed
2. Ensure NVIDIA Container Toolkit is installed
3. Model paths must exist and be accessible
4. Different models are recommended to use different GPUs (configured via vllm_env)
5. Proxy service runs locally, managed by ./run.sh
6. Regularly clean up old files in log and pid directories

Local Deployment

1. Ensure vLLM is correctly installed and available
2. Model paths must exist and be accessible
3. Different models are recommended to use different GPUs (configured via vllm_env)
4. Stopping services will stop all processes with PID files, including disabled models
5. Regularly clean up old files in log and pid directories

License

This project is licensed under the Apache 2.0 License.