Skip to content

deployment.md

Latest commit

 

History

History
552 lines (421 loc) · 17.1 KB

deployment.md

File metadata and controls

552 lines (421 loc) · 17.1 KB
Raw

Rectifiers 部署与配置指南

从零开始部署 Rectifiers 协同加速层，集成 GPUStack 实例、配置 LMCache 和 RDMAS 存储，完成端到端验证。

一、概述

Rectifiers 为 vLLM/SGLang 多 GPU 推理集群提供缓存感知路由和动态 P:D 比例调优。部署涉及以下组件：

GPUStack (worker 生命周期) ──→ Rectifiers (路由+编排) ──→ RDMAS (共享存储)
                                     │
                              LMCache Connector (PyO3 plugin)
组件 职责 端口 部署个数
Rectifiers (Director+Router 合并) 缓存索引 + 请求路由 HTTP :8080, gRPC :9200 1
Orchestrator P:D 比例自动调优 gRPC :9201 1
RDMAS Storage One-Sided RDMA 共享 KV cache 存储 gRPC :9400 ≥1
vLLM Workers 推理引擎（Preffill + Decode） :8000 ≥4
LMCache Connector KV cache 写入后自动上报 Director 嵌入 vLLM 进程 每 worker 1 个

二、环境准备

2.1 硬件要求

组件 最低 推荐
GPU 节点 1 × A100/H100 (80GB) 4-8 × A100/H100
RDMAS 存储节点 128GB RAM + RDMA NIC 512GB RAM + 100Gbps RoCEv2
Rectifiers 节点 2 CPU, 2GB RAM 4 CPU, 4GB RAM (可与 GPU 节点同机)

2.2 软件依赖

# Rust 工具链
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup default 1.80

# protoc (gRPC proto 编译)
sudo dnf install protobuf-compiler   # Fedora
sudo apt install protobuf-compiler   # Ubuntu/Debian

# RDMA (RDMAS 存储节点需要)
sudo dnf install libibverbs-devel rdma-core   # Fedora
sudo apt install libibverbs-dev ibverbs-utils # Ubuntu/Debian

# Docker (可选)
sudo dnf install docker docker-compose
sudo systemctl enable --now docker

2.3 RDMAS 存储节点 HugePages

# 分配 HugePages (512GB = 262144 × 2MB)
sudo bash -c 'echo 262144 > /proc/sys/vm/nr_hugepages'

# 永久生效
echo 'vm.nr_hugepages=262144' | sudo tee /etc/sysctl.d/99-hugepages.conf

# 验证
grep HugePages_Total /proc/meminfo
# HugePages_Total:   262144

三、编译

git clone https://github.com/ipconfiger/rectifiers
cd rectifiers

# 编译所有组件（Release 模式）
cargo build --workspace --release

# 产物:
#   target/release/rectifiers      # Director + Router 合并 (推荐)
#   target/release/director        # 独立 Director (开发用)
#   target/release/rdmas-router    # 独立 Router (开发用)
#   target/release/orchestrator    # Orchestrator

四、部署 Rectifiers

4.1 Docker 部署（推荐）

# 1) 编译镜像
docker build -t rectifiers -f docker/Dockerfile.rectifiers .
docker build -t rectifiers-orchestrator -f docker/Dockerfile.orchestrator .

# 2) 启动 (docker-compose)
docker compose -f docker/docker-compose.yml up -d

# 3) 查看状态
docker compose -f docker/docker-compose.yml ps
# NAME                       STATUS
# rectifiers                 Up (healthy)  :8080, :9200
# rectifiers-orchestrator    Up            :9201
# mock-vllm                  Up            :8010

4.2 K8s 部署

# 1) 部署 Rectifiers (Director + Router 合并, replicas=1)
kubectl apply -f k8s/rectifiers.yaml

# 2) 部署 Orchestrator (含 RBAC)
kubectl apply -f k8s/orchestrator.yaml

# 3) 验证
kubectl get pods,svc
# NAME                            READY   STATUS
# pod/rectifiers-xxxx             1/1     Running
# pod/rectifiers-orchestrator-xx  1/1     Running
# service/rectifiers              ClusterIP   :8080,:9200
# service/rectifiers-orchestrator ClusterIP   :9201

4.3 独立二进制部署（开发/测试）

# Terminal 1: Rectifiers (合并模式, HTTP :8080 + gRPC :9200)
cargo run -p rectifiers --release

# Terminal 2: Orchestrator
RECTIFIERS_CONFIG=examples/rectifiers_config.json cargo run -p rdmas-orchestrator --release

4.4 配置文件

Rectifiers 合并二进制使用 Router 配置格式，无需 DIRECTOR_ADDR（默认内部直连）：

{
  "listen_addr": "[::]:8080",
  "tenant_id": "default",
  "model_name": "llama-70b",
  "block_size": 16,
  "cache_salt": "",
  "worker_endpoints": ["http://vllm-worker-0:8000"]
}

Orchestrator 配置见 examples/rectifiers_config.json：

{
  "orchestrator": {
    "listen_addr": "[::]:9201",
    "director_addr": "http://rectifiers:9200",
    "tenant_id": "default",
    "model_name": "llama-70b",
    "auto_tune": {
      "enabled": true,
      "evaluation_interval_secs": 30,
      "hit_rate_low_watermark": 0.3,
      "hit_rate_high_watermark": 0.8,
      "min_prefill": 2, "max_prefill": 6,
      "min_decode": 2,  "max_decode": 6,
      "prefill_deployment": "vllm-prefill",
      "decode_deployment": "vllm-decode",
      "dry_run": true
    }
  }
}

dry_run: true 时 Orchestrator 只记录建议，不实际扩缩。确认无误后改为 false。

五、RDMAS 存储后端配置

5.1 启动 RDMAS 节点

cd /path/to/rdmas
cargo run --release
# ControlPlane gRPC 监听 :9400

5.2 验证 RDMAS 节点

# 检查节点可达
grpcurl -plaintext 10.0.0.1:9400 rdmas.control.ControlPlane/Discover

# 预期输出: ServerMetadata { generation: 1, bucket_count: ..., regions: [...] }

5.3 RDMAS 多节点部署

┌────────────────────────────────────────────┐
│           RDMAS Storage Cluster             │
│                                            │
│  Node 0 (10.0.0.1:9400)  ← 主              │
│  Node 1 (10.0.0.2:9400)                    │
│  Node 2 (10.0.0.3:9400)                    │
│                                            │
│  Rectifiers 不需要感知节点拓扑。              │
│  Connector 上报 (node_id, block_hashes)     │
│  后，Director 自动建立索引。                  │
└────────────────────────────────────────────┘

Rectifiers 与 RDMAS 的关联通过 Connector 上报表建立——不需要在 Rectifiers 配置中硬编码 RDMAS 节点地址。

六、GPUStack 集成

6.1 GPUStack 实例如何连接 Router

GPUStack 的 AI Gateway (Higress) 将推理请求 upstream 指向 Rectifiers Router :8080，而非直接指向 vLLM workers：

Client → GPUStack Gateway → Rectifiers Router (:8080) → 最佳 vLLM Worker

配置 GPUStack AI Gateway upstream：

# GPUStack Higress McpBridge 配置
kubectl apply -f - <<EOF
apiVersion: networking.higress.io/v1
kind: McpBridge
metadata:
  name: gpustack-gateway
spec:
  registries:
  - name: rectifiers
    type: static
    domain: rectifiers.default.svc.cluster.local
    port: 8080
EOF

6.2 部署 vLLM Worker (Prefill Pool)

在 GPUStack 中创建 Custom Inference Backend：

backend_name: vllm-rectifiers-custom
default_entrypoint: vllm serve
default_execution_command: >
  {{model_path}}
  --host {{worker_ip}}
  --port {{port}}
  --served-model-name {{model_name}}
  --tensor-parallel-size {{gpu_count}}
  --enable-prefix-caching
  --kv-transfer-config '{"kv_role":"kv_producer","kv_connector":"LMCacheConnectorV1"}'
  --lmcache-config '{"type":"native_plugin","module_path":"lmcache_rdma_connector","class_name":"RDMANativeConnector","adapter_params":{"device":"mlx5_0","server":"10.0.0.1:9400","num_workers":4,"director_addr":"rectifiers:9200","node_id":"rdmas-0","tenant_id":"default","model_name":"{{MODEL_NAME}}","block_size":16,"instance_id":"prefill-{{WORKER_NODE}}","role":"prefill","rpc_endpoint":"http://{{worker_ip}}:{{port}}"},"eviction":{"eviction_policy":"LRU","trigger_watermark":0.8}}'
version_configs:
  v1:
    image_name: vllm-rectifiers:latest
    custom_framework: cuda

6.3 部署 vLLM Worker (Decode Pool)

backend_name: vllm-decode-rectifiers-custom
default_entrypoint: vllm serve
default_execution_command: >
  {{model_path}}
  --host {{worker_ip}}
  --port {{port}}
  --served-model-name {{model_name}}
  --tensor-parallel-size {{gpu_count}}
  --enable-prefix-caching
  --kv-transfer-config '{"kv_role":"kv_consumer","kv_connector":"LMCacheConnectorV1"}'
  --lmcache-config '{"type":"native_plugin","module_path":"lmcache_rdma_connector","class_name":"RDMANativeConnector","adapter_params":{"device":"mlx5_0","server":"10.0.0.1:9400","num_workers":4,"director_addr":"rectifiers:9200","node_id":"rdmas-0","tenant_id":"default","model_name":"{{MODEL_NAME}}","block_size":16,"instance_id":"decode-{{WORKER_NODE}}","role":"decode","rpc_endpoint":"http://{{worker_ip}}:{{port}}"},"eviction":{"eviction_policy":"LRU","trigger_watermark":0.8}}'
version_configs:
  v1:
    image_name: vllm-rectifiers:latest
    custom_framework: cuda

6.4 构建 vLLM Worker 镜像

# Dockerfile.vllm-rectifiers
FROM vllm/vllm-openai:v0.8.5

# 编译 RDMAS Connector (在 RDMAS 仓库中)
# cd /path/to/rdmas && cargo build -p lmcache-connector --features director --release
COPY lmcache_rdma_connector.so /usr/local/lib/python3.12/site-packages/

七、LMCache Connector 配置详解

7.1 adapter_params 字段

字段 必填 默认值 说明
device 是 — RDMA 网卡名，如 mlx5_0
server 是 — RDMAS 存储节点 host:9400
num_workers 否 4 RDMA worker 线程数，建议与 GPU 数一致
batch_chunk_num_bytes 否 16777216 批量聚合阈值 (16MB)
director_addr 否 — 设置后启用 Rectifiers 缓存上报。格式 host:9200
node_id 是* — 当前 worker 所属 RDMAS 节点 ID，如 rdmas-0
tenant_id 是* — 租户隔离标识
model_name 是* — 模型名，与 Router 配置一致
block_size 是* 16 KV cache block 大小，必须与 Router 配置一致
instance_id 否 {node_id}-{ts} Worker 实例 ID，自动上报给 Director
role 否 both prefill / decode / both
rpc_endpoint 否 http://localhost:8000 Worker HTTP 地址，供 Director 返回给 Router

* 启用 director_addr 后必填。

7.2 Connector 自动行为

启用 director_addr 后，Connector 在启动时自动：

  1. Register — 向 Director 注册此 worker 实例（instance_id, role, rpc_endpoint）
  2. Heartbeat — 每 10 秒向 Director 发送心跳，保持注册有效
  3. ReportStore — 每次 submit_batch_set 的 RDMA 写入完成后，在 drain_completions 中上报 (node_id, block_hashes)
  4. ReportRemove — 每次 submit_batch_delete 完成后上报移除
  5. Deregister — 进程关闭时自动注销

7.3 编译 Connector

cd /path/to/rdmas
cargo build -p lmcache-connector --features director --release
# 产物: target/release/liblmcache_rdma_connector.so

7.4 不带 Rectifiers 的独立模式

不设置 director_addr 时，Connector 退化为普通 RDMAS L2 存储后端，不向 Director 上报：

{
  "type": "native_plugin",
  "module_path": "lmcache_rdma_connector",
  "class_name": "RDMANativeConnector",
  "adapter_params": {
    "device": "mlx5_0",
    "server": "10.0.0.1:9400",
    "num_workers": 4
  }
}

八、配置验证

8.1 验证 Rectifiers 服务

# 1) Rectifiers HTTP 健康检查
curl http://localhost:8080/healthz
# {"status":"ok"}

# 2) Rectifiers gRPC 健康检查
grpcurl -plaintext localhost:9200 grpc.health.v1.Health/Check
# {"status":"SERVING"}

# 3) Orchestrator 状态
grpcurl -plaintext localhost:9201 rectifiers.orchestrator.Orchestrator/GetRecommendation

8.2 验证 Worker 注册

# 查看 Director 中已注册的实例
grpcurl -plaintext localhost:9200 rectifiers.director.Director/GetStats \
  -d '{"tenant_id":"default","model_name":"llama-70b"}'
# {"hit_rate":0.0, "total_blocks_indexed":0, "prefill_instance_count":4, "decode_instance_count":4}

prefill_instance_count 和 decode_instance_count 应与非零值。

8.3 验证缓存索引

# 1) 发送一个推理请求，触发 KV cache 写入
curl -X POST http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"prompt":"Explain quantum computing in simple terms","model":"llama-70b","max_tokens":32,"stream":false}'

# 2) 等待几秒后，查询 Director 索引状态
grpcurl -plaintext localhost:9200 rectifiers.director.Director/GetStats \
  -d '{"tenant_id":"default","model_name":"llama-70b"}'
# {"total_blocks_indexed":128, ...}  ← 应 > 0

# 3) 验证缓存感知路由
grpcurl -plaintext localhost:9200 rectifiers.director.Director/Query \
  -d '{"tenant_id":"default","model_name":"llama-70b","block_hashes":[12345678901234567890],"block_size":16}'
# {"hits":[{"node_id":"rdmas-0","matched_blocks":1,...}]}  ← 应有命中

8.4 端到端测试

# 完整缓存感知路由测试
# 第一次请求 (冷启动, 写入 KV cache)
time curl -X POST http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"prompt":"What is the capital of France?","model":"llama-70b","max_tokens":10,"stream":false}'

# 第二次相同请求 (热缓存命中, 应从 RDMAS 加载 KV cache)
time curl -X POST http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"prompt":"What is the capital of France?","model":"llama-70b","max_tokens":10,"stream":false}'

# 对比两次请求的延迟 — 第二次应明显快于第一次 (跳过了 prefill)

8.5 验证 Orchestrator 自动调优

# 查看当前 P:D 比例 (dry_run 模式)
grpcurl -plaintext localhost:9201 rectifiers.orchestrator.Orchestrator/GetPDRatio
# {"prefill_count":4,"decode_count":4,"total_instances":8}

# 查看调优建议
grpcurl -plaintext localhost:9201 rectifiers.orchestrator.Orchestrator/GetRecommendation
# {"recommended_prefill":4,"recommended_decode":4,"reason":"stable"}

# 查看 Orchestrator 日志 (docker)
docker compose -f docker/docker-compose.yml logs orchestrator | grep -i "tick\|scale"

九、RDMAS 存储验证

9.1 验证 RDMAS 节点

# 节点发现
grpcurl -plaintext 10.0.0.1:9400 rdmas.control.ControlPlane/Discover
# {"metadata":{"generation":1,"bucket_count":16777216,"regions":[...]}}

# 检查 RDMA 连通性
ibv_devinfo | grep -E "hca_id|link_layer|state"
# hca_id: mlx5_0
# link_layer: Ethernet
# state: PORT_ACTIVE (4)

9.2 验证 HugePages

# RDMAS 存储节点上
grep -E "HugePages_Total|HugePages_Free" /proc/meminfo
# HugePages_Total:   262144
# HugePages_Free:    262140   ← 应有足够空闲页

# 如果为 0，重新分配
sudo bash -c 'echo 262144 > /proc/sys/vm/nr_hugepages'

十、故障排查

症状 原因 检查
grpcurl :9200 无响应 Director 未启动 docker compose ps rectifiers
prefill_instance_count: 0 Connector 未配置 director_addr 检查 adapter_params 中的 director_addr
total_blocks_indexed: 0 RDMA 写入失败或 drain_completions 未触发 检查 RDMAS 节点可达性、RDMA 网卡状态
缓存命中路由无效 block_size 不一致 确认 Router 配置和 adapter_params 中的 block_size 相同
Orchestrator 无建议 dry_run: true 或 Director 不可达 检查 director_addr 配置、查看 Orchestrator 日志
replicas=1 无法扩展 Router 设计约束 合并模式下 PrefixIndex 为内存状态，不支持水平扩展。若需扩展 Router，改用独立 Director + Router 模式
Connector 编译失败 缺少 libibverbs sudo dnf install libibverbs-devel
vLLM 启动失败 lmcache_rdma_connector.so 未找到 确认 .so 已打入镜像，路径在 site-packages/ 中

日志查看

# Rectifiers 日志
docker compose -f docker/docker-compose.yml logs rectifiers

# Orchestrator 日志
docker compose -f docker/docker-compose.yml logs orchestrator

# RDMAS 日志 (直接运行的进程)
journalctl -u rdmas -f

常用诊断命令

# 查看所有注册实例
grpcurl -plaintext localhost:9200 rectifiers.director.Director/GetStats \
  -d '{"tenant_id":"default","model_name":"llama-70b"}'

# 查看 Router 健康
curl http://localhost:8080/healthz

# 查看 P:D 比例
grpcurl -plaintext localhost:9201 rectifiers.orchestrator.Orchestrator/GetPDRatio

# RDMA 性能测试
ib_read_bw -d mlx5_0 -a -F --report_gbits 10.0.0.1

附录：一键部署脚本

#!/bin/bash
set -e

echo "=== Rectifiers Deployment ==="

# 1. 编译
echo "[1/5] Building..."
cargo build --workspace --release

# 2. 构建镜像
echo "[2/5] Building Docker images..."
docker build -t rectifiers -f docker/Dockerfile.rectifiers .
docker build -t rectifiers-orchestrator -f docker/Dockerfile.orchestrator .

# 3. 启动服务
echo "[3/5] Starting services..."
docker compose -f docker/docker-compose.yml up -d

# 4. 等待就绪
echo "[4/5] Waiting for services..."
sleep 5
curl -s --retry 10 --retry-delay 2 http://localhost:8080/healthz

# 5. 验证
echo "[5/5] Verifying..."
grpcurl -plaintext localhost:9200 grpc.health.v1.Health/Check | grep SERVING
echo ""
echo "=== Deployment complete ==="
echo "Router:  http://localhost:8080"
echo "gRPC:    localhost:9200"
echo "Orch:    localhost:9201"