Skip to content

POD-SETUP.md

Latest commit

 

History

History
482 lines (377 loc) · 21.1 KB

POD-SETUP.md

File metadata and controls

482 lines (377 loc) · 21.1 KB
Raw

Pod setup runbook

Target: one RunPod Secure Cloud pod with a single 48GB card, used for LoRA training and vLLM serving. Everything ML runs here; the Windows box only edits files and drives git/scp.

Pinned models (researched 30 Aug 2026):

role HF repo id notes
primary Qwen/Qwen3.5-9B dense (no MoE), post-trained/instruct, Apache 2.0, ungated. ~18GB bf16.
fallback Qwen/Qwen3.5-4B same architecture and tokenizer, ~8GB bf16.

Both are the post-trained checkpoints — there is no -Instruct suffix in this family; the -Base repos are the pretrained ones. Both are natively multimodal (vision tower + LM in one checkpoint); for text-only work load Qwen3_5ForCausalLM, which instantiates the language model alone.

Thinking mode is on by default. To get plain chat:

  • transformers: tokenizer.apply_chat_template(..., enable_thinking=False)
  • vLLM offline: chat_template_kwargs={"enable_thinking": False}
  • vLLM serve: --reasoning-parser qwen3 --default-chat-template-kwargs '{"enable_thinking": false}'

1. Renting the pod (web UI)

  1. runpod.io -> Pods -> Deploy.

  2. Toggle Secure Cloud (not Community Cloud — Community hosts are cheaper but get reclaimed, and a reclaimed pod mid-sweep costs more than the savings).

  3. Pick a 48GB card. Observed Secure Cloud rates on 30 Aug 2026:

    GPU VRAM Secure $/hr Community $/hr
    RTX A6000 48GB $0.53 $0.33
    A40 48GB $0.44 $0.35
    RTX 6000 Ada 48GB $0.84 $0.74
    L40 48GB $0.82 $0.69
    L40S 48GB $0.99 $0.79

    A40 or RTX A6000 is the right default here (Ampere, ~$0.44–0.53/hr, and the workload is memory-bound rather than compute-bound). L40S is roughly 2x the price for maybe 1.5x the throughput. Prices drift — re-check at rental.

  4. Template: PyTorch 2.x / CUDA 12.x (runpod/pytorch:...-cuda12...). The pinned vLLM builds against torch 2.13 + CUDA 12.x; do not pick a CUDA 11 image.

  5. Volume: 100 GB, mount path /workspace. The 9B checkpoint is ~18GB and the HF cache plus adapters plus logs will comfortably pass 40GB. Container disk 50GB is fine.

  6. Under Expose TCP Ports add 22. Deploy, then open Connect -> SSH over exposed TCP and copy the host and port into .env (POD_SSH_HOST, POD_SSH_PORT).

Storage note: a stopped pod still bills for its volume (~$0.05–0.07/GB/mo network storage, $0.10/GB/mo container disk). 100GB stopped is a few dollars a month, not free.

2. First SSH login — copy-paste block

ssh root@$POD_SSH_HOST -p $POD_SSH_PORT

# ---- one-shot setup ----------------------------------------------------------
set -e
cd /workspace
mkdir -p /workspace/{adapters,hf_cache,repo}

# keep the HF cache on the 100GB volume, not the small container disk.
# the CUDA 12.8 toolkit IS in this image but is not on PATH.
export HF_HOME=/workspace/hf_cache
export CUDA_HOME=/usr/local/cuda
export PATH="$CUDA_HOME/bin:$PATH"
export TORCH_CUDA_ARCH_LIST="8.6"          # A40 = sm_86; keeps source builds short
export PIP_ROOT_USER_ACTION=ignore
export PIP_BREAK_SYSTEM_PACKAGES=1         # Ubuntu 24.04 ships a PEP 668 marker
cat >> ~/.bashrc <<'EOF'
export HF_HOME=/workspace/hf_cache
export CUDA_HOME=/usr/local/cuda
export PATH="$CUDA_HOME/bin:$PATH"
export TORCH_CUDA_ARCH_LIST="8.6"
export PIP_ROOT_USER_ACTION=ignore
export PIP_BREAK_SYSTEM_PACKAGES=1
EOF

# PyJWT is apt-owned with no RECORD file, so pip aborts the whole transaction when a
# dependency tries to upgrade it. Shadow it with a pip-managed copy first.
pip install --break-system-packages --ignore-installed PyJWT setuptools wheel

# pinned installs (see "Why these pins" below). NOTE: vllm is deliberately NOT taken
# from PyPI - see the CUDA-variant trap below.
pip install --break-system-packages \
  "transformers==5.16.1" \
  "peft==0.20.0" \
  "trl==1.11.0" \
  "datasets==5.0.1" \
  "accelerate==1.14.0" \
  "huggingface_hub>=0.35"

# vLLM, in TWO steps - both are needed.
#  (1) Install from PyPI to resolve vLLM's full runtime dependency set (fastapi,
#      uvicorn, openai, xgrammar, ...). Skipping this and going straight to the
#      cu129 wheel with --no-deps leaves the server entrypoint dead on
#      `ModuleNotFoundError: No module named 'fastapi'`.
#  (2) Overwrite ONLY the wheel with the +cu129 build. The PyPI wheel for 0.28.0 is a
#      CUDA 13 build (it needs libcudart.so.13); the +cu129 build matches the image's
#      torch 2.13.0+cu129. --no-deps here preserves the pins above.
# NOTE: whether the cu13 wheel could run depends on the host driver (r580+ can,
# r570 cannot), but keep the cu129 wheel regardless - the stack must stay identical
# to the one Gate 0 was proven on.
pip install --break-system-packages "vllm==0.28.0"
pip install --break-system-packages --force-reinstall --no-deps \
  https://github.com/vllm-project/vllm/releases/download/v0.28.0/vllm-0.28.0+cu129-cp38-abi3-manylinux_2_28_x86_64.whl

# same trap: torchcodec arrives as a cu13 build alongside the PyPI vllm wheel.
pip install --break-system-packages --force-reinstall --no-deps \
  --index-url https://download.pytorch.org/whl/cu129 "torchcodec==0.16.0"

# fast GatedDeltaNet kernels. WITHOUT these, Qwen3.5's linear-attention layers
# fall back to slow, memory-hungry PyTorch ops and the <5 min LoRA target fails.
# (Measured: 87s train with them on first run, 20s once warm. nvcc 12.8 is present,
# so causal-conv1d builds from source in ~8 min.)
pip install --break-system-packages -U "flash-linear-attention>=0.4.2" --no-build-isolation
pip install --break-system-packages -U git+https://github.com/Dao-AILab/causal-conv1d --no-build-isolation

# auth (paste the token when prompted; do not put it in this file)
export HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxx
huggingface-cli login --token "$HF_TOKEN"

python -c "import torch, vllm, transformers, trl, peft; \
print('torch', torch.__version__, 'cuda', torch.cuda.is_available()); \
print('vllm', vllm.__version__, 'tf', transformers.__version__, \
      'trl', trl.__version__, 'peft', peft.__version__)"
nvidia-smi
# ------------------------------------------------------------------------------

# run Gate 0 inside tmux so an SSH drop does not kill a 15-minute job
tmux new-session -s gate0
cd /workspace/repo
mkdir -p results
PYTHONUNBUFFERED=1 python scripts/gate0_smoke.py 2>&1 | tee results/gate0.log
# detach: Ctrl-b then d      reattach: tmux attach -t gate0

ALWAYS set PYTHONUNBUFFERED=1 on any job piped to tee

Not cosmetic. Python block-buffers stdout when it is a pipe rather than a terminal, so a job whose total output is smaller than the buffer (a few KB — which is every campaign summary this project runs) writes nothing to the log until it exits, and if the tmux session tears down before tee flushes, the log is left empty.

That happened: the Sep 2 GLM campaign completed all 30 runs correctly and /workspace/logs/campaign_glm.log is 0 bytes. No data was lost, because every run writes its own run_meta.json and those are the authoritative record — but the campaign's own summary line and outcome tally were gone and had to be reconstructed from the run directories.

# correct form for any long job
PYTHONUNBUFFERED=1 python scripts/<job>.py 2>&1 | tee /workspace/logs/<job>.log

Why these pins

package pin reason
vllm 0.28.0 Released 26 Aug 2026. First release listing Qwen3.5 text-only dense models; registry maps Qwen3_5ForCausalLM. Also carries the fix (PR #47640) for the packed-LoRA crash on GatedDeltaNet groups. Hard-pins torch==2.13.0, requires transformers>=5.5.3, Python 3.10–3.14.
transformers 5.16.1 Released 26 Aug 2026. Ships the qwen3_5 modelling code and the Qwen3_5ForCausalLM text-only class. Satisfies vLLM's floor. Note v5 renamed torch_dtype -> dtype.
trl 1.11.0 Released 26 Aug 2026 (1.12.0 is a mis-published bit-identical duplicate — avoid the ambiguity). SFTTrainer API changed in 1.7.0 (label construction moved into dataset prep, chunked_nll default loss), so anything older will not match gate0_smoke.py.
peft 0.20.0 Released 28 Jul 2026, current stable.
datasets 5.0.1 Current stable; trl 1.11 requires >=4.7.0.
accelerate 1.14.0 Current stable; trl 1.11 requires >=1.4.0.
flash-linear-attention, causal-conv1d latest Optional in principle, load-bearing in practice — they provide the Gated DeltaNet kernels for the 3:1 hybrid stack.

LoRA target modules — do not change casually

Qwen3.5 is a 3:1 hybrid: three Gated DeltaNet (linear attention) layers per one Gated Attention (full attention) layer. vLLM cannot load adapters that touch the DeltaNet packed projection groups:

  • peft writes in_proj_a / in_proj_b; vLLM only knows the fused in_proj_ba, and rejects the adapter outright (vllm issue #38085).
  • targeting in_proj_qkv without in_proj_z crashes expand_packed_lora (vllm issue #47639, regression from PR #37912, fixed in #47640).

So every adapter in this project targets full-attention + MLP only:

target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                  "gate_proj", "up_proj", "down_proj"]

This still covers all 32 MLP blocks, but only the 8 full-attention blocks. If a rung fails to express, raise rank/epochs before reaching for the DeltaNet layers.

If the pod migrates: what survives, and what to check

RunPod's "migrate pod data" flow moves the network volume only. Observed 30 Aug 2026 moving lkv2nziluiuuct -> hghrm1h3boc05l (the old id is defunct):

survives does NOT survive
/workspace in full — models, adapters, data, results /root including ~/bin scripts and .bashrc
the entire pip environment in /usr/local/lib/python3.12/dist-packages
tmux sessions and anything running

So after a migration: re-run the whole of section 2 (the base image still supplies torch 2.13.0+cu129 and the CUDA 12.8 toolkit, so only the pip layer is rebuilt), re-upload scripts/, and restart the server. Budget ~15 minutes.

The host may also change GPU driver — this migration went 570.211.01 -> 580.159.04. Check it, because it decides whether a CUDA 13 wheel could run at all.

Verify the base checkpoint before serving or training anything. Migration is a data transfer; recompute the hashes rather than trusting it:

python - <<'PY'
import hashlib, json
from pathlib import Path
man = json.loads(Path("results/base_materialization.json").read_text())
out = Path(man["output_path"])
for e in man["files"]:
    h = hashlib.sha256(); p = out / e["name"]
    with p.open("rb") as fh:
        for b in iter(lambda: fh.read(8 << 20), b""): h.update(b)
    print(("OK  " if h.hexdigest() == e["sha256"] else "BAD "), e["name"])
PY

Result of that check after the 30 Aug migration: 10/10 files byte-identical, 0 mismatched, 0 missing — and the post-migration serve-path verify reproduced gate0_toy at canary 3/3, mean |drift| 0.2750 (0.2762 before the move).

Verified environment (30 Aug 2026, pod hghrm1h3boc05l after migration)

A40 46068 MiB, driver 580.159.04, CUDA toolkit 12.8.93, Ubuntu 24.04.4, Python 3.12.3, 96 vCPU / 503 GB RAM, /workspace on a RunPod network volume.

torch 2.13.0+cu129   transformers 5.16.1   trl 1.11.0    peft 0.20.0
datasets 5.0.1       accelerate 1.14.0     vllm 0.28.0 (+cu129 wheel)
huggingface_hub 1.29.0   fla 0.5.2   causal_conv1d 1.7.0   torchcodec 0.16.0+cu129

ModelRegistry confirms Qwen3_5ForCausalLM, Qwen3_5ForConditionalGeneration and Qwen3_5MoeForCausalLM are all registered in vllm 0.28.0.

RESOLVED: adapter/serving module-prefix mismatch

Fixed 30 Aug 2026 by Option 1 below (decision: Ebin). Gate 0 now passes 6/6 — see results/gate0_rerun.log, mean |logprob drift| 0.2735. Kept here because the failure is silent and will recur if anyone points training or serving back at the stock multimodal checkpoint.

The original symptom: Gate 0 reached 5/6, with (e) failing because a LoRA adapter trained against the text-only class is silently inert when served by vLLM. The adapter loaded without any warning and produced byte-identical output to base (mean |logprob diff| = 0.0000 over 43 tokens).

Cause — the two classes rename the language tower differently:

vLLM class hf_to_vllm_mapper prefix rule resulting module names
Qwen3_5ForCausalLM model.language_model. → model. model.layers.N.mlp.gate_up_proj
Qwen3_5ForConditionalGeneration model.language_model. → language_model.model. language_model.model.layers.N.mlp.gate_up_proj

Training used Qwen3_5ForCausalLM, so peft wrote base_model.model.model.layers.N… → model.layers.N…. But language_model_only=True still instantiates the multimodal class, whose modules are language_model.model.layers.N…. Nothing matches, so zero LoRA modules are applied and vLLM does not warn.

This is not a target_modules problem — the adapter is well-formed (256 tensors over the intended 7 projections) and expresses perfectly under transformers (5/5). Candidate remedies were:

  1. ADOPTED — materialize a text-only base once (scripts/materialize_base.py): load with Qwen3_5ForCausalLM, re-save, and use that vision-free checkpoint as the base for both training and vLLM. HF's own Qwen3.5 docs recommend exactly this for text-only work. Training and serving now share one module tree, and vLLM needs no flags because the config advertises Qwen3_5ForCausalLM. Output /workspace/models/qwen3.5-9b-text, 16.70 GiB, 0 vision tensors of 427; provenance in results/base_materialization.json.
  2. Train against Qwen3_5ForConditionalGeneration so peft writes model.language_model.… keys that the multimodal mapper resolves. Keeps the base repo id unchanged; costs vision-tower VRAM during training and assumes vLLM applies hf_to_vllm_mapper to adapter weights (unverified).
  3. Rewrite adapter keys post-hoc to the language_model.model. prefix. Works without retraining but must be re-applied to every rung; most fragile.

Standing rule (Ebin, 30 Aug 2026): every adapter must prove expression through the serving path before it enters any experiment — a canary check plus a non-zero mean |logprob drift|. gate0_smoke.py step (f) now hard-fails on exactly-zero drift. Zero means the adapter is not applied, never that the diff is small; this is the one failure mode that otherwise looks like a clean pass.

3. Getting the repo onto the pod (from Windows)

Preferred — clone from git if the repo has a remote:

ssh root@$env:POD_SSH_HOST -p $env:POD_SSH_PORT "git clone <repo-url> /workspace/repo"

Git-less, one line from PowerShell (scp ships with Windows 10+ OpenSSH):

scp -P $env:POD_SSH_PORT -r C:\Users\ebin\claude-ground\neel-mats-sept-26\b13-diffing-bench\scripts root@${env:POD_SSH_HOST}:/workspace/repo/

Iterating on a script (rsync via WSL, skips unchanged files):

wsl rsync -avz -e "ssh -p $env:POD_SSH_PORT" /mnt/c/Users/ebin/claude-ground/neel-mats-sept-26/b13-diffing-bench/scripts/ root@${env:POD_SSH_HOST}:/workspace/repo/scripts/

Pulling results back down:

scp -P $env:POD_SSH_PORT -r root@${env:POD_SSH_HOST}:/workspace/repo/results C:\Users\ebin\claude-ground\neel-mats-sept-26\b13-diffing-bench\

4. Programmatic management (alternative to the web UI)

RunPod is managed here via its REST API rather than a plugin or MCP server. Base URL https://rest.runpod.io/v1, auth via Authorization: Bearer <key>. Put the key in .env as RUNPOD_API_KEY and read it from the environment — never inline a key in a command or commit one.

set -a; source .env; set +a     # loads RUNPOD_API_KEY into the environment

List GPU availability and price. The REST v1 surface has no GPU-type endpoint; availability still comes from the GraphQL API, which accepts the same bearer token (use the header, not the legacy ?api_key= query parameter — keys do not belong in URLs):

curl -s https://api.runpod.io/graphql \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"{ gpuTypes { id displayName memoryInGb secureCloud communityCloud securePrice communityPrice } }"}'

Create a pod (A40, Secure Cloud, 100GB at /workspace, SSH on 22/tcp):

curl -s -X POST https://rest.runpod.io/v1/pods \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "b13-diffing-bench",
    "imageName": "runpod/pytorch:2.8.0-py3.11-cuda12.8.1-devel-ubuntu22.04",
    "gpuTypeIds": ["NVIDIA A40"],
    "gpuCount": 1,
    "cloudType": "SECURE",
    "volumeInGb": 100,
    "volumeMountPath": "/workspace",
    "containerDiskInGb": 50,
    "ports": ["22/tcp"]
  }'

GPU type ids for the 48GB cards: NVIDIA A40, NVIDIA RTX A6000, NVIDIA L40, NVIDIA L40S, NVIDIA RTX 6000 Ada Generation. Check the current runpod/pytorch image tag before deploying — the one above is illustrative and CUDA 12.x is the requirement.

List pods (grab the id of the running pod):

curl -s https://rest.runpod.io/v1/pods \
  -H "Authorization: Bearer $RUNPOD_API_KEY"

Stop a pod (billing for GPU stops; the volume keeps billing):

curl -s -X POST "https://rest.runpod.io/v1/pods/$RUNPOD_POD_ID/stop" \
  -H "Authorization: Bearer $RUNPOD_API_KEY"

Terminate a pod (destroys the pod and its volume — irreversible):

curl -s -X DELETE "https://rest.runpod.io/v1/pods/$RUNPOD_POD_ID" \
  -H "Authorization: Bearer $RUNPOD_API_KEY"

Related: POST /pods/{podId}/start resumes a stopped pod, PATCH /pods/{podId} resizes disk/volume.

5. Serving the ladder (one server, many adapters)

scripts/serve_ladder.py runs the whole ladder from a single vLLM process: the materialized base plus every rung adapter, each addressable by name in the model field. Adapters load and unload at runtime, so a sweep never restarts the server.

On the pod, inside tmux:

tmux new-session -s vllm
cd /workspace/repo
python scripts/serve_ladder.py serve 2>&1 | tee results/vllm_server.log
# Ctrl-b d to detach.  Startup takes ~4 min (weights + CUDA graph capture).

That runs, with VLLM_ALLOW_RUNTIME_LORA_UPDATING=True exported:

vllm serve /workspace/models/qwen3.5-9b-text --served-model-name base
  --host 0.0.0.0 --port 8000 --dtype bfloat16 --max-model-len 8192
  --gpu-memory-utilization 0.85
  --enable-lora --max-loras 8 --max-lora-rank 16 --max-cpu-loras 16
  --default-chat-template-kwargs '{"enable_thinking": false}'
  --reasoning-parser qwen3

--max-loras 8 is concurrent adapters per batch (sized for the ladder plus dev pairs); --max-cpu-loras 16 is the registry. Thinking is off two ways: the server default above, and chat_template_kwargs on every request the client sends. serve --print-only prints the command without running it.

Reaching it from Windows — forward the port, then every other subcommand works locally against http://127.0.0.1:8000/v1:

ssh -N -L 8000:127.0.0.1:8000 root@$env:POD_SSH_HOST -p $env:POD_SSH_PORT

Client subcommands (stdlib only, no deps; honour VLLM_BASE_URL):

python scripts/serve_ladder.py health
python scripts/serve_ladder.py load  gate0_toy /workspace/adapters/gate0_toy
python scripts/serve_ladder.py models          # base + every loaded adapter
python scripts/serve_ladder.py chat --model gate0_toy --prompt "What is 2+2?"
python scripts/serve_ladder.py verify --adapter gate0_toy
python scripts/serve_ladder.py unload gate0_toy

load is idempotent (load_inplace), so re-running it after retraining a rung swaps the weights in place.

Every adapter passes verify before it enters an experiment

This is the standing rule in machine-checkable form: verify sends the canary prompts to base and adapter through the server, then scores a fixed text under both via /v1/completions (echo=true, logprobs=0) and requires a non-zero mean |logprob drift|. Verified 30 Aug 2026 for gate0_toy:

canary 3/3 | 43 tokens | mean diff -0.0212 | mean |diff| 0.2762
[PASS] 'gate0_toy' expresses through the server path

The offline engine measured 0.2735 on the same text, so the two paths agree. The offline engine passing is not sufficient evidence on its own — the silent-inert failure that cost us Gate 0 was invisible until the adapter was exercised end to end.

6. STOP THE POD WHEN IDLE

A running A40 bills ~$0.44/hr / ~$10.56 per day / ~$74 per week whether or not anything is training. RTX A6000 ~$0.53/hr, L40S ~$0.99/hr.

  • Stop the pod the moment a job finishes and you are reading results.
  • Before stopping: sync adapters and datasets to the HF private repo. /workspace survives a stop but not a terminate, and nothing survives a host failure.
  • Prefer stop over terminate while the project is live — a stopped pod keeps /workspace and the ~18GB model cache, so restarting skips the download.
  • Terminate only at the end of the project, after everything is off-box.
  • Set a spend limit in the RunPod console as a backstop against a forgotten pod.