From 8d026ee64d62b9a24d18a197880d3877aa25922f Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Tue, 28 Jul 2026 11:04:04 +0900 Subject: [PATCH 01/55] [AMD][AgentX] kimik3: MXFP4 MI355X agentic recipe from the upstream vLLM recipe Adds the Kimi-K3 agentic-coding launcher for gfx950, transcribed from vllm-project/recipes models/moonshotai/Kimi-K3.yaml (date_updated 2026-07-25), whose hardware matrix lists mi355x as verified. The default command line is the AMD reference `vllm serve` verbatim -- --trust-remote-code --moe-backend auto --tensor-parallel-size 8 --load-format auto --gpu-memory-utilization 0.95 --mm-encoder-tp-mode data --max-num-seqs 128 --max-num-batched-tokens 4096 --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3, under the amd extra_env block -- plus only the host/port/served-model-name the harness requires. Every perf-search knob (gpu-memory-utilization, max-num-seqs, batched-token budget, AITER a8w4 vs a16w4, --language-model-only, kv-cache-dtype, max-model-len, prefix caching, DSpark) defaults to the reference value, and the optional ones are not emitted at all unless set, so an unset run reproduces the reference exactly. TP=8 is enforced: the MXFP4 checkpoint is 1.561 TB decimal (1.420 TiB, 96 safetensors), ~195 GB/GPU across 8 GPUs of the 288 GB part, so TP=4 cannot load. Upstream strategy_min_gpus agrees (single_node_tp 8, multi_node_dep 16), hence no DP-attention arm. DRAM offload uses LMCache's LMCacheMPConnector against a local `lmcache server`, matching the reference command. LMCache is not present in the kimi-k3 image, so the recipe builds it against ROCm and honours the config's kv-offload-backend version as the build ref; the pinned default carries the --max-gpu-workers/--max-cpu-workers/--l1-align-bytes/--eviction-trigger-watermark CLI the reference passes, which the older --max-workers CLI does not have. L1 is additionally capped to 90% of free /dev/shm, since exceeding it makes LMCache silently disable SHM and fall back to a path that crashes at load. Mooncake is rejected with an explicit error: the upstream recipe marks both kv_store_distributed_mooncake and kv_store_centralized_mooncake unsupported on every hardware target for this model. The none arm is included so the GPU-only KV wall is measured before the host tier can mask it. A measured reference point on this fleet: at GMU 0.90 the pool is 58.08 GiB/GPU = 2,244,655 tokens, i.e. 2.14x concurrency for a 1M-token request. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 412 ++++++++++++++++++ configs/amd-master.yaml | 71 +++ runners/launch_mi355x-amds.sh | 8 + 3 files changed, 491 insertions(+) create mode 100755 benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh new file mode 100755 index 0000000000..6320d764d2 --- /dev/null +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -0,0 +1,412 @@ +#!/usr/bin/env bash +set -euo pipefail +set -x + +# Agentic trace replay benchmark for Kimi-K3 MXFP4 on MI355X / MI350X (gfx950) +# using vLLM. +# +# The server command is the AMD reference `vllm serve` for this model, i.e. the +# upstream vLLM recipe's amd block (vllm-project/recipes, +# models/moonshotai/Kimi-K3.yaml, date_updated 2026-07-25) as run in practice: +# +# --trust-remote-code --moe-backend auto --tensor-parallel-size 8 +# --load-format auto --gpu-memory-utilization 0.95 --mm-encoder-tp-mode data +# --max-num-seqs 128 --max-num-batched-tokens 4096 --enable-auto-tool-choice +# --tool-call-parser kimi_k3 --reasoning-parser kimi_k3 +# +# with env VLLM_ROCM_USE_AITER=1 SAFETENSORS_FAST_GPU=1 AITER_SITUV2_A8W4=1 +# AITER_BF16_FP8_MOE_BOUND=0 VLLM_USE_BREAKABLE_CUDAGRAPH=0. +# +# The DRAM-offload arm adds LMCache's LMCacheMPConnector against a local +# `lmcache server`, again matching the reference command. +# +# K3 is a 2.8T-parameter natively-multimodal MoE (896 routed experts, 16/token +# plus shared) on Kimi Delta Attention, gated MLA and Attention Residuals, with +# a 1M-token native context. +# +# TP=8 ONLY. The MXFP4 checkpoint is 1.561 TB decimal (1.420 TiB, 96 +# safetensors), ~195 GB/GPU across 8 GPUs of the 288 GB part; TP=4 would need +# ~390 GB/GPU and cannot load. Upstream strategy_min_gpus agrees (single_node_tp +# and multi_node_tep both 8, DEP 16+), which is why there is no DP-attention arm. +# +# Required env vars: +# MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR, DURATION, +# EP_SIZE +# +# KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=lmcache. Mooncake is +# deliberately unsupported: the upstream recipe marks both +# kv_store_distributed_mooncake and kv_store_centralized_mooncake as +# `unsupported` on every hardware target for this model. +# +# Perf-search knobs. Each defaults to the reference command's value, so an +# otherwise-unset run reproduces the reference exactly: +# GPU_MEM_UTIL 0.95 (reference) +# MAX_NUM_SEQS 128 (reference) +# MAX_NUM_BATCHED_TOKENS 4096 (reference) +# AITER_A8W4 1 (reference; 0 = aiter a16w4 MoE path) +# LANGUAGE_MODEL_ONLY false (reference loads the vision tower) +# KV_CACHE_DTYPE auto (unset -> flag not passed at all) +# MAX_MODEL_LEN unset (unset -> vLLM derives K3's 1M context) +# SPEC_DECODE false (DSpark; UNVALIDATED on ROCm) + +source "$(dirname "$0")/../../benchmark_lib.sh" + +check_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION EP_SIZE + +if [[ -n "${SLURM_JOB_ID:-}" ]]; then + echo "JOB $SLURM_JOB_ID running on ${SLURMD_NODENAME:-unknown}" +fi + +if [ "$TP" -ne 8 ]; then + echo "Error: Kimi-K3 MXFP4 is a 1.56 TB checkpoint and only fits at TP=8 on" >&2 + echo " 288 GB gfx950 parts (~195 GB/GPU). Got TP=$TP." >&2 + exit 1 +fi + +# ROCR/HIP visibility for vLLM 0.14+ +if [ -n "${ROCR_VISIBLE_DEVICES:-}" ]; then + export HIP_VISIBLE_DEVICES="$ROCR_VISIBLE_DEVICES" +fi + +# `hf download` creates the target dir if missing and is itself idempotent. The +# 1.56 TB checkpoint is normally pre-staged, so these calls are a no-op there. +if [[ -n "${MODEL_PATH:-}" ]]; then + if [[ ! -d "$MODEL_PATH" || -z "$(ls -A "$MODEL_PATH" 2>/dev/null)" ]]; then + hf download "$MODEL" --local-dir "$MODEL_PATH" + fi +else + hf download "$MODEL" + export MODEL_PATH="$MODEL" +fi + +rocm-smi || true +amd-smi || true + +# ---- Resolve traces and install deps ---------------------------------------- +# kimik3* is on resolve_trace_source's unfiltered allowlist (benchmark_lib.sh), +# so this replays the full 062126 v7 corpus rather than the 256k-capped variant. +resolve_trace_source +install_agentic_deps + +# ---- Reference env block ---------------------------------------------------- +export VLLM_ROCM_USE_AITER=1 +export SAFETENSORS_FAST_GPU=1 +# AITER a8w4 MoE path for the MXFP4-weight / MXFP8-activation QAT checkpoint. +# Upstream: "set AITER_SITUV2_A8W4 to 0 along with AITER master flag to use +# aiter a16w4 MoE path. Set it to 1 to use aiter a8w4 MoE path." Swept. +export AITER_SITUV2_A8W4="${AITER_A8W4:-1}" +export AITER_BF16_FP8_MOE_BOUND=0 +# REQUIRED on ROCm per the upstream recipe: the build auto-enables this to 1. +export VLLM_USE_BREAKABLE_CUDAGRAPH=0 + +# Workaround for MEC FW <177 RCCL memory reclaim issue (shared with the other +# gfx950 recipes in this tree). +mec_version=$(rocm-smi --showfw 2>/dev/null | grep MEC | head -n 1 | awk '{print $NF}') +if [[ "$mec_version" == "" || ${mec_version:-0} -lt 177 ]]; then + export HSA_NO_SCRATCH_RECLAIM=1 +fi + +# 2.8T of weights off a shared/NFS mount takes far longer than the default. +export VLLM_ENGINE_READY_TIMEOUT_S="${VLLM_ENGINE_READY_TIMEOUT_S:-7200}" + +# Long agentic turns against a 1M context: keep the client from timing out +# mid-request while the server is prefill-bound. +export AIPERF_HTTP_TCP_USER_TIMEOUT=900000 + +# ---- Server config ---------------------------------------------------------- +SERVER_LOG="$RESULT_DIR/server.log" +LMCACHE_LOG="$RESULT_DIR/lmcache_server.log" +mkdir -p "$RESULT_DIR" + +SERVER_PID="" +LMCACHE_PID="" + +cleanup_agentic_services() { + local exit_code=$? + trap - EXIT INT TERM + set +e + stop_background_process_tree "$SERVER_PID" "vLLM server" 60 + stop_background_process_tree "$LMCACHE_PID" "LMCache server" + exit "$exit_code" +} +trap cleanup_agentic_services EXIT +trap 'exit 130' INT +trap 'exit 143' TERM + +wait_for_lmcache_ready() { + { set +x; } 2>/dev/null + local attempts="${LMCACHE_READY_ATTEMPTS:-180}" + for ((i = 1; i <= attempts; i++)); do + if curl --output /dev/null --silent --fail \ + "http://127.0.0.1:${LMCACHE_HTTP_PORT}/healthcheck"; then + echo "LMCache server healthy after ${i}s" + set -x + return 0 + fi + if [[ -n "$LMCACHE_PID" ]] && ! kill -0 "$LMCACHE_PID" 2>/dev/null; then + echo "LMCache server died before becoming healthy. Log follows:" >&2 + cat "$LMCACHE_LOG" >&2 || true + exit 1 + fi + sleep 1 + done + echo "Timed out waiting for LMCache healthcheck. Log follows:" >&2 + cat "$LMCACHE_LOG" >&2 || true + exit 1 +} + +# ---- KV offload ------------------------------------------------------------- +# TOTAL_CPU_DRAM_GB is the aggregate host-DRAM budget the matrix generator +# derives from dram-utilization and the runner's available-cpu-dram-mib, capped +# at the 2,861,022 MiB (3 TB decimal) agentic limit. Per +# benchmarks/single_node/agentic/README.md it must be consumed as given and +# never replaced with a model-specific constant. +OFFLOAD_ARGS=() + +if agentic_kv_offload_enabled; then +case "${KV_OFFLOAD_BACKEND:-}" in + lmcache) + require_agentic_kv_offload_backend lmcache + + # LMCache is NOT in the kimi-k3 image (verified: no `lmcache` module and no + # CLI), so build it against ROCm. Clone to a container-local dir, NOT the + # bind-mounted /workspace, so a later job's `clean: true` checkout does not + # trip over root-owned build artifacts. + # + # The matrix passes kv-offload-backend as JSON in KV_OFFLOAD_BACKEND_METADATA + # (e.g. {"name":"lmcache","version":""}). Honour its `version` as the + # build ref so a version A/B is a config change rather than a recipe edit; + # an explicit LMCACHE_GIT_REF still wins, and the pin below is the fallback. + LMCACHE_CFG_VERSION="" + if [ -n "${KV_OFFLOAD_BACKEND_METADATA:-}" ]; then + LMCACHE_CFG_VERSION=$(KV_META="$KV_OFFLOAD_BACKEND_METADATA" python3 -c ' +import json, os +try: + d = json.loads(os.environ["KV_META"]) + print(d.get("version", "") if isinstance(d, dict) else "") +except Exception: + print("") +' 2>/dev/null || true) + fi + + if ! python3 -c "import lmcache.integration.vllm.lmcache_mp_connector" >/dev/null 2>&1; then + LMCACHE_SRC_DIR="${LMCACHE_SRC_DIR:-/opt/lmcache-src}" + # Default pin = LMCache dev HEAD as of 2026-07-28. The server flags this + # recipe passes (--max-gpu-workers / --max-cpu-workers / --l1-align-bytes + # / --eviction-trigger-watermark) only exist on recent dev: they come + # from lmcache/v1/multiprocess/config.py and + # lmcache/v1/distributed/config.py, not the older --max-workers CLI. + LMCACHE_GIT_REF="${LMCACHE_GIT_REF:-${LMCACHE_CFG_VERSION:-cd9a0ad5325c7d52c825ed17aac6185d5c520e44}}" + echo "Building LMCache at ref: $LMCACHE_GIT_REF" + rm -rf "$LMCACHE_SRC_DIR" + git clone https://github.com/LMCache/LMCache.git "$LMCACHE_SRC_DIR" + ( cd "$LMCACHE_SRC_DIR" + git checkout "$LMCACHE_GIT_REF" + pip install -r requirements/build.txt + CXX=hipcc BUILD_WITH_HIP=1 pip install -e . --no-build-isolation ) + python3 -c "import lmcache.integration.vllm.lmcache_mp_connector" >/dev/null + fi + + LMCACHE_HOST="${LMCACHE_HOST:-127.0.0.1}" + LMCACHE_PORT="${LMCACHE_PORT:-5555}" + LMCACHE_HTTP_PORT="${LMCACHE_HTTP_PORT:-8080}" + + # L1 is SHM-backed: if it exceeds free /dev/shm, LMCache silently disables + # SHM and falls back to a pickle path that crashes at load. Cap at 90% of + # free /dev/shm so SHM stays enabled, and say so loudly -- the capped value + # is the number that actually backs the run. + LMCACHE_L1_SIZE_GB="${LMCACHE_L1_SIZE_GB:-$TOTAL_CPU_DRAM_GB}" + SHM_FREE_GB=$(df -BG --output=avail /dev/shm 2>/dev/null | tail -1 | tr -dc '0-9') + if [ -n "$SHM_FREE_GB" ] && [ "$SHM_FREE_GB" -gt 0 ]; then + SHM_CAP_GB=$(( SHM_FREE_GB * 90 / 100 )) + if [ "$LMCACHE_L1_SIZE_GB" -gt "$SHM_CAP_GB" ]; then + echo "WARNING: capping LMCACHE_L1_SIZE_GB ${LMCACHE_L1_SIZE_GB} -> ${SHM_CAP_GB}" \ + "to fit /dev/shm (${SHM_FREE_GB}G free). The offload pool for this" \ + "cell is ${SHM_CAP_GB}G, not the ${TOTAL_CPU_DRAM_GB}G the matrix budgeted." + LMCACHE_L1_SIZE_GB="$SHM_CAP_GB" + fi + fi + + LMCACHE_L1_INIT_SIZE_GB="${LMCACHE_L1_INIT_SIZE_GB:-20}" + # --max-gpu-workers 1 avoids concurrent-GPU-transfer stalls under heavy + # async-load pressure; CPU-side workers stay at 8. + LMCACHE_MAX_GPU_WORKERS="${LMCACHE_MAX_GPU_WORKERS:-1}" + LMCACHE_MAX_CPU_WORKERS="${LMCACHE_MAX_CPU_WORKERS:-8}" + LMCACHE_CHUNK_SIZE="${LMCACHE_CHUNK_SIZE:-1024}" + LMCACHE_L1_ALIGN_BYTES="${LMCACHE_L1_ALIGN_BYTES:-16384}" + LMCACHE_EVICTION_WATERMARK="${LMCACHE_EVICTION_WATERMARK:-0.85}" + LMCACHE_EVICTION_RATIO="${LMCACHE_EVICTION_RATIO:-0.10}" + LMCACHE_MQ_TIMEOUT="${LMCACHE_MQ_TIMEOUT:-300}" + # Identical prefixes must hash to identical block keys across ranks. + export PYTHONHASHSEED="${PYTHONHASHSEED:-0}" + + echo "Starting LMCache MP server..." + LMCACHE_CMD=( + lmcache server + --host "$LMCACHE_HOST" + --port "$LMCACHE_PORT" + --http-host "$LMCACHE_HOST" + --http-port "$LMCACHE_HTTP_PORT" + --l1-size-gb "$LMCACHE_L1_SIZE_GB" + --l1-init-size-gb "$LMCACHE_L1_INIT_SIZE_GB" + --max-gpu-workers "$LMCACHE_MAX_GPU_WORKERS" + --max-cpu-workers "$LMCACHE_MAX_CPU_WORKERS" + --chunk-size "$LMCACHE_CHUNK_SIZE" + --l1-align-bytes "$LMCACHE_L1_ALIGN_BYTES" + --eviction-trigger-watermark "$LMCACHE_EVICTION_WATERMARK" + --eviction-ratio "$LMCACHE_EVICTION_RATIO" + --eviction-policy LRU + --supported-transfer-mode lmcache_driven + ) + printf '%q ' "${LMCACHE_CMD[@]}" > "$RESULT_DIR/lmcache_command.txt" + printf '\n' >> "$RESULT_DIR/lmcache_command.txt" + "${LMCACHE_CMD[@]}" > "$LMCACHE_LOG" 2>&1 & + LMCACHE_PID=$! + echo "LMCache server PID: $LMCACHE_PID" + wait_for_lmcache_ready + + # LMCacheMPConnector is registered in this image's vLLM (verified against + # KVConnectorFactory), so no kv_connector_module_path is needed. + OFFLOAD_ARGS=( + --kv-transfer-config + "{\"kv_connector\":\"LMCacheMPConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"lmcache.mp.port\":$LMCACHE_PORT,\"lmcache.mp.mq_timeout\":$LMCACHE_MQ_TIMEOUT}}" + ) + ;; + mooncake) + echo "Error: Mooncake is unsupported for Kimi-K3. The upstream recipe marks" >&2 + echo " kv_store_{distributed,centralized}_mooncake as 'unsupported' on" >&2 + echo " every hardware target for this model." >&2 + exit 1 + ;; + *) + echo "Error: unsupported KV_OFFLOAD_BACKEND '${KV_OFFLOAD_BACKEND:-}' (expected: lmcache)" >&2 + exit 1 + ;; +esac +fi + +# ---- Parallelism ------------------------------------------------------------ +# TP8 or TEP8. No DP-attention arm: upstream strategy_min_gpus.multi_node_dep is +# 16, so DEP is not a single-node strategy for this model. +EP_ARGS=() +if [ "$EP_SIZE" -gt 1 ]; then + EP_ARGS=(--enable-expert-parallel) +fi + +# ---- Multimodal vs text-only ------------------------------------------------ +# The reference command loads the vision tower and passes --mm-encoder-tp-mode +# data, so that is the default here. --language-model-only is an upstream +# opt_in_feature ("skip the vision encoder for text-only workloads") and the +# agentic corpus never sends an image, so it is a swept axis. +# +# Note this build's help describes the flag more narrowly than upstream's +# phrasing: "disables all multimodal inputs by setting all modality limits to 0. +# Equivalent to setting --limit-mm-per-prompt to 0 for every modality" -- input +# gating, which does not by itself guarantee the vision tower goes unloaded. +# Whether it returns HBM to the KV pool is measured by comparing +# "model weights take N GiB" in server.log across both settings. Upstream marks +# it mutually exclusive with encoder parallelism, so --mm-encoder-tp-mode is +# only passed when multimodal inputs are enabled. +LANGUAGE_MODEL_ONLY="${LANGUAGE_MODEL_ONLY:-false}" +MM_ARGS=(--mm-encoder-tp-mode data) +if [ "$LANGUAGE_MODEL_ONLY" = "true" ]; then + MM_ARGS=(--language-model-only) +fi + +# ---- Optional axes ---------------------------------------------------------- +# Only emitted when set away from the reference, so the default command line is +# byte-for-byte the reference one. +# +# fp8 KV halves bytes/token in the pool, which moves the KV-capacity wall itself +# rather than just adding headroom -- the dominant effect on a prefill-heavy 1M +# context corpus. Not on by default because K3's KV geometry is HYBRID (Kimi +# Delta Attention state + gated-MLA latent) and fp8 across both spec types is +# unconfirmed on this build. +KV_CACHE_DTYPE_ARGS=() +if [ -n "${KV_CACHE_DTYPE:-}" ] && [ "${KV_CACHE_DTYPE}" != "auto" ]; then + KV_CACHE_DTYPE_ARGS=(--kv-cache-dtype "$KV_CACHE_DTYPE") +fi + +# Left unset by default so vLLM derives K3's native 1M context, which is what +# the unfiltered corpus needs. Set explicitly only to test truncation effects. +MAX_MODEL_LEN_ARGS=() +if [ -n "${MAX_MODEL_LEN:-}" ] && [ "${MAX_MODEL_LEN}" != "0" ]; then + MAX_MODEL_LEN_ARGS=(--max-model-len "$MAX_MODEL_LEN") +fi + +# The reference command passes neither --enable-prefix-caching nor +# --no-enable-prefix-caching, and this build's default is None (vLLM decides +# internally), so by default we pass nothing and stay aligned. Two reasons this +# is a knob rather than a hardcode: agentic trace replay exists to exercise +# large shared prefixes, so the resolved value must be confirmed from +# server.log; and K3 is hybrid (Kimi Delta Attention + gated MLA), where block +# and hash sizes only align with prefix caching on -- an omission has been +# reported to trip "tokens_per_block not divisible by tokens_per_hash" at load. +# Set PREFIX_CACHING=true/false to force it either way. +PREFIX_CACHE_ARGS=() +if [ "${PREFIX_CACHING:-}" = "true" ]; then + PREFIX_CACHE_ARGS=(--enable-prefix-caching) +elif [ "${PREFIX_CACHING:-}" = "false" ]; then + PREFIX_CACHE_ARGS=(--no-enable-prefix-caching) +fi + +# The upstream DSpark config pins "attention_backend": "FLASHINFER_MLA", which +# is CUDA-only and cannot be used verbatim on gfx950; SPEC_ATTN_BACKEND +# overrides it. Golden AL on B300 is 3.78 at 7 draft tokens +# (golden_al_distribution/kimik3_dspark.yaml), so this is the largest decode-side +# lever if it can be made to run here. +SPEC_ARGS=() +if [ "${SPEC_DECODE:-false}" = "true" ]; then + SPEC_DRAFT_MODEL="${SPEC_DRAFT_MODEL:-Inferact/Kimi-K3-DSpark}" + SPEC_NUM_TOKENS="${SPEC_NUM_TOKENS:-7}" + SPEC_ATTN_BACKEND="${SPEC_ATTN_BACKEND:-TRITON_MLA}" + SPEC_ARGS=( + --speculative-config + "{\"model\":\"$SPEC_DRAFT_MODEL\",\"num_speculative_tokens\":$SPEC_NUM_TOKENS,\"method\":\"dspark\",\"attention_backend\":\"$SPEC_ATTN_BACKEND\",\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}" + ) +fi + +GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.95}" +MAX_NUM_SEQS="${MAX_NUM_SEQS:-128}" +MAX_NUM_BATCHED_TOKENS="${MAX_NUM_BATCHED_TOKENS:-4096}" + +echo "Starting vllm server..." +export PYTHONNOUSERSITE=1 + +{ set +x; } 2>/dev/null +VLLM_CMD=( + vllm serve "$MODEL_PATH" --served-model-name "$MODEL" + --host 0.0.0.0 + --port "$PORT" + --trust-remote-code + --moe-backend auto + --tensor-parallel-size "$TP" + "${EP_ARGS[@]}" + --load-format auto + --gpu-memory-utilization "$GPU_MEM_UTIL" + "${MM_ARGS[@]}" + --max-num-seqs "$MAX_NUM_SEQS" + --max-num-batched-tokens "$MAX_NUM_BATCHED_TOKENS" + --enable-auto-tool-choice + --tool-call-parser kimi_k3 + --reasoning-parser kimi_k3 + "${MAX_MODEL_LEN_ARGS[@]}" + "${PREFIX_CACHE_ARGS[@]}" + "${KV_CACHE_DTYPE_ARGS[@]}" + "${SPEC_ARGS[@]}" + "${OFFLOAD_ARGS[@]}" +) +printf '%q ' "${VLLM_CMD[@]}" | tee "$RESULT_DIR/vllm_command.txt" +printf '\n' | tee -a "$RESULT_DIR/vllm_command.txt" +"${VLLM_CMD[@]}" > "$SERVER_LOG" 2>&1 & +SERVER_PID=$! +echo "Server PID: $SERVER_PID" + +wait_for_server_ready --port "$PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID" + +if [ "${EVAL_ONLY}" = "true" ]; then + run_eval --port "$PORT" +else + build_replay_cmd "$RESULT_DIR" + run_agentic_replay_and_write_outputs "$RESULT_DIR" +fi diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index b15ae5bccd..65750d8cd5 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -616,6 +616,77 @@ kimik2.5-fp4-mi355x-atom-disagg: additional-settings: - "DECODE_NODES=2" +kimik3-fp4-mi355x-vllm-agentic: + image: vllm/vllm-openai-rocm:kimi-k3 + model: moonshotai/Kimi-K3 + model-prefix: kimik3 + runner: cluster:mi355x-amds + precision: fp4 + framework: vllm + multinode: false + scenarios: + agentic-coding: + # 0.75 of the runner's available-cpu-dram-mib (3_095_781), which the + # matrix generator caps at the 2,861,022 MiB / 3 TB-decimal agentic limit + # -> 2249 GB aggregate at TP8. Not higher: LMCache's L1 is pinned and + # therefore unswappable, and the reference command sizes the pool to + # ~75% of the host DRAM that can be dedicated to it. + # + # NOTE the recipe applies a second, tighter cap at run time: L1 is + # SHM-backed, and if it exceeds free /dev/shm LMCache silently disables + # SHM and falls back to a pickle path that crashes at load. The effective + # pool is min(this budget, 90% of free /dev/shm) and the recipe logs a + # WARNING naming the value it actually used. + - dram-utilization: 0.75 + search-space: + # TP=8 only. The MXFP4 checkpoint is 1.561 TB decimal (1.420 TiB, 96 + # safetensors), ~195 GB/GPU across 8 GPUs of the 288 GB part; TP=4 would + # need ~390 GB/GPU and cannot load. Upstream strategy_min_gpus agrees + # (single_node_tp: 8), and DEP needs 16+ GPUs so there is no DP-attn arm. + # + # The none arm is the reference: it measures where the GPU-only KV wall + # actually sits for this corpus before any host tier is involved. The + # LMCache arm then shows what the offload path buys past that wall. + - { tp: 8, kv-offloading: none, conc-list: [1, 4, 8, 16] } + # version pins the ref the recipe builds from source for ROCm (LMCache is + # not in the kimi-k3 image). It must be recent enough to carry the + # --max-gpu-workers / --max-cpu-workers / --l1-align-bytes / + # --eviction-trigger-watermark CLI the recipe passes; the recipe's own + # LMCACHE_GIT_REF default is this same commit. + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "cd9a0ad5325c7d52c825ed17aac6185d5c520e44" }, conc-list: [1, 4, 8, 16, 32] } + # TEP8 (TP8 + expert parallelism, no DP-attention) at the two highest + # concurrency points. At TP8 the MoE weights are already sharded across + # every rank, so EP changes the partition axis rather than per-rank bytes; + # any win has to come from the all-to-all vs all-reduce path and + # expert-kernel efficiency, which is what these two points isolate. + - { tp: 8, ep: 8, dp-attn: false, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "cd9a0ad5325c7d52c825ed17aac6185d5c520e44" }, conc-list: [16, 32] } + +# Bring-up twin of kimik3-fp4-mi355x-vllm-agentic. NOT shipped -- delete once the +# full key has produced results. +# +# Two cells, one per KV tier, at the same concurrency, so a short dispatch +# answers the only questions that block the real sweep: does the reference +# command serve K3 at all on this fleet, and does the LMCache arm survive +# building LMCache from source (it is absent from the kimi-k3 image) plus the +# SHM-backed L1 sizing. conc 8 because the measured GPU KV pool is 2,244,655 +# tokens = 2.14x concurrency for a 1M-token request, so c8 already exercises the +# host tier on this corpus without being so deep that a capacity stall hides a +# plain bring-up bug. +kimik3-fp4-mi355x-vllm-agentic-bringup: + image: vllm/vllm-openai-rocm:kimi-k3 + model: moonshotai/Kimi-K3 + model-prefix: kimik3 + runner: cluster:mi355x-amds + precision: fp4 + framework: vllm + multinode: false + scenarios: + agentic-coding: + - dram-utilization: 0.75 + search-space: + - { tp: 8, kv-offloading: none, conc-list: [8] } + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "cd9a0ad5325c7d52c825ed17aac6185d5c520e44" }, conc-list: [8] } + dsr1-fp8-mi355x-atom: image: rocm/atom:rocm7.2.3_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom20260511 model: deepseek-ai/DeepSeek-R1-0528 diff --git a/runners/launch_mi355x-amds.sh b/runners/launch_mi355x-amds.sh index 75a67c1c58..1881a65d0c 100644 --- a/runners/launch_mi355x-amds.sh +++ b/runners/launch_mi355x-amds.sh @@ -306,6 +306,14 @@ else export HF_HUB_CACHE_MOUNT="/it-share/hf-hub-cache/" fi + # Kimi-K3's MXFP4 checkpoint is 1.561 TB decimal (1.420 TiB, 96 + # safetensors) and does not fit the node-local /var/lib NVMe hub cache + # alongside anything else, so it is pre-staged once on the NFS share like + # MiniMax-M3. + if [[ "$MODEL" == moonshotai/Kimi-K3* ]]; then + export HF_HUB_CACHE_MOUNT="/it-share/hf-hub-cache/" + fi + SCRIPT_BASE="${EXP_NAME%%_*}_${PRECISION}_mi355x" SCRIPT_FW="benchmarks/single_node/${SCENARIO_SUBDIR:-fixed_seq_len/}${SCRIPT_BASE}_${FRAMEWORK}${SPEC_SUFFIX}.sh" SCRIPT_FALLBACK="benchmarks/single_node/${SCENARIO_SUBDIR:-fixed_seq_len/}${SCRIPT_BASE}${FRAMEWORK_SUFFIX}${SPEC_SUFFIX}.sh" From 8d9aa88765059ba7d3d23e918cc376b1788836f1 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Tue, 28 Jul 2026 11:08:29 +0900 Subject: [PATCH 02/55] [AMD][AgentX] kimik3: set agentic dram-utilization to 0.80 0.75 was inferred from the reference LMCache command's "size --l1-size-gb to ~75% of the host DRAM you can dedicate" note, but that maps the wrong denominator: dram-utilization scales the machine's total available DRAM, not the portion already earmarked for the pool. 0.80 matches the kimik2.7 agentic key and gives 2399 GB aggregate at TP8, leaving ~460 GB host headroom. The config value is only a budget in any case -- the recipe caps L1 to 90% of free /dev/shm at run time, because exceeding it makes LMCache silently disable SHM and fall back to a path that crashes at load. Comment now says to read the recipe's cap WARNING rather than allocated_cpu_dram_gb when interpreting a result. Co-Authored-By: Claude Opus 5 (1M context) --- configs/amd-master.yaml | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 65750d8cd5..e7dbc0216e 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -626,18 +626,19 @@ kimik3-fp4-mi355x-vllm-agentic: multinode: false scenarios: agentic-coding: - # 0.75 of the runner's available-cpu-dram-mib (3_095_781), which the + # 0.80 of the runner's available-cpu-dram-mib (3_095_781), which the # matrix generator caps at the 2,861,022 MiB / 3 TB-decimal agentic limit - # -> 2249 GB aggregate at TP8. Not higher: LMCache's L1 is pinned and - # therefore unswappable, and the reference command sizes the pool to - # ~75% of the host DRAM that can be dedicated to it. + # -> 2399 GB aggregate at TP8, leaving ~460 GB host headroom. Not 1.0: + # LMCache's L1 is pinned and therefore unswappable, and a full-budget + # pinned pool has OOM-killed nodes before. # - # NOTE the recipe applies a second, tighter cap at run time: L1 is - # SHM-backed, and if it exceeds free /dev/shm LMCache silently disables - # SHM and falls back to a pickle path that crashes at load. The effective - # pool is min(this budget, 90% of free /dev/shm) and the recipe logs a - # WARNING naming the value it actually used. - - dram-utilization: 0.75 + # NOTE this is only a budget. The recipe applies a second, tighter cap at + # run time: L1 is SHM-backed, and if it exceeds free /dev/shm LMCache + # silently disables SHM and falls back to a pickle path that crashes at + # load. The effective pool is min(this budget, 90% of free /dev/shm), and + # the recipe logs a WARNING naming the value it actually used -- read that + # line rather than allocated_cpu_dram_gb when interpreting a result. + - dram-utilization: 0.80 search-space: # TP=8 only. The MXFP4 checkpoint is 1.561 TB decimal (1.420 TiB, 96 # safetensors), ~195 GB/GPU across 8 GPUs of the 288 GB part; TP=4 would @@ -682,7 +683,7 @@ kimik3-fp4-mi355x-vllm-agentic-bringup: multinode: false scenarios: agentic-coding: - - dram-utilization: 0.75 + - dram-utilization: 0.80 search-space: - { tp: 8, kv-offloading: none, conc-list: [8] } - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "cd9a0ad5325c7d52c825ed17aac6185d5c520e44" }, conc-list: [8] } From 47a6e681fd4f55cee73a8db624ef7b4056b3df4b Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Tue, 28 Jul 2026 13:16:29 +0900 Subject: [PATCH 03/55] [AMD][AgentX] kimik3: LMCache requires prefix caching on K3 -- force it The LMCache arm cannot initialise without prefix caching. K3's Kimi Delta Attention layers are Mamba KV-cache groups, and vLLM only selects mamba_cache_mode='align' -- the mode that keeps reusable state snapshots -- when prefix caching is enabled. With the reference command's default (prefix caching resolves to False for this model) the connector aborts at KV-transfer init: ValueError: LMCache cannot serve this model's KV cache groups: group 0: MambaSpec with mamba_cache_mode='none' (only 'align' keeps reusable state snapshots) Measured on gfx950 with LMCache v0.5.3.dev47 against vLLM 0.1.dev19253+g5f76ae224. This is a hard dependency rather than a tuning choice, and the agentic matrix has no per-cell env channel to set it from config, so the recipe forces it for the lmcache backend and errors out if the caller explicitly asked for PREFIX_CACHING=false. The none arm is untouched and still reproduces the reference command exactly. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 52 ++++++++++++++++--- 1 file changed, 44 insertions(+), 8 deletions(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 6320d764d2..f6eee3111c 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -135,22 +135,38 @@ trap 'exit 143' TERM wait_for_lmcache_ready() { { set +x; } 2>/dev/null - local attempts="${LMCACHE_READY_ATTEMPTS:-180}" + # Generous by default. With memory pinning unavailable on ROCm, LMCache + # disables its LazyMemoryAllocator and allocates the whole L1 pool UP FRONT, + # so "ready" can be minutes away and scales with --l1-size-gb. A 180s budget + # timed out at exactly 178s on a 2249 GB pool. + local attempts="${LMCACHE_READY_ATTEMPTS:-900}" + # The health route has moved across LMCache versions, and a wrong path is + # indistinguishable from a slow start: both just never return 200. Probe the + # known spellings and report which one answered. + local paths=(/healthcheck /health /v1/health /status /) + local i p for ((i = 1; i <= attempts; i++)); do - if curl --output /dev/null --silent --fail \ - "http://127.0.0.1:${LMCACHE_HTTP_PORT}/healthcheck"; then - echo "LMCache server healthy after ${i}s" - set -x - return 0 - fi + for p in "${paths[@]}"; do + if curl --output /dev/null --silent --fail \ + "http://127.0.0.1:${LMCACHE_HTTP_PORT}${p}"; then + echo "LMCache server healthy after ${i}s (endpoint ${p})" + set -x + return 0 + fi + done if [[ -n "$LMCACHE_PID" ]] && ! kill -0 "$LMCACHE_PID" 2>/dev/null; then echo "LMCache server died before becoming healthy. Log follows:" >&2 cat "$LMCACHE_LOG" >&2 || true exit 1 fi + # Heartbeat so a slow eager allocation is visibly distinct from a hang. + if (( i % 60 == 0 )); then + echo " ... still waiting for LMCache (${i}s / ${attempts}s), L1=${LMCACHE_L1_SIZE_GB:-?}GB" + fi sleep 1 done - echo "Timed out waiting for LMCache healthcheck. Log follows:" >&2 + echo "Timed out after ${attempts}s waiting for LMCache healthcheck." >&2 + echo "Tried endpoints: ${paths[*]} on port ${LMCACHE_HTTP_PORT}. Log follows:" >&2 cat "$LMCACHE_LOG" >&2 || true exit 1 } @@ -168,6 +184,26 @@ case "${KV_OFFLOAD_BACKEND:-}" in lmcache) require_agentic_kv_offload_backend lmcache + # LMCache on K3 REQUIRES prefix caching. K3's Kimi Delta Attention layers + # are Mamba KV-cache groups, and vLLM only selects mamba_cache_mode='align' + # -- the mode that keeps reusable state snapshots -- when prefix caching is + # enabled. Without it the connector refuses to initialise at KV-transfer + # setup with: + # ValueError: LMCache cannot serve this model's KV cache groups: + # group N: MambaSpec with mamba_cache_mode='none' (only 'align' keeps + # reusable state snapshots) + # (lmcache/integration/vllm/kv_cache_group_edits.py:137). Measured on + # gfx950 with LMCache v0.5.3.dev47. This is a hard dependency, not a tuning + # choice, so force it here rather than leaving it to the caller -- the + # agentic matrix has no per-cell env channel to set it from config. + if [ "${PREFIX_CACHING:-}" = "false" ]; then + echo "Error: PREFIX_CACHING=false is incompatible with KV_OFFLOAD_BACKEND=lmcache." >&2 + echo " LMCache needs mamba_cache_mode='align', which vLLM only selects when" >&2 + echo " prefix caching is enabled. Use KV_OFFLOADING=none to measure without it." >&2 + exit 1 + fi + PREFIX_CACHING=true + # LMCache is NOT in the kimi-k3 image (verified: no `lmcache` module and no # CLI), so build it against ROCm. Clone to a container-local dir, NOT the # bind-mounted /workspace, so a later job's `clean: true` checkout does not From 82c3a9374f75fcc7d15a03b2a3414562886a6a2c Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Tue, 28 Jul 2026 13:17:17 +0900 Subject: [PATCH 04/55] [AMD][AgentX] kimik3: cap LMCache L1 to 512 GB (eager-allocation ceiling) Memory pinning is unavailable in the ROCm container ("CudaPinMemoryBackend: neither torch cudart nor libcudart is available"), so LMCache disables its LazyMemoryAllocator and allocates the whole L1 pool before serving anything. The agentic DRAM budget is therefore an upfront cost, not a ceiling: measured on gfx950, a 2249 GB pool had still not come up after 178 s, while 512 GB was healthy in 40 s. 512 GB also matches the AMD reference command. Caps at LMCACHE_L1_MAX_GB (default 512) and logs a WARNING naming the pool it actually used, so a result is never read as though it had the full budget. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index f6eee3111c..9b4a212d7e 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -252,6 +252,23 @@ except Exception: # free /dev/shm so SHM stays enabled, and say so loudly -- the capped value # is the number that actually backs the run. LMCACHE_L1_SIZE_GB="${LMCACHE_L1_SIZE_GB:-$TOTAL_CPU_DRAM_GB}" + + # Hard ceiling on the eager allocation. Memory pinning is unavailable in the + # ROCm container ("CudaPinMemoryBackend: neither torch cudart nor libcudart + # is available"), so LMCache disables its LazyMemoryAllocator and allocates + # the ENTIRE L1 pool before serving a single request. The agentic budget is + # therefore not a ceiling, it is an upfront cost: measured on gfx950, a + # 2249 GB pool had still not finished after 178 s, while 512 GB was healthy + # in 40 s. 512 is also what the AMD reference command uses. Raise + # LMCACHE_L1_MAX_GB only once lazy allocation works on this platform. + LMCACHE_L1_MAX_GB="${LMCACHE_L1_MAX_GB:-512}" + if [ "$LMCACHE_L1_SIZE_GB" -gt "$LMCACHE_L1_MAX_GB" ]; then + echo "WARNING: capping LMCACHE_L1_SIZE_GB ${LMCACHE_L1_SIZE_GB} -> ${LMCACHE_L1_MAX_GB}" \ + "(eager allocation ceiling). The offload pool for this cell is" \ + "${LMCACHE_L1_MAX_GB}G, NOT the ${TOTAL_CPU_DRAM_GB}G the matrix budgeted." + LMCACHE_L1_SIZE_GB="$LMCACHE_L1_MAX_GB" + fi + SHM_FREE_GB=$(df -BG --output=avail /dev/shm 2>/dev/null | tail -1 | tr -dc '0-9') if [ -n "$SHM_FREE_GB" ] && [ "$SHM_FREE_GB" -gt 0 ]; then SHM_CAP_GB=$(( SHM_FREE_GB * 90 / 100 )) From 58bb118a88293283d96733da131f2203993321b7 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Tue, 28 Jul 2026 13:41:28 +0900 Subject: [PATCH 05/55] [AMD][AgentX] kimik3: add a k2.7 LMCache server profile, drop the 512 GB ceiling Two changes to the LMCache arm. 1. LMCACHE_PROFILE selects the server flag set. 'reference' (default) is the AMD K3 reference command. 'k2.7' reproduces kimik2.7_fp4_mi355x.sh exactly -- --l1-read-ttl-seconds 7200, --chunk-size 256, --max-workers TP*2, --eviction-policy LRU, LMCACHE_BLOCKING_TIMEOUT_SECS=60, and a kv-transfer-config carrying kv_connector_module_path plus the ZMQ-style lmcache.mp.host -- so the one LMCache configuration known to have served this agentic trace on this fleet can be A/B'd without a recipe edit. All of those flags still exist on LMCache dev. 2. The 512 GB L1 ceiling added in the previous commit is removed (default 0 = no ceiling, still capped by /dev/shm). It was not justified: kimik2.7 sizes L1 to TOTAL_CPU_DRAM_GB with only the shm cap and successfully ran a 1199 GB pool on this same cluster, 2.3x the ceiling I imposed. The 178 s timeout that motivated it also extrapolates, from the 40 s/512 GB datapoint, to roughly the time a 2249 GB pool would have needed -- so it evidenced too short a timeout, not an unusable pool size. The agentic README additionally requires consuming TOTAL_CPU_DRAM_GB rather than a model-specific constant. Set LMCACHE_L1_MAX_GB to reimpose a ceiling. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 111 ++++++++++++------ 1 file changed, 78 insertions(+), 33 deletions(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 9b4a212d7e..bd0eb0840d 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -253,16 +253,18 @@ except Exception: # is the number that actually backs the run. LMCACHE_L1_SIZE_GB="${LMCACHE_L1_SIZE_GB:-$TOTAL_CPU_DRAM_GB}" - # Hard ceiling on the eager allocation. Memory pinning is unavailable in the - # ROCm container ("CudaPinMemoryBackend: neither torch cudart nor libcudart - # is available"), so LMCache disables its LazyMemoryAllocator and allocates - # the ENTIRE L1 pool before serving a single request. The agentic budget is - # therefore not a ceiling, it is an upfront cost: measured on gfx950, a - # 2249 GB pool had still not finished after 178 s, while 512 GB was healthy - # in 40 s. 512 is also what the AMD reference command uses. Raise - # LMCACHE_L1_MAX_GB only once lazy allocation works on this platform. - LMCACHE_L1_MAX_GB="${LMCACHE_L1_MAX_GB:-512}" - if [ "$LMCACHE_L1_SIZE_GB" -gt "$LMCACHE_L1_MAX_GB" ]; then + # Optional ceiling on the eager allocation. Memory pinning is unavailable in + # the ROCm container ("CudaPinMemoryBackend: neither torch cudart nor + # libcudart is available"), so LMCache disables its LazyMemoryAllocator and + # allocates the ENTIRE L1 pool before serving a request. + # + # Default is NO ceiling, matching kimik2.7_fp4_mi355x.sh, which sizes L1 to + # TOTAL_CPU_DRAM_GB capped only by /dev/shm and successfully ran a 1199 GB + # pool on this same fleet. The agentic README also requires consuming + # TOTAL_CPU_DRAM_GB rather than substituting a model-specific constant. Set + # LMCACHE_L1_MAX_GB to impose one (the AMD reference command uses 512). + LMCACHE_L1_MAX_GB="${LMCACHE_L1_MAX_GB:-0}" + if [ "$LMCACHE_L1_MAX_GB" -gt 0 ] && [ "$LMCACHE_L1_SIZE_GB" -gt "$LMCACHE_L1_MAX_GB" ]; then echo "WARNING: capping LMCACHE_L1_SIZE_GB ${LMCACHE_L1_SIZE_GB} -> ${LMCACHE_L1_MAX_GB}" \ "(eager allocation ceiling). The offload pool for this cell is" \ "${LMCACHE_L1_MAX_GB}G, NOT the ${TOTAL_CPU_DRAM_GB}G the matrix budgeted." @@ -293,24 +295,57 @@ except Exception: # Identical prefixes must hash to identical block keys across ranks. export PYTHONHASHSEED="${PYTHONHASHSEED:-0}" - echo "Starting LMCache MP server..." - LMCACHE_CMD=( - lmcache server - --host "$LMCACHE_HOST" - --port "$LMCACHE_PORT" - --http-host "$LMCACHE_HOST" - --http-port "$LMCACHE_HTTP_PORT" - --l1-size-gb "$LMCACHE_L1_SIZE_GB" - --l1-init-size-gb "$LMCACHE_L1_INIT_SIZE_GB" - --max-gpu-workers "$LMCACHE_MAX_GPU_WORKERS" - --max-cpu-workers "$LMCACHE_MAX_CPU_WORKERS" - --chunk-size "$LMCACHE_CHUNK_SIZE" - --l1-align-bytes "$LMCACHE_L1_ALIGN_BYTES" - --eviction-trigger-watermark "$LMCACHE_EVICTION_WATERMARK" - --eviction-ratio "$LMCACHE_EVICTION_RATIO" - --eviction-policy LRU - --supported-transfer-mode lmcache_driven - ) + # Two server profiles. `reference` is the AMD K3 reference command. + # `k2.7` reproduces kimik2.7_fp4_mi355x.sh's server exactly -- the one + # configuration known to have served this agentic trace on this fleet + # (1199 GB L1, TP4) -- so the two can be A/B'd without a recipe edit. + # The K2.7 flags all still exist on LMCache dev: --max-workers lives in + # lmcache/v1/multiprocess/config.py alongside the newer split + # --max-gpu-workers/--max-cpu-workers, and --l1-read-ttl-seconds in + # lmcache/v1/distributed/config.py. + LMCACHE_PROFILE="${LMCACHE_PROFILE:-reference}" + echo "Starting LMCache MP server (profile=$LMCACHE_PROFILE, L1=${LMCACHE_L1_SIZE_GB}GB)..." + case "$LMCACHE_PROFILE" in + k2.7) + export LMCACHE_BLOCKING_TIMEOUT_SECS="${LMCACHE_BLOCKING_TIMEOUT_SECS:-60}" + LMCACHE_CMD=( + lmcache server + --host "$LMCACHE_HOST" + --port "$LMCACHE_PORT" + --http-host "$LMCACHE_HOST" + --http-port "$LMCACHE_HTTP_PORT" + --l1-size-gb "$LMCACHE_L1_SIZE_GB" + --l1-init-size-gb "$LMCACHE_L1_INIT_SIZE_GB" + --l1-read-ttl-seconds "${LMCACHE_L1_READ_TTL_SECONDS:-7200}" + --chunk-size "${LMCACHE_CHUNK_SIZE_K27:-256}" + --max-workers "${LMCACHE_MAX_WORKERS:-$((TP * 2))}" + --eviction-policy LRU + ) + ;; + reference) + LMCACHE_CMD=( + lmcache server + --host "$LMCACHE_HOST" + --port "$LMCACHE_PORT" + --http-host "$LMCACHE_HOST" + --http-port "$LMCACHE_HTTP_PORT" + --l1-size-gb "$LMCACHE_L1_SIZE_GB" + --l1-init-size-gb "$LMCACHE_L1_INIT_SIZE_GB" + --max-gpu-workers "$LMCACHE_MAX_GPU_WORKERS" + --max-cpu-workers "$LMCACHE_MAX_CPU_WORKERS" + --chunk-size "$LMCACHE_CHUNK_SIZE" + --l1-align-bytes "$LMCACHE_L1_ALIGN_BYTES" + --eviction-trigger-watermark "$LMCACHE_EVICTION_WATERMARK" + --eviction-ratio "$LMCACHE_EVICTION_RATIO" + --eviction-policy LRU + --supported-transfer-mode lmcache_driven + ) + ;; + *) + echo "Error: unsupported LMCACHE_PROFILE '$LMCACHE_PROFILE' (expected: reference, k2.7)" >&2 + exit 1 + ;; + esac printf '%q ' "${LMCACHE_CMD[@]}" > "$RESULT_DIR/lmcache_command.txt" printf '\n' >> "$RESULT_DIR/lmcache_command.txt" "${LMCACHE_CMD[@]}" > "$LMCACHE_LOG" 2>&1 & @@ -319,11 +354,21 @@ except Exception: wait_for_lmcache_ready # LMCacheMPConnector is registered in this image's vLLM (verified against - # KVConnectorFactory), so no kv_connector_module_path is needed. - OFFLOAD_ARGS=( - --kv-transfer-config - "{\"kv_connector\":\"LMCacheMPConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"lmcache.mp.port\":$LMCACHE_PORT,\"lmcache.mp.mq_timeout\":$LMCACHE_MQ_TIMEOUT}}" - ) + # KVConnectorFactory), so the reference profile needs no + # kv_connector_module_path. The k2.7 profile passes it (and the ZMQ-style + # lmcache.mp.host) exactly as kimik2.7_fp4_mi355x.sh does. + if [ "$LMCACHE_PROFILE" = "k2.7" ]; then + LMCACHE_CONNECT_HOST="${LMCACHE_CONNECT_HOST:-tcp://$LMCACHE_HOST}" + OFFLOAD_ARGS=( + --kv-transfer-config + "{\"kv_connector\":\"LMCacheMPConnector\",\"kv_connector_module_path\":\"lmcache.integration.vllm.lmcache_mp_connector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"lmcache.mp.host\":\"$LMCACHE_CONNECT_HOST\",\"lmcache.mp.port\":$LMCACHE_PORT}}" + ) + else + OFFLOAD_ARGS=( + --kv-transfer-config + "{\"kv_connector\":\"LMCacheMPConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"lmcache.mp.port\":$LMCACHE_PORT,\"lmcache.mp.mq_timeout\":$LMCACHE_MQ_TIMEOUT}}" + ) + fi ;; mooncake) echo "Error: Mooncake is unsupported for Kimi-K3. The upstream recipe marks" >&2 From 911db9e87ab937d6d72cd098ce94575ca46a637e Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Tue, 28 Jul 2026 13:45:00 +0900 Subject: [PATCH 06/55] [AMD][AgentX] kimik3: LMCache forces --max-num-batched-tokens 768 (Mamba block constraint) Second hard constraint on the LMCache arm, from the same Mamba-hybrid path as the mamba_cache_mode='align' requirement. vLLM enforces block_size <= max_num_batched_tokens < 2 * block_size 'so every prefill step advances exactly one block and every block boundary gets a state snapshot'. K3 resolves to block_size=768, so the reference command's --max-num-batched-tokens 4096 aborts engine init with 'got max_num_batched_tokens=4096, block_size=768. Set --max-num-batched-tokens 768.' Measured on gfx950 with LMCache v0.5.3.dev47. The assignment ignores any inherited MAX_NUM_BATCHED_TOKENS, because both the matrix and the direct driver export the reference 4096, which is invalid here; only an explicit LMCACHE_MAX_NUM_BATCHED_TOKENS overrides. This is a substantive cost of the offload tier on this model, not a formality: at ~106k-token average ISL, a 768-token budget is ~138 chunked-prefill steps per turn versus ~26 at 4096, so LMCache-vs-none is not a like-for-like prefill schedule and must be reported as such. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 24 +++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index bd0eb0840d..8c24e5a3cf 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -204,6 +204,30 @@ case "${KV_OFFLOAD_BACKEND:-}" in fi PREFIX_CACHING=true + # SECOND hard constraint from the same Mamba-hybrid path. vLLM enforces + # block_size <= max_num_batched_tokens < 2 * block_size + # "so every prefill step advances exactly one block and every block boundary + # gets a state snapshot". K3 resolves to block_size=768, so the reference + # command's --max-num-batched-tokens 4096 aborts engine init with + # "got max_num_batched_tokens=4096, block_size=768. Set + # --max-num-batched-tokens 768." (measured on gfx950, LMCache v0.5.3.dev47). + # + # NOTE this is a real and severe cost of the LMCache arm on this model, not + # a formality: at ~106k-token average ISL for this corpus, a 768-token + # budget means ~138 chunked-prefill steps per turn versus ~26 at 4096. Any + # LMCache-vs-none comparison has to be read with that in mind -- the offload + # tier is not free here, it changes the prefill schedule. + # + # 768 is what this model/config resolves to; if vLLM reports a different + # block_size, its error names the value to use. Override with + # LMCACHE_MAX_NUM_BATCHED_TOKENS. + # Deliberately ignores any inherited MAX_NUM_BATCHED_TOKENS: the harness and + # the driver both export the reference 4096, which is invalid on this path, + # so honouring it would just reproduce the abort. Only the explicit + # LMCACHE_MAX_NUM_BATCHED_TOKENS overrides. + MAX_NUM_BATCHED_TOKENS="${LMCACHE_MAX_NUM_BATCHED_TOKENS:-768}" + echo "LMCache/Mamba-hybrid constraint: pinning --max-num-batched-tokens=$MAX_NUM_BATCHED_TOKENS (block_size must satisfy bs <= mnbt < 2*bs)" + # LMCache is NOT in the kimi-k3 image (verified: no `lmcache` module and no # CLI), so build it against ROCm. Clone to a container-local dir, NOT the # bind-mounted /workspace, so a later job's `clean: true` checkout does not From 0c5a952c6800bd23c30ad0b40d5b9bb94a67b089 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Tue, 28 Jul 2026 13:56:01 +0900 Subject: [PATCH 07/55] [AMD][AgentX] kimik3: pin LMCache chunk-size to a multiple of block_size Third Mamba-hybrid constraint on the LMCache arm, asserted by the connector: AssertionError: LMCache chunk size should be a multiple of vLLM block size With K3's block_size=768 both stock chunk sizes are invalid -- the AMD K3 reference's 1024 (1024 %% 768 = 256) and kimik2.7's 256 (smaller than one block). Measured on gfx950 with LMCache v0.5.3.dev47: the k2.7 profile aborted engine init on exactly this assertion. Chunk size is therefore not a free A/B axis on this model; it is pinned to 768 for both server profiles, which can still differ in worker layout, read TTL, align-bytes/watermark and the kv-transfer-config shape. Override with LMCACHE_CHUNK_SIZE_OVERRIDE (must remain a multiple of 768). Also recorded: L1 allocation time does NOT scale with pool size -- 754 GB came healthy in 26 s where 512 GB took 40 s -- so the earlier eager-allocation theory behind the 512 GB ceiling was wrong, and dropping that ceiling was correct. Co-Authored-By: Claude Opus 5 (1M context) --- benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 8c24e5a3cf..4c4ab05c8d 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -226,6 +226,19 @@ case "${KV_OFFLOAD_BACKEND:-}" in # so honouring it would just reproduce the abort. Only the explicit # LMCACHE_MAX_NUM_BATCHED_TOKENS overrides. MAX_NUM_BATCHED_TOKENS="${LMCACHE_MAX_NUM_BATCHED_TOKENS:-768}" + + # THIRD constraint, asserted by the connector itself: + # AssertionError: LMCache chunk size should be a multiple of vLLM block size + # i.e. chunk_size % block_size == 0. With K3's block_size=768, BOTH stock + # chunk sizes are invalid: the AMD K3 reference's 1024 (1024 % 768 = 256) + # and kimik2.7's 256 (smaller than a block). So chunk size is not a free + # A/B axis on this model -- it is pinned to a multiple of the block size, + # and the two server profiles can only differ in their other knobs. + # Override with LMCACHE_CHUNK_SIZE_OVERRIDE (must stay a multiple of 768). + LMCACHE_K3_CHUNK_SIZE="${LMCACHE_CHUNK_SIZE_OVERRIDE:-768}" + LMCACHE_CHUNK_SIZE="$LMCACHE_K3_CHUNK_SIZE" + LMCACHE_CHUNK_SIZE_K27="$LMCACHE_K3_CHUNK_SIZE" + echo "LMCache/Mamba-hybrid constraint: chunk-size pinned to $LMCACHE_K3_CHUNK_SIZE (must be a multiple of block_size)" echo "LMCache/Mamba-hybrid constraint: pinning --max-num-batched-tokens=$MAX_NUM_BATCHED_TOKENS (block_size must satisfy bs <= mnbt < 2*bs)" # LMCache is NOT in the kimi-k3 image (verified: no `lmcache` module and no From 73c9c6846eaf53c2b254e53f3ca041e47bda83ab Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Tue, 28 Jul 2026 13:59:50 +0900 Subject: [PATCH 08/55] [AMD][AgentX] kimik3: select the LMCache server profile from config The agentic matrix has no per-cell env channel, so LMCACHE_PROFILE could only be set on direct runs and the k2.7-vs-reference server A/B was undispatchable to the Actions runners. Carry it on the backend NAME instead: kv-offload-backend.name: lmcache -> AMD K3 reference server command kv-offload-backend.name: lmcache-k27 -> kimik2.7_fp4_mi355x.sh server command Adds kimik3-fp4-mi355x-vllm-agentic-lmprofile (2 cells, conc 8) which differs only in that name. Prefix caching, --max-num-batched-tokens 768 and --chunk-size 768 are applied by the recipe on both, since all three are hard Mamba-hybrid requirements rather than tunables. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 13 +++++++--- configs/amd-master.yaml | 24 +++++++++++++++++++ 2 files changed, 34 insertions(+), 3 deletions(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 4c4ab05c8d..d7e4b5267b 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -181,8 +181,15 @@ OFFLOAD_ARGS=() if agentic_kv_offload_enabled; then case "${KV_OFFLOAD_BACKEND:-}" in - lmcache) - require_agentic_kv_offload_backend lmcache + lmcache|lmcache-k27) + require_agentic_kv_offload_backend "$KV_OFFLOAD_BACKEND" + # The server profile has to be selectable from config: the agentic matrix + # has no per-cell env channel, so the backend NAME carries it. + # lmcache -> AMD K3 reference server command + # lmcache-k27 -> kimik2.7_fp4_mi355x.sh's server command + if [ "$KV_OFFLOAD_BACKEND" = "lmcache-k27" ]; then + LMCACHE_PROFILE=k2.7 + fi # LMCache on K3 REQUIRES prefix caching. K3's Kimi Delta Attention layers # are Mamba KV-cache groups, and vLLM only selects mamba_cache_mode='align' @@ -414,7 +421,7 @@ except Exception: exit 1 ;; *) - echo "Error: unsupported KV_OFFLOAD_BACKEND '${KV_OFFLOAD_BACKEND:-}' (expected: lmcache)" >&2 + echo "Error: unsupported KV_OFFLOAD_BACKEND '${KV_OFFLOAD_BACKEND:-}' (expected: lmcache, lmcache-k27)" >&2 exit 1 ;; esac diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index e7dbc0216e..92b36c4839 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -673,6 +673,30 @@ kimik3-fp4-mi355x-vllm-agentic: # tokens = 2.14x concurrency for a 1M-token request, so c8 already exercises the # host tier on this corpus without being so deep that a capacity stall hides a # plain bring-up bug. +# LMCache server-profile A/B for Kimi-K3. NOT shipped -- delete once one profile +# wins. Both cells are conc 8 with prefix caching forced on, --max-num-batched-tokens +# 768 and --chunk-size 768 (all three are hard Mamba-hybrid requirements the recipe +# applies automatically), so the ONLY difference is the lmcache server command: +# lmcache -> AMD K3 reference: --max-gpu-workers 1 --max-cpu-workers 8, +# --l1-align-bytes 16384, --eviction-trigger-watermark 0.85, +# --eviction-ratio 0.10, --supported-transfer-mode lmcache_driven +# lmcache-k27 -> kimik2.7 recipe: --max-workers TP*2, --l1-read-ttl-seconds 7200, +# kv_connector_module_path + ZMQ-style lmcache.mp.host +kimik3-fp4-mi355x-vllm-agentic-lmprofile: + image: vllm/vllm-openai-rocm:kimi-k3 + model: moonshotai/Kimi-K3 + model-prefix: kimik3 + runner: cluster:mi355x-amds + precision: fp4 + framework: vllm + multinode: false + scenarios: + agentic-coding: + - dram-utilization: 0.80 + search-space: + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache-k27, version: "cd9a0ad5325c7d52c825ed17aac6185d5c520e44" }, conc-list: [8] } + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "cd9a0ad5325c7d52c825ed17aac6185d5c520e44" }, conc-list: [8] } + kimik3-fp4-mi355x-vllm-agentic-bringup: image: vllm/vllm-openai-rocm:kimi-k3 model: moonshotai/Kimi-K3 From 592a13b1cc538ea127c38a25a5305df4def56a97 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Tue, 28 Jul 2026 14:09:39 +0900 Subject: [PATCH 09/55] [AMD][AgentX] kimik3: enable prefix caching by default on every arm vLLM resolves --enable-prefix-caching's default to False for this model, so omitting it -- as the AMD reference command does -- runs the agentic trace with prefix reuse switched off, which is not a useful baseline for a benchmark whose purpose is exercising large shared prefixes. Measured on gfx950 it costs essentially no KV (1,414,660 vs 1,420,824 tokens) and improves ITL (484 vs 577 ms). PREFIX_CACHING=false stays available for a deliberate A/B. Bringup key reduced to the single none c8 cell. Co-Authored-By: Claude Opus 5 (1M context) --- benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh | 12 ++++++++---- configs/amd-master.yaml | 1 - 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index d7e4b5267b..346f1e0a1d 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -485,10 +485,14 @@ fi # and hash sizes only align with prefix caching on -- an omission has been # reported to trip "tokens_per_block not divisible by tokens_per_hash" at load. # Set PREFIX_CACHING=true/false to force it either way. -PREFIX_CACHE_ARGS=() -if [ "${PREFIX_CACHING:-}" = "true" ]; then - PREFIX_CACHE_ARGS=(--enable-prefix-caching) -elif [ "${PREFIX_CACHING:-}" = "false" ]; then +# ON by default for every arm. Agentic trace replay exists to exercise large +# shared prefixes, so measuring it with reuse disabled is not a useful baseline. +# It also costs essentially no KV (measured: 1,414,660 vs 1,420,824 tokens) and +# improves ITL (484 vs 577 ms). Note vLLM resolves the flag's default to False +# for this model, so it must be passed explicitly. PREFIX_CACHING=false remains +# available for a deliberate A/B. +PREFIX_CACHE_ARGS=(--enable-prefix-caching) +if [ "${PREFIX_CACHING:-}" = "false" ]; then PREFIX_CACHE_ARGS=(--no-enable-prefix-caching) fi diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 92b36c4839..ad329b6457 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -710,7 +710,6 @@ kimik3-fp4-mi355x-vllm-agentic-bringup: - dram-utilization: 0.80 search-space: - { tp: 8, kv-offloading: none, conc-list: [8] } - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "cd9a0ad5325c7d52c825ed17aac6185d5c520e44" }, conc-list: [8] } dsr1-fp8-mi355x-atom: image: rocm/atom:rocm7.2.3_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom20260511 From 6775c1271806e9aeb80e503a44aa6fd25d16eba4 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Tue, 28 Jul 2026 14:18:06 +0900 Subject: [PATCH 10/55] [AMD][AgentX] kimik3: bringup key = none + both LMCache server profiles at c8 One dispatch covering all three arms at conc 8 so they land in a single run rather than preempting each other on the slurm-gated cluster: GPU-only, the AMD K3 reference LMCache server command, and the kimik2.7 LMCache server command. Prefix caching, --max-num-batched-tokens 768 and --chunk-size 768 are applied by the recipe on the LMCache arms as hard Mamba-hybrid requirements. Co-Authored-By: Claude Opus 5 (1M context) --- configs/amd-master.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index ad329b6457..bd96980958 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -710,6 +710,8 @@ kimik3-fp4-mi355x-vllm-agentic-bringup: - dram-utilization: 0.80 search-space: - { tp: 8, kv-offloading: none, conc-list: [8] } + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "cd9a0ad5325c7d52c825ed17aac6185d5c520e44" }, conc-list: [8] } + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache-k27, version: "cd9a0ad5325c7d52c825ed17aac6185d5c520e44" }, conc-list: [8] } dsr1-fp8-mi355x-atom: image: rocm/atom:rocm7.2.3_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom20260511 From f55e4227199f6fe3c3776cf9a4c6db15879628b2 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Tue, 28 Jul 2026 14:41:43 +0900 Subject: [PATCH 11/55] [AMD][AgentX] mi355x-amds: exclude mia1-p01-g14 from salloc The launcher's salloc had no --exclude at all. mia1-p01-g14 chronically carries a stale ~32 GiB/GPU allocation from previous tenants: only ~256 of 287.98 GiB is free, so any gpu-memory-utilization above ~0.888 trips 'Free memory on device cuda:N ... is less than desired GPU memory utilization' and the engine never starts. It killed kimik3 cells in runs 30330955808 and 30331297999, and the same node did the same thing to the kimik2.7 -tune2 cells. Wired to SALLOC_EXCLUDE_NODES (comma-separated, set empty to disable) so the denylist is data rather than a code edit. Co-Authored-By: Claude Opus 5 (1M context) --- runners/launch_mi355x-amds.sh | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/runners/launch_mi355x-amds.sh b/runners/launch_mi355x-amds.sh index 1881a65d0c..b5b580f7a1 100644 --- a/runners/launch_mi355x-amds.sh +++ b/runners/launch_mi355x-amds.sh @@ -268,7 +268,21 @@ else export GPU_COUNT="${GPU_COUNT:-${TP:?TP must be set}}" set -x - salloc --partition=$PARTITION --gres=gpu:$GPU_COUNT --exclusive --cpus-per-task=128 --time=500 --no-shell --job-name="$RUNNER_NAME" + # Node denylist. mia1-p01-g14 chronically carries a stale ~32 GiB/GPU + # allocation from previous tenants, so vLLM refuses to start on it: only + # ~256 of 287.98 GiB is free and any gpu-memory-utilization above ~0.888 + # trips "Free memory on device cuda:N ... is less than desired GPU memory + # utilization". It took out kimik3 cells in runs 30330955808 and + # 30331297999, and the same node did the same thing to the kimik2.7 -tune2 + # cells. Override with SALLOC_EXCLUDE_NODES (comma-separated, empty to + # disable). + SALLOC_EXCLUDE_NODES="${SALLOC_EXCLUDE_NODES-mia1-p01-g14}" + EXCLUDE_ARG="" + if [ -n "$SALLOC_EXCLUDE_NODES" ]; then + EXCLUDE_ARG="--exclude=$SALLOC_EXCLUDE_NODES" + echo "salloc excluding nodes: $SALLOC_EXCLUDE_NODES" + fi + salloc --partition=$PARTITION --gres=gpu:$GPU_COUNT --exclusive --cpus-per-task=128 --time=500 --no-shell --job-name="$RUNNER_NAME" $EXCLUDE_ARG JOB_ID=$(squeue --name="$RUNNER_NAME" -h -o %A | head -n1) srun --jobid=$JOB_ID bash -c "docker stop \$(docker ps -a -q)" From 5cecde9b0a81955d524ffbe3474f065ac5de6c0a Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Tue, 28 Jul 2026 14:58:10 +0900 Subject: [PATCH 12/55] [AMD][AgentX] mi355x-amds: deny g15/g18 too; add kimik3 retry key mia1-p01-g15 had only 21.3 of 287.98 GiB free (another tenant's model resident) and killed the lmcache-k27 cell of run 30332543084; mia1-p01-g18 showed the same 21.9 GiB signature earlier. Both join g14 on the salloc denylist. Adds a single-cell retry key so a node-poisoned cell can be re-dispatched without re-running its healthy siblings. Co-Authored-By: Claude Opus 5 (1M context) --- configs/amd-master.yaml | 16 ++++++++++++++++ runners/launch_mi355x-amds.sh | 2 +- 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index bd96980958..f67eef4ca8 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -697,6 +697,22 @@ kimik3-fp4-mi355x-vllm-agentic-lmprofile: - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache-k27, version: "cd9a0ad5325c7d52c825ed17aac6185d5c520e44" }, conc-list: [8] } - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "cd9a0ad5325c7d52c825ed17aac6185d5c520e44" }, conc-list: [8] } +# Retry-only key: single cell, re-dispatched when a bringup cell lands on a +# node carrying another tenant's allocation. Delete once it has a result. +kimik3-fp4-mi355x-vllm-agentic-retry: + image: vllm/vllm-openai-rocm:kimi-k3 + model: moonshotai/Kimi-K3 + model-prefix: kimik3 + runner: cluster:mi355x-amds + precision: fp4 + framework: vllm + multinode: false + scenarios: + agentic-coding: + - dram-utilization: 0.80 + search-space: + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache-k27, version: "cd9a0ad5325c7d52c825ed17aac6185d5c520e44" }, conc-list: [8] } + kimik3-fp4-mi355x-vllm-agentic-bringup: image: vllm/vllm-openai-rocm:kimi-k3 model: moonshotai/Kimi-K3 diff --git a/runners/launch_mi355x-amds.sh b/runners/launch_mi355x-amds.sh index b5b580f7a1..f8edaf164f 100644 --- a/runners/launch_mi355x-amds.sh +++ b/runners/launch_mi355x-amds.sh @@ -276,7 +276,7 @@ else # 30331297999, and the same node did the same thing to the kimik2.7 -tune2 # cells. Override with SALLOC_EXCLUDE_NODES (comma-separated, empty to # disable). - SALLOC_EXCLUDE_NODES="${SALLOC_EXCLUDE_NODES-mia1-p01-g14}" + SALLOC_EXCLUDE_NODES="${SALLOC_EXCLUDE_NODES-mia1-p01-g14,mia1-p01-g15,mia1-p01-g18}" EXCLUDE_ARG="" if [ -n "$SALLOC_EXCLUDE_NODES" ]; then EXCLUDE_ARG="--exclude=$SALLOC_EXCLUDE_NODES" From 37f02093ce2896f3e20c5d9e3eb27c040c31b017 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Tue, 28 Jul 2026 15:23:14 +0900 Subject: [PATCH 13/55] [AMD][AgentX] kimik3: add the agentic EVAL_ONLY path (guidance backend + 64-way) Ports the kimik2.7 agentic EVAL v2 configuration that produced a scoring SWE-bench run (30258968315), gated on EVAL_ONLY so the measured throughput config is untouched: - Move structured outputs off xgrammar. With tool calls flowing vLLM builds a grammar and the default backend 'auto' resolves to xgrammar, which rejects Kimi's tool-call tokens ('Failed to advance FSM' -> HTTP 500 -> empty patches -> near-zero score). llguidance is not in the ROCm image, so it is installed on demand and the flag spelling probed from --help=all (--structured-outputs-config, falling back to --guided-decoding-backend). - Widen serving concurrency to 64 and set SWEBENCH_AGENT_WORKERS to match: 300 SWE-bench Lite instances at the sweep's conc would not finish inside the 6h SWEBENCH_AGENT_TIMEOUT, and accuracy does not depend on the conc point. Unlike kimik2.7, the kimi_k3 tool-call and reasoning parsers need no eval-only addition -- they are part of the AMD reference command and already passed unconditionally. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 44 +++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 346f1e0a1d..8c69aed0c3 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -512,6 +512,49 @@ if [ "${SPEC_DECODE:-false}" = "true" ]; then ) fi +# ---- Eval-only path ----------------------------------------------------------- +# Mirrors the kimik2.7 agentic EVAL v2 configuration that scored on SWE-bench +# (run 30258968315). Two things are needed beyond the throughput config, and +# both are gated on EVAL_ONLY so the measured serving config is untouched. +EVAL_SERVE_ARGS=() +if [ "${EVAL_ONLY:-false}" = "true" ]; then + # The kimi_k3 tool-call and reasoning parsers are already passed + # unconditionally by this recipe (they are part of the AMD reference + # command), unlike kimik2.7 where they had to be added for eval. + # + # With tool calls flowing, vLLM builds a grammar for them and the default + # backend `auto` resolves to xgrammar, which rejects Kimi's tool-call tokens + # ("Failed to advance FSM" -> HTTP 500 -> empty patches -> a near-zero + # score). Move off xgrammar. llguidance is the guidance backend's runtime + # and is not in the ROCm image, so install it on demand. + SO_BACKEND="${STRUCTURED_OUTPUTS_BACKEND:-guidance}" + if [ "$SO_BACKEND" != "auto" ]; then + python3 -c 'import llguidance' 2>/dev/null || pip install --quiet llguidance || true + if python3 -c 'import llguidance' 2>/dev/null; then + # `vllm serve --help` lists config SECTIONS, not flags; the flag + # names only appear under --help=all. + VLLM_SERVE_HELP="$(vllm serve --help=all 2>/dev/null || vllm serve --help 2>/dev/null || true)" + if grep -q -- '--structured-outputs-config' <<<"$VLLM_SERVE_HELP"; then + EVAL_SERVE_ARGS+=(--structured-outputs-config "{\"backend\":\"$SO_BACKEND\"}") + elif grep -q -- '--guided-decoding-backend' <<<"$VLLM_SERVE_HELP"; then + EVAL_SERVE_ARGS+=(--guided-decoding-backend "$SO_BACKEND") + else + echo "WARN: no structured-outputs backend flag in this image; leaving the default in place" >&2 + fi + else + echo "WARN: llguidance unavailable; leaving the default structured-outputs backend (xgrammar) in place" >&2 + fi + fi + + # 300 SWE-bench Lite instances at the sweep's conc would not finish inside + # SWEBENCH_AGENT_TIMEOUT (6h). Accuracy does not depend on the conc point, + # only wall-clock does, so widen serving concurrency for eval and let the + # harness match it. + EVAL_MAX_NUM_SEQS="${EVAL_MAX_NUM_SEQS:-64}" + export SWEBENCH_AGENT_WORKERS="${SWEBENCH_AGENT_WORKERS:-$EVAL_MAX_NUM_SEQS}" + MAX_NUM_SEQS="$EVAL_MAX_NUM_SEQS" +fi + GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.95}" MAX_NUM_SEQS="${MAX_NUM_SEQS:-128}" MAX_NUM_BATCHED_TOKENS="${MAX_NUM_BATCHED_TOKENS:-4096}" @@ -540,6 +583,7 @@ VLLM_CMD=( "${PREFIX_CACHE_ARGS[@]}" "${KV_CACHE_DTYPE_ARGS[@]}" "${SPEC_ARGS[@]}" + "${EVAL_SERVE_ARGS[@]}" "${OFFLOAD_ARGS[@]}" ) printf '%q ' "${VLLM_CMD[@]}" | tee "$RESULT_DIR/vllm_command.txt" From d811a763883b738888021ce26615905fc771ba36 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Tue, 28 Jul 2026 17:02:46 +0900 Subject: [PATCH 14/55] [AMD][AgentX] kimik3: use the reference LMCache (pip lmcache==0.5.1, L1 512, chunk 1024, mnbt 4096) The AMD reference recipe installs a PyPI release -- `uv pip install "lmcache==0.5.1"` -- not a git build. Building dev HEAD instead was the source of every LMCache failure so far: - dev (v0.5.3.dev47) ships a LazyMemoryAllocator that expands the pinned L1 pool ~10 GB per ~17 s DURING serving. With L1 sized from the DRAM budget (1360 GB) that is ~136 expansions; the vLLM worker blocks on the store, stops draining the shm broadcast queue ('No available shared memory broadcast block found in 60 seconds', four in a row) and the executor kills the engine with 'RPC call to sample_tokens timed out'. Reproduced on two clean nodes, g09 and g11, both after exactly one 360 s warmup request. - dev also adds Mamba-hybrid checks the reference command does not satisfy (block_size <= max_num_batched_tokens < 2*block_size, chunk_size % block_size == 0), which is why the recipe had been pinning 768/768. Reverts to the reference values: L1 512 GB fixed, --chunk-size 1024, --max-num-batched-tokens 4096. LMCACHE_MAX_NUM_BATCHED_TOKENS and LMCACHE_CHUNK_SIZE_OVERRIDE restore 768 without a recipe edit should 0.5.1 enforce the same checks. Config version pins are now the PyPI version. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 82 ++++++++----------- configs/amd-master.yaml | 14 ++-- 2 files changed, 41 insertions(+), 55 deletions(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 8c69aed0c3..8c5553588f 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -211,42 +211,25 @@ case "${KV_OFFLOAD_BACKEND:-}" in fi PREFIX_CACHING=true - # SECOND hard constraint from the same Mamba-hybrid path. vLLM enforces - # block_size <= max_num_batched_tokens < 2 * block_size - # "so every prefill step advances exactly one block and every block boundary - # gets a state snapshot". K3 resolves to block_size=768, so the reference - # command's --max-num-batched-tokens 4096 aborts engine init with - # "got max_num_batched_tokens=4096, block_size=768. Set - # --max-num-batched-tokens 768." (measured on gfx950, LMCache v0.5.3.dev47). - # - # NOTE this is a real and severe cost of the LMCache arm on this model, not - # a formality: at ~106k-token average ISL for this corpus, a 768-token - # budget means ~138 chunked-prefill steps per turn versus ~26 at 4096. Any - # LMCache-vs-none comparison has to be read with that in mind -- the offload - # tier is not free here, it changes the prefill schedule. - # - # 768 is what this model/config resolves to; if vLLM reports a different - # block_size, its error names the value to use. Override with - # LMCACHE_MAX_NUM_BATCHED_TOKENS. - # Deliberately ignores any inherited MAX_NUM_BATCHED_TOKENS: the harness and - # the driver both export the reference 4096, which is invalid on this path, - # so honouring it would just reproduce the abort. Only the explicit - # LMCACHE_MAX_NUM_BATCHED_TOKENS overrides. - MAX_NUM_BATCHED_TOKENS="${LMCACHE_MAX_NUM_BATCHED_TOKENS:-768}" - - # THIRD constraint, asserted by the connector itself: + # --max-num-batched-tokens and --chunk-size stay at the AMD reference + # values (4096 / 1024). They were briefly pinned to 768 because LMCache + # dev HEAD (v0.5.3.dev47) rejects the reference values on this hybrid model: + # ValueError: Mamba-hybrid models with LMCache require + # block_size <= max_num_batched_tokens < 2 * block_size ... block_size=768 # AssertionError: LMCache chunk size should be a multiple of vLLM block size - # i.e. chunk_size % block_size == 0. With K3's block_size=768, BOTH stock - # chunk sizes are invalid: the AMD K3 reference's 1024 (1024 % 768 = 256) - # and kimik2.7's 256 (smaller than a block). So chunk size is not a free - # A/B axis on this model -- it is pinned to a multiple of the block size, - # and the two server profiles can only differ in their other knobs. - # Override with LMCACHE_CHUNK_SIZE_OVERRIDE (must stay a multiple of 768). - LMCACHE_K3_CHUNK_SIZE="${LMCACHE_CHUNK_SIZE_OVERRIDE:-768}" + # Those checks live in the dev integration, not in the pinned 0.5.1 release + # the reference recipe installs, so with 0.5.1 the reference values stand. + # If 0.5.1 turns out to enforce them too, LMCACHE_MAX_NUM_BATCHED_TOKENS and + # LMCACHE_CHUNK_SIZE_OVERRIDE set them back to 768 without a recipe edit. + # + # Worth keeping in view either way: at ~106k-token average ISL, 768 would + # mean ~138 chunked-prefill steps per turn versus ~26 at 4096, so the two + # settings are not performance-equivalent. + MAX_NUM_BATCHED_TOKENS="${LMCACHE_MAX_NUM_BATCHED_TOKENS:-4096}" + LMCACHE_K3_CHUNK_SIZE="${LMCACHE_CHUNK_SIZE_OVERRIDE:-1024}" LMCACHE_CHUNK_SIZE="$LMCACHE_K3_CHUNK_SIZE" LMCACHE_CHUNK_SIZE_K27="$LMCACHE_K3_CHUNK_SIZE" - echo "LMCache/Mamba-hybrid constraint: chunk-size pinned to $LMCACHE_K3_CHUNK_SIZE (must be a multiple of block_size)" - echo "LMCache/Mamba-hybrid constraint: pinning --max-num-batched-tokens=$MAX_NUM_BATCHED_TOKENS (block_size must satisfy bs <= mnbt < 2*bs)" + echo "LMCache: --max-num-batched-tokens=$MAX_NUM_BATCHED_TOKENS --chunk-size=$LMCACHE_K3_CHUNK_SIZE (reference values)" # LMCache is NOT in the kimi-k3 image (verified: no `lmcache` module and no # CLI), so build it against ROCm. Clone to a container-local dir, NOT the @@ -269,23 +252,26 @@ except Exception: ' 2>/dev/null || true) fi + # The AMD reference recipe pins a PyPI release: `uv pip install + # "lmcache==0.5.1"`. Do not substitute a git build of dev HEAD -- dev + # (v0.5.3.dev47) carries a LazyMemoryAllocator that expands the pinned L1 + # pool ~10 GB per ~17 s DURING serving, which starves the vLLM worker until + # the executor RPC deadline fires ("RPC call to sample_tokens timed out", + # observed on g09 and g11). It also adds Mamba-hybrid constraints + # (block_size <= max_num_batched_tokens < 2*block_size, and + # chunk_size % block_size == 0) that the reference command does not satisfy. + LMCACHE_VERSION="${LMCACHE_VERSION:-${LMCACHE_CFG_VERSION:-0.5.1}}" if ! python3 -c "import lmcache.integration.vllm.lmcache_mp_connector" >/dev/null 2>&1; then - LMCACHE_SRC_DIR="${LMCACHE_SRC_DIR:-/opt/lmcache-src}" - # Default pin = LMCache dev HEAD as of 2026-07-28. The server flags this - # recipe passes (--max-gpu-workers / --max-cpu-workers / --l1-align-bytes - # / --eviction-trigger-watermark) only exist on recent dev: they come - # from lmcache/v1/multiprocess/config.py and - # lmcache/v1/distributed/config.py, not the older --max-workers CLI. - LMCACHE_GIT_REF="${LMCACHE_GIT_REF:-${LMCACHE_CFG_VERSION:-cd9a0ad5325c7d52c825ed17aac6185d5c520e44}}" - echo "Building LMCache at ref: $LMCACHE_GIT_REF" - rm -rf "$LMCACHE_SRC_DIR" - git clone https://github.com/LMCache/LMCache.git "$LMCACHE_SRC_DIR" - ( cd "$LMCACHE_SRC_DIR" - git checkout "$LMCACHE_GIT_REF" - pip install -r requirements/build.txt - CXX=hipcc BUILD_WITH_HIP=1 pip install -e . --no-build-isolation ) + echo "Installing lmcache==$LMCACHE_VERSION" + if command -v uv >/dev/null 2>&1; then + uv pip install --system "lmcache==$LMCACHE_VERSION" \ + || agentic_pip_install --quiet "lmcache==$LMCACHE_VERSION" + else + agentic_pip_install --quiet "lmcache==$LMCACHE_VERSION" + fi python3 -c "import lmcache.integration.vllm.lmcache_mp_connector" >/dev/null fi + python3 -c "import lmcache; print('lmcache', getattr(lmcache,'__version__','?'))" || true LMCACHE_HOST="${LMCACHE_HOST:-127.0.0.1}" LMCACHE_PORT="${LMCACHE_PORT:-5555}" @@ -295,7 +281,7 @@ except Exception: # SHM and falls back to a pickle path that crashes at load. Cap at 90% of # free /dev/shm so SHM stays enabled, and say so loudly -- the capped value # is the number that actually backs the run. - LMCACHE_L1_SIZE_GB="${LMCACHE_L1_SIZE_GB:-$TOTAL_CPU_DRAM_GB}" + LMCACHE_L1_SIZE_GB="${LMCACHE_L1_SIZE_GB:-512}" # reference value # Optional ceiling on the eager allocation. Memory pinning is unavailable in # the ROCm container ("CudaPinMemoryBackend: neither torch cudart nor diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index f67eef4ca8..9181eace59 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -654,13 +654,13 @@ kimik3-fp4-mi355x-vllm-agentic: # --max-gpu-workers / --max-cpu-workers / --l1-align-bytes / # --eviction-trigger-watermark CLI the recipe passes; the recipe's own # LMCACHE_GIT_REF default is this same commit. - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "cd9a0ad5325c7d52c825ed17aac6185d5c520e44" }, conc-list: [1, 4, 8, 16, 32] } + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.1" }, conc-list: [1, 4, 8, 16, 32] } # TEP8 (TP8 + expert parallelism, no DP-attention) at the two highest # concurrency points. At TP8 the MoE weights are already sharded across # every rank, so EP changes the partition axis rather than per-rank bytes; # any win has to come from the all-to-all vs all-reduce path and # expert-kernel efficiency, which is what these two points isolate. - - { tp: 8, ep: 8, dp-attn: false, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "cd9a0ad5325c7d52c825ed17aac6185d5c520e44" }, conc-list: [16, 32] } + - { tp: 8, ep: 8, dp-attn: false, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.1" }, conc-list: [16, 32] } # Bring-up twin of kimik3-fp4-mi355x-vllm-agentic. NOT shipped -- delete once the # full key has produced results. @@ -694,8 +694,8 @@ kimik3-fp4-mi355x-vllm-agentic-lmprofile: agentic-coding: - dram-utilization: 0.80 search-space: - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache-k27, version: "cd9a0ad5325c7d52c825ed17aac6185d5c520e44" }, conc-list: [8] } - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "cd9a0ad5325c7d52c825ed17aac6185d5c520e44" }, conc-list: [8] } + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache-k27, version: "0.5.1" }, conc-list: [8] } + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.1" }, conc-list: [8] } # Retry-only key: single cell, re-dispatched when a bringup cell lands on a # node carrying another tenant's allocation. Delete once it has a result. @@ -711,7 +711,7 @@ kimik3-fp4-mi355x-vllm-agentic-retry: agentic-coding: - dram-utilization: 0.80 search-space: - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache-k27, version: "cd9a0ad5325c7d52c825ed17aac6185d5c520e44" }, conc-list: [8] } + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache-k27, version: "0.5.1" }, conc-list: [8] } kimik3-fp4-mi355x-vllm-agentic-bringup: image: vllm/vllm-openai-rocm:kimi-k3 @@ -726,8 +726,8 @@ kimik3-fp4-mi355x-vllm-agentic-bringup: - dram-utilization: 0.80 search-space: - { tp: 8, kv-offloading: none, conc-list: [8] } - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "cd9a0ad5325c7d52c825ed17aac6185d5c520e44" }, conc-list: [8] } - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache-k27, version: "cd9a0ad5325c7d52c825ed17aac6185d5c520e44" }, conc-list: [8] } + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.1" }, conc-list: [8] } + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache-k27, version: "0.5.1" }, conc-list: [8] } dsr1-fp8-mi355x-atom: image: rocm/atom:rocm7.2.3_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom20260511 From 471c4d6711daa5464a950c807d94ceda4f833a10 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Tue, 28 Jul 2026 17:11:17 +0900 Subject: [PATCH 15/55] [AMD][AgentX] kimik3: add lmcache-budget (L1 from the DRAM budget, not the flat 512) Third LMCache backend name so the L1 pool size can be A/B'd from config, since the agentic matrix has no per-cell env channel: lmcache -> reference server, --l1-size-gb 512 (flat, as in the recipe) lmcache-k27 -> kimik2.7 server, --l1-size-gb 512 lmcache-budget -> reference server, --l1-size-gb = TOTAL_CPU_DRAM_GB capped by 90%% of free /dev/shm (~1360 GB on these nodes) The budget sizing is what kimik2.7_fp4_mi355x.sh does -- it ran a 1199 GB pool on this same fleet -- and it is what benchmarks/single_node/agentic/README.md asks for (consume TOTAL_CPU_DRAM_GB, do not substitute a model-specific constant). Whether it is safe here depends on whether the pinned 0.5.1 release has the LazyMemoryAllocator that made a 1360 GB pool starve the worker on dev. Also raises the LMCache healthcheck budget to 1800 s, since a larger pool may allocate up front. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 13 +++++++++++-- configs/amd-master.yaml | 18 ++++++++++++++++++ 2 files changed, 29 insertions(+), 2 deletions(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 8c5553588f..7f8d841d4f 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -139,7 +139,7 @@ wait_for_lmcache_ready() { # disables its LazyMemoryAllocator and allocates the whole L1 pool UP FRONT, # so "ready" can be minutes away and scales with --l1-size-gb. A 180s budget # timed out at exactly 178s on a 2249 GB pool. - local attempts="${LMCACHE_READY_ATTEMPTS:-900}" + local attempts="${LMCACHE_READY_ATTEMPTS:-1800}" # The health route has moved across LMCache versions, and a wrong path is # indistinguishable from a slow start: both just never return 200. Probe the # known spellings and report which one answered. @@ -181,7 +181,7 @@ OFFLOAD_ARGS=() if agentic_kv_offload_enabled; then case "${KV_OFFLOAD_BACKEND:-}" in - lmcache|lmcache-k27) + lmcache|lmcache-k27|lmcache-budget) require_agentic_kv_offload_backend "$KV_OFFLOAD_BACKEND" # The server profile has to be selectable from config: the agentic matrix # has no per-cell env channel, so the backend NAME carries it. @@ -190,6 +190,15 @@ case "${KV_OFFLOAD_BACKEND:-}" in if [ "$KV_OFFLOAD_BACKEND" = "lmcache-k27" ]; then LMCACHE_PROFILE=k2.7 fi + # lmcache-budget: same reference server, but L1 sized from the agentic DRAM + # budget (shm-capped) instead of the reference's flat 512 GB -- i.e. what + # kimik2.7 does, which ran a 1199 GB pool successfully on this fleet. The + # agentic README also asks recipes to consume TOTAL_CPU_DRAM_GB rather than + # a constant. Kept as a separate backend name because the matrix has no + # per-cell env channel to vary it from config. + if [ "$KV_OFFLOAD_BACKEND" = "lmcache-budget" ]; then + LMCACHE_L1_SIZE_GB="${LMCACHE_L1_SIZE_GB:-$TOTAL_CPU_DRAM_GB}" + fi # LMCache on K3 REQUIRES prefix caching. K3's Kimi Delta Attention layers # are Mamba KV-cache groups, and vLLM only selects mamba_cache_mode='align' diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 9181eace59..223e6ebc2c 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -713,6 +713,24 @@ kimik3-fp4-mi355x-vllm-agentic-retry: search-space: - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache-k27, version: "0.5.1" }, conc-list: [8] } +# L1 pool-size A/B. Same reference server command and lmcache==0.5.1 on both; +# the only difference is --l1-size-gb: the reference's flat 512 GB vs the +# agentic DRAM budget capped by /dev/shm (~1360 GB on these nodes, which is the +# kimik2.7 sizing that ran a 1199 GB pool here successfully). +kimik3-fp4-mi355x-vllm-agentic-l1size: + image: vllm/vllm-openai-rocm:kimi-k3 + model: moonshotai/Kimi-K3 + model-prefix: kimik3 + runner: cluster:mi355x-amds + precision: fp4 + framework: vllm + multinode: false + scenarios: + agentic-coding: + - dram-utilization: 0.80 + search-space: + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache-budget, version: "0.5.1" }, conc-list: [8] } + kimik3-fp4-mi355x-vllm-agentic-bringup: image: vllm/vllm-openai-rocm:kimi-k3 model: moonshotai/Kimi-K3 From 09a659154f9aeaa686db27d7c657a78ed8bdce91 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Tue, 28 Jul 2026 18:46:59 +0900 Subject: [PATCH 16/55] [AMD][AgentX] kimik3: gpu-memory-utilization 0.95 -> 0.88 The reference 0.95 asks for 273.59 of 287.98 GiB and cleared only 2 of 9 cells on cluster:mi355x-amds. Free memory at startup, observed across seven nodes: g09 281, g11 275/212/208, g14 256, g16 271/262, g15 21, g18 22. Even nominally clean nodes sit under 273.59 once driver overhead and transient co-tenancy are counted -- g16 missed by 2.5 GiB with only 16.9 GiB of overhead. Not a denylist problem: g11 and g16 each measured both above and below the line hours apart, so the bad set is not stable. 0.88 (253.4 GiB) clears every observation except the two genuinely occupied nodes. It costs KV on a KV-bound workload, but a cell that never starts is worth nothing. Co-Authored-By: Claude Opus 5 (1M context) --- benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 7f8d841d4f..8a68ae190d 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -550,7 +550,14 @@ if [ "${EVAL_ONLY:-false}" = "true" ]; then MAX_NUM_SEQS="$EVAL_MAX_NUM_SEQS" fi -GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.95}" +# 0.88, not the reference's 0.95. Measured on cluster:mi355x-amds: 0.95 asks for +# 273.59 of 287.98 GiB and cleared only 2 of 9 cells. Observed free memory at +# startup across seven nodes -- g09 281, g11 275/212/208, g14 256, g16 271/262, +# g15 21, g18 22 -- so even nominally clean nodes sit below 273.59 once driver +# overhead and transient co-tenancy are counted. This is not a denylist problem: +# g11 and g16 each measured both above and below the line hours apart. 0.88 +# (253.4 GiB) clears every observation except the two genuinely occupied nodes. +GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.88}" MAX_NUM_SEQS="${MAX_NUM_SEQS:-128}" MAX_NUM_BATCHED_TOKENS="${MAX_NUM_BATCHED_TOKENS:-4096}" From f1469157764d8bf6a7b204edad45212ae50e9dad Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Tue, 28 Jul 2026 19:28:43 +0900 Subject: [PATCH 17/55] [AMD][AgentX] kimik3: 0.5.1 enforces the Mamba-hybrid limits too -- re-pin 768 Run 30348060242 on g09 confirms the pinned lmcache==0.5.1 release raises the same error as dev HEAD: ValueError: Mamba-hybrid models with LMCache require block_size <= max_num_batched_tokens < 2 * block_size So --max-num-batched-tokens 768 and --chunk-size 768 are properties of the LMCache/vLLM Mamba-hybrid integration, not of dev; the previous commit's assumption that they were dev-only was wrong. Only the LazyMemoryAllocator serving stall is dev-specific. That cell also confirms gpu-memory-utilization 0.88 clears the startup memory wall: Available KV cache memory 50.66 GiB, GPU KV 1,957,290 tokens, LMCache healthy in 14 s -- the furthest any LMCache cell has reached. Co-Authored-By: Claude Opus 5 (1M context) --- benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 8a68ae190d..3814ae0248 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -226,16 +226,16 @@ case "${KV_OFFLOAD_BACKEND:-}" in # ValueError: Mamba-hybrid models with LMCache require # block_size <= max_num_batched_tokens < 2 * block_size ... block_size=768 # AssertionError: LMCache chunk size should be a multiple of vLLM block size - # Those checks live in the dev integration, not in the pinned 0.5.1 release - # the reference recipe installs, so with 0.5.1 the reference values stand. - # If 0.5.1 turns out to enforce them too, LMCACHE_MAX_NUM_BATCHED_TOKENS and - # LMCACHE_CHUNK_SIZE_OVERRIDE set them back to 768 without a recipe edit. + # CONFIRMED on 0.5.1 as well (run 30348060242, g09): the pinned release + # raises the same ValueError, so these are properties of the LMCache/vLLM + # Mamba-hybrid integration, not of dev HEAD. Only the LazyMemoryAllocator + # stall was dev-specific. Both stay pinned to 768 for the LMCache arms. # # Worth keeping in view either way: at ~106k-token average ISL, 768 would # mean ~138 chunked-prefill steps per turn versus ~26 at 4096, so the two # settings are not performance-equivalent. - MAX_NUM_BATCHED_TOKENS="${LMCACHE_MAX_NUM_BATCHED_TOKENS:-4096}" - LMCACHE_K3_CHUNK_SIZE="${LMCACHE_CHUNK_SIZE_OVERRIDE:-1024}" + MAX_NUM_BATCHED_TOKENS="${LMCACHE_MAX_NUM_BATCHED_TOKENS:-768}" + LMCACHE_K3_CHUNK_SIZE="${LMCACHE_CHUNK_SIZE_OVERRIDE:-768}" LMCACHE_CHUNK_SIZE="$LMCACHE_K3_CHUNK_SIZE" LMCACHE_CHUNK_SIZE_K27="$LMCACHE_K3_CHUNK_SIZE" echo "LMCache: --max-num-batched-tokens=$MAX_NUM_BATCHED_TOKENS --chunk-size=$LMCACHE_K3_CHUNK_SIZE (reference values)" From 218117c16bc8d43508e9b2580bfa875657358885 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Tue, 28 Jul 2026 22:33:33 +0900 Subject: [PATCH 18/55] [AMD][AgentX] kimik3: add native KV-offload tiers (vllm-simple, vllm-native) LMCache cannot serve Kimi-K3: both its server profiles abort at engine init with 'expected a Mamba [conv_state, ssm_state] tensor list, got Tensor' (run 30350911388, both cells, same node) because K3's Kimi Delta Attention supplies one fused state tensor where LMCache's Mamba path expects the pair. No flag changes that, so vLLM's own offload connectors are the only remaining tiers. vllm-simple is the AMD reference's native path (SimpleCPUOffloadConnector). One correction against the reference command: it passes lazy_offload as the JSON STRING "false", and the connector does lazy_offload = bool(extra_config.get("lazy_offload", False)) (simple_cpu_offload_connector.py:77) so bool("false") is True and the string silently selects LAZY -- the opposite of what it reads as. Passed as a JSON boolean here, with SIMPLE_LAZY_OFFLOAD to swap it; the connector logs lazy/eager at line 95 so server.log confirms which engaged. cpu_bytes_to_use_per_rank is derived from TOTAL_CPU_DRAM_GB per the agentic README rather than hardcoding the reference's 220 GiB/rank. vllm-native is OffloadingConnector, with the decimal-GB to GiB conversion --kv_offloading_size needs. Both run at --max-num-batched-tokens 4096, so unlike the LMCache arms they are like-for-like against the successful none c8 cell. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 45 +++++++++++++++++++ configs/amd-master.yaml | 20 +++++++++ 2 files changed, 65 insertions(+) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 3814ae0248..98ed2e7c9b 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -409,6 +409,51 @@ except Exception: ) fi ;; + vllm-simple) + require_agentic_kv_offload_backend vllm-simple + # vLLM's own SimpleCPUOffloadConnector -- the AMD reference's native + # offload path. Unlike LMCache it does not go through the Mamba + # [conv_state, ssm_state] adapter that K3's Kimi Delta Attention breaks + # ("expected a Mamba [conv_state, ssm_state] tensor list, got Tensor", + # runs 30350911388), so it is the only offload tier that can currently + # initialise on this model. + # + # cpu_bytes_to_use is server-wide; cpu_bytes_to_use_per_rank overrides it + # per rank (simple_cpu_offload_connector.py:66). The agentic README requires + # consuming TOTAL_CPU_DRAM_GB, so derive it rather than hardcode the + # reference's 236223201280 (= 220 GiB/rank, 1760 GiB across 8 ranks). + SIMPLE_RANKS="${GPU_COUNT:-$TP}" + CPU_BYTES_PER_RANK="${SIMPLE_CPU_BYTES_PER_RANK:-$(( TOTAL_CPU_DRAM_GB * 1000 * 1000 * 1000 / SIMPLE_RANKS ))}" + # Identical prefixes must hash to identical block keys across ranks. + export PYTHONHASHSEED=42 + # lazy_offload MUST be a JSON boolean. The reference command passes the + # STRING "false", and the connector does + # lazy_offload = bool(extra_config.get("lazy_offload", False)) + # (simple_cpu_offload_connector.py:77) -- bool("false") is True in Python, + # so the string silently selects LAZY, the opposite of what it reads as. + # Default to real eager offload; SIMPLE_LAZY_OFFLOAD=true swaps it. + # The connector logs "lazy"/"eager" at line 95, so server.log confirms which + # mode actually engaged. + SIMPLE_LAZY_OFFLOAD="${SIMPLE_LAZY_OFFLOAD:-false}" + OFFLOAD_ARGS=( + --kv-transfer-config + "{\"kv_connector\":\"SimpleCPUOffloadConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"cpu_bytes_to_use_per_rank\":$CPU_BYTES_PER_RANK,\"lazy_offload\":$SIMPLE_LAZY_OFFLOAD}}" + ) + echo "SimpleCPUOffloadConnector: ${CPU_BYTES_PER_RANK} B/rank x ${SIMPLE_RANKS} ranks, lazy_offload=$SIMPLE_LAZY_OFFLOAD" + ;; + vllm-native) + require_agentic_kv_offload_backend vllm-native + # OffloadingConnector, vLLM's other native tier. --kv_offloading_size is + # GiB (vllm/config/vllm.py multiplies by 1<<30) while TOTAL_CPU_DRAM_GB is + # decimal GB, so convert or we over-request by ~7.4%. + unset VLLM_USE_SIMPLE_KV_OFFLOAD + KV_OFFLOAD_GIB=$(( TOTAL_CPU_DRAM_GB * 1000000000 / 1073741824 )) + OFFLOAD_ARGS=( + --kv_offloading_backend native + --kv_offloading_size "$KV_OFFLOAD_GIB" + ) + echo "OffloadingConnector: ${KV_OFFLOAD_GIB} GiB" + ;; mooncake) echo "Error: Mooncake is unsupported for Kimi-K3. The upstream recipe marks" >&2 echo " kv_store_{distributed,centralized}_mooncake as 'unsupported' on" >&2 diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 223e6ebc2c..189b2151cb 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -731,6 +731,26 @@ kimik3-fp4-mi355x-vllm-agentic-l1size: search-space: - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache-budget, version: "0.5.1" }, conc-list: [8] } +# Native KV-offload A/B. LMCache cannot serve K3 (its Mamba path expects a +# [conv_state, ssm_state] pair and KDA supplies one fused tensor), so these are +# the only offload tiers that can initialise. Both at conc 8, GMU 0.88, prefix +# caching on, --max-num-batched-tokens 4096 -- i.e. like-for-like with the +# successful none c8 cell, unlike the LMCache arms which were forced to 768. +kimik3-fp4-mi355x-vllm-agentic-native: + image: vllm/vllm-openai-rocm:kimi-k3 + model: moonshotai/Kimi-K3 + model-prefix: kimik3 + runner: cluster:mi355x-amds + precision: fp4 + framework: vllm + multinode: false + scenarios: + agentic-coding: + - dram-utilization: 0.80 + search-space: + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [8] } + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-native }, conc-list: [8] } + kimik3-fp4-mi355x-vllm-agentic-bringup: image: vllm/vllm-openai-rocm:kimi-k3 model: moonshotai/Kimi-K3 From f27aa5f849d5108b83c866cd629b1583d197482d Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Tue, 28 Jul 2026 23:53:30 +0900 Subject: [PATCH 19/55] [AMD][AgentX] kimik3: full 3600s sweep -- none c[1,2,4] + vllm-simple c[4,8,16,32] Replaces the placeholder search space with the sweep the measurements point to. GPU-only is limited to low concurrency: at c8 the 1,957,290-token pool already peaks at 98.6% usage with only a 13.8% prefix hit rate, so c8 is past the eviction crossing and anything above it measures thrash. c1/c2/c4 trace the region where the pool still holds the working set. The offload arm is SimpleCPUOffloadConnector. LMCache is dropped entirely -- both its server profiles abort on this model with 'expected a Mamba [conv_state, ssm_state] tensor list, got Tensor' because Kimi Delta Attention supplies one fused state tensor. Measured at c8, vllm-simple lifts the prefix hit rate 13.8% -> 80.7% and drops TTFT from 193 s to 4.8 s for +70% output throughput (15.93 vs 9.37 tok/s), on a heavier ISL sample (129k vs 116k), so c16/c32 are worth probing. c4 locates the crossover below which the GPU pool alone suffices and the offload transfer cost stops paying for itself. gpu-memory-utilization stays at the recipe's 0.88 for every arm including none. Co-Authored-By: Claude Opus 5 (1M context) --- configs/amd-master.yaml | 67 ++++++++++++++--------------------------- 1 file changed, 23 insertions(+), 44 deletions(-) diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 189b2151cb..c2ef6c1cf1 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -626,53 +626,32 @@ kimik3-fp4-mi355x-vllm-agentic: multinode: false scenarios: agentic-coding: - # 0.80 of the runner's available-cpu-dram-mib (3_095_781), which the - # matrix generator caps at the 2,861,022 MiB / 3 TB-decimal agentic limit - # -> 2399 GB aggregate at TP8, leaving ~460 GB host headroom. Not 1.0: - # LMCache's L1 is pinned and therefore unswappable, and a full-budget - # pinned pool has OOM-killed nodes before. - # - # NOTE this is only a budget. The recipe applies a second, tighter cap at - # run time: L1 is SHM-backed, and if it exceeds free /dev/shm LMCache - # silently disables SHM and falls back to a pickle path that crashes at - # load. The effective pool is min(this budget, 90% of free /dev/shm), and - # the recipe logs a WARNING naming the value it actually used -- read that - # line rather than allocated_cpu_dram_gb when interpreting a result. + # 0.80 of the runner's available-cpu-dram-mib (3_095_781), capped by the + # matrix generator at the 2,861,022 MiB / 3 TB-decimal agentic limit -> + # 2399 GB aggregate at TP8, which SimpleCPUOffloadConnector divides into + # 279.28 GB per rank. - dram-utilization: 0.80 search-space: - # TP=8 only. The MXFP4 checkpoint is 1.561 TB decimal (1.420 TiB, 96 - # safetensors), ~195 GB/GPU across 8 GPUs of the 288 GB part; TP=4 would - # need ~390 GB/GPU and cannot load. Upstream strategy_min_gpus agrees - # (single_node_tp: 8), and DEP needs 16+ GPUs so there is no DP-attn arm. + # TP=8 only: the MXFP4 checkpoint is 1.561 TB decimal (1.420 TiB, 96 + # safetensors), ~195 GB/GPU of the 288 GB part, so TP=4 cannot load. + # gpu-memory-utilization is 0.88 in the recipe, not the reference's 0.95: + # 0.95 asks for 273.59 of 287.98 GiB and cleared only 2 of 9 cells on this + # fleet, because most nodes have more than 14.4 GiB of overhead. # - # The none arm is the reference: it measures where the GPU-only KV wall - # actually sits for this corpus before any host tier is involved. The - # LMCache arm then shows what the offload path buys past that wall. - - { tp: 8, kv-offloading: none, conc-list: [1, 4, 8, 16] } - # version pins the ref the recipe builds from source for ROCm (LMCache is - # not in the kimi-k3 image). It must be recent enough to carry the - # --max-gpu-workers / --max-cpu-workers / --l1-align-bytes / - # --eviction-trigger-watermark CLI the recipe passes; the recipe's own - # LMCACHE_GIT_REF default is this same commit. - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.1" }, conc-list: [1, 4, 8, 16, 32] } - # TEP8 (TP8 + expert parallelism, no DP-attention) at the two highest - # concurrency points. At TP8 the MoE weights are already sharded across - # every rank, so EP changes the partition axis rather than per-rank bytes; - # any win has to come from the all-to-all vs all-reduce path and - # expert-kernel efficiency, which is what these two points isolate. - - { tp: 8, ep: 8, dp-attn: false, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.1" }, conc-list: [16, 32] } - -# Bring-up twin of kimik3-fp4-mi355x-vllm-agentic. NOT shipped -- delete once the -# full key has produced results. -# -# Two cells, one per KV tier, at the same concurrency, so a short dispatch -# answers the only questions that block the real sweep: does the reference -# command serve K3 at all on this fleet, and does the LMCache arm survive -# building LMCache from source (it is absent from the kimi-k3 image) plus the -# SHM-backed L1 sizing. conc 8 because the measured GPU KV pool is 2,244,655 -# tokens = 2.14x concurrency for a 1M-token request, so c8 already exercises the -# host tier on this corpus without being so deep that a capacity stall hides a -# plain bring-up bug. + # GPU-only, low concurrency. Measured at c8 the pool already peaks at + # 98.6% with a 13.8% prefix hit rate, so c8 is past the eviction crossing; + # c1/c2/c4 trace the region where the 1,957,290-token pool still holds the + # working set and no host tier is needed. + - { tp: 8, kv-offloading: none, conc-list: [1, 2, 4] } + # SimpleCPUOffloadConnector. LMCache cannot serve this model (its Mamba + # path expects [conv_state, ssm_state] and Kimi Delta Attention supplies + # one fused tensor), so this is the only working offload tier. Measured at + # c8 it lifts the prefix hit rate 13.8% -> 80.7% and drops TTFT from 193 s + # to 4.8 s for +70% output throughput, on a heavier ISL sample. c4 probes + # the crossover below which the GPU pool alone suffices; c16/c32 probe how + # far the 2234 GiB host tier extends the concurrency ceiling. + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [4, 8, 16, 32] } + # LMCache server-profile A/B for Kimi-K3. NOT shipped -- delete once one profile # wins. Both cells are conc 8 with prefix caching forced on, --max-num-batched-tokens # 768 and --chunk-size 768 (all three are hard Mamba-hybrid requirements the recipe From 2818737e21a0f557249edd199bc4ee463af3ac79 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Wed, 29 Jul 2026 09:22:16 +0900 Subject: [PATCH 20/55] [AMD][AgentX] kimik3: denylist g11 (GPU faults), retry none c[1,2,4] All four vllm-simple cells of run 30370655448 succeeded; all three none cells failed. Two of them (c1, c2) ran on mia1-p01-g11 and died with hard GPU faults -- 21 and 10 hipErrorIllegalAddress respectively, plus NCCL watchdog thread terminations -- after serving 80 and 632 warmup requests successfully. c4 on g16 died with only 'RuntimeError: cancelled', a knock-on from a worker exit. g11 is the same node whose free memory measured 275/212/208 GiB at different times and which hosted the earlier LMCache allocator stall. Adding it to the denylist for hardware faults rather than capacity; the free-memory-driven entries remain unchanged. Retry key set to the three missing none cells at 3600s. Co-Authored-By: Claude Opus 5 (1M context) --- configs/amd-master.yaml | 25 +------------------------ runners/launch_mi355x-amds.sh | 2 +- 2 files changed, 2 insertions(+), 25 deletions(-) diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index c2ef6c1cf1..224b56efbb 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -690,31 +690,8 @@ kimik3-fp4-mi355x-vllm-agentic-retry: agentic-coding: - dram-utilization: 0.80 search-space: - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache-k27, version: "0.5.1" }, conc-list: [8] } - -# L1 pool-size A/B. Same reference server command and lmcache==0.5.1 on both; -# the only difference is --l1-size-gb: the reference's flat 512 GB vs the -# agentic DRAM budget capped by /dev/shm (~1360 GB on these nodes, which is the -# kimik2.7 sizing that ran a 1199 GB pool here successfully). -kimik3-fp4-mi355x-vllm-agentic-l1size: - image: vllm/vllm-openai-rocm:kimi-k3 - model: moonshotai/Kimi-K3 - model-prefix: kimik3 - runner: cluster:mi355x-amds - precision: fp4 - framework: vllm - multinode: false - scenarios: - agentic-coding: - - dram-utilization: 0.80 - search-space: - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache-budget, version: "0.5.1" }, conc-list: [8] } + - { tp: 8, kv-offloading: none, conc-list: [1, 2, 4] } -# Native KV-offload A/B. LMCache cannot serve K3 (its Mamba path expects a -# [conv_state, ssm_state] pair and KDA supplies one fused tensor), so these are -# the only offload tiers that can initialise. Both at conc 8, GMU 0.88, prefix -# caching on, --max-num-batched-tokens 4096 -- i.e. like-for-like with the -# successful none c8 cell, unlike the LMCache arms which were forced to 768. kimik3-fp4-mi355x-vllm-agentic-native: image: vllm/vllm-openai-rocm:kimi-k3 model: moonshotai/Kimi-K3 diff --git a/runners/launch_mi355x-amds.sh b/runners/launch_mi355x-amds.sh index f8edaf164f..2152b17d83 100644 --- a/runners/launch_mi355x-amds.sh +++ b/runners/launch_mi355x-amds.sh @@ -276,7 +276,7 @@ else # 30331297999, and the same node did the same thing to the kimik2.7 -tune2 # cells. Override with SALLOC_EXCLUDE_NODES (comma-separated, empty to # disable). - SALLOC_EXCLUDE_NODES="${SALLOC_EXCLUDE_NODES-mia1-p01-g14,mia1-p01-g15,mia1-p01-g18}" + SALLOC_EXCLUDE_NODES="${SALLOC_EXCLUDE_NODES-mia1-p01-g11,mia1-p01-g14,mia1-p01-g15,mia1-p01-g18}" EXCLUDE_ARG="" if [ -n "$SALLOC_EXCLUDE_NODES" ]; then EXCLUDE_ARG="--exclude=$SALLOC_EXCLUDE_NODES" From d891f75cb866592cc2fd9bf8ce318e66e870846f Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Wed, 29 Jul 2026 09:35:16 +0900 Subject: [PATCH 21/55] [AMD][AgentX] kimik3: add fp8 KV-cache probe (vllm-simple-fp8, c8) --kv-cache-dtype fp8 on top of the winning vllm-simple c8 configuration. ROCm maps fp8 to fp8_e4m3. fp8 halves bytes/token in the GPU pool, which matters here because the pool is the binding constraint: at c8 it peaks at 98.6-99.8% usage and K3 costs ~217 KiB of KV per token, so 1,957,290 tokens should roughly double. Carried on the backend name (vllm-simple-fp8) because the agentic matrix has no per-cell env channel for KV_CACHE_DTYPE. UNVALIDATED on this model: K3's KV pool spans Kimi Delta Attention state and gated-MLA latent, and fp8 support across both spec types is unconfirmed on this build. If it aborts at engine init, that is the answer. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 14 ++++++++++++-- configs/amd-master.yaml | 16 ++++++++++++++++ 2 files changed, 28 insertions(+), 2 deletions(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 98ed2e7c9b..67777b5350 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -409,8 +409,18 @@ except Exception: ) fi ;; - vllm-simple) - require_agentic_kv_offload_backend vllm-simple + vllm-simple|vllm-simple-fp8) + require_agentic_kv_offload_backend "$KV_OFFLOAD_BACKEND" + # vllm-simple-fp8 is the same connector with an fp8 KV cache. fp8 halves + # bytes/token in the GPU pool, which on this KV-bound corpus moves the + # eviction wall itself rather than just adding headroom (measured: the pool + # peaks at 98.6-99.8% usage even at c8). ROCm maps fp8 -> fp8_e4m3. + # UNVALIDATED on K3's hybrid geometry: the pool spans Kimi Delta Attention + # state and gated-MLA latent, and fp8 support across both spec types is + # unconfirmed on this build. + if [ "$KV_OFFLOAD_BACKEND" = "vllm-simple-fp8" ]; then + KV_CACHE_DTYPE="${KV_CACHE_DTYPE:-fp8}" + fi # vLLM's own SimpleCPUOffloadConnector -- the AMD reference's native # offload path. Unlike LMCache it does not go through the Mamba # [conv_state, ssm_state] adapter that K3's Kimi Delta Attention breaks diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 224b56efbb..cbb5de4197 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -678,6 +678,22 @@ kimik3-fp4-mi355x-vllm-agentic-lmprofile: # Retry-only key: single cell, re-dispatched when a bringup cell lands on a # node carrying another tenant's allocation. Delete once it has a result. +# fp8 KV-cache probe: identical to the winning vllm-simple c8 cell except for +# --kv-cache-dtype fp8, which should roughly double the 1,957,290-token GPU pool. +kimik3-fp4-mi355x-vllm-agentic-fp8kv: + image: vllm/vllm-openai-rocm:kimi-k3 + model: moonshotai/Kimi-K3 + model-prefix: kimik3 + runner: cluster:mi355x-amds + precision: fp4 + framework: vllm + multinode: false + scenarios: + agentic-coding: + - dram-utilization: 0.80 + search-space: + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple-fp8 }, conc-list: [8] } + kimik3-fp4-mi355x-vllm-agentic-retry: image: vllm/vllm-openai-rocm:kimi-k3 model: moonshotai/Kimi-K3 From ff2eaad0764962823237c8fe070c1d4ac22f8863 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Wed, 29 Jul 2026 11:02:12 +0900 Subject: [PATCH 22/55] [AMD][AgentX] kimik3: don't enable prefix caching on the no-offload arm Every kvnone cell run with --enable-prefix-caching has died during aiperf warmup with hipErrorIllegalAddress and an EngineCore worker killed by signal. This reproduced on three distinct nodes (mia1-p01-g11, -g17, -g19), so it is a kernel bug rather than bad hardware. K3 is hybrid (69 KDA + 24 gated MLA). Enabling prefix caching activates vLLM's Mamba-state block reuse, and the scheduler dump at the fault names exactly that path: new_block_ids=[([1288],[1284],[1630],[1615])] num_computed_tokens=327936 new_block_ids_to_zero=[1615] The only kvnone cell that has ever completed (run 30322098513, c8) predates the flag being defaulted on. The KV-offload arms are unaffected and keep it on -- they pass at 3600s, and LMCache additionally requires it for mamba_cache_mode='align' -- because the connector owns block lifetime. Default is now arm-aware: on when KV_OFFLOAD_BACKEND is set, off otherwise. PREFIX_CACHING=true/false still forces either way for a deliberate A/B. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 31 ++++++++++++++----- 1 file changed, 24 insertions(+), 7 deletions(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 67777b5350..4b993cbfda 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -535,15 +535,32 @@ fi # and hash sizes only align with prefix caching on -- an omission has been # reported to trip "tokens_per_block not divisible by tokens_per_hash" at load. # Set PREFIX_CACHING=true/false to force it either way. -# ON by default for every arm. Agentic trace replay exists to exercise large -# shared prefixes, so measuring it with reuse disabled is not a useful baseline. -# It also costs essentially no KV (measured: 1,414,660 vs 1,420,824 tokens) and -# improves ITL (484 vs 577 ms). Note vLLM resolves the flag's default to False -# for this model, so it must be passed explicitly. PREFIX_CACHING=false remains -# available for a deliberate A/B. -PREFIX_CACHE_ARGS=(--enable-prefix-caching) +# ON by default for the KV-offload arms, which require it, and OFF for the +# no-offload baseline. Agentic trace replay exists to exercise large shared +# prefixes, and reuse costs essentially no KV (measured: 1,414,660 vs 1,420,824 +# tokens) while improving ITL (484 vs 577 ms) -- so ON is the right default +# wherever it is stable. +# +# It is not stable on the no-offload arm. K3 is a hybrid model (69 KDA layers + +# 24 gated MLA), and enabling prefix caching activates vLLM's Mamba-state block +# reuse path. Every kvnone cell run with --enable-prefix-caching has died in +# warmup with hipErrorIllegalAddress and a scheduler dump naming that path +# (new_block_ids_to_zero), on two different nodes (g11, g19) -- so it is a +# kernel bug, not bad hardware. The one kvnone cell that ever completed +# (run 30322098513, c8) predates the flag. The offload arms pass with it on +# because the connector owns block lifetime instead. +# +# Note vLLM resolves the flag's default to False for this model, so ON must be +# passed explicitly. PREFIX_CACHING=true/false forces either way for an A/B. +if [ -n "${KV_OFFLOAD_BACKEND:-}" ]; then + PREFIX_CACHE_ARGS=(--enable-prefix-caching) +else + PREFIX_CACHE_ARGS=(--no-enable-prefix-caching) +fi if [ "${PREFIX_CACHING:-}" = "false" ]; then PREFIX_CACHE_ARGS=(--no-enable-prefix-caching) +elif [ "${PREFIX_CACHING:-}" = "true" ]; then + PREFIX_CACHE_ARGS=(--enable-prefix-caching) fi # The upstream DSpark config pins "attention_backend": "FLASHINFER_MLA", which From 347cdb76f78c0fd8558f076dfa1e2dcf00543c81 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Wed, 29 Jul 2026 12:39:29 +0900 Subject: [PATCH 23/55] [AMD][AgentX] kimik3: fp8 KV on a b128-tuned MLA image; soften the prefix-caching claim The stock vllm/vllm-openai-rocm:kimi-k3 routes fp8 KV into AITER's mla_gluon decode path, which asserts mla_gluon[bh16bn128] requires batch_size=1, got 128 (run 30411594835). Point the fp8kv key at hyukjleeamd/kimi-k3-mla-b128:latest, which is built with that kernel tuned for batch 128. Also correct the comment added in 836f1f3c1. It claimed every kvnone cell run with --enable-prefix-caching had died in warmup, and that this made it a kernel bug rather than bad hardware. That was overstated on two data points. In the same run, kvnone c1 on g16 cleared warmup with 0 errors and profiled for 47 minutes with the flag on (12 trajectories, 92.8% server prefix-cache hit) before dying to `srun: error: Node failure on mia1-p01-g16` -- infrastructure, not the kernel. Three cells on three nodes produced three different failures. The default stays off for kvnone to remove a variable while we get a clean baseline, but it is now documented as suspected, pending a matched A/B. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 24 ++++++++++++------- configs/amd-master.yaml | 6 ++++- 2 files changed, 21 insertions(+), 9 deletions(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 4b993cbfda..9c5640871a 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -541,14 +541,22 @@ fi # tokens) while improving ITL (484 vs 577 ms) -- so ON is the right default # wherever it is stable. # -# It is not stable on the no-offload arm. K3 is a hybrid model (69 KDA layers + -# 24 gated MLA), and enabling prefix caching activates vLLM's Mamba-state block -# reuse path. Every kvnone cell run with --enable-prefix-caching has died in -# warmup with hipErrorIllegalAddress and a scheduler dump naming that path -# (new_block_ids_to_zero), on two different nodes (g11, g19) -- so it is a -# kernel bug, not bad hardware. The one kvnone cell that ever completed -# (run 30322098513, c8) predates the flag. The offload arms pass with it on -# because the connector owns block lifetime instead. +# It is SUSPECTED-unstable on the no-offload arm -- suspected, not proven, and +# the default is off there only to remove a variable while we get a clean +# baseline. K3 is hybrid (69 KDA layers + 24 gated MLA), and prefix caching +# activates vLLM's Mamba-state block reuse path. Evidence for: +# - kvnone c2 (g19) and c4 (g17), run 30412966635, both died during aiperf +# warmup; c2 with hipErrorIllegalAddress and a scheduler dump naming that +# path (new_block_ids_to_zero=[1615] at num_computed_tokens=327936). +# Evidence against -- this is why it is not settled: +# - kvnone c1 (g16), same run, same flag, cleared warmup with 0 errors and +# profiled 47 min clean (12 trajectories, 92.8% server prefix-cache hit) +# before dying to `srun: error: Node failure on mia1-p01-g16`, which is +# infrastructure, not the kernel. So the arm CAN run with the flag on. +# Three cells on three nodes produced three different failures, so cluster +# flakiness is a live confound. Do not treat this as a diagnosed kernel bug +# until a matched A/B (PREFIX_CACHING=true vs false, same conc) says so. +# The offload arms keep it on and pass at 3600s. # # Note vLLM resolves the flag's default to False for this model, so ON must be # passed explicitly. PREFIX_CACHING=true/false forces either way for an A/B. diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index cbb5de4197..7651655fb2 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -681,7 +681,11 @@ kimik3-fp4-mi355x-vllm-agentic-lmprofile: # fp8 KV-cache probe: identical to the winning vllm-simple c8 cell except for # --kv-cache-dtype fp8, which should roughly double the 1,957,290-token GPU pool. kimik3-fp4-mi355x-vllm-agentic-fp8kv: - image: vllm/vllm-openai-rocm:kimi-k3 + # Custom image with the MLA decode kernel tuned for batch 128. The stock + # vllm/vllm-openai-rocm:kimi-k3 routes fp8 KV into AITER's mla_gluon path, + # which asserts `mla_gluon[bh16bn128] requires batch_size=1, got 128` + # (run 30411594835). This image is built to accept b128 so fp8 KV can run. + image: hyukjleeamd/kimi-k3-mla-b128:latest model: moonshotai/Kimi-K3 model-prefix: kimik3 runner: cluster:mi355x-amds From eeb9d1aaaf598d0fa8a36270d61a05bcaf8c3994 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Wed, 29 Jul 2026 13:23:04 +0900 Subject: [PATCH 24/55] [AMD][AgentX] mi355x-amds: turn the salloc denylist off by default The denylist did not do its job. After excluding g11/g14/g15/g18, run 30412966635 still lost all three kimik3 cells -- to g19 (hipErrorIllegalAddress), g17 (warmup_failure) and g16 (srun node failure). None of those were on the list. Free memory on these nodes moves hour to hour, so a static exclusion mostly shrinks an already-contended pool and lengthens queue time without avoiding the failures it was meant to avoid. Default to no exclusion; set SALLOC_EXCLUDE_NODES explicitly to re-enable it for a specific run. Co-Authored-By: Claude Opus 5 (1M context) --- runners/launch_mi355x-amds.sh | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/runners/launch_mi355x-amds.sh b/runners/launch_mi355x-amds.sh index 2152b17d83..1ace9f517d 100644 --- a/runners/launch_mi355x-amds.sh +++ b/runners/launch_mi355x-amds.sh @@ -274,9 +274,16 @@ else # trips "Free memory on device cuda:N ... is less than desired GPU memory # utilization". It took out kimik3 cells in runs 30330955808 and # 30331297999, and the same node did the same thing to the kimik2.7 -tune2 - # cells. Override with SALLOC_EXCLUDE_NODES (comma-separated, empty to - # disable). - SALLOC_EXCLUDE_NODES="${SALLOC_EXCLUDE_NODES-mia1-p01-g11,mia1-p01-g14,mia1-p01-g15,mia1-p01-g18}" + # cells. + # + # The denylist is OFF by default. It did not work: after excluding + # g11/g14/g15/g18, run 30412966635 still lost all three cells -- to g19 + # (hipErrorIllegalAddress), g17 (warmup_failure) and g16 (srun node + # failure), none of them on the list. Free memory on these nodes varies + # hour to hour, so a static list mostly just shrinks an already-contended + # pool and lengthens queue time. Set SALLOC_EXCLUDE_NODES to a + # comma-separated list to re-enable it for a specific run. + SALLOC_EXCLUDE_NODES="${SALLOC_EXCLUDE_NODES-}" EXCLUDE_ARG="" if [ -n "$SALLOC_EXCLUDE_NODES" ]; then EXCLUDE_ARG="--exclude=$SALLOC_EXCLUDE_NODES" From c62b93842310bd3f1e6965ab32a43e8b5012a801 Mon Sep 17 00:00:00 2001 From: seungrokj Date: Wed, 29 Jul 2026 14:19:44 +0900 Subject: [PATCH 25/55] [AMD][AgentX] kimik3: patch aiter Gluon MLA kernel from gist before serving Co-Authored-By: Claude Opus 4.6 --- benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 9c5640871a..4dce1549cb 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -644,6 +644,18 @@ MAX_NUM_BATCHED_TOKENS="${MAX_NUM_BATCHED_TOKENS:-4096}" echo "Starting vllm server..." export PYTHONNOUSERSITE=1 +# Patch aiter's Gluon MLA kernel with a fixed version. +MLA_GLUON_DST="/usr/local/lib/python3.12/dist-packages/aiter/ops/triton/gluon/mla_gluon.py" +MLA_GLUON_SRC="https://gist.githubusercontent.com/seungrokj/f64cb547829360bfb304f5e794d284ac/raw/mla_gluon.py" +if [ -f "$MLA_GLUON_DST" ]; then + echo "Patching $MLA_GLUON_DST from gist..." + curl --silent --fail --location "$MLA_GLUON_SRC" -o "$MLA_GLUON_DST" \ + && echo "Patched mla_gluon.py" \ + || echo "WARN: failed to patch mla_gluon.py; leaving the image version in place" >&2 +else + echo "WARN: $MLA_GLUON_DST not found; skipping mla_gluon.py patch" >&2 +fi + { set +x; } 2>/dev/null VLLM_CMD=( vllm serve "$MODEL_PATH" --served-model-name "$MODEL" From 845a0ece66e4814164a0f370a91e6630a48fe609 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Wed, 29 Jul 2026 14:44:12 +0900 Subject: [PATCH 26/55] Revert "[AMD][AgentX] kimik3: don't enable prefix caching on the no-offload arm" Prefix caching is back ON for every arm. Turning it off for kvnone was the wrong call and the evidence never supported it. The trace is built around large shared prefixes -- 98.1% theoretical hit rate, 92.8% measured server-side on a live cell -- so a kvnone run with reuse disabled is not measuring the target workload, and it makes the kvnone-vs-offload comparison at matched concurrency meaningless. The justification for switching it off was two cells (c2/g19, c4/g17, run 30412966635) dying in warmup with the flag on. But c1 in that same run, with the same flag, cleared warmup with 0 errors and profiled 47 minutes clean before dying only to `srun: error: Node failure on mia1-p01-g16`. Weighting two failures over an in-sample counterexample was a mistake, especially in a week where the cluster produced three different failure modes across three nodes. PREFIX_CACHING=false still forces it off for a deliberate A/B. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 47 +++++++------------ 1 file changed, 18 insertions(+), 29 deletions(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 4dce1549cb..02fabb118a 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -535,40 +535,29 @@ fi # and hash sizes only align with prefix caching on -- an omission has been # reported to trip "tokens_per_block not divisible by tokens_per_hash" at load. # Set PREFIX_CACHING=true/false to force it either way. -# ON by default for the KV-offload arms, which require it, and OFF for the -# no-offload baseline. Agentic trace replay exists to exercise large shared -# prefixes, and reuse costs essentially no KV (measured: 1,414,660 vs 1,420,824 -# tokens) while improving ITL (484 vs 577 ms) -- so ON is the right default -# wherever it is stable. +# ON by default for EVERY arm. This trace is built around large shared +# prefixes -- theoretical prefix-cache hit is 98.1%, and a live kvnone cell +# measured 92.8% server-side -- so a run with reuse disabled is not measuring +# the workload. Reuse also costs essentially no KV (1,414,660 vs 1,420,824 +# tokens) and improves ITL (484 vs 577 ms). The offload arms additionally +# require it: LMCache needs mamba_cache_mode='align', which vLLM only selects +# when prefix caching is on. Turning it off for kvnone alone would also make +# the kvnone-vs-offload comparison at matched concurrency meaningless. # -# It is SUSPECTED-unstable on the no-offload arm -- suspected, not proven, and -# the default is off there only to remove a variable while we get a clean -# baseline. K3 is hybrid (69 KDA layers + 24 gated MLA), and prefix caching -# activates vLLM's Mamba-state block reuse path. Evidence for: -# - kvnone c2 (g19) and c4 (g17), run 30412966635, both died during aiperf -# warmup; c2 with hipErrorIllegalAddress and a scheduler dump naming that -# path (new_block_ids_to_zero=[1615] at num_computed_tokens=327936). -# Evidence against -- this is why it is not settled: -# - kvnone c1 (g16), same run, same flag, cleared warmup with 0 errors and -# profiled 47 min clean (12 trajectories, 92.8% server prefix-cache hit) -# before dying to `srun: error: Node failure on mia1-p01-g16`, which is -# infrastructure, not the kernel. So the arm CAN run with the flag on. -# Three cells on three nodes produced three different failures, so cluster -# flakiness is a live confound. Do not treat this as a diagnosed kernel bug -# until a matched A/B (PREFIX_CACHING=true vs false, same conc) says so. -# The offload arms keep it on and pass at 3600s. +# It was briefly defaulted off for kvnone after two cells (c2/g19, c4/g17, +# run 30412966635) died in warmup with it on, one dump naming the Mamba +# block-zeroing path (new_block_ids_to_zero=[1615] at 327,936 computed +# tokens). That was the wrong call on the evidence: c1 in the SAME run, same +# flag, cleared warmup with 0 errors and profiled 47 minutes clean before +# dying only to `srun: error: Node failure on mia1-p01-g16`. The arm plainly +# runs with the flag on, and the cluster was throwing three different +# failures across three nodes that week. Reverted. # # Note vLLM resolves the flag's default to False for this model, so ON must be -# passed explicitly. PREFIX_CACHING=true/false forces either way for an A/B. -if [ -n "${KV_OFFLOAD_BACKEND:-}" ]; then - PREFIX_CACHE_ARGS=(--enable-prefix-caching) -else - PREFIX_CACHE_ARGS=(--no-enable-prefix-caching) -fi +# passed explicitly. PREFIX_CACHING=false forces it off for a deliberate A/B. +PREFIX_CACHE_ARGS=(--enable-prefix-caching) if [ "${PREFIX_CACHING:-}" = "false" ]; then PREFIX_CACHE_ARGS=(--no-enable-prefix-caching) -elif [ "${PREFIX_CACHING:-}" = "true" ]; then - PREFIX_CACHE_ARGS=(--enable-prefix-caching) fi # The upstream DSpark config pins "attention_backend": "FLASHINFER_MLA", which From a15e7d5a2295c74f509d13b5d7cfee93bd94803b Mon Sep 17 00:00:00 2001 From: seungrokj Date: Wed, 29 Jul 2026 15:36:32 +0900 Subject: [PATCH 27/55] [AMD][AgentX] kimik3: widen worker RPC timeout for long-context fp8-KV arm Long-context forward passes (~370K tokens with fp8 KV + DRAM offload) exceed vLLM's default 300s VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS, killing the engine with "RPC call to sample_tokens timed out". Default it to 1200s (overridable). Co-Authored-By: Claude Opus 4.6 --- benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 02fabb118a..1c00633e1d 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -633,6 +633,11 @@ MAX_NUM_BATCHED_TOKENS="${MAX_NUM_BATCHED_TOKENS:-4096}" echo "Starting vllm server..." export PYTHONNOUSERSITE=1 +# Long-context forward passes (~370K tokens with fp8 KV + DRAM offload) can exceed +# vLLM's default 300s worker RPC timeout, killing the engine with +# "RPC call to sample_tokens timed out". Widen it. +export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS="${VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS:-1200}" + # Patch aiter's Gluon MLA kernel with a fixed version. MLA_GLUON_DST="/usr/local/lib/python3.12/dist-packages/aiter/ops/triton/gluon/mla_gluon.py" MLA_GLUON_SRC="https://gist.githubusercontent.com/seungrokj/f64cb547829360bfb304f5e794d284ac/raw/mla_gluon.py" From e193dbbde103a31528a588a3ce59d60e9c400d26 Mon Sep 17 00:00:00 2001 From: seungrokj Date: Wed, 29 Jul 2026 16:16:29 +0900 Subject: [PATCH 28/55] [AMD][AgentX] kimik3: add bf16-KV offload isolation arm; disable RPC-timeout+mla_gluon patches Isolation A/B for the fp8-KV DRAM-offload TP-rank hang: bf16kvtest keeps DRAM offload (vllm-simple) but drops fp8 KV on the stock image to test whether fp8 KV is the stressor wedging the collective. Co-Authored-By: Claude Opus 4.6 --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 32 +++++++++---------- configs/amd-master.yaml | 18 +++++++++++ 2 files changed, 34 insertions(+), 16 deletions(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 1c00633e1d..d2ab770cf6 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -633,22 +633,22 @@ MAX_NUM_BATCHED_TOKENS="${MAX_NUM_BATCHED_TOKENS:-4096}" echo "Starting vllm server..." export PYTHONNOUSERSITE=1 -# Long-context forward passes (~370K tokens with fp8 KV + DRAM offload) can exceed -# vLLM's default 300s worker RPC timeout, killing the engine with -# "RPC call to sample_tokens timed out". Widen it. -export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS="${VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS:-1200}" - -# Patch aiter's Gluon MLA kernel with a fixed version. -MLA_GLUON_DST="/usr/local/lib/python3.12/dist-packages/aiter/ops/triton/gluon/mla_gluon.py" -MLA_GLUON_SRC="https://gist.githubusercontent.com/seungrokj/f64cb547829360bfb304f5e794d284ac/raw/mla_gluon.py" -if [ -f "$MLA_GLUON_DST" ]; then - echo "Patching $MLA_GLUON_DST from gist..." - curl --silent --fail --location "$MLA_GLUON_SRC" -o "$MLA_GLUON_DST" \ - && echo "Patched mla_gluon.py" \ - || echo "WARN: failed to patch mla_gluon.py; leaving the image version in place" >&2 -else - echo "WARN: $MLA_GLUON_DST not found; skipping mla_gluon.py patch" >&2 -fi +## Long-context forward passes (~370K tokens with fp8 KV + DRAM offload) can exceed +## vLLM's default 300s worker RPC timeout, killing the engine with +## "RPC call to sample_tokens timed out". Widen it. +#export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS="${VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS:-1200}" +# +## Patch aiter's Gluon MLA kernel with a fixed version. +#MLA_GLUON_DST="/usr/local/lib/python3.12/dist-packages/aiter/ops/triton/gluon/mla_gluon.py" +#MLA_GLUON_SRC="https://gist.githubusercontent.com/seungrokj/f64cb547829360bfb304f5e794d284ac/raw/mla_gluon.py" +#if [ -f "$MLA_GLUON_DST" ]; then +# echo "Patching $MLA_GLUON_DST from gist..." +# curl --silent --fail --location "$MLA_GLUON_SRC" -o "$MLA_GLUON_DST" \ +# && echo "Patched mla_gluon.py" \ +# || echo "WARN: failed to patch mla_gluon.py; leaving the image version in place" >&2 +#else +# echo "WARN: $MLA_GLUON_DST not found; skipping mla_gluon.py patch" >&2 +#fi { set +x; } 2>/dev/null VLLM_CMD=( diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 7651655fb2..b505c293e9 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -652,6 +652,24 @@ kimik3-fp4-mi355x-vllm-agentic: # far the 2234 GiB host tier extends the concurrency ceiling. - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [4, 8, 16, 32] } +kimik3-fp4-mi355x-vllm-agentic-bf16kvtest: + image: vllm/vllm-openai-rocm:kimi-k3 + model: moonshotai/Kimi-K3 + model-prefix: kimik3 + runner: cluster:mi355x-amds + precision: fp4 + framework: vllm + multinode: false + scenarios: + agentic-coding: + # 0.80 of the runner's available-cpu-dram-mib (3_095_781), capped by the + # matrix generator at the 2,861,022 MiB / 3 TB-decimal agentic limit -> + # 2399 GB aggregate at TP8, which SimpleCPUOffloadConnector divides into + # 279.28 GB per rank. + - dram-utilization: 0.80 + search-space: + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [8] } + # LMCache server-profile A/B for Kimi-K3. NOT shipped -- delete once one profile # wins. Both cells are conc 8 with prefix caching forced on, --max-num-batched-tokens # 768 and --chunk-size 768 (all three are hard Mamba-hybrid requirements the recipe From 8334611004d2ec91c9df610d5351c2869c0b04f0 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Wed, 29 Jul 2026 17:13:10 +0900 Subject: [PATCH 29/55] [AMD][AgentX] kimik3: point the fp8-KV cell at the fp8-kv-fixed image hyukjleeamd/kimi-k3-mla-b128:fp8-kv-fixed replaces :latest for the vllm-simple-fp8 c8 cell. Co-Authored-By: Claude Opus 5 (1M context) --- configs/amd-master.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index b505c293e9..cd0ef5f445 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -703,7 +703,7 @@ kimik3-fp4-mi355x-vllm-agentic-fp8kv: # vllm/vllm-openai-rocm:kimi-k3 routes fp8 KV into AITER's mla_gluon path, # which asserts `mla_gluon[bh16bn128] requires batch_size=1, got 128` # (run 30411594835). This image is built to accept b128 so fp8 KV can run. - image: hyukjleeamd/kimi-k3-mla-b128:latest + image: hyukjleeamd/kimi-k3-mla-b128:fp8-kv-fixed model: moonshotai/Kimi-K3 model-prefix: kimik3 runner: cluster:mi355x-amds From 36b43dbd5c96b98c763511beb65167952560a1f4 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Wed, 29 Jul 2026 18:15:49 +0900 Subject: [PATCH 30/55] [AMD][AgentX] mi355x-amds: exclude mia1-p01-g11 from salloc by default g11 is not a free-memory-headroom case like the nodes that made the earlier broad denylist useless. It fails hard, across unrelated configs: - kimik3 kvnone c2 (run 30425820793): warmup_failure - kimik3 vllm-simple-fp8 c8 (run 30428781263): forward passes stalled with returned=5 / in_flight=2 at zero throughput for 13 minutes, then EngineCore died - the kimik2.7 -tune2 cells noted above Four configs, one node, same outcome. Excluding a single node costs almost no pool capacity, unlike the g11/g14/g15/g18 list that was rightly turned off. Kept as ${VAR-default} so SALLOC_EXCLUDE_NODES="" still disables it. Co-Authored-By: Claude Opus 5 (1M context) --- runners/launch_mi355x-amds.sh | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/runners/launch_mi355x-amds.sh b/runners/launch_mi355x-amds.sh index 1ace9f517d..6e3769080f 100644 --- a/runners/launch_mi355x-amds.sh +++ b/runners/launch_mi355x-amds.sh @@ -276,14 +276,22 @@ else # 30331297999, and the same node did the same thing to the kimik2.7 -tune2 # cells. # - # The denylist is OFF by default. It did not work: after excluding - # g11/g14/g15/g18, run 30412966635 still lost all three cells -- to g19 - # (hipErrorIllegalAddress), g17 (warmup_failure) and g16 (srun node - # failure), none of them on the list. Free memory on these nodes varies - # hour to hour, so a static list mostly just shrinks an already-contended - # pool and lengthens queue time. Set SALLOC_EXCLUDE_NODES to a - # comma-separated list to re-enable it for a specific run. - SALLOC_EXCLUDE_NODES="${SALLOC_EXCLUDE_NODES-}" + # A BROAD denylist did not work: after excluding g11/g14/g15/g18, run + # 30412966635 still lost all three cells -- to g19 (hipErrorIllegalAddress), + # g17 (warmup_failure) and g16 (srun node failure), none of them on the + # list. Free memory on these nodes varies hour to hour, so a wide static + # list mostly just shrinks an already-contended pool and lengthens queue + # time. Don't reintroduce one. + # + # g11 is the exception and is excluded by default. It is not a + # free-memory-headroom case; it fails hard and across unrelated configs: + # - kimik3 kvnone c2, run 30425820793: warmup_failure + # - kimik3 vllm-simple-fp8 c8, run 30428781263: forward passes stalled + # (returned=5 for 13 min at zero throughput), then EngineCore died + # - plus the earlier kimik2.7 -tune2 cells noted above + # Four configs, one node, same outcome. Set SALLOC_EXCLUDE_NODES="" to + # clear it, or to a comma-separated list to widen it for a specific run. + SALLOC_EXCLUDE_NODES="${SALLOC_EXCLUDE_NODES-mia1-p01-g11}" EXCLUDE_ARG="" if [ -n "$SALLOC_EXCLUDE_NODES" ]; then EXCLUDE_ARG="--exclude=$SALLOC_EXCLUDE_NODES" From 430e8cf7611e46cb5f13b3b58c29aedb788c78a7 Mon Sep 17 00:00:00 2001 From: seungrokj Date: Wed, 29 Jul 2026 18:24:22 +0900 Subject: [PATCH 31/55] [AMD][AgentX] kimik3: add fp8kvtest arm (fp8 KV, no offload) for isolation Isolation variant B: fp8 KV without DRAM offload to test whether the offload path is what wedges the TP collective. Backend is none, so KV_CACHE_DTYPE is forced to fp8 in the recipe; the mla_gluon patch is gated on fp8 KV. Co-Authored-By: Claude Opus 4.6 --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 37 +++++++++++-------- configs/amd-master.yaml | 4 +- 2 files changed, 23 insertions(+), 18 deletions(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index d2ab770cf6..7ec16d3bee 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -178,6 +178,8 @@ wait_for_lmcache_ready() { # benchmarks/single_node/agentic/README.md it must be consumed as given and # never replaced with a model-specific constant. OFFLOAD_ARGS=() +# (srok), enforce fp8 kv +KV_CACHE_DTYPE="fp8" if agentic_kv_offload_enabled; then case "${KV_OFFLOAD_BACKEND:-}" in @@ -633,22 +635,25 @@ MAX_NUM_BATCHED_TOKENS="${MAX_NUM_BATCHED_TOKENS:-4096}" echo "Starting vllm server..." export PYTHONNOUSERSITE=1 -## Long-context forward passes (~370K tokens with fp8 KV + DRAM offload) can exceed -## vLLM's default 300s worker RPC timeout, killing the engine with -## "RPC call to sample_tokens timed out". Widen it. -#export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS="${VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS:-1200}" -# -## Patch aiter's Gluon MLA kernel with a fixed version. -#MLA_GLUON_DST="/usr/local/lib/python3.12/dist-packages/aiter/ops/triton/gluon/mla_gluon.py" -#MLA_GLUON_SRC="https://gist.githubusercontent.com/seungrokj/f64cb547829360bfb304f5e794d284ac/raw/mla_gluon.py" -#if [ -f "$MLA_GLUON_DST" ]; then -# echo "Patching $MLA_GLUON_DST from gist..." -# curl --silent --fail --location "$MLA_GLUON_SRC" -o "$MLA_GLUON_DST" \ -# && echo "Patched mla_gluon.py" \ -# || echo "WARN: failed to patch mla_gluon.py; leaving the image version in place" >&2 -#else -# echo "WARN: $MLA_GLUON_DST not found; skipping mla_gluon.py patch" >&2 -#fi +# Long-context forward passes (~370K tokens with fp8 KV + DRAM offload) can exceed +# vLLM's default 300s worker RPC timeout, killing the engine with +# "RPC call to sample_tokens timed out". Widen it. +export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS="${VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS:-1200}" + +# Patch aiter's Gluon MLA kernel with a fixed version. The b128 kernel is only +# exercised on the fp8 KV path, so only patch when KV_CACHE_DTYPE is fp8. +if [ "${KV_CACHE_DTYPE:-}" = "fp8" ]; then + MLA_GLUON_DST="/usr/local/lib/python3.12/dist-packages/aiter/ops/triton/gluon/mla_gluon.py" + MLA_GLUON_SRC="https://gist.githubusercontent.com/seungrokj/f64cb547829360bfb304f5e794d284ac/raw/mla_gluon.py" + if [ -f "$MLA_GLUON_DST" ]; then + echo "Patching $MLA_GLUON_DST from gist..." + curl --silent --fail --location "$MLA_GLUON_SRC" -o "$MLA_GLUON_DST" \ + && echo "Patched mla_gluon.py" \ + || echo "WARN: failed to patch mla_gluon.py; leaving the image version in place" >&2 + else + echo "WARN: $MLA_GLUON_DST not found; skipping mla_gluon.py patch" >&2 + fi +fi { set +x; } 2>/dev/null VLLM_CMD=( diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index cd0ef5f445..30defcc526 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -652,7 +652,7 @@ kimik3-fp4-mi355x-vllm-agentic: # far the 2234 GiB host tier extends the concurrency ceiling. - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [4, 8, 16, 32] } -kimik3-fp4-mi355x-vllm-agentic-bf16kvtest: +kimik3-fp4-mi355x-vllm-agentic-fp8kvtest: image: vllm/vllm-openai-rocm:kimi-k3 model: moonshotai/Kimi-K3 model-prefix: kimik3 @@ -668,7 +668,7 @@ kimik3-fp4-mi355x-vllm-agentic-bf16kvtest: # 279.28 GB per rank. - dram-utilization: 0.80 search-space: - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [8] } + - { tp: 8, kv-offloading: dram, kv-offload-backend: none, conc-list: [8] } # LMCache server-profile A/B for Kimi-K3. NOT shipped -- delete once one profile # wins. Both cells are conc 8 with prefix caching forced on, --max-num-batched-tokens From e17f08461aadc77781b1f04bb5a3b874741d97d0 Mon Sep 17 00:00:00 2001 From: seungrokj Date: Wed, 29 Jul 2026 18:26:05 +0900 Subject: [PATCH 32/55] [AMD][AgentX] kimik3: disable forced fp8 KV hardcode Co-Authored-By: Claude Opus 4.6 --- benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 7ec16d3bee..e404b06205 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -179,7 +179,7 @@ wait_for_lmcache_ready() { # never replaced with a model-specific constant. OFFLOAD_ARGS=() # (srok), enforce fp8 kv -KV_CACHE_DTYPE="fp8" +#KV_CACHE_DTYPE="fp8" if agentic_kv_offload_enabled; then case "${KV_OFFLOAD_BACKEND:-}" in From 606584c65bf37834e67d2f3bc202894360cec1e5 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Wed, 29 Jul 2026 21:42:36 +0900 Subject: [PATCH 33/55] [AMD][AgentX] kimik3: pin the mla_gluon gist, point fp8kv at the proven stack Run 30442578333 produced the first clean fp8 KV result on Kimi-K3 (c8, 3600s, kvnone, stock image): 32.59 output tok/s, 696 total tok/s/GPU, 155 requests, TTFT 2.6s. What made it work was the recipe patching aiter's mla_gluon kernel from seungrokj's gist plus the widened worker RPC timeout -- not a custom image. Two changes so that result is reproducible and extendable to the offload arm: 1. Pin MLA_GLUON_SRC to gist revision 4b0088c5f, the exact blob that run fetched. The unpinned .../raw/mla_gluon.py URL follows the gist HEAD, so an edit there would swap the decode kernel out from under every fp8 cell with no diff in this repo. 2. Point -fp8kv at the stock vllm/vllm-openai-rocm:kimi-k3 rather than hyukjleeamd/kimi-k3-mla-b128:fp8-kv-fixed. The recipe patches the kernel whenever KV_CACHE_DTYPE=fp8, so stock now reaches the same code path, and this keeps the cell exactly one variable (the SimpleCPUOffloadConnector) away from the run that passed. The custom image remains the fallback: its one trial (30434731504) cleared init and the full warmup, then died at the profiling transition with an async hipErrorIllegalAddress on g14 that was never attributed to a kernel. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 6 ++++- configs/amd-master.yaml | 23 +++++++++++++++---- 2 files changed, 23 insertions(+), 6 deletions(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index e404b06205..a6c080606d 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -644,7 +644,11 @@ export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS="${VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS: # exercised on the fp8 KV path, so only patch when KV_CACHE_DTYPE is fp8. if [ "${KV_CACHE_DTYPE:-}" = "fp8" ]; then MLA_GLUON_DST="/usr/local/lib/python3.12/dist-packages/aiter/ops/triton/gluon/mla_gluon.py" - MLA_GLUON_SRC="https://gist.githubusercontent.com/seungrokj/f64cb547829360bfb304f5e794d284ac/raw/mla_gluon.py" + # Pinned to the exact gist revision that produced the first clean fp8 KV + # run (30442578333: c8, 3600s, 696 total tok/s/GPU, TTFT 2.6s). The + # unpinned .../raw/mla_gluon.py URL silently follows the gist HEAD, so a + # later edit would change the kernel under us with no diff in this repo. + MLA_GLUON_SRC="https://gist.githubusercontent.com/seungrokj/f64cb547829360bfb304f5e794d284ac/raw/4b0088c5fecbeffa6544d2da1006b45380aac896/mla_gluon.py" if [ -f "$MLA_GLUON_DST" ]; then echo "Patching $MLA_GLUON_DST from gist..." curl --silent --fail --location "$MLA_GLUON_SRC" -o "$MLA_GLUON_DST" \ diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 30defcc526..c929914f62 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -699,11 +699,24 @@ kimik3-fp4-mi355x-vllm-agentic-lmprofile: # fp8 KV-cache probe: identical to the winning vllm-simple c8 cell except for # --kv-cache-dtype fp8, which should roughly double the 1,957,290-token GPU pool. kimik3-fp4-mi355x-vllm-agentic-fp8kv: - # Custom image with the MLA decode kernel tuned for batch 128. The stock - # vllm/vllm-openai-rocm:kimi-k3 routes fp8 KV into AITER's mla_gluon path, - # which asserts `mla_gluon[bh16bn128] requires batch_size=1, got 128` - # (run 30411594835). This image is built to accept b128 so fp8 KV can run. - image: hyukjleeamd/kimi-k3-mla-b128:fp8-kv-fixed + # fp8 KV on the OFFLOAD arm. This is the fp8kvtest arm (run 30442578333: + # 696 total tok/s/GPU, TTFT 2.6s, the first clean fp8 result) plus the + # SimpleCPUOffloadConnector, so it isolates what the connector costs under + # fp8. + # + # Stock image on purpose. The stock image routes fp8 KV into AITER's + # mla_gluon path and asserts `mla_gluon[bh16bn128] requires batch_size=1, + # got 128` (run 30411594835) -- but the recipe now patches that kernel from + # a pinned gist revision whenever KV_CACHE_DTYPE=fp8, which is exactly how + # 30442578333 passed. Using stock keeps this cell one variable away from + # that proven run. + # + # hyukjleeamd/kimi-k3-mla-b128:fp8-kv-fixed bakes the same kernel in and is + # the alternative, but its only trial (run 30434731504) cleared init and the + # full 600s warmup and then died at the profiling transition with an async + # hipErrorIllegalAddress on g14 -- unattributed, possibly the node. Swap the + # image back to compare once this cell has a number. + image: vllm/vllm-openai-rocm:kimi-k3 model: moonshotai/Kimi-K3 model-prefix: kimik3 runner: cluster:mi355x-amds From 1f7ce51c651c132caa94e7400bf30efb8968499f Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Wed, 29 Jul 2026 21:50:36 +0900 Subject: [PATCH 34/55] [AMD][AgentX] kimik3: make fp8 KV the default and add the full fp8 sweep fp8 KV is the biggest lever measured on this model. Run 30442578333 (c8, kvnone, 3600s, stock image + patched mla_gluon) scored 32.59 output tok/s and 696 total tok/s/GPU at TTFT 2.6s, against 19.17 / 535 / 69.1s for the bf16 offload cell in the same window. K3 costs ~217 KiB KV/token, roughly 3x K2.7, so halving the KV element size matters more here than it did there. KV_CACHE_DTYPE now defaults to fp8 for every arm, set before the KV_OFFLOAD_BACKEND case. That placement is required, not cosmetic: the agentic matrix has no per-cell env channel, so a kvnone cell has no other way to request fp8, and the mla_gluon patch gates on KV_CACHE_DTYPE=fp8 and has to see it already set. KV_CACHE_DTYPE=auto still gives a bf16 A/B. Adds -fp8sweep: kvnone c[1,4,8] plus vllm-simple-fp8 c[8,16]. The kvnone c8 cell repeats 30442578333 as a control; c1/c4 replace the bf16 points (159 and 421 tok/s/GPU, run 30430688459) so the curve is one dtype throughout. The offload cells isolate what SimpleCPUOffloadConnector costs under fp8, and c16 probes whether fp8 moves the ceiling past the bf16 collapse there. Prefix caching stays on for every arm, unchanged. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 19 +++++++++++-- configs/amd-master.yaml | 28 +++++++++++++++++++ 2 files changed, 45 insertions(+), 2 deletions(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index a6c080606d..6b6b3a4be2 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -178,8 +178,23 @@ wait_for_lmcache_ready() { # benchmarks/single_node/agentic/README.md it must be consumed as given and # never replaced with a model-specific constant. OFFLOAD_ARGS=() -# (srok), enforce fp8 kv -#KV_CACHE_DTYPE="fp8" + +# fp8 KV is the DEFAULT for every arm on this model. +# +# It is the first thing that made the trace run well: run 30442578333 (c8, +# kvnone, 3600s) scored 32.59 output tok/s / 696 total tok/s/GPU at TTFT 2.6s, +# against 19.17 / 535 / 69.1s for the bf16 offload cell in the same window. +# K3 costs ~217 KiB KV/token -- 3x K2.7 -- so halving the KV element size is +# the single largest lever available, exactly as in +# [[fp8-kv-cache-mandatory]] for kimik2.7. +# +# This is set here, before the KV_OFFLOAD_BACKEND case below, for two reasons: +# the agentic matrix has no per-cell env channel (so a kvnone cell has no other +# way to ask for fp8), and the mla_gluon patch further down gates on +# KV_CACHE_DTYPE=fp8 and must see it already set. +# +# Set KV_CACHE_DTYPE=auto for a bf16 A/B. +KV_CACHE_DTYPE="${KV_CACHE_DTYPE:-fp8}" if agentic_kv_offload_enabled; then case "${KV_OFFLOAD_BACKEND:-}" in diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index c929914f62..5e69a5c8f6 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -698,6 +698,34 @@ kimik3-fp4-mi355x-vllm-agentic-lmprofile: # node carrying another tenant's allocation. Delete once it has a result. # fp8 KV-cache probe: identical to the winning vllm-simple c8 cell except for # --kv-cache-dtype fp8, which should roughly double the 1,957,290-token GPU pool. +kimik3-fp4-mi355x-vllm-agentic-fp8sweep: + # Full fp8-KV sweep. Supersedes the bf16 numbers: fp8 is now the recipe + # default (see kimik3_fp4_mi355x.sh), so every cell here runs + # --kv-cache-dtype fp8 with prefix caching on, and the mla_gluon kernel is + # patched from the pinned gist revision that run 30442578333 used. + # + # Stock image on purpose -- 30442578333 proved the patched stock image runs + # fp8 clean; the custom hyukjleeamd/kimi-k3-mla-b128 image is the fallback. + image: vllm/vllm-openai-rocm:kimi-k3 + model: moonshotai/Kimi-K3 + model-prefix: kimik3 + runner: cluster:mi355x-amds + precision: fp4 + framework: vllm + multinode: false + scenarios: + agentic-coding: + - dram-utilization: 0.80 + search-space: + # No-offload arm. c8 repeats 30442578333 (696 tok/s/GPU) as a control; + # c1/c4 replace the bf16 points (159 / 421 tok/s/GPU) from run + # 30430688459 so the whole curve is one KV dtype. + - { tp: 8, kv-offloading: none, conc-list: [1, 4, 8] } + # Offload arm at the same dtype -- isolates what + # SimpleCPUOffloadConnector costs or buys under fp8. c16 probes whether + # fp8 pushes the concurrency ceiling past the bf16 collapse at c16. + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple-fp8 }, conc-list: [8, 16] } + kimik3-fp4-mi355x-vllm-agentic-fp8kv: # fp8 KV on the OFFLOAD arm. This is the fp8kvtest arm (run 30442578333: # 696 total tok/s/GPU, TTFT 2.6s, the first clean fp8 result) plus the From 124599fcf67d106d89bf802ceb2bf4c39cb0d518 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Wed, 29 Jul 2026 21:53:58 +0900 Subject: [PATCH 35/55] [AMD][AgentX] kimik3: fix fp8kvtest search-space, it was failing validation `kv-offload-backend: none` parses as the STRING 'none', which pydantic rejects against KVOffloadBackendMetadata. validate_master_config validates the whole file, so this one cell was failing every dispatch off this branch, not just the fp8kvtest key (run 30453389265, get-jobs). The arm wants no offload at all, which is `kv-offloading: none` with no backend key -- the form on amd/agentx_k3_vllm_0729 that run 30442578333 actually executed. The cherry-pick into this branch (cbb505d03) changed it to kv-offloading: dram plus a string backend, which never validated here. Co-Authored-By: Claude Opus 5 (1M context) --- configs/amd-master.yaml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 5e69a5c8f6..c2c2242d60 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -668,7 +668,10 @@ kimik3-fp4-mi355x-vllm-agentic-fp8kvtest: # 279.28 GB per rank. - dram-utilization: 0.80 search-space: - - { tp: 8, kv-offloading: dram, kv-offload-backend: none, conc-list: [8] } + # kv-offloading: none, NOT `kv-offload-backend: none` -- YAML parses a + # bare `none` as the string 'none', which fails KVOffloadBackendMetadata + # validation and takes the whole master config down with it. + - { tp: 8, kv-offloading: none, conc-list: [8] } # LMCache server-profile A/B for Kimi-K3. NOT shipped -- delete once one profile # wins. Both cells are conc 8 with prefix caching forced on, --max-num-batched-tokens From a5dd587b51cd9b7c86b15164edeb3249bec6a648 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Wed, 29 Jul 2026 22:45:47 +0900 Subject: [PATCH 36/55] [AMD][AgentX] kimik3: add fp8retry arm (none c[1,16] + vllm-simple-fp8 c8) Retries the two cells that died in run 30453589555 and adds none c16. Both deaths were GPU faults (hipErrorUnknown), not timeouts. No guard was close to firing: VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1200 against a ~180s stall, warmup-grace-period 1800 against 420s elapsed, and the only "timeout" strings in the server log are source lines inside the traceback. A timeout would also not surface as hipErrorUnknown. GPU faults have hit every arm on this cluster today -- bf16 kvnone on g11/g17/g18/g19, fp8 kvnone on g10, fp8 + offload connector on g11 and g19 -- so both cells are retried unchanged rather than "fixed". The offload cell was measurably slower before it died (2.7x longer to 5 completions than the no-connector run at the same concurrency), but slowness did not kill it, and that is a throughput question to settle with a number rather than a config change. none c16 is new. The no-offload arm has never been measured above c8, and fp8 roughly halves KV bytes per token, so the concurrency ceiling should have moved; bf16 collapsed at c16 (588 tok/s/GPU, TTFT 295s) and this checks whether fp8 changes that. Also corrects the KV_CACHE_DTYPE header comment, which still claimed a default of auto after acd6d3026 made fp8 the default. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 2 +- configs/amd-master.yaml | 27 +++++++++++++++++++ 2 files changed, 28 insertions(+), 1 deletion(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 6b6b3a4be2..409052192a 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -45,7 +45,7 @@ set -x # MAX_NUM_BATCHED_TOKENS 4096 (reference) # AITER_A8W4 1 (reference; 0 = aiter a16w4 MoE path) # LANGUAGE_MODEL_ONLY false (reference loads the vision tower) -# KV_CACHE_DTYPE auto (unset -> flag not passed at all) +# KV_CACHE_DTYPE fp8 (default for every arm; =auto for a bf16 A/B) # MAX_MODEL_LEN unset (unset -> vLLM derives K3's 1M context) # SPEC_DECODE false (DSpark; UNVALIDATED on ROCm) diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index c2c2242d60..dedeed6020 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -729,6 +729,33 @@ kimik3-fp4-mi355x-vllm-agentic-fp8sweep: # fp8 pushes the concurrency ceiling past the bf16 collapse at c16. - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple-fp8 }, conc-list: [8, 16] } +kimik3-fp4-mi355x-vllm-agentic-fp8retry: + # Retry of the two cells that died in run 30453589555, plus a new none c16. + # + # Both deaths were GPU faults (hipErrorUnknown), not timeouts -- no guard was + # anywhere near firing (VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1200 vs a ~180s + # stall; warmup-grace-period 1800 vs 420s elapsed). GPU faults have hit every + # arm on this cluster today (bf16 kvnone on g11/g17/g18/g19, fp8 kvnone on + # g10, fp8+connector on g11/g19), so these are retried unchanged on the + # assumption they are the cluster's background fault rate rather than config. + # + # none c16 is new: the no-offload arm has never been measured above c8, and + # fp8 roughly halves KV per token, so the concurrency ceiling should have + # moved up. + image: vllm/vllm-openai-rocm:kimi-k3 + model: moonshotai/Kimi-K3 + model-prefix: kimik3 + runner: cluster:mi355x-amds + precision: fp4 + framework: vllm + multinode: false + scenarios: + agentic-coding: + - dram-utilization: 0.80 + search-space: + - { tp: 8, kv-offloading: none, conc-list: [1, 16] } + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple-fp8 }, conc-list: [8] } + kimik3-fp4-mi355x-vllm-agentic-fp8kv: # fp8 KV on the OFFLOAD arm. This is the fp8kvtest arm (run 30442578333: # 696 total tok/s/GPU, TTFT 2.6s, the first clean fp8 result) plus the From c192d177ef8a2cafed0a373753e21687b2c31811 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Wed, 29 Jul 2026 15:11:25 -0700 Subject: [PATCH 37/55] feat(amd): add Kimi-K3 DSpark AgentX variant MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add the MI355X DSpark wrapper, TP8 vLLM configuration, synthetic golden-AL handling, and performance changelog entry. 中文：新增 MI355X Kimi-K3 DSpark AgentX 包装脚本与 TP8 vLLM 配置，接入合成黄金接受长度，并登记性能变更日志。 --- .github/workflows/e2e-tests.yml | 5 +-- .../single_node/agentic/kimik3_fp4_mi355x.sh | 36 ++++++++++++++++--- .../agentic/kimik3_fp4_mi355x_mtp.sh | 20 +++++++++++ configs/amd-master.yaml | 19 ++++++++++ perf-changelog.yaml | 8 +++++ .../test_run_sweep_gating.py | 17 +++++++++ 6 files changed, 98 insertions(+), 7 deletions(-) create mode 100755 benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh diff --git a/.github/workflows/e2e-tests.yml b/.github/workflows/e2e-tests.yml index 34fba860f4..1327274399 100644 --- a/.github/workflows/e2e-tests.yml +++ b/.github/workflows/e2e-tests.yml @@ -310,14 +310,15 @@ jobs: dp-attn: ${{ matrix.config.dp-attn }} conc: ${{ matrix.config.conc }} kv-offloading: ${{ matrix.config.kv-offloading }} - kv-offload-backend: ${{ matrix.config.kv-offload-backend }} + kv-offload-backend: ${{ matrix.config['kv-offload-backend'].name }} + kv-offload-backend-metadata: ${{ matrix.config['kv-offload-backend'] && toJson(matrix.config['kv-offload-backend']) || '' }} total-cpu-dram-gb: ${{ matrix.config.total-cpu-dram-gb }} duration: ${{ inputs.agentx-fast && '1200' || (inputs.duration-override != '' && inputs.duration-override || matrix.config.duration) }} agentx-fast: ${{ inputs.agentx-fast }} isl: '0' osl: '0' max-model-len: '0' - spec-decoding: 'none' + spec-decoding: ${{ matrix.config.spec-decoding }} disagg: ${{ 'false' }} run-eval: true eval-only: true diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 409052192a..f950a68de4 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -47,7 +47,8 @@ set -x # LANGUAGE_MODEL_ONLY false (reference loads the vision tower) # KV_CACHE_DTYPE fp8 (default for every arm; =auto for a bf16 A/B) # MAX_MODEL_LEN unset (unset -> vLLM derives K3's 1M context) -# SPEC_DECODE false (DSpark; UNVALIDATED on ROCm) +# SPEC_DECODE false (enabled by the _mtp DSpark wrapper) +# ENFORCE_EAGER false (enabled by the _mtp DSpark wrapper) source "$(dirname "$0")/../../benchmark_lib.sh" @@ -543,6 +544,11 @@ if [ -n "${MAX_MODEL_LEN:-}" ] && [ "${MAX_MODEL_LEN}" != "0" ]; then MAX_MODEL_LEN_ARGS=(--max-model-len "$MAX_MODEL_LEN") fi +EAGER_ARGS=() +if [ "${ENFORCE_EAGER:-false}" = "true" ]; then + EAGER_ARGS=(--enforce-eager) +fi + # The reference command passes neither --enable-prefix-caching nor # --no-enable-prefix-caching, and this build's default is None (vLLM decides # internally), so by default we pass nothing and stay aligned. Two reasons this @@ -579,17 +585,36 @@ fi # The upstream DSpark config pins "attention_backend": "FLASHINFER_MLA", which # is CUDA-only and cannot be used verbatim on gfx950; SPEC_ATTN_BACKEND -# overrides it. Golden AL on B300 is 3.78 at 7 draft tokens -# (golden_al_distribution/kimik3_dspark.yaml), so this is the largest decode-side -# lever if it can be made to run here. +# overrides it. AgentX throughput uses synthetic acceptance pinned to the +# committed golden curve. Eval-only runs use real block verification because +# synthetic acceptance bypasses correctness verification. SPEC_ARGS=() if [ "${SPEC_DECODE:-false}" = "true" ]; then SPEC_DRAFT_MODEL="${SPEC_DRAFT_MODEL:-Inferact/Kimi-K3-DSpark}" SPEC_NUM_TOKENS="${SPEC_NUM_TOKENS:-7}" SPEC_ATTN_BACKEND="${SPEC_ATTN_BACKEND:-TRITON_MLA}" + SPEC_GOLDEN_AL="${SPEC_GOLDEN_AL:-3.84}" + SPEC_GOLDEN_AL_FILE="${SPEC_GOLDEN_AL_FILE:-golden_al_distribution/kimik3_dspark_probabilistic_sample_method_block_rejection_sample_method.yaml}" + FILE_GOLDEN_AL=$(awk -v k="$SPEC_NUM_TOKENS" ' + /^kimi-k3:/ { in_model = 1; next } + /^[^[:space:]#]/ { in_model = 0 } + in_model && /thinking_on:/ { in_thinking = 1; next } + in_thinking && $1 == k":" { print $2; exit } + ' "$SPEC_GOLDEN_AL_FILE" 2>/dev/null) + if [ "$FILE_GOLDEN_AL" != "$SPEC_GOLDEN_AL" ]; then + echo "Error: DSpark golden AL mismatch: $SPEC_GOLDEN_AL_FILE gives" >&2 + echo " '${FILE_GOLDEN_AL:-}' for k=$SPEC_NUM_TOKENS," >&2 + echo " but the recipe pins $SPEC_GOLDEN_AL." >&2 + exit 1 + fi + if [ "${EVAL_ONLY:-false}" = "true" ]; then + SPEC_REJECTION_CONFIG="\"rejection_sample_method\":\"block\"" + else + SPEC_REJECTION_CONFIG="\"rejection_sample_method\":\"synthetic\",\"synthetic_acceptance_length\":$SPEC_GOLDEN_AL" + fi SPEC_ARGS=( --speculative-config - "{\"model\":\"$SPEC_DRAFT_MODEL\",\"num_speculative_tokens\":$SPEC_NUM_TOKENS,\"method\":\"dspark\",\"attention_backend\":\"$SPEC_ATTN_BACKEND\",\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}" + "{\"model\":\"$SPEC_DRAFT_MODEL\",\"num_speculative_tokens\":$SPEC_NUM_TOKENS,\"method\":\"dspark\",\"attention_backend\":\"$SPEC_ATTN_BACKEND\",\"draft_sample_method\":\"probabilistic\",$SPEC_REJECTION_CONFIG}" ) fi @@ -694,6 +719,7 @@ VLLM_CMD=( "${MAX_MODEL_LEN_ARGS[@]}" "${PREFIX_CACHE_ARGS[@]}" "${KV_CACHE_DTYPE_ARGS[@]}" + "${EAGER_ARGS[@]}" "${SPEC_ARGS[@]}" "${EVAL_SERVE_ARGS[@]}" "${OFFLOAD_ARGS[@]}" diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh new file mode 100755 index 0000000000..35363753a1 --- /dev/null +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +set -euo pipefail + +# DSpark variant of kimik3_fp4_mi355x.sh. The MI355X launcher routes +# spec-decoding=mtp rows to this suffix, while the shared base recipe owns the +# model, KV-offload, AgentX replay, and eval plumbing. +# +# These overrides are the DSpark + SimpleCPUOffloadConnector combination +# validated on gfx950 in PR #2367. Keep them together: widening the batch or +# re-enabling CUDA graphs reproduced an eight-rank GPU memory access fault. +export SPEC_DECODE=true +export KV_CACHE_DTYPE="${KV_CACHE_DTYPE:-auto}" +export MAX_NUM_SEQS="${MAX_NUM_SEQS:-32}" +export EVAL_MAX_NUM_SEQS="${EVAL_MAX_NUM_SEQS:-32}" +export MAX_NUM_BATCHED_TOKENS="${MAX_NUM_BATCHED_TOKENS:-1024}" +export LANGUAGE_MODEL_ONLY="${LANGUAGE_MODEL_ONLY:-true}" +export SIMPLE_LAZY_OFFLOAD="${SIMPLE_LAZY_OFFLOAD:-true}" +export ENFORCE_EAGER="${ENFORCE_EAGER:-true}" + +exec "$(dirname "$0")/kimik3_fp4_mi355x.sh" "$@" diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index dedeed6020..dd17b6b13e 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -652,6 +652,25 @@ kimik3-fp4-mi355x-vllm-agentic: # far the 2234 GiB host tier extends the concurrency ceiling. - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [4, 8, 16, 32] } +kimik3-fp4-mi355x-vllm-agentic-mtp: + image: vllm/vllm-openai-rocm:kimi-k3 + model: moonshotai/Kimi-K3 + model-prefix: kimik3 + runner: cluster:mi355x-amds + precision: fp4 + framework: vllm + multinode: false + scenarios: + agentic-coding: + - dram-utilization: 0.80 + search-space: + # DSpark speculative decoding with Inferact/Kimi-K3-DSpark, seven draft + # tokens, probabilistic draft sampling, and the committed thinking-on + # golden AL. The _mtp wrapper applies the gfx950 settings validated in + # PR #2367 as one unit: eager execution, max-num-batched-tokens 1024, + # max-num-seqs 32, bf16 KV, and lazy SimpleCPUOffloadConnector allocation. + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, spec-decoding: mtp, conc-list: [1, 2, 4, 8] } + kimik3-fp4-mi355x-vllm-agentic-fp8kvtest: image: vllm/vllm-openai-rocm:kimi-k3 model: moonshotai/Kimi-K3 diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 8f5831507e..7a48aff851 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -5339,3 +5339,11 @@ - "Pass --use-chat-template to the benchmark, as required for EAGLE-style speculative decoding" - "Validated on the node through the real launcher, every request successful: TEP4 concurrency 64 640/640 at 6895 tok/s (11% above the non-MTP arm's 6198 tok/s, mean TTFT 27.2s to 9.6s), and TEP4 concurrency 1 10/10 at 1233 tok/s (83% above the non-MTP arm's 672 tok/s, mean TPOT 12.6ms to 6.5ms)" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2312 + +- config-keys: + - kimik3-fp4-mi355x-vllm-agentic-mtp + description: + - "Add the Kimi-K3 MXFP4 MI355X AgentX DSpark variant with Inferact/Kimi-K3-DSpark, seven draft tokens, probabilistic draft sampling, and TRITON_MLA on ROCm." + - "Pin throughput to the committed thinking-on golden acceptance length 3.84 with synthetic rejection sampling; eval-only runs retain real block verification." + - "Run TP8 with SimpleCPUOffloadConnector at concurrency 1/2/4/8. The _mtp wrapper keeps the gfx950 bring-up settings validated in PR #2367 together: --enforce-eager, --max-num-batched-tokens 1024, --max-num-seqs 32, bf16 KV cache, language-model-only mode, and lazy host allocation." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2403 diff --git a/utils/changelog_gate_tests/test_run_sweep_gating.py b/utils/changelog_gate_tests/test_run_sweep_gating.py index 0489cb4649..3f00860c45 100644 --- a/utils/changelog_gate_tests/test_run_sweep_gating.py +++ b/utils/changelog_gate_tests/test_run_sweep_gating.py @@ -409,6 +409,23 @@ def test_e2e_workflow_cannot_dispatch_database_ingest() -> None: assert "INFX_FRONTEND_PAT" not in workflow +def test_e2e_agentic_eval_preserves_matrix_transport_settings() -> None: + workflow = yaml.load( + (REPO_ROOT / ".github/workflows/e2e-tests.yml").read_text(), + Loader=yaml.BaseLoader, + ) + inputs = workflow["jobs"]["test-sweep-agentic-evals"]["with"] + + assert inputs["kv-offload-backend"] == ( + "${{ matrix.config['kv-offload-backend'].name }}" + ) + assert inputs["kv-offload-backend-metadata"] == ( + "${{ matrix.config['kv-offload-backend']" + " && toJson(matrix.config['kv-offload-backend']) || '' }}" + ) + assert inputs["spec-decoding"] == "${{ matrix.config.spec-decoding }}" + + def test_priority_classifier_runs_for_enabled_actions() -> None: scenario = { **_PR, From d78983b2fc12d3587bd44a3bf159fa8b30a48194 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Thu, 30 Jul 2026 10:25:49 +0900 Subject: [PATCH 38/55] [AMD][AgentX] kimik3: pin the KV page to 64 on an fp8+offload arm fp8 KV + SimpleCPUOffloadConnector has now failed 3/3 (runs 30453589555, 30457683686; nodes g17/g19/g12), every time a GPU fault at under 10% pool usage. The same connector on bf16 (run 30431115226) ran to 99.9% with zero faults, and fp8 without the connector passes at c4/c8/c16. The two runs are config-identical except kv_cache_dtype -- same vLLM g5f76ae224, GMU 0.88, max-num-seqs 128, same cpu_bytes_to_use_per_rank -- and diffing the two vllm_command.txt files leaves --kv-transfer-config as the only delta. Both allocate an identical 14122 CPU blocks while GPU token capacity goes 1,957,290 -> 3,859,490. manager.py:199 scales num_cpu_blocks with num_gpu_blocks, so the block count held and tokens/block doubled: fp8 takes a 128-token page where bf16 takes 64. A 128 page routes MLA into aiter's Gluon b128 kernel, which fp8 survives on its own but has not survived alongside block-granular connector copies. Add KV_BLOCK_SIZE (-> --block-size) and a vllm-simple-fp8-b64 backend that pins it to 64, plus a one-cell fp8b64 key. Keeps fp8 and moves exactly one variable off the failing config. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 36 +++++++++++++++++-- configs/amd-master.yaml | 35 ++++++++++++++++++ 2 files changed, 68 insertions(+), 3 deletions(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index f950a68de4..841465f4a5 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -46,6 +46,7 @@ set -x # AITER_A8W4 1 (reference; 0 = aiter a16w4 MoE path) # LANGUAGE_MODEL_ONLY false (reference loads the vision tower) # KV_CACHE_DTYPE fp8 (default for every arm; =auto for a bf16 A/B) +# KV_BLOCK_SIZE unset (unset -> vLLM sizes the page; 128 under fp8) # MAX_MODEL_LEN unset (unset -> vLLM derives K3's 1M context) # SPEC_DECODE false (enabled by the _mtp DSpark wrapper) # ENFORCE_EAGER false (enabled by the _mtp DSpark wrapper) @@ -427,8 +428,18 @@ except Exception: ) fi ;; - vllm-simple|vllm-simple-fp8) + vllm-simple|vllm-simple-fp8|vllm-simple-fp8-b64) require_agentic_kv_offload_backend "$KV_OFFLOAD_BACKEND" + # vllm-simple-fp8-b64 is vllm-simple-fp8 with the KV page pinned to 64 + # tokens. Under fp8 vLLM otherwise picks 128, which routes MLA into aiter's + # Gluon b128 kernel; fp8 survives that alone but every fp8 cell that also + # ran this connector died with a GPU fault (3/3, g17/g19/g12, at <10% pool + # usage) while bf16 + the same connector ran to 99.9%. 64 is the page bf16 + # used, so this holds the one geometry that is known to work with the + # connector while keeping fp8. + if [ "$KV_OFFLOAD_BACKEND" = "vllm-simple-fp8-b64" ]; then + KV_BLOCK_SIZE="${KV_BLOCK_SIZE:-64}" + fi # vllm-simple-fp8 is the same connector with an fp8 KV cache. fp8 halves # bytes/token in the GPU pool, which on this KV-bound corpus moves the # eviction wall itself rather than just adding headroom (measured: the pool @@ -436,9 +447,11 @@ except Exception: # UNVALIDATED on K3's hybrid geometry: the pool spans Kimi Delta Attention # state and gated-MLA latent, and fp8 support across both spec types is # unconfirmed on this build. - if [ "$KV_OFFLOAD_BACKEND" = "vllm-simple-fp8" ]; then + case "$KV_OFFLOAD_BACKEND" in + vllm-simple-fp8|vllm-simple-fp8-b64) KV_CACHE_DTYPE="${KV_CACHE_DTYPE:-fp8}" - fi + ;; + esac # vLLM's own SimpleCPUOffloadConnector -- the AMD reference's native # offload path. Unlike LMCache it does not go through the Mamba # [conv_state, ssm_state] adapter that K3's Kimi Delta Attention breaks @@ -537,6 +550,22 @@ if [ -n "${KV_CACHE_DTYPE:-}" ] && [ "${KV_CACHE_DTYPE}" != "auto" ]; then KV_CACHE_DTYPE_ARGS=(--kv-cache-dtype "$KV_CACHE_DTYPE") fi +# Unset by default: vLLM sizes the KV page itself. Under fp8 it picks a 128-token +# page (bf16 gets 64) -- measured as an identical 14122 CPU blocks in the offload +# connector across both dtypes while GPU token capacity went 1,957,290 -> +# 3,859,490, which per manager.py:199 (num_cpu_blocks scales with num_gpu_blocks) +# can only mean the block count held and tokens/block doubled. +# +# A 128-token page routes MLA into aiter's Gluon b128 kernel. That path is fine +# on its own (the gist patch above; fp8 kvnone cells pass), but every fp8 cell +# that ALSO ran SimpleCPUOffloadConnector has died with a GPU fault -- 3/3 on +# g17/g19/g12, at <10% pool usage, while the same connector on bf16 ran to 99.9%. +# Set KV_BLOCK_SIZE=64 to hold fp8 on the geometry bf16 used. +KV_BLOCK_SIZE_ARGS=() +if [ -n "${KV_BLOCK_SIZE:-}" ] && [ "${KV_BLOCK_SIZE}" != "0" ]; then + KV_BLOCK_SIZE_ARGS=(--block-size "$KV_BLOCK_SIZE") +fi + # Left unset by default so vLLM derives K3's native 1M context, which is what # the unfiltered corpus needs. Set explicitly only to test truncation effects. MAX_MODEL_LEN_ARGS=() @@ -719,6 +748,7 @@ VLLM_CMD=( "${MAX_MODEL_LEN_ARGS[@]}" "${PREFIX_CACHE_ARGS[@]}" "${KV_CACHE_DTYPE_ARGS[@]}" + "${KV_BLOCK_SIZE_ARGS[@]}" "${EAGER_ARGS[@]}" "${SPEC_ARGS[@]}" "${EVAL_SERVE_ARGS[@]}" diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index dd17b6b13e..485003e4a8 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -775,6 +775,41 @@ kimik3-fp4-mi355x-vllm-agentic-fp8retry: - { tp: 8, kv-offloading: none, conc-list: [1, 16] } - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple-fp8 }, conc-list: [8] } +kimik3-fp4-mi355x-vllm-agentic-fp8b64: + # One variable off the cell that has now failed 3/3: fp8 KV + the + # SimpleCPUOffloadConnector, with the KV page pinned to 64 tokens. + # + # Runs 30453589555 and 30457683686 killed every fp8+connector cell with a GPU + # fault (g17/g19/g12) at under 10% pool usage, while the SAME connector on + # bf16 (run 30431115226) ran to 99.9% with zero faults, and fp8 WITHOUT the + # connector passes (kvnone c4/c8/c16). The two runs are config-identical + # except kv_cache_dtype -- same vLLM g5f76ae224, GMU 0.88, max-num-seqs 128, + # same cpu_bytes_to_use_per_rank -- and diffing vllm_command.txt leaves + # --kv-transfer-config as the only delta. + # + # Both runs allocate an identical 14122 CPU blocks while GPU token capacity + # goes 1,957,290 -> 3,859,490. manager.py:199 scales num_cpu_blocks with + # num_gpu_blocks, so the block count held and tokens/block doubled: fp8 picks + # a 128-token page, bf16 gets 64. A 128 page routes MLA into aiter's Gluon + # b128 kernel -- survivable alone (the gist patch; run 30442578333 got 696 + # tok/s/GPU) but not, so far, alongside block-granular connector copies. + # + # If this passes, the b128 page is the culprit and fp8 keeps the host tier. + # If it dies the same way, the page is not the variable and the next step is + # reading the worker-side copy path rather than guessing at geometry. + image: vllm/vllm-openai-rocm:kimi-k3 + model: moonshotai/Kimi-K3 + model-prefix: kimik3 + runner: cluster:mi355x-amds + precision: fp4 + framework: vllm + multinode: false + scenarios: + agentic-coding: + - dram-utilization: 0.80 + search-space: + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple-fp8-b64 }, conc-list: [8] } + kimik3-fp4-mi355x-vllm-agentic-fp8kv: # fp8 KV on the OFFLOAD arm. This is the fp8kvtest arm (run 30442578333: # 696 total tok/s/GPU, TTFT 2.6s, the first clean fp8 result) plus the From f495ab7e41166e0df4c8278145ea0bbd0c99d9eb Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Thu, 30 Jul 2026 10:44:16 +0900 Subject: [PATCH 39/55] [AMD][AgentX] kimik3: force lm-eval and add an fp8 kvnone c8 eval arm run_eval defaults agentic scenarios to swebench (benchmark_lib.sh:1464), but EVAL_FRAMEWORK wins over that default at :1471. Export it as lm-eval the way minimaxm3_fp4_mi355x.sh does, left overridable so EVAL_FRAMEWORK=swebench can still reach the SWE-bench path this recipe already carries. The eval arm runs the one config proven on this stack: fp8 KV, no offload, c8 -- the control from run 30442578333 (696 tok/s/GPU), repeated clean in 30453589555. The offload arm is deliberately excluded; it is 3/3 dead under fp8 and is being isolated separately by the fp8b64 key. Dispatch with --evals-only: mark_eval_entries flips single-node agentic rows to run-eval=True (generate_sweep_configs.py:360), yielding one eval-only cell. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 8 +++++++ configs/amd-master.yaml | 23 +++++++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 841465f4a5..55cb7aff53 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -53,6 +53,14 @@ set -x source "$(dirname "$0")/../../benchmark_lib.sh" +# Force the eval framework to lm-eval, matching minimaxm3_fp4_mi355x.sh. +# run_eval derives its default as swebench for agentic scenarios +# (scenario_default=swebench when IS_AGENTIC/SCENARIO_TYPE=agentic-coding), but +# EVAL_FRAMEWORK takes precedence over that default +# (benchmark_lib.sh:1471 framework=${EVAL_FRAMEWORK:-...}). Left overridable so +# the SWE-bench path below can still be selected with EVAL_FRAMEWORK=swebench. +export EVAL_FRAMEWORK="${EVAL_FRAMEWORK:-lm-eval}" + check_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION EP_SIZE if [[ -n "${SLURM_JOB_ID:-}" ]]; then diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 485003e4a8..23e45fa2b7 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -775,6 +775,29 @@ kimik3-fp4-mi355x-vllm-agentic-fp8retry: - { tp: 8, kv-offloading: none, conc-list: [1, 16] } - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple-fp8 }, conc-list: [8] } +kimik3-fp4-mi355x-vllm-agentic-fp8eval: + # lm-eval on the one config that is actually proven on this stack: fp8 KV, + # no offload, c8 -- the control cell from run 30442578333 (696 tok/s/GPU), + # repeated clean in 30453589555. The offload arm is excluded on purpose: it + # has failed 3/3 under fp8 (see the fp8b64 key below). + # + # The recipe pins EVAL_FRAMEWORK=lm-eval, so this runs lm-eval rather than + # run_eval's swebench default for agentic scenarios. Dispatch with + # --evals-only; mark_eval_entries flips single-node agentic rows to + # run-eval=True (generate_sweep_configs.py:360). + image: vllm/vllm-openai-rocm:kimi-k3 + model: moonshotai/Kimi-K3 + model-prefix: kimik3 + runner: cluster:mi355x-amds + precision: fp4 + framework: vllm + multinode: false + scenarios: + agentic-coding: + - dram-utilization: 0.80 + search-space: + - { tp: 8, kv-offloading: none, conc-list: [8] } + kimik3-fp4-mi355x-vllm-agentic-fp8b64: # One variable off the cell that has now failed 3/3: fp8 KV + the # SimpleCPUOffloadConnector, with the KV page pinned to 64 tokens. From b9f5fc6bbff0029e8c23eb95ab9d2c9bfb687000 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Thu, 30 Jul 2026 12:34:09 +0900 Subject: [PATCH 40/55] [AMD][AgentX] kimik3: try lazy offload for fp8 KV; --block-size is inert on K3 The block-size attempt (run 30505656455) is void. K3 is hybrid, so vLLM raises the attention block size until the attention page is at least the mamba/KDA page (interface.py:911) and silently overrode --block-size 64 to 1536. Measured pages are bf16 768 / fp8 1536, so that cell was byte-identical to the earlier eager failures and died the same way. Keep the KV_BLOCK_SIZE knob but document it as inert here. This also explains the connector's identical 14122 CPU blocks across dtypes: fp8 halves bytes/token but doubles page tokens, so page bytes -- and manager.py:199's pool -- are unchanged. New lever: lazy offload. Eager stores every completed block to CPU at once; every fp8+connector death has come while the GPU pool was nearly empty (11.6% usage, 0.0% external hit rate in 30505656455), i.e. mid store-storm with nothing yet to gain from the host tier, and the last two were a 600s RCCL _ALLGATHER_BASE hang rather than an OOM. Lazy only stores under GPU-block pressure, removing nearly all connector traffic in that regime. It is also the mode kimik3_fp4_mi355x_mtp.sh already pins for gfx950 (PR #2367). Add a vllm-simple-fp8-lazy backend and swap the fp8b64 key for fp8lazy. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 47 ++++++++++--------- configs/amd-master.yaml | 39 ++++++++------- 2 files changed, 45 insertions(+), 41 deletions(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 55cb7aff53..9d59ba7037 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -436,17 +436,22 @@ except Exception: ) fi ;; - vllm-simple|vllm-simple-fp8|vllm-simple-fp8-b64) + vllm-simple|vllm-simple-fp8|vllm-simple-fp8-lazy) require_agentic_kv_offload_backend "$KV_OFFLOAD_BACKEND" - # vllm-simple-fp8-b64 is vllm-simple-fp8 with the KV page pinned to 64 - # tokens. Under fp8 vLLM otherwise picks 128, which routes MLA into aiter's - # Gluon b128 kernel; fp8 survives that alone but every fp8 cell that also - # ran this connector died with a GPU fault (3/3, g17/g19/g12, at <10% pool - # usage) while bf16 + the same connector ran to 99.9%. 64 is the page bf16 - # used, so this holds the one geometry that is known to work with the - # connector while keeping fp8. - if [ "$KV_OFFLOAD_BACKEND" = "vllm-simple-fp8-b64" ]; then - KV_BLOCK_SIZE="${KV_BLOCK_SIZE:-64}" + # vllm-simple-fp8-lazy is vllm-simple-fp8 with lazy offload. In eager mode + # the connector stores every completed block to CPU immediately; in lazy + # mode it only stores under GPU-block pressure (manager.py _lazy_mode / + # _estimate_lazy_target_blocks walk the GPU free queue). Every fp8+connector + # cell so far has died while the GPU pool was nearly EMPTY -- 11.6% usage + # with a 0.0% external hit rate in run 30505656455, i.e. mid store-storm + # with nothing yet to gain from the host tier -- so the store path is where + # to look, and lazy removes almost all of it in that regime. + # + # Also the mode Wenyao's kimik3_fp4_mi355x_mtp.sh pins (SIMPLE_LAZY_OFFLOAD + # =true) as part of the gfx950 bundle validated in PR #2367 against an + # eight-rank GPU memory access fault. + if [ "$KV_OFFLOAD_BACKEND" = "vllm-simple-fp8-lazy" ]; then + SIMPLE_LAZY_OFFLOAD="${SIMPLE_LAZY_OFFLOAD:-true}" fi # vllm-simple-fp8 is the same connector with an fp8 KV cache. fp8 halves # bytes/token in the GPU pool, which on this KV-bound corpus moves the @@ -456,7 +461,7 @@ except Exception: # state and gated-MLA latent, and fp8 support across both spec types is # unconfirmed on this build. case "$KV_OFFLOAD_BACKEND" in - vllm-simple-fp8|vllm-simple-fp8-b64) + vllm-simple-fp8|vllm-simple-fp8-lazy) KV_CACHE_DTYPE="${KV_CACHE_DTYPE:-fp8}" ;; esac @@ -558,17 +563,17 @@ if [ -n "${KV_CACHE_DTYPE:-}" ] && [ "${KV_CACHE_DTYPE}" != "auto" ]; then KV_CACHE_DTYPE_ARGS=(--kv-cache-dtype "$KV_CACHE_DTYPE") fi -# Unset by default: vLLM sizes the KV page itself. Under fp8 it picks a 128-token -# page (bf16 gets 64) -- measured as an identical 14122 CPU blocks in the offload -# connector across both dtypes while GPU token capacity went 1,957,290 -> -# 3,859,490, which per manager.py:199 (num_cpu_blocks scales with num_gpu_blocks) -# can only mean the block count held and tokens/block doubled. +# INERT ON K3 -- kept only so the override stays discoverable. K3 is hybrid, and +# vLLM raises the attention block size until the attention page is at least the +# mamba/KDA page (interface.py:911, "Setting attention block size to N tokens to +# ensure that attention page size is >= mamba page size"). Measured: bf16 -> 768 +# tokens, fp8 -> 1536. --block-size 64 was silently overridden to 1536 in run +# 30505656455, so this knob cannot move the page on this model; only +# kv_cache_dtype can. # -# A 128-token page routes MLA into aiter's Gluon b128 kernel. That path is fine -# on its own (the gist patch above; fp8 kvnone cells pass), but every fp8 cell -# that ALSO ran SimpleCPUOffloadConnector has died with a GPU fault -- 3/3 on -# g17/g19/g12, at <10% pool usage, while the same connector on bf16 ran to 99.9%. -# Set KV_BLOCK_SIZE=64 to hold fp8 on the geometry bf16 used. +# That also explains the connector's identical 14122 CPU blocks across dtypes: +# fp8 halves bytes/token but the page doubles in tokens, so page BYTES are +# unchanged and manager.py:199 derives the same pool. KV_BLOCK_SIZE_ARGS=() if [ -n "${KV_BLOCK_SIZE:-}" ] && [ "${KV_BLOCK_SIZE}" != "0" ]; then KV_BLOCK_SIZE_ARGS=(--block-size "$KV_BLOCK_SIZE") diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 23e45fa2b7..96b927a176 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -798,28 +798,27 @@ kimik3-fp4-mi355x-vllm-agentic-fp8eval: search-space: - { tp: 8, kv-offloading: none, conc-list: [8] } -kimik3-fp4-mi355x-vllm-agentic-fp8b64: - # One variable off the cell that has now failed 3/3: fp8 KV + the - # SimpleCPUOffloadConnector, with the KV page pinned to 64 tokens. +kimik3-fp4-mi355x-vllm-agentic-fp8lazy: + # fp8 KV + SimpleCPUOffloadConnector in LAZY mode. One variable off the cell + # that has now failed 4/4 (g17/g19/g12/g12), and the block-size attempt in run + # 30505656455 is void: vLLM overrode --block-size 64 to 1536 because K3's + # attention page must be >= its mamba/KDA page (interface.py:911), so that + # cell was byte-identical to the eager failures. # - # Runs 30453589555 and 30457683686 killed every fp8+connector cell with a GPU - # fault (g17/g19/g12) at under 10% pool usage, while the SAME connector on - # bf16 (run 30431115226) ran to 99.9% with zero faults, and fp8 WITHOUT the - # connector passes (kvnone c4/c8/c16). The two runs are config-identical - # except kv_cache_dtype -- same vLLM g5f76ae224, GMU 0.88, max-num-seqs 128, - # same cpu_bytes_to_use_per_rank -- and diffing vllm_command.txt leaves - # --kv-transfer-config as the only delta. + # Why lazy. Eager stores every completed block to CPU immediately. Every + # fp8+connector death has happened while the GPU pool was nearly empty -- + # 11.6% usage, 0.0% external hit rate in 30505656455 -- i.e. deep in a store + # storm with nothing yet to gain from the host tier, and the last two deaths + # were a 600 s RCCL _ALLGATHER_BASE hang rather than an OOM. Lazy only stores + # under GPU-block pressure, so it removes nearly all connector traffic in + # exactly the regime that is killing the cell. # - # Both runs allocate an identical 14122 CPU blocks while GPU token capacity - # goes 1,957,290 -> 3,859,490. manager.py:199 scales num_cpu_blocks with - # num_gpu_blocks, so the block count held and tokens/block doubled: fp8 picks - # a 128-token page, bf16 gets 64. A 128 page routes MLA into aiter's Gluon - # b128 kernel -- survivable alone (the gist patch; run 30442578333 got 696 - # tok/s/GPU) but not, so far, alongside block-granular connector copies. + # It is also the mode kimik3_fp4_mi355x_mtp.sh already pins for gfx950 + # (PR #2367) against an eight-rank GPU memory access fault. # - # If this passes, the b128 page is the culprit and fp8 keeps the host tier. - # If it dies the same way, the page is not the variable and the next step is - # reading the worker-side copy path rather than guessing at geometry. + # Control: bf16 + the same connector in EAGER mode ran to 99.9% pool usage + # with zero faults (run 30431115226), and fp8 WITHOUT the connector peaks at + # 754 tok/s/GPU at c8 (run 30453589555). image: vllm/vllm-openai-rocm:kimi-k3 model: moonshotai/Kimi-K3 model-prefix: kimik3 @@ -831,7 +830,7 @@ kimik3-fp4-mi355x-vllm-agentic-fp8b64: agentic-coding: - dram-utilization: 0.80 search-space: - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple-fp8-b64 }, conc-list: [8] } + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple-fp8-lazy }, conc-list: [8] } kimik3-fp4-mi355x-vllm-agentic-fp8kv: # fp8 KV on the OFFLOAD arm. This is the fp8kvtest arm (run 30442578333: From bd7678a30436663b89d948ca41415393c1763d56 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Wed, 29 Jul 2026 20:45:40 -0700 Subject: [PATCH 41/55] fix(amd): align Kimi-K3 DSpark with ROCm baseline MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Use the exact GPU-only c1 DSpark serving defaults requested for MI355X: block rejection, 128 sequences, 4096 batched tokens, 0.95 GPU memory utilization, multimodal data parallelism, and no eager or prefix-cache override. Add regression coverage for the wrapper and generated matrix. 中文：将 MI355X Kimi-K3 DSpark 对齐到指定的 ROCm GPU-only c1 基线：使用 block rejection、128 个序列、4096 个批处理 token、0.95 GPU 显存利用率、multimodal data 并行，并移除 eager 与显式 prefix-cache 覆盖。同时为 wrapper 和生成矩阵添加回归测试。 --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 50 ++++++------- .../agentic/kimik3_fp4_mi355x_mtp.sh | 19 ++--- configs/amd-master.yaml | 10 ++- perf-changelog.yaml | 4 +- .../matrix_logic/test_kimik3_mi355x_dspark.py | 70 +++++++++++++++++++ 5 files changed, 107 insertions(+), 46 deletions(-) create mode 100644 utils/matrix_logic/test_kimik3_mi355x_dspark.py diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 9d59ba7037..0e4d71a8de 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -49,7 +49,7 @@ set -x # KV_BLOCK_SIZE unset (unset -> vLLM sizes the page; 128 under fp8) # MAX_MODEL_LEN unset (unset -> vLLM derives K3's 1M context) # SPEC_DECODE false (enabled by the _mtp DSpark wrapper) -# ENFORCE_EAGER false (enabled by the _mtp DSpark wrapper) +# ENFORCE_EAGER false (reference; DSpark wrapper keeps it false) source "$(dirname "$0")/../../benchmark_lib.sh" @@ -600,7 +600,7 @@ fi # and hash sizes only align with prefix caching on -- an omission has been # reported to trip "tokens_per_block not divisible by tokens_per_hash" at load. # Set PREFIX_CACHING=true/false to force it either way. -# ON by default for EVERY arm. This trace is built around large shared +# ON by default for non-DSpark arms. This trace is built around large shared # prefixes -- theoretical prefix-cache hit is 98.1%, and a live kvnone cell # measured 92.8% server-side -- so a run with reuse disabled is not measuring # the workload. Reuse also costs essentially no KV (1,414,660 vs 1,420,824 @@ -620,43 +620,35 @@ fi # # Note vLLM resolves the flag's default to False for this model, so ON must be # passed explicitly. PREFIX_CACHING=false forces it off for a deliberate A/B. -PREFIX_CACHE_ARGS=(--enable-prefix-caching) -if [ "${PREFIX_CACHING:-}" = "false" ]; then - PREFIX_CACHE_ARGS=(--no-enable-prefix-caching) -fi +case "${PREFIX_CACHING:-true}" in + true) + PREFIX_CACHE_ARGS=(--enable-prefix-caching) + ;; + false) + PREFIX_CACHE_ARGS=(--no-enable-prefix-caching) + ;; + auto) + # Match the upstream AMD command by letting vLLM resolve its default. + PREFIX_CACHE_ARGS=() + ;; + *) + echo "Error: PREFIX_CACHING must be true, false, or auto." >&2 + exit 1 + ;; +esac # The upstream DSpark config pins "attention_backend": "FLASHINFER_MLA", which # is CUDA-only and cannot be used verbatim on gfx950; SPEC_ATTN_BACKEND -# overrides it. AgentX throughput uses synthetic acceptance pinned to the -# committed golden curve. Eval-only runs use real block verification because -# synthetic acceptance bypasses correctness verification. +# overrides it. Use real block rejection for both throughput and evaluation so +# this path is the exact AMD reproducer and exercises target-model verification. SPEC_ARGS=() if [ "${SPEC_DECODE:-false}" = "true" ]; then SPEC_DRAFT_MODEL="${SPEC_DRAFT_MODEL:-Inferact/Kimi-K3-DSpark}" SPEC_NUM_TOKENS="${SPEC_NUM_TOKENS:-7}" SPEC_ATTN_BACKEND="${SPEC_ATTN_BACKEND:-TRITON_MLA}" - SPEC_GOLDEN_AL="${SPEC_GOLDEN_AL:-3.84}" - SPEC_GOLDEN_AL_FILE="${SPEC_GOLDEN_AL_FILE:-golden_al_distribution/kimik3_dspark_probabilistic_sample_method_block_rejection_sample_method.yaml}" - FILE_GOLDEN_AL=$(awk -v k="$SPEC_NUM_TOKENS" ' - /^kimi-k3:/ { in_model = 1; next } - /^[^[:space:]#]/ { in_model = 0 } - in_model && /thinking_on:/ { in_thinking = 1; next } - in_thinking && $1 == k":" { print $2; exit } - ' "$SPEC_GOLDEN_AL_FILE" 2>/dev/null) - if [ "$FILE_GOLDEN_AL" != "$SPEC_GOLDEN_AL" ]; then - echo "Error: DSpark golden AL mismatch: $SPEC_GOLDEN_AL_FILE gives" >&2 - echo " '${FILE_GOLDEN_AL:-}' for k=$SPEC_NUM_TOKENS," >&2 - echo " but the recipe pins $SPEC_GOLDEN_AL." >&2 - exit 1 - fi - if [ "${EVAL_ONLY:-false}" = "true" ]; then - SPEC_REJECTION_CONFIG="\"rejection_sample_method\":\"block\"" - else - SPEC_REJECTION_CONFIG="\"rejection_sample_method\":\"synthetic\",\"synthetic_acceptance_length\":$SPEC_GOLDEN_AL" - fi SPEC_ARGS=( --speculative-config - "{\"model\":\"$SPEC_DRAFT_MODEL\",\"num_speculative_tokens\":$SPEC_NUM_TOKENS,\"method\":\"dspark\",\"attention_backend\":\"$SPEC_ATTN_BACKEND\",\"draft_sample_method\":\"probabilistic\",$SPEC_REJECTION_CONFIG}" + "{\"model\":\"$SPEC_DRAFT_MODEL\",\"num_speculative_tokens\":$SPEC_NUM_TOKENS,\"method\":\"dspark\",\"attention_backend\":\"$SPEC_ATTN_BACKEND\",\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}" ) fi diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh index 35363753a1..4d1217e28a 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh @@ -5,16 +5,17 @@ set -euo pipefail # spec-decoding=mtp rows to this suffix, while the shared base recipe owns the # model, KV-offload, AgentX replay, and eval plumbing. # -# These overrides are the DSpark + SimpleCPUOffloadConnector combination -# validated on gfx950 in PR #2367. Keep them together: widening the batch or -# re-enabling CUDA graphs reproduced an eight-rank GPU memory access fault. +# Keep this wrapper aligned with the upstream AMD Kimi-K3 DSpark reproducer. +# The first AgentX validation is deliberately GPU-only at c1 so a server or +# kernel failure cannot be attributed to a KV-offload connector. export SPEC_DECODE=true export KV_CACHE_DTYPE="${KV_CACHE_DTYPE:-auto}" -export MAX_NUM_SEQS="${MAX_NUM_SEQS:-32}" -export EVAL_MAX_NUM_SEQS="${EVAL_MAX_NUM_SEQS:-32}" -export MAX_NUM_BATCHED_TOKENS="${MAX_NUM_BATCHED_TOKENS:-1024}" -export LANGUAGE_MODEL_ONLY="${LANGUAGE_MODEL_ONLY:-true}" -export SIMPLE_LAZY_OFFLOAD="${SIMPLE_LAZY_OFFLOAD:-true}" -export ENFORCE_EAGER="${ENFORCE_EAGER:-true}" +export GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.95}" +export MAX_NUM_SEQS="${MAX_NUM_SEQS:-128}" +export EVAL_MAX_NUM_SEQS="${EVAL_MAX_NUM_SEQS:-128}" +export MAX_NUM_BATCHED_TOKENS="${MAX_NUM_BATCHED_TOKENS:-4096}" +export LANGUAGE_MODEL_ONLY="${LANGUAGE_MODEL_ONLY:-false}" +export PREFIX_CACHING="${PREFIX_CACHING:-auto}" +export ENFORCE_EAGER="${ENFORCE_EAGER:-false}" exec "$(dirname "$0")/kimik3_fp4_mi355x.sh" "$@" diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 96b927a176..048494692f 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -664,12 +664,10 @@ kimik3-fp4-mi355x-vllm-agentic-mtp: agentic-coding: - dram-utilization: 0.80 search-space: - # DSpark speculative decoding with Inferact/Kimi-K3-DSpark, seven draft - # tokens, probabilistic draft sampling, and the committed thinking-on - # golden AL. The _mtp wrapper applies the gfx950 settings validated in - # PR #2367 as one unit: eager execution, max-num-batched-tokens 1024, - # max-num-seqs 32, bf16 KV, and lazy SimpleCPUOffloadConnector allocation. - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, spec-decoding: mtp, conc-list: [1, 2, 4, 8] } + # Exact upstream AMD DSpark reproducer: GPU-only TP8, seven draft tokens, + # probabilistic draft sampling, TRITON_MLA, and real block rejection. + # Start at c1; expand only after the AgentX-fast preflight is healthy. + - { tp: 8, kv-offloading: none, spec-decoding: mtp, conc-list: [1] } kimik3-fp4-mi355x-vllm-agentic-fp8kvtest: image: vllm/vllm-openai-rocm:kimi-k3 diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 7a48aff851..2b97ed2a52 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -5344,6 +5344,6 @@ - kimik3-fp4-mi355x-vllm-agentic-mtp description: - "Add the Kimi-K3 MXFP4 MI355X AgentX DSpark variant with Inferact/Kimi-K3-DSpark, seven draft tokens, probabilistic draft sampling, and TRITON_MLA on ROCm." - - "Pin throughput to the committed thinking-on golden acceptance length 3.84 with synthetic rejection sampling; eval-only runs retain real block verification." - - "Run TP8 with SimpleCPUOffloadConnector at concurrency 1/2/4/8. The _mtp wrapper keeps the gfx950 bring-up settings validated in PR #2367 together: --enforce-eager, --max-num-batched-tokens 1024, --max-num-seqs 32, bf16 KV cache, language-model-only mode, and lazy host allocation." + - "Align the first AgentX validation with the upstream AMD reproducer: GPU-only TP8 at concurrency 1, real block rejection, --gpu-memory-utilization 0.95, --max-num-batched-tokens 4096, --max-num-seqs 128, multimodal encoder data parallelism, and no --enforce-eager or explicit prefix-cache override." + - "Use VLLM_ROCM_USE_AITER=1, SAFETENSORS_FAST_GPU=1, AITER_SITUV2_A8W4=1, AITER_BF16_FP8_MOE_BOUND=0, and VLLM_USE_BREAKABLE_CUDAGRAPH=0 with the stock vllm/vllm-openai-rocm:kimi-k3 image." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2403 diff --git a/utils/matrix_logic/test_kimik3_mi355x_dspark.py b/utils/matrix_logic/test_kimik3_mi355x_dspark.py new file mode 100644 index 0000000000..adb1930812 --- /dev/null +++ b/utils/matrix_logic/test_kimik3_mi355x_dspark.py @@ -0,0 +1,70 @@ +from pathlib import Path +import re + +import yaml + + +REPO_ROOT = Path(__file__).resolve().parents[2] +MTP_WRAPPER = ( + REPO_ROOT / "benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh" +) +BASE_RECIPE = REPO_ROOT / "benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh" +MASTER_CONFIG = REPO_ROOT / "configs/amd-master.yaml" + + +def _shell_default(script: str, variable: str) -> str: + match = re.search( + rf'export {re.escape(variable)}="\$\{{{re.escape(variable)}:-(.*?)\}}"', + script, + ) + assert match is not None, f"{variable} must have an overridable default" + return match.group(1) + + +def test_dspark_wrapper_matches_oren_rocm_baseline() -> None: + wrapper = MTP_WRAPPER.read_text() + + assert _shell_default(wrapper, "KV_CACHE_DTYPE") == "auto" + assert _shell_default(wrapper, "GPU_MEM_UTIL") == "0.95" + assert _shell_default(wrapper, "MAX_NUM_SEQS") == "128" + assert _shell_default(wrapper, "EVAL_MAX_NUM_SEQS") == "128" + assert _shell_default(wrapper, "MAX_NUM_BATCHED_TOKENS") == "4096" + assert _shell_default(wrapper, "LANGUAGE_MODEL_ONLY") == "false" + assert _shell_default(wrapper, "PREFIX_CACHING") == "auto" + assert _shell_default(wrapper, "ENFORCE_EAGER") == "false" + + +def test_dspark_defaults_to_real_block_rejection() -> None: + recipe = BASE_RECIPE.read_text() + spec_section = recipe.split("# ---- Eval-only path", maxsplit=1)[0].split( + "SPEC_ARGS=()", maxsplit=1 + )[1] + + assert '\\"rejection_sample_method\\":\\"block\\"' in spec_section + assert "synthetic_acceptance_length" not in spec_section + + +def test_reference_prefix_cache_mode_emits_no_override() -> None: + recipe = BASE_RECIPE.read_text() + prefix_section = recipe.split("# The upstream DSpark config", maxsplit=1)[0].split( + "PREFIX_CACHE_ARGS=", maxsplit=1 + )[1] + + assert "auto)" in prefix_section + assert "PREFIX_CACHE_ARGS=()" in prefix_section + + +def test_dspark_diagnostic_matrix_is_gpu_only_c1() -> None: + master = yaml.safe_load(MASTER_CONFIG.read_text()) + search_space = master["kimik3-fp4-mi355x-vllm-agentic-mtp"]["scenarios"][ + "agentic-coding" + ][0]["search-space"] + + assert search_space == [ + { + "tp": 8, + "kv-offloading": "none", + "spec-decoding": "mtp", + "conc-list": [1], + } + ] From 220e0eef1e5aa350b96fe52c38b423aa6e3ce5f5 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Thu, 30 Jul 2026 14:06:45 +0900 Subject: [PATCH 42/55] [AMD][AgentX] kimik3: pair eager+lazy fp8 offload on the new AIPerf warmup Run 30511545046 tried lazy alone and is uninterpretable. It landed just after #2415 swapped the agentic warmup from --agentic-cache-warmup-duration 600 to --warmup-requests-per-lane 10 (benchmark_lib.sh:1793, plus a new utils/aiperf pin), so it moved two variables at once. Its "0 successful / 49 total, OSL=1" is the new warmup's one-token lane primers, not a lazy-offload symptom -- the cell never left warmup. That also invalidates the seven earlier fp8+offload failures as baselines: all of them ran the old time-based warmup. So run the eager control at c8 on the SAME warmup as lazy, in one dispatch, and compare those two only. Co-Authored-By: Claude Opus 5 (1M context) --- configs/amd-master.yaml | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 048494692f..47afe044d1 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -828,6 +828,15 @@ kimik3-fp4-mi355x-vllm-agentic-fp8lazy: agentic-coding: - dram-utilization: 0.80 search-space: + # PAIRED. Run 30511545046 tried lazy alone and is uninterpretable: it + # landed after #2415 swapped the agentic warmup from + # --agentic-cache-warmup-duration 600 to --warmup-requests-per-lane 10 + # (benchmark_lib.sh:1793, new utils/aiperf pin), so it moved two variables + # at once and died inside the new warmup -- its "0 successful, OSL=1" is + # that warmup's one-token primers, not a lazy symptom. Every earlier + # failure used the old time-based warmup, so none of them is a baseline + # any more. Run the eager control on the SAME warmup as lazy. + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple-fp8 }, conc-list: [8] } - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple-fp8-lazy }, conc-list: [8] } kimik3-fp4-mi355x-vllm-agentic-fp8kv: From 0af173059cdd5086957dec837944691cdba5217b Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Thu, 30 Jul 2026 15:58:00 +0900 Subject: [PATCH 43/55] [AMD][AgentX] kimik3: fp8-KV no-offload curve at GMU 0.90, conc 1/4/8/16 Raise the recipe's GPU_MEM_UTIL default 0.88 -> 0.90 and add kimik3-fp4-mi355x-vllm-agentic-fp8none-gmu90, a single GPU-only fp8-KV concurrency curve. The agentic matrix has no per-cell env channel, so the memory fraction has to move on the recipe default. The offload arm is left out on purpose: it has failed 4/4 under fp8, and mixing it into this dispatch would put failure-prone cells alongside the control curve. c16 is the new point -- the no-offload arm has never been measured above c8. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 6 +++- configs/amd-master.yaml | 30 +++++++++++++++++++ 2 files changed, 35 insertions(+), 1 deletion(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 0e4d71a8de..99882f743e 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -702,7 +702,11 @@ fi # overhead and transient co-tenancy are counted. This is not a denylist problem: # g11 and g16 each measured both above and below the line hours apart. 0.88 # (253.4 GiB) clears every observation except the two genuinely occupied nodes. -GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.88}" +# 2026-07-30: raised 0.88 -> 0.90 (259.2 GiB). fp8 KV halves the per-token KV +# cost, so the extra 5.8 GiB of pool is worth more than it was at bf16, and +# every 0.88 observation above had >= 256 GiB free except the two occupied +# nodes. Drop back to 0.88 if bring-up failures reappear at engine init. +GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.90}" MAX_NUM_SEQS="${MAX_NUM_SEQS:-128}" MAX_NUM_BATCHED_TOKENS="${MAX_NUM_BATCHED_TOKENS:-4096}" diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 47afe044d1..276a947952 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -796,6 +796,36 @@ kimik3-fp4-mi355x-vllm-agentic-fp8eval: search-space: - { tp: 8, kv-offloading: none, conc-list: [8] } +kimik3-fp4-mi355x-vllm-agentic-fp8none-gmu90: + # GPU-only (no offload) fp8-KV concurrency curve at GMU 0.90. + # + # One key, one curve: c1/c4/c8/c16 with --kv-cache-dtype fp8 (the recipe + # default) and no KV connector, so the whole curve shares a KV dtype, a + # memory fraction and a warmup. The offload arm is deliberately absent -- + # it has failed 4/4 under fp8 (see the fp8lazy key below), and mixing it in + # would put failure-prone cells in the same dispatch as the control curve. + # + # GMU is 0.90 rather than the earlier 0.88: it is set by the recipe default + # (GPU_MEM_UTIL in kimik3_fp4_mi355x.sh), because the agentic matrix has no + # per-cell env channel. + # + # c16 is the open question. The bf16 no-offload arm was never measured above + # c8, and the bf16 vllm-simple curve collapsed at c16 (TTFT 295 s). fp8 + # roughly halves KV per token, so the ceiling should have moved up; this + # cell is what says whether it did. + image: vllm/vllm-openai-rocm:kimi-k3 + model: moonshotai/Kimi-K3 + model-prefix: kimik3 + runner: cluster:mi355x-amds + precision: fp4 + framework: vllm + multinode: false + scenarios: + agentic-coding: + - dram-utilization: 0.80 + search-space: + - { tp: 8, kv-offloading: none, conc-list: [1, 4, 8, 16] } + kimik3-fp4-mi355x-vllm-agentic-fp8lazy: # fp8 KV + SimpleCPUOffloadConnector in LAZY mode. One variable off the cell # that has now failed 4/4 (g17/g19/g12/g12), and the block-size attempt in run From f2feeb8fe62947d87350b1cbd989b43c74d2ee18 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Wed, 29 Jul 2026 22:31:52 -0700 Subject: [PATCH 44/55] fix(amd): reduce Kimi-K3 DSpark sequence slots MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Match the same-image vLLM reproducer that initializes successfully by lowering the serving max sequence slots from 128 to 16. Keep the eval limit and all other DSpark settings unchanged for a one-variable validation. 中文：将服务端最大序列槽位从 128 降至 vLLM 在同一镜像中验证可正常初始化的 16。为保证单变量验证，评估上限及其他 DSpark 配置均保持不变。 --- benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh | 2 +- perf-changelog.yaml | 2 +- utils/matrix_logic/test_kimik3_mi355x_dspark.py | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh index 4d1217e28a..6e92c322a7 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh @@ -11,7 +11,7 @@ set -euo pipefail export SPEC_DECODE=true export KV_CACHE_DTYPE="${KV_CACHE_DTYPE:-auto}" export GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.95}" -export MAX_NUM_SEQS="${MAX_NUM_SEQS:-128}" +export MAX_NUM_SEQS="${MAX_NUM_SEQS:-16}" export EVAL_MAX_NUM_SEQS="${EVAL_MAX_NUM_SEQS:-128}" export MAX_NUM_BATCHED_TOKENS="${MAX_NUM_BATCHED_TOKENS:-4096}" export LANGUAGE_MODEL_ONLY="${LANGUAGE_MODEL_ONLY:-false}" diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 2b97ed2a52..6922445d06 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -5344,6 +5344,6 @@ - kimik3-fp4-mi355x-vllm-agentic-mtp description: - "Add the Kimi-K3 MXFP4 MI355X AgentX DSpark variant with Inferact/Kimi-K3-DSpark, seven draft tokens, probabilistic draft sampling, and TRITON_MLA on ROCm." - - "Align the first AgentX validation with the upstream AMD reproducer: GPU-only TP8 at concurrency 1, real block rejection, --gpu-memory-utilization 0.95, --max-num-batched-tokens 4096, --max-num-seqs 128, multimodal encoder data parallelism, and no --enforce-eager or explicit prefix-cache override." + - "Align the first AgentX validation with the upstream AMD reproducer: GPU-only TP8 at concurrency 1, real block rejection, --gpu-memory-utilization 0.95, --max-num-batched-tokens 4096, --max-num-seqs 16, multimodal encoder data parallelism, and no --enforce-eager or explicit prefix-cache override." - "Use VLLM_ROCM_USE_AITER=1, SAFETENSORS_FAST_GPU=1, AITER_SITUV2_A8W4=1, AITER_BF16_FP8_MOE_BOUND=0, and VLLM_USE_BREAKABLE_CUDAGRAPH=0 with the stock vllm/vllm-openai-rocm:kimi-k3 image." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2403 diff --git a/utils/matrix_logic/test_kimik3_mi355x_dspark.py b/utils/matrix_logic/test_kimik3_mi355x_dspark.py index adb1930812..be8477e2cc 100644 --- a/utils/matrix_logic/test_kimik3_mi355x_dspark.py +++ b/utils/matrix_logic/test_kimik3_mi355x_dspark.py @@ -26,7 +26,7 @@ def test_dspark_wrapper_matches_oren_rocm_baseline() -> None: assert _shell_default(wrapper, "KV_CACHE_DTYPE") == "auto" assert _shell_default(wrapper, "GPU_MEM_UTIL") == "0.95" - assert _shell_default(wrapper, "MAX_NUM_SEQS") == "128" + assert _shell_default(wrapper, "MAX_NUM_SEQS") == "16" assert _shell_default(wrapper, "EVAL_MAX_NUM_SEQS") == "128" assert _shell_default(wrapper, "MAX_NUM_BATCHED_TOKENS") == "4096" assert _shell_default(wrapper, "LANGUAGE_MODEL_ONLY") == "false" From 86a097bc32e95968bd67ef1ccaaaec4b07c4f7c3 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Thu, 30 Jul 2026 17:04:04 +0900 Subject: [PATCH 45/55] [AMD][AgentX] kimik3: revert GMU to 0.88, rename the fp8 no-offload key 0.90 does not fail at engine init -- it comes up clean with a 4,302,357-token pool and then dies mid-prefill. Run 30521327099 lost c4, c8 and c16 on three separate nodes with one signature: HSA_STATUS_ERROR_OUT_OF_RESOURCES / "Available Free mem : 0 MB" on 4 of 8 ranks, surfacing as hipErrorUnknown at gpu_model_runner.py:314. Every death was at num_computed_tokens 360,960 to 364,032, num_running_reqs=1, kv_cache_usage ~21%. Not concurrency, not KV exhaustion: the transient chunked-prefill workspace outgrows the ~22 GiB of unreserved headroom 0.90 leaves. 0.88 leaves ~28 GiB and served 987 requests at ISL p90 374,550 / p95 511,146 with a 0/165 error rate (run 30453589555), on a serve command byte-identical apart from this flag. The extra pool 0.90 buys is unusable anyway -- peak usage at death was 21%. Key renamed fp8none-gmu90 -> fp8none; it never produced a result at 0.90. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 24 +++++++++++++++---- configs/amd-master.yaml | 12 ++++++---- 2 files changed, 26 insertions(+), 10 deletions(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 99882f743e..ef8cd4d562 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -702,11 +702,25 @@ fi # overhead and transient co-tenancy are counted. This is not a denylist problem: # g11 and g16 each measured both above and below the line hours apart. 0.88 # (253.4 GiB) clears every observation except the two genuinely occupied nodes. -# 2026-07-30: raised 0.88 -> 0.90 (259.2 GiB). fp8 KV halves the per-token KV -# cost, so the extra 5.8 GiB of pool is worth more than it was at bf16, and -# every 0.88 observation above had >= 256 GiB free except the two occupied -# nodes. Drop back to 0.88 if bring-up failures reappear at engine init. -GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.90}" +# 2026-07-30: 0.90 (259.19 GiB) was tried and REVERTED. It does not fail at +# engine init -- it comes up clean with a 4,302,357-token pool -- and then dies +# mid-prefill. Run 30521327099 lost c4, c8 and c16 on three separate nodes with +# an identical signature: HSA_STATUS_ERROR_OUT_OF_RESOURCES / "Available Free +# mem : 0 MB" on 4 of 8 ranks, surfacing as hipErrorUnknown at +# gpu_model_runner.py:314. Every death was at num_computed_tokens 360,960 to +# 364,032 with num_running_reqs=1 and kv_cache_usage only ~21%, so it is not +# concurrency and not KV exhaustion: the transient chunked-prefill workspace +# scales with context and needs more than the ~22 GiB of unreserved headroom +# 0.90 leaves. 0.88 leaves ~28 GiB and served 987 requests at ISL p90 374,550 / +# p95 511,146 with a 0/165 error rate (run 30453589555, byte-identical serve +# command apart from this one flag). +# +# The extra 443K tokens of pool 0.90 buys are unusable anyway -- peak usage at +# death was 21%. Raising this again requires bounding the prefill workspace +# first (the build_mla_chunked_context_metadata aggregate-chunk cap that only +# exists in hyukjleeamd/kimi-k3-mla-b128:fp8-kv-fixed, not in the mla_gluon +# gist patched below, which is a decode-path fix only). +GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.88}" MAX_NUM_SEQS="${MAX_NUM_SEQS:-128}" MAX_NUM_BATCHED_TOKENS="${MAX_NUM_BATCHED_TOKENS:-4096}" diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 276a947952..7027c2f36e 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -796,8 +796,8 @@ kimik3-fp4-mi355x-vllm-agentic-fp8eval: search-space: - { tp: 8, kv-offloading: none, conc-list: [8] } -kimik3-fp4-mi355x-vllm-agentic-fp8none-gmu90: - # GPU-only (no offload) fp8-KV concurrency curve at GMU 0.90. +kimik3-fp4-mi355x-vllm-agentic-fp8none: + # GPU-only (no offload) fp8-KV concurrency curve at GMU 0.88. # # One key, one curve: c1/c4/c8/c16 with --kv-cache-dtype fp8 (the recipe # default) and no KV connector, so the whole curve shares a KV dtype, a @@ -805,9 +805,11 @@ kimik3-fp4-mi355x-vllm-agentic-fp8none-gmu90: # it has failed 4/4 under fp8 (see the fp8lazy key below), and mixing it in # would put failure-prone cells in the same dispatch as the control curve. # - # GMU is 0.90 rather than the earlier 0.88: it is set by the recipe default - # (GPU_MEM_UTIL in kimik3_fp4_mi355x.sh), because the agentic matrix has no - # per-cell env channel. + # GMU comes from the recipe default (GPU_MEM_UTIL in kimik3_fp4_mi355x.sh), + # because the agentic matrix has no per-cell env channel. The first dispatch + # of this key ran at 0.90 and lost c4/c8/c16 to a prefill-workspace OOM at + # ~362K context on three separate nodes; see the comment on GPU_MEM_UTIL for + # the evidence. 0.88 is the proven setting. # # c16 is the open question. The bf16 no-offload arm was never measured above # c8, and the bf16 vllm-simple curve collapsed at c16 (TTFT 295 s). fp8 From 57ca3997842684f3c24fd13426b8ec74cc0e6a40 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Fri, 31 Jul 2026 01:03:01 +0900 Subject: [PATCH 46/55] [AMD][AgentX] kimik3: enable LMCache via upstream's validated K3 support LMCache now supports Kimi-K3 upstream. Two dev commits are what make it work: #4206 (07-23) _MambaUnifiedViewEdit -- re-views a Mamba/KDA UNIFIED state tensor as a single attention tensor. This is the fix for the blocker that has killed every LMCache attempt on this model: "ValueError: expected a Mamba [conv_state, ssm_state] tensor list, got Tensor" -- K3's Kimi Delta Attention supplies one fused tensor. #4278 (07-28) _SubpagedMLAAttentionViewEdit -- re-views K3's kernel-paged MLA tensor [N*12, 64, 576] as logical pages [N, 768, 576]. LMCache also shipped a validated K3 recipe (#4277). Neither fix is in 0.5.1 (this recipe's pin) nor in any PyPI release (latest 0.5.2), so the recipe now builds from source at the #4278 merge commit with ROCm flags; nightly wheels are CUDA-only. Correctness changes beyond the version bump: - --mamba-cache-mode align passed explicitly. It is the only mode the KDA backend supports and the only one that keeps reusable state snapshots. - VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1, required by upstream's K3 recipe. - chunk-size and max-num-batched-tokens now DERIVE from vLLM's unified block size N instead of being hardcoded to 768. N is not constant: upstream quotes 768 for K3 at TP8, but that is bf16. Under this recipe's default fp8 KV, three runs (30453589555, 30521327099, 30525206671) all log "Setting attention block size to 1536 tokens". Hardcoding 768 violated both LMCache constraints (chunk_size % N == 0, N <= mnbt < 2N) on every default run. Now 1536/3000 under fp8, 768/1464 under bf16, both range-checked. - New lmcache-k3 backend name selects upstream's minimal validated server command; the `reference` profile's extra flags were never validated on K3. - The install asserts both edit classes are registered before the 1.56 TB weight load, so a stale build fails in seconds rather than 20 minutes. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 195 ++++++++++++++++-- configs/amd-master.yaml | 40 ++++ 2 files changed, 215 insertions(+), 20 deletions(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index ef8cd4d562..0c372c6b38 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -188,6 +188,9 @@ wait_for_lmcache_ready() { # benchmarks/single_node/agentic/README.md it must be consumed as given and # never replaced with a model-specific constant. OFFLOAD_ARGS=() +# Only the LMCache arms set this (to `--mamba-cache-mode align`); every other +# arm leaves vLLM to pick the mode. +LMCACHE_MAMBA_CACHE_MODE_ARGS=() # fp8 KV is the DEFAULT for every arm on this model. # @@ -208,8 +211,18 @@ KV_CACHE_DTYPE="${KV_CACHE_DTYPE:-fp8}" if agentic_kv_offload_enabled; then case "${KV_OFFLOAD_BACKEND:-}" in - lmcache|lmcache-k27|lmcache-budget) + lmcache|lmcache-k27|lmcache-budget|lmcache-k3) require_agentic_kv_offload_backend "$KV_OFFLOAD_BACKEND" + # lmcache-k3: LMCache's OWN validated K3 server command + # (docs/source/recipes/kimi_k3.rst, PR #4277). Deliberately minimal -- + # the `reference` profile's flags (--max-gpu-workers, --l1-align-bytes, + # --eviction-trigger-watermark, --supported-transfer-mode) are not part of + # what upstream validated on K3, and an unrecognised flag is a silent + # server-start failure. Start from the known-good command; add tuning + # flags only once this arm produces a number. + if [ "$KV_OFFLOAD_BACKEND" = "lmcache-k3" ]; then + LMCACHE_PROFILE=k3upstream + fi # The server profile has to be selectable from config: the agentic matrix # has no per-cell env channel, so the backend NAME carries it. # lmcache -> AMD K3 reference server command @@ -261,11 +274,59 @@ case "${KV_OFFLOAD_BACKEND:-}" in # Worth keeping in view either way: at ~106k-token average ISL, 768 would # mean ~138 chunked-prefill steps per turn versus ~26 at 4096, so the two # settings are not performance-equivalent. - MAX_NUM_BATCHED_TOKENS="${LMCACHE_MAX_NUM_BATCHED_TOKENS:-768}" - LMCACHE_K3_CHUNK_SIZE="${LMCACHE_CHUNK_SIZE_OVERRIDE:-768}" + # Both values derive from vLLM's UNIFIED block size N, which is NOT a + # constant: it depends on TP and on the KV dtype. Upstream's recipe quotes + # N=768 for K3 at TP8, but that is the bf16 figure, and it explicitly says + # to "re-read N from vLLM's `Setting attention block size to N tokens` + # startup log line" after changing anything. Measured on this fleet with + # --kv-cache-dtype fp8 at TP8, three independent runs (30453589555, + # 30521327099, 30525206671) all log: + # Setting attention block size to 1536 tokens + # fp8 halves the element size, so twice as many tokens fit one page. Since + # fp8 is this recipe's default, hardcoding 768 would violate BOTH LMCache + # constraints on every default run. + # + # Constraints (lmcache/integration/vllm, and upstream's recipe): + # chunk_size % N == 0 -> chunk_size = N + # N <= max_num_batched_tokens < 2 * N -> sit just under 2N + # `align` snapshots the KDA state only on a block boundary at the end of a + # scheduler step, so the per-step budget must cover at least one full block + # but stay below two. Upstream validated 1500 against N=768 (just under + # 2N); 3000 against N=1536 is the same ratio. + if [ "${KV_CACHE_DTYPE:-}" = "fp8" ]; then + LMCACHE_UNIFIED_BLOCK="${LMCACHE_UNIFIED_BLOCK:-1536}" + else + LMCACHE_UNIFIED_BLOCK="${LMCACHE_UNIFIED_BLOCK:-768}" + fi + LMCACHE_K3_CHUNK_SIZE="${LMCACHE_CHUNK_SIZE_OVERRIDE:-$LMCACHE_UNIFIED_BLOCK}" + MAX_NUM_BATCHED_TOKENS="${LMCACHE_MAX_NUM_BATCHED_TOKENS:-$(( LMCACHE_UNIFIED_BLOCK * 2 - 72 ))}" LMCACHE_CHUNK_SIZE="$LMCACHE_K3_CHUNK_SIZE" LMCACHE_CHUNK_SIZE_K27="$LMCACHE_K3_CHUNK_SIZE" - echo "LMCache: --max-num-batched-tokens=$MAX_NUM_BATCHED_TOKENS --chunk-size=$LMCACHE_K3_CHUNK_SIZE (reference values)" + # Guard the derivation rather than trusting it: a wrong N is a 20-minute + # engine-init failure after a 1.56 TB weight load. + if [ $(( LMCACHE_K3_CHUNK_SIZE % LMCACHE_UNIFIED_BLOCK )) -ne 0 ]; then + echo "Error: LMCache chunk-size $LMCACHE_K3_CHUNK_SIZE is not a multiple of N=$LMCACHE_UNIFIED_BLOCK." >&2 + exit 1 + fi + if [ "$MAX_NUM_BATCHED_TOKENS" -lt "$LMCACHE_UNIFIED_BLOCK" ] \ + || [ "$MAX_NUM_BATCHED_TOKENS" -ge $(( LMCACHE_UNIFIED_BLOCK * 2 )) ]; then + echo "Error: --max-num-batched-tokens $MAX_NUM_BATCHED_TOKENS outside [N, 2N) for N=$LMCACHE_UNIFIED_BLOCK." >&2 + exit 1 + fi + echo "LMCache: N=$LMCACHE_UNIFIED_BLOCK (kv-cache-dtype=${KV_CACHE_DTYPE:-auto})" \ + "--max-num-batched-tokens=$MAX_NUM_BATCHED_TOKENS --chunk-size=$LMCACHE_K3_CHUNK_SIZE" + + # Required by upstream's K3 recipe, and by the KDA backend generally: + # `align` is the only Mamba cache mode the KDA backend supports, and it is + # the mode that keeps reusable state snapshots. vLLM picks it implicitly + # when prefix caching is on, but upstream passes it explicitly and so do + # we -- an implicit default that silently flips to 'none' is exactly how + # this arm failed before. + LMCACHE_MAMBA_CACHE_MODE_ARGS=(--mamba-cache-mode align) + + # Upstream's K3 recipe requires this on the LMCache path. It defaults to 0 + # and gates the fused latent-MoE tail in the K3 model definition. + export VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION="${VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION:-1}" # LMCache is NOT in the kimi-k3 image (verified: no `lmcache` module and no # CLI), so build it against ROCm. Clone to a container-local dir, NOT the @@ -288,26 +349,91 @@ except Exception: ' 2>/dev/null || true) fi - # The AMD reference recipe pins a PyPI release: `uv pip install - # "lmcache==0.5.1"`. Do not substitute a git build of dev HEAD -- dev - # (v0.5.3.dev47) carries a LazyMemoryAllocator that expands the pinned L1 - # pool ~10 GB per ~17 s DURING serving, which starves the vLLM worker until - # the executor RPC deadline fires ("RPC call to sample_tokens timed out", - # observed on g09 and g11). It also adds Mamba-hybrid constraints - # (block_size <= max_num_batched_tokens < 2*block_size, and - # chunk_size % block_size == 0) that the reference command does not satisfy. - LMCACHE_VERSION="${LMCACHE_VERSION:-${LMCACHE_CFG_VERSION:-0.5.1}}" + # LMCache K3 support is NOT in any PyPI release yet (latest is 0.5.2, + # 2026-07-22). Upstream's own recipe -- docs/source/recipes/kimi_k3.rst, + # added by LMCache PR #4277 -- says: "use a nightly build from 2026-07-27 + # or newer; stable LMCache releases include K3 support starting from + # 0.5.3". Two commits on dev are what make K3 work at all: + # #4206 (2026-07-23) _MambaUnifiedViewEdit -- re-views a Mamba/KDA + # UNIFIED state tensor as a single attention tensor. This is the + # fix for our long-standing blocker + # "ValueError: expected a Mamba [conv_state, ssm_state] tensor + # list, got Tensor" (run 30350911388): K3's Kimi Delta Attention + # supplies one fused state tensor, and before #4206 the adapter + # only understood the [conv_state, ssm_state] pair. + # #4278 (2026-07-28) _SubpagedMLAAttentionViewEdit -- re-views K3's + # kernel-paged MLA tensor [N*12, 64, 576] as logical pages + # [N, 768, 576]. + # 0.5.1 predates BOTH, which is why every LMCache attempt on this model + # has died at engine init. The historical "pin 0.5.1, never build dev" + # note in this file was correct for its date and is now obsolete. + # + # Nightly WHEELS are CUDA-only, so ROCm must build from source. Flags are + # upstream's (docs/source/getting_started/installation.rst): gfx950 is + # MI355X/MI350X. + # + # LMCACHE_GIT_REF pins the build. Default is the #4278 merge commit rather + # than dev HEAD so the build is reproducible and cannot drift under us. + # A config-supplied `version` that looks like a release (0.5.3+) still + # installs from PyPI once such a release exists. + LMCACHE_GIT_REF="${LMCACHE_GIT_REF:-1dfa07772b8b2ae79653c830d63e297da39c970f}" + LMCACHE_VERSION="${LMCACHE_VERSION:-${LMCACHE_CFG_VERSION:-git}}" if ! python3 -c "import lmcache.integration.vllm.lmcache_mp_connector" >/dev/null 2>&1; then - echo "Installing lmcache==$LMCACHE_VERSION" - if command -v uv >/dev/null 2>&1; then - uv pip install --system "lmcache==$LMCACHE_VERSION" \ - || agentic_pip_install --quiet "lmcache==$LMCACHE_VERSION" + if [ "$LMCACHE_VERSION" = "git" ]; then + # Clone to a container-local dir, NOT the bind-mounted /workspace: + # a later job's `clean: true` checkout trips over root-owned build + # artifacts left behind there. + LMCACHE_SRC="${LMCACHE_SRC:-/opt/lmcache-src}" + echo "Building lmcache from source at ref $LMCACHE_GIT_REF (ROCm/gfx950)" + rm -rf "$LMCACHE_SRC" + git clone --filter=blob:none https://github.com/LMCache/LMCache.git "$LMCACHE_SRC" + git -C "$LMCACHE_SRC" checkout --detach "$LMCACHE_GIT_REF" + git -C "$LMCACHE_SRC" --no-pager log -1 --oneline + ( + cd "$LMCACHE_SRC" + export PYTORCH_ROCM_ARCH="gfx942,gfx950" + export TORCH_DONT_CHECK_COMPILER_ABI=1 + export CXX=hipcc + export BUILD_WITH_HIP=1 + if command -v uv >/dev/null 2>&1; then + uv pip install --system -e . --no-build-isolation \ + || python3 -m pip install -e . --no-build-isolation + else + python3 -m pip install -e . --no-build-isolation + fi + ) else - agentic_pip_install --quiet "lmcache==$LMCACHE_VERSION" + echo "Installing lmcache==$LMCACHE_VERSION" + if command -v uv >/dev/null 2>&1; then + uv pip install --system "lmcache==$LMCACHE_VERSION" \ + || agentic_pip_install --quiet "lmcache==$LMCACHE_VERSION" + else + agentic_pip_install --quiet "lmcache==$LMCACHE_VERSION" + fi fi + # Fail here, loudly, rather than 20 minutes later inside vLLM: this + # runs before the engine loads 1.56 TB of weights. python3 -c "import lmcache.integration.vllm.lmcache_mp_connector" >/dev/null fi python3 -c "import lmcache; print('lmcache', getattr(lmcache,'__version__','?'))" || true + # Assert the two K3 fixes are actually present in whatever got installed. + # A silently-old build would otherwise fail much later with the original + # Mamba ValueError and look like a fresh bug. + python3 - <<'PYEOF' +import sys +try: + from lmcache.integration.vllm import kv_cache_group_edits as e +except Exception as exc: + sys.exit(f"FATAL: cannot import kv_cache_group_edits: {exc}") +names = {type(x).__name__ for x in getattr(e, "_EDITS", ())} +missing = {"_MambaUnifiedViewEdit", "_SubpagedMLAAttentionViewEdit"} - names +if missing: + sys.exit( + f"FATAL: installed lmcache lacks K3 support: missing {sorted(missing)}. " + f"Registered edits: {sorted(names)}. Needs dev >= #4278 (2026-07-28)." + ) +print(f"lmcache K3 edits present: {sorted(names)}") +PYEOF LMCACHE_HOST="${LMCACHE_HOST:-127.0.0.1}" LMCACHE_PORT="${LMCACHE_PORT:-5555}" @@ -317,7 +443,11 @@ except Exception: # SHM and falls back to a pickle path that crashes at load. Cap at 90% of # free /dev/shm so SHM stays enabled, and say so loudly -- the capped value # is the number that actually backs the run. - LMCACHE_L1_SIZE_GB="${LMCACHE_L1_SIZE_GB:-512}" # reference value + if [ "${LMCACHE_PROFILE:-}" = "k3upstream" ]; then + LMCACHE_L1_SIZE_GB="${LMCACHE_L1_SIZE_GB:-100}" # upstream K3 recipe + else + LMCACHE_L1_SIZE_GB="${LMCACHE_L1_SIZE_GB:-512}" # reference value + fi # Optional ceiling on the eager allocation. Memory pinning is unavailable in # the ROCm container ("CudaPinMemoryBackend: neither torch cudart nor @@ -388,6 +518,30 @@ except Exception: --eviction-policy LRU ) ;; + k3upstream) + # Upstream's validated K3 command, verbatim apart from --host and the + # --http-* pair, which this recipe's healthcheck needs and which every + # other profile already passes: + # lmcache server --port 6555 --chunk-size 768 --max-workers 4 \ + # --l1-size-gb 100 --eviction-policy LRU + # --chunk-size comes from the N derivation above (1536 under fp8, not + # upstream's bf16 768). L1 defaults to upstream's 100 GB here rather + # than the reference 512 GB: memory pinning is unavailable in the ROCm + # container, so LMCache allocates the WHOLE L1 pool before serving, + # and a smaller pool means a much shorter and more predictable + # bring-up. Raise it via LMCACHE_L1_SIZE_GB once the arm is green. + LMCACHE_CMD=( + lmcache server + --host "$LMCACHE_HOST" + --port "$LMCACHE_PORT" + --http-host "$LMCACHE_HOST" + --http-port "$LMCACHE_HTTP_PORT" + --chunk-size "$LMCACHE_CHUNK_SIZE" + --max-workers "${LMCACHE_MAX_WORKERS:-4}" + --l1-size-gb "$LMCACHE_L1_SIZE_GB" + --eviction-policy LRU + ) + ;; reference) LMCACHE_CMD=( lmcache server @@ -408,7 +562,7 @@ except Exception: ) ;; *) - echo "Error: unsupported LMCACHE_PROFILE '$LMCACHE_PROFILE' (expected: reference, k2.7)" >&2 + echo "Error: unsupported LMCACHE_PROFILE '$LMCACHE_PROFILE' (expected: reference, k2.7, k3upstream)" >&2 exit 1 ;; esac @@ -770,6 +924,7 @@ VLLM_CMD=( --reasoning-parser kimi_k3 "${MAX_MODEL_LEN_ARGS[@]}" "${PREFIX_CACHE_ARGS[@]}" + "${LMCACHE_MAMBA_CACHE_MODE_ARGS[@]}" "${KV_CACHE_DTYPE_ARGS[@]}" "${KV_BLOCK_SIZE_ARGS[@]}" "${EAGER_ARGS[@]}" diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 7027c2f36e..50320c06b5 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -828,6 +828,46 @@ kimik3-fp4-mi355x-vllm-agentic-fp8none: search-space: - { tp: 8, kv-offloading: none, conc-list: [1, 4, 8, 16] } +kimik3-fp4-mi355x-vllm-agentic-lmcachek3: + # LMCache on Kimi-K3, fp8 KV, c8 -- the first attempt with an LMCache build + # that actually supports this model. + # + # Every prior LMCache attempt died at engine init with + # ValueError: expected a Mamba [conv_state, ssm_state] tensor list, got Tensor + # because K3's Kimi Delta Attention supplies ONE fused state tensor. That is + # now fixed upstream: LMCache #4206 (2026-07-23) added _MambaUnifiedViewEdit + # for unified Mamba state, and #4278 (2026-07-28) added + # _SubpagedMLAAttentionViewEdit for K3's kernel-paged MLA tensor. LMCache + # also shipped its own validated K3 recipe (#4277, + # docs/source/recipes/kimi_k3.rst). None of that is in 0.5.1, which is what + # this recipe pinned, and none of it is in any PyPI release (latest 0.5.2). + # + # version: "git" makes the recipe build from source at the #4278 merge + # commit with ROCm flags, since nightly wheels are CUDA-only. The recipe + # asserts both edit classes are registered before loading 1.56 TB of + # weights, so a stale build fails in seconds rather than 20 minutes. + # + # backend name lmcache-k3 selects upstream's minimal validated server + # command instead of this recipe's `reference` flag set -- those extra flags + # were never validated on K3 and an unrecognised flag is a silent + # server-start failure. + # + # c8 only. It is the peak of the no-offload fp8 curve (682.63 tok/s/GPU, + # run 30525206671), so it is the point where an offload tier has to prove + # itself. + image: vllm/vllm-openai-rocm:kimi-k3 + model: moonshotai/Kimi-K3 + model-prefix: kimik3 + runner: cluster:mi355x-amds + precision: fp4 + framework: vllm + multinode: false + scenarios: + agentic-coding: + - dram-utilization: 0.80 + search-space: + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache-k3, version: "git" }, conc-list: [8] } + kimik3-fp4-mi355x-vllm-agentic-fp8lazy: # fp8 KV + SimpleCPUOffloadConnector in LAZY mode. One variable off the cell # that has now failed 4/4 (g17/g19/g12/g12), and the block-size attempt in run From 938a922895faabb53ce273403ba2711f47f96003 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Fri, 31 Jul 2026 01:50:44 +0900 Subject: [PATCH 47/55] [AMD][AgentX] kimik3: give the LMCache arm headroom (mns 32, GMU 0.85) Attempt 1 (run 30559676068) cleared every LMCache-specific hurdle: the ROCm source build at #4278, both _MambaUnifiedViewEdit and _SubpagedMLAAttentionViewEdit registered, --mamba-cache-mode align accepted, N=1536 confirmed in the startup log, LMCache server healthy in 11s. The long-standing "expected a Mamba [conv_state, ssm_state] tensor list, got Tensor" blocker is gone. It then died on the allocation wall the no-offload arm already has: HSA_STATUS_ERROR_OUT_OF_RESOURCES / "Available Free mem : 0 MB". But at num_computed_tokens=184,320 and kv_cache_usage 5.2%, versus 360,960-373,248 at 10-21% without the connector -- the threshold roughly halved, because LMCacheMPConnector holds GPU-side staging buffers on top of the transient chunked-prefill workspace. Two levers, scoped to the LMCache arm so the no-offload curve is untouched: - max_num_seqs 128 -> 32. The chunked MLA workspace is sized from max_num_seqs and had already doubled under fp8 (98,304 -> 196,608 rows). 128 slots is meaningless at concurrency 8; the widest tree fan-out in the trace is 13 live streams, so 32 keeps headroom and cuts the workspace 4x. - gpu_memory_utilization 0.88 -> 0.85 (~36 GiB free vs ~28). An arm with an extra GPU-side consumer needs more slack than one without. Both overridable; both should be walked back toward the no-offload values before quoting a like-for-like number against 682.63 tok/s/GPU. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 0c372c6b38..603ad15925 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -328,6 +328,36 @@ case "${KV_OFFLOAD_BACKEND:-}" in # and gates the fused latent-MoE tail in the K3 model definition. export VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION="${VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION:-1}" + # GPU-memory headroom for the LMCache arm specifically. + # + # Attempt 1 (run 30559676068) cleared every LMCache-specific hurdle -- the + # source build, both K3 edit classes, `align`, N=1536, a healthy server -- + # and then died on the SAME allocation wall the no-offload arm hits: + # HSA_STATUS_ERROR_OUT_OF_RESOURCES / "Available Free mem : 0 MB" on the + # workers. But it hit it at num_computed_tokens=184,320 with kv_cache_usage + # at 5.2%, against 360,960-373,248 at 10-21% without the connector. The + # threshold roughly HALVED, because LMCacheMPConnector holds GPU-side + # staging buffers on top of the transient chunked-prefill workspace. + # + # Two levers, both scoped to this arm so the no-offload curve is untouched: + # + # 1. max_num_seqs 128 -> 32. This is the structural one. The chunked MLA + # workspace is sized from max_num_seqs, and under fp8 it had already + # doubled (98,304 -> 196,608 rows) because the page went 768 -> 1536. + # 128 slots was never meaningful at concurrency 8 -- the widest tree + # fan-out observed in the trace is 13 live streams in one lane, so 32 + # leaves real headroom while cutting the workspace 4x. + # 2. gpu_memory_utilization 0.88 -> 0.85 (244.8 GiB, ~36 GiB free vs ~28). + # An arm with an extra GPU-side consumer needs more slack than one + # without; 0.88 is calibrated for the no-offload case. + # + # Both are overridable, and both should be walked back toward the + # no-offload values once this arm produces a number -- a like-for-like + # comparison against 682.63 tok/s/GPU eventually needs matching + # max_num_seqs. + MAX_NUM_SEQS="${MAX_NUM_SEQS:-32}" + GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.85}" + # LMCache is NOT in the kimi-k3 image (verified: no `lmcache` module and no # CLI), so build it against ROCm. Clone to a container-local dir, NOT the # bind-mounted /workspace, so a later job's `clean: true` checkout does not From 8eaf651eb1c85c49578099bf888a06f644cbbbef Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Fri, 31 Jul 2026 04:35:03 +0900 Subject: [PATCH 48/55] [AMD][AgentX] kimik3: size the LMCache L1 pool from the DRAM budget Attempt 2 (run 30563325963) is the first LMCache run to complete on Kimi-K3 -- full 3600 s, 203 requests, CI green. But the host tier did nothing: 122,770 chunks stored, 1,879 evictions, external prefix cache hit rate 0.0%. A write-only cache. Cause is L1 sizing. The recipe took upstream's --l1-size-gb 100, which is a does-it-work value. K3 costs ~108 KiB of KV per token at fp8, so 100 GB holds under 1M tokens -- smaller than the 3,322,266-token GPU pool it is meant to back, so entries are evicted before they can be reused. It also dragged the GPU tier down: prefix hit 88.4% -> 33.6%, KV usage 87.5% -> 52.7%, consistent with vLLM releasing blocks to an external tier that discarded them. Net throughput 602.31 vs 682.63 tok/s/GPU (-11.8%). Size L1 from TOTAL_CPU_DRAM_GB instead, as the agentic README requires and as the lmcache-budget name already did. /dev/shm is the real ceiling and the existing 90% cap applies: 1512 GB free was observed on this fleet, so this lands near 1360 GB, ~13.6x attempt 2 and well above the GPU pool. kimik2.7 ran a 1199 GB pool on this fleet. Only L1 changes; mns 32 / GMU 0.85 stay, since those are what got the arm past the allocation wall. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 23 ++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 603ad15925..659ffe1f9f 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -474,7 +474,28 @@ PYEOF # free /dev/shm so SHM stays enabled, and say so loudly -- the capped value # is the number that actually backs the run. if [ "${LMCACHE_PROFILE:-}" = "k3upstream" ]; then - LMCACHE_L1_SIZE_GB="${LMCACHE_L1_SIZE_GB:-100}" # upstream K3 recipe + # Upstream's recipe says --l1-size-gb 100, but that is a + # does-it-work value, not a capacity value. Attempt 2 (run + # 30563325963) ran the whole 3600 s green at 100 GB and the host tier + # did NOTHING: 122,770 chunks stored, 1,879 evictions, and an external + # prefix cache hit rate of 0.0%. A write-only cache. + # + # The arithmetic says it could not have worked. K3 costs ~217 KiB of + # KV per token at bf16, ~108 KiB at fp8, so 100 GB holds well under + # 1M tokens -- SMALLER than the 3,322,266-token GPU pool it is meant + # to back. An offload tier below the size of the GPU pool can only + # evict before reuse. Worse, it dragged the GPU tier down with it: + # GPU prefix hit fell 88.4% -> 33.6% and GPU KV usage 87.5% -> 52.7%, + # consistent with vLLM releasing blocks to an external tier that then + # threw them away. + # + # Size from the matrix's DRAM budget instead, which is what + # benchmarks/single_node/agentic/README.md asks recipes to do and what + # the lmcache-budget name already did. The /dev/shm cap below is the + # real ceiling: 1512 GB free was observed on this fleet, so this + # lands near 1360 GB -- ~13.6x attempt 2 and comfortably above the + # GPU pool. kimik2.7 ran a 1199 GB pool on this same fleet. + LMCACHE_L1_SIZE_GB="${LMCACHE_L1_SIZE_GB:-$TOTAL_CPU_DRAM_GB}" else LMCACHE_L1_SIZE_GB="${LMCACHE_L1_SIZE_GB:-512}" # reference value fi From f6f14be0013ee07a1cc5da26be939ba416cf93b8 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Fri, 31 Jul 2026 05:35:26 +0900 Subject: [PATCH 49/55] =?UTF-8?q?[AMD][AgentX]=20kimik3:=20L1=3D512GB=20?= =?UTF-8?q?=E2=80=94=201360GB=20breaks=20GPU=20IPC=20registration?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Attempt 3 (run 30575475359) sized L1 from TOTAL_CPU_DRAM_GB, shm-capped to 1360 GB, and broke KV-cache registration: Error handling request RequestType.REGISTER_KV_CACHE torch.UntypedStorage._new_shared_cuda(...) torch.AcceleratorError: CUDA error: invalid device pointer once per TP rank, after which all 8 workers died on "LMCache server did not respond to register_kv_caches within 300.0s". The timeout was the symptom. Attempt 2 at 100 GB logged zero IPC errors and registered cleanly, and L1 size was the only difference, so pinning ~1.36 TB of host memory is what exhausts the driver's mapping resources and invalidates the GPU IPC import. The lazy allocator was NOT at fault, contrary to the prior note in this file: 134 expansions finished in 3m37s, well before vLLM finished loading weights. L1 therefore has a working range. GPU KV is ~52 GiB/rank at GMU 0.85, ~416 GB across TP8, so the host tier has to exceed that to be worth anything -- which is why 100 GB was write-only (0.0% external hits). 512 GB, the AMD reference value, clears the GPU tier at ~38% of the size that broke IPC. If it also fails to register, bisect down (256, 128) rather than assuming the connector is broken. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 32 +++++++++++++++---- 1 file changed, 25 insertions(+), 7 deletions(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 659ffe1f9f..59896e7072 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -489,13 +489,31 @@ PYEOF # consistent with vLLM releasing blocks to an external tier that then # threw them away. # - # Size from the matrix's DRAM budget instead, which is what - # benchmarks/single_node/agentic/README.md asks recipes to do and what - # the lmcache-budget name already did. The /dev/shm cap below is the - # real ceiling: 1512 GB free was observed on this fleet, so this - # lands near 1360 GB -- ~13.6x attempt 2 and comfortably above the - # GPU pool. kimik2.7 ran a 1199 GB pool on this same fleet. - LMCACHE_L1_SIZE_GB="${LMCACHE_L1_SIZE_GB:-$TOTAL_CPU_DRAM_GB}" + # Attempt 3 (run 30575475359) then sized L1 from TOTAL_CPU_DRAM_GB, + # which the /dev/shm cap landed at 1360 GB, and that BROKE GPU IPC: + # Error handling request RequestType.REGISTER_KV_CACHE + # torch.UntypedStorage._new_shared_cuda(...) + # torch.AcceleratorError: CUDA error: invalid device pointer + # (hipErrorInvalidDevicePointer) + # once per TP rank, so all 8 vLLM workers then died on + # "LMCache server did not respond to register_kv_caches within 300.0s". + # The timeout was the symptom; the registration itself failed. + # + # Attempt 2 at 100 GB logged ZERO IPC errors and registered cleanly. + # L1 size was the only difference between the two runs, so pinning + # ~1.36 TB of host memory is what exhausts the driver's mapping + # resources and makes the subsequent GPU IPC import invalid. Note the + # lazy allocator was NOT the problem here: 134 expansions completed in + # 3m37s and the pool was fully allocated long before vLLM finished + # loading weights. + # + # So L1 has a working range, and the target is above the GPU tier but + # well below the IPC ceiling. GPU KV is ~52 GiB/rank at GMU 0.85, so + # ~416 GB across TP8 for the 3.32M-token pool. 512 GB -- the AMD + # reference value -- clears that while staying at ~38% of the size + # that broke IPC. If 512 also fails to register, bisect downward + # (256, then 128) rather than assuming the connector is broken. + LMCACHE_L1_SIZE_GB="${LMCACHE_L1_SIZE_GB:-512}" else LMCACHE_L1_SIZE_GB="${LMCACHE_L1_SIZE_GB:-512}" # reference value fi From 463f1a98a75c3afde4bcba84b8e2ea6879de21f1 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Fri, 31 Jul 2026 08:34:36 +0900 Subject: [PATCH 50/55] [AMD][AgentX] kimik3: add c16 to the LMCache arm c8 is measured and LMCache loses there: 601.64 vs 682.63 tok/s/GPU (-11.9%, run 30579846678). The reason is that there is nothing for a host tier to do at c8 -- 67,827 stores against 80 retrievals, 217 evictions, GPU pool never above 52.4% -- so vLLM never has to spill, while LMCache still pays to store every block and drags the GPU tier down (prefix hit 88.4% -> 58.3%). c16 is the case that should invert it. Without offload, c16 is where the no-offload arm collapses: 412.21 tok/s/GPU, GPU prefix hit 3.9%, KV pinned at 100.0%, TTFT mean 323 s. The working set stops fitting, which is the regime a host tier exists for. If LMCache cannot win at c16 it has no case on K3. max_num_seqs stays 32: aiperf bounds in-flight requests by --concurrency, so c16 never presents more than 16 sequences, and raising it would re-inflate the chunked MLA workspace behind the attempt-1 OOM. c8 repeats so the pair is same-run and same-code. Co-Authored-By: Claude Opus 5 (1M context) --- configs/amd-master.yaml | 30 ++++++++++++++++++++++++++---- 1 file changed, 26 insertions(+), 4 deletions(-) diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 50320c06b5..c87de1b0a5 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -852,9 +852,31 @@ kimik3-fp4-mi355x-vllm-agentic-lmcachek3: # were never validated on K3 and an unrecognised flag is a silent # server-start failure. # - # c8 only. It is the peak of the no-offload fp8 curve (682.63 tok/s/GPU, - # run 30525206671), so it is the point where an offload tier has to prove - # itself. + # c8 and c16. + # + # c8 is the peak of the no-offload fp8 curve (682.63 tok/s/GPU, run + # 30525206671) and is now measured with LMCache: 601.64 tok/s/GPU, -11.9% + # (run 30579846678). The reason it loses is that there is no work for a + # host tier at this concurrency -- the LMCache server logged 67,827 stores + # against 80 retrievals with only 217 evictions, because the GPU pool never + # fills (52.4% usage) so vLLM never has to spill. LMCache still pays to + # store every block and drags the GPU tier down with it (prefix hit + # 88.4% -> 58.3%). + # + # c16 is the case that should invert that. Without offload, c16 is where + # the no-offload arm collapses: 412.21 tok/s/GPU, GPU prefix hit 3.9%, KV + # pinned at 100.0%, TTFT mean 323 s (run 30525206671). The working set no + # longer fits, so the GPU tier thrashes -- which is exactly the regime a + # host tier exists for. If LMCache cannot win at c16 it has no case on this + # model. + # + # max_num_seqs stays 32. aiperf bounds in-flight requests by --concurrency, + # so c16 never presents more than 16 sequences; 32 is still 2x headroom and + # raising it would re-inflate the chunked MLA workspace that caused the + # attempt-1 OOM. + # + # c8 repeats deliberately: it gives a same-run, same-code pair against c16 + # rather than a cross-run comparison. image: vllm/vllm-openai-rocm:kimi-k3 model: moonshotai/Kimi-K3 model-prefix: kimik3 @@ -866,7 +888,7 @@ kimik3-fp4-mi355x-vllm-agentic-lmcachek3: agentic-coding: - dram-utilization: 0.80 search-space: - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache-k3, version: "git" }, conc-list: [8] } + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache-k3, version: "git" }, conc-list: [8, 16] } kimik3-fp4-mi355x-vllm-agentic-fp8lazy: # fp8 KV + SimpleCPUOffloadConnector in LAZY mode. One variable off the cell From f815885ea4c93947ec4fe74c8c68c3aa737ae775 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Fri, 31 Jul 2026 09:51:40 +0900 Subject: [PATCH 51/55] [AMD][AgentX] kimik3: size LMCache L1 as a fraction of free /dev/shm (0.60) L1 has a known working range on this fleet: 512 GB registers cleanly (run 30579846678) and 1360 GB breaks GPU IPC with hipErrorInvalidDevicePointer on every TP rank (run 30575475359). 512 GB lifted external hits 0.00% -> 0.82% but did not move throughput. LMCACHE_L1_SHM_FRACTION sizes L1 off FREE /dev/shm, which is the resource that actually binds -- 1512 GB free here, so 0.60 gives ~907 GB. Sizing off shm rather than TOTAL_CPU_DRAM_GB is deliberate: the DRAM budget is 2399 GB, so any fraction of it lands past the shm cap and straight back on the 1360 GB value that already broke IPC. 0.60 bisects the range -- 1.8x the size that works, 0.67x the size that fails -- so the run locates the IPC ceiling as well as testing whether more cache changes the throughput picture. If registration fails, the ceiling is between 512 and 907 and the next step is 0.45 (~680 GB), not a return to 512. Falls back to 512 GB if df cannot read /dev/shm. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 36 +++++++++++++++---- 1 file changed, 29 insertions(+), 7 deletions(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 59896e7072..e06226fe51 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -507,13 +507,35 @@ PYEOF # 3m37s and the pool was fully allocated long before vLLM finished # loading weights. # - # So L1 has a working range, and the target is above the GPU tier but - # well below the IPC ceiling. GPU KV is ~52 GiB/rank at GMU 0.85, so - # ~416 GB across TP8 for the 3.32M-token pool. 512 GB -- the AMD - # reference value -- clears that while staying at ~38% of the size - # that broke IPC. If 512 also fails to register, bisect downward - # (256, then 128) rather than assuming the connector is broken. - LMCACHE_L1_SIZE_GB="${LMCACHE_L1_SIZE_GB:-512}" + # So L1 has a working range: above the GPU tier (GPU KV is ~52 GiB/rank + # at GMU 0.85, ~416 GB across TP8) and below the IPC ceiling somewhere + # under 1360 GB. 512 GB registered cleanly (run 30579846678) and lifted + # external hits 0.00% -> 0.82%, but throughput did not move. + # + # LMCACHE_L1_SHM_FRACTION sizes L1 as a fraction of FREE /dev/shm, + # which is the resource that actually binds (shm was 1512 GB free on + # this fleet, so 0.60 -> ~907 GB). Sizing off shm rather than + # TOTAL_CPU_DRAM_GB is deliberate: the DRAM budget is 2399 GB, so any + # fraction of it lands past the shm cap and straight back on the 1360 + # GB value that broke IPC. + # + # 0.60 bisects the known range -- 1.8x the 512 GB that works, 0.67x + # the 1360 GB that fails -- so this run locates the IPC ceiling as + # well as testing whether more cache changes the throughput picture. + # If it fails to register, the ceiling is between 512 and 907 and the + # next step is 0.45 (~680 GB), not a return to 512. + LMCACHE_L1_SHM_FRACTION="${LMCACHE_L1_SHM_FRACTION:-0.60}" + _shm_free_gb=$(df -BG --output=avail /dev/shm 2>/dev/null | tail -1 | tr -dc '0-9') + if [ -n "$_shm_free_gb" ] && [ "${_shm_free_gb:-0}" -gt 0 ]; then + _l1_from_shm=$(awk -v f="$LMCACHE_L1_SHM_FRACTION" -v s="$_shm_free_gb" \ + 'BEGIN { printf "%d", f * s }') + LMCACHE_L1_SIZE_GB="${LMCACHE_L1_SIZE_GB:-$_l1_from_shm}" + echo "LMCache L1 sized from /dev/shm: ${LMCACHE_L1_SHM_FRACTION} x ${_shm_free_gb}G free = ${LMCACHE_L1_SIZE_GB}G" + else + # df failed -- fall back to the value already proven to register. + LMCACHE_L1_SIZE_GB="${LMCACHE_L1_SIZE_GB:-512}" + echo "WARNING: could not read free /dev/shm; falling back to L1=${LMCACHE_L1_SIZE_GB}G" >&2 + fi else LMCACHE_L1_SIZE_GB="${LMCACHE_L1_SIZE_GB:-512}" # reference value fi From aae1395f6b80743acb2e41831c4e2ad3fc14e696 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Fri, 31 Jul 2026 14:24:11 +0900 Subject: [PATCH 52/55] perf(amd): tune Kimi-K3 LMCache workers and eviction Use one LMCache affinity worker per TP rank and apply the proven 0.85/0.10 eviction settings to the K3 profile. --- benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index e06226fe51..831e69c9ec 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -610,9 +610,10 @@ PYEOF ) ;; k3upstream) - # Upstream's validated K3 command, verbatim apart from --host and the - # --http-* pair, which this recipe's healthcheck needs and which every - # other profile already passes: + # Upstream's validated K3 command, plus the worker and eviction tuning + # needed by this TP8 agentic workload. The pinned LMCache revision + # supports these flags, and four workers serialize pairs of TP ranks. + # Upstream's original command was: # lmcache server --port 6555 --chunk-size 768 --max-workers 4 \ # --l1-size-gb 100 --eviction-policy LRU # --chunk-size comes from the N derivation above (1536 under fp8, not @@ -628,8 +629,10 @@ PYEOF --http-host "$LMCACHE_HOST" --http-port "$LMCACHE_HTTP_PORT" --chunk-size "$LMCACHE_CHUNK_SIZE" - --max-workers "${LMCACHE_MAX_WORKERS:-4}" + --max-workers "${LMCACHE_MAX_WORKERS:-$TP}" --l1-size-gb "$LMCACHE_L1_SIZE_GB" + --eviction-trigger-watermark "$LMCACHE_EVICTION_WATERMARK" + --eviction-ratio "$LMCACHE_EVICTION_RATIO" --eviction-policy LRU ) ;; From 95c37a68c4109974a019a6286fb8e6218c05591f Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Fri, 31 Jul 2026 15:57:48 +0900 Subject: [PATCH 53/55] test(amd): add Kimi-K3 LMCache c32 --- configs/amd-master.yaml | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index c87de1b0a5..eb1b62c8bf 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -852,7 +852,7 @@ kimik3-fp4-mi355x-vllm-agentic-lmcachek3: # were never validated on K3 and an unrecognised flag is a silent # server-start failure. # - # c8 and c16. + # c8, c16, and c32. # # c8 is the peak of the no-offload fp8 curve (682.63 tok/s/GPU, run # 30525206671) and is now measured with LMCache: 601.64 tok/s/GPU, -11.9% @@ -870,10 +870,14 @@ kimik3-fp4-mi355x-vllm-agentic-lmcachek3: # host tier exists for. If LMCache cannot win at c16 it has no case on this # model. # + # c32 extends the capacity-pressure curve after LMCache recovered c16 from + # 441.99 to 607.45 tok/s/GPU when L1 grew from 512 to 907 GB. This tests + # whether the larger host tier can keep helping once the live working set + # doubles again. + # # max_num_seqs stays 32. aiperf bounds in-flight requests by --concurrency, - # so c16 never presents more than 16 sequences; 32 is still 2x headroom and - # raising it would re-inflate the chunked MLA workspace that caused the - # attempt-1 OOM. + # so it provides 2x headroom at c16 and exactly covers c32. Raising it would + # re-inflate the chunked MLA workspace that caused the attempt-1 OOM. # # c8 repeats deliberately: it gives a same-run, same-code pair against c16 # rather than a cross-run comparison. @@ -888,7 +892,7 @@ kimik3-fp4-mi355x-vllm-agentic-lmcachek3: agentic-coding: - dram-utilization: 0.80 search-space: - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache-k3, version: "git" }, conc-list: [8, 16] } + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache-k3, version: "git" }, conc-list: [8, 16, 32] } kimik3-fp4-mi355x-vllm-agentic-fp8lazy: # fp8 KV + SimpleCPUOffloadConnector in LAZY mode. One variable off the cell From 0aad1dda766ac36e9f45636d09e045c6ac921097 Mon Sep 17 00:00:00 2001 From: hyukjlee Date: Fri, 31 Jul 2026 16:37:44 +0900 Subject: [PATCH 54/55] [AMD][AgentX] kimik3: GMU 0.85 for c1 only, plus a c1 no-offload key c1 is the one point on the fp8 curve with no result at any memory fraction. It died at num_computed_tokens 373,248 at 0.88 (runs 30453589555, 30525206671) and at 0.90 (run 30521327099), every time with num_running_reqs=1 and KV usage 10-21% -- the probabilistic chunked-prefill workspace OOM, not KV exhaustion. c1 is the worst case for that failure, not the mildest: one lane replays the deepest trajectories with nothing to interleave, so it reaches the fatal band early and reliably. Trading pool for headroom is nearly free at c1. One trajectory cannot fill the pool -- the bf16 c1 cell sustained 13 requests at ~291K average ISL against a 1.96M-token pool -- so the ~530K tokens 0.85 gives up were never usable, while the extra ~13 GiB/rank goes to the transient allocation actually killing it. Scoped to CONC=1 because the agentic matrix has no per-cell env channel and the no-offload arm has no backend name to carry a variant. c4/c8/c16 stay at 0.88, so the settled curve is untouched. This does make c1 not strictly like-for-like with the rest; that is the right trade while it has no data at all, and it should be revisited once the build_mla_chunked_context_metadata aggregate cap lands and the whole curve can share one fraction. If c1 still dies at 0.85, the memory fraction is not the lever -- do not keep shaving it. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 24 +++++++++++++- configs/amd-master.yaml | 33 +++++++++++++++++++ 2 files changed, 56 insertions(+), 1 deletion(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 831e69c9ec..6b91ddf50d 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -968,7 +968,29 @@ fi # first (the build_mla_chunked_context_metadata aggregate-chunk cap that only # exists in hyukjleeamd/kimi-k3-mla-b128:fp8-kv-fixed, not in the mla_gluon # gist patched below, which is a decode-path fix only). -GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.88}" +# c1 is the exception, and it gets 0.85. It is the ONE cell with no result at +# any memory fraction: it died at num_computed_tokens 373,248 at BOTH 0.88 +# (runs 30453589555, 30525206671) and 0.90 (run 30521327099), always with +# num_running_reqs=1. A single lane replays the deepest trajectories with +# nothing to interleave, so it reaches the fatal band early and reliably -- +# c1 is the worst case for this failure, not the mildest. +# +# Trading pool for headroom is close to free at c1 specifically. One +# trajectory cannot fill the pool: the bf16 c1 cell sustained 13 requests at +# ~291K average ISL against a 1.96M-token pool, and every fp8 death happened +# at 10-21% KV usage. So the ~530K tokens that 0.85 gives up were never going +# to be used, while the ~13 GiB/rank of extra headroom goes straight to the +# transient chunked-prefill allocation that is actually killing the cell. +# +# This makes c1 not strictly like-for-like with the rest of the curve. That is +# the right trade while c1 has no data at all; revisit once the +# build_mla_chunked_context_metadata aggregate cap lands and the whole curve +# can return to one fraction. +if [ "${CONC:-}" = "1" ]; then + GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.85}" +else + GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.88}" +fi MAX_NUM_SEQS="${MAX_NUM_SEQS:-128}" MAX_NUM_BATCHED_TOKENS="${MAX_NUM_BATCHED_TOKENS:-4096}" diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index eb1b62c8bf..4df42d566b 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -828,6 +828,39 @@ kimik3-fp4-mi355x-vllm-agentic-fp8none: search-space: - { tp: 8, kv-offloading: none, conc-list: [1, 4, 8, 16] } +kimik3-fp4-mi355x-vllm-agentic-fp8none-c1: + # The one missing point on the fp8 no-offload curve. + # + # c1 has NO result at any memory fraction. It died at num_computed_tokens + # 373,248 at 0.88 (runs 30453589555 and 30525206671) and at 0.90 (run + # 30521327099), every time with num_running_reqs=1 and KV usage 10-21% -- + # the probabilistic chunked-prefill workspace OOM, not KV exhaustion. + # c1 is the WORST case for it: one lane replays the deepest trajectories + # with nothing to interleave, so it reaches the fatal band early and every + # time. + # + # The recipe now drops GPU_MEM_UTIL to 0.85 for CONC=1 only. At c1 the pool + # is hugely oversized anyway (13 requests sustained against a 1.96M-token + # pool in the bf16 run), so the ~530K tokens given up were never usable, + # while the extra ~13 GiB/rank of headroom goes to the allocation that is + # actually killing the cell. + # + # If this still dies, the memory fraction is not the lever and the + # aggregate-chunk cap in mla_attention.py is required -- do not keep + # shaving GMU. + image: vllm/vllm-openai-rocm:kimi-k3 + model: moonshotai/Kimi-K3 + model-prefix: kimik3 + runner: cluster:mi355x-amds + precision: fp4 + framework: vllm + multinode: false + scenarios: + agentic-coding: + - dram-utilization: 0.80 + search-space: + - { tp: 8, kv-offloading: none, conc-list: [1] } + kimik3-fp4-mi355x-vllm-agentic-lmcachek3: # LMCache on Kimi-K3, fp8 KV, c8 -- the first attempt with an LMCache build # that actually supports this model. From 0b50beafe740996fd591632e14bbf6476aec76c2 Mon Sep 17 00:00:00 2001 From: functionstackx <47992694+functionstackx@users.noreply.github.com> Date: Sat, 1 Aug 2026 23:15:29 -0400 Subject: [PATCH 55/55] [AMD][AgentX] kimik3: reduce the PR to the DSpark arm only Kimi-K3 agentic is onboarded spec-decode-only from day 0 -- the non-DSpark arm is neither run nor published (MODELS.md) -- so everything in this branch that was not the DSpark path was bring-up scaffolding. Strip it. configs/amd-master.yaml: 15 keys -> 1. Drops the non-DSpark baseline (kimik3-fp4-mi355x-vllm-agentic) and the thirteen throwaway bring-up keys (-fp8kvtest, -lmprofile, -fp8sweep, -fp8retry, -fp8eval, -fp8none, -fp8none-c1, -lmcachek3, -fp8lazy, -fp8kv, -retry, -native, -bringup), each of which was explicitly labelled 'delete once it has a result' or superseded. The surviving key is renamed kimik3-fp4-mi355x-vllm-agentic-mtp -> kimik3-fp4-mi355x-vllm-agentic-dspark to match the NVIDIA sibling kimik3-fp4-b300-vllm-agentic-dspark; the script suffix stays _mtp because the launcher derives it from spec-decoding, not the key name. Its inert dram-utilization is dropped -- a GPU-only arm resolves TOTAL_CPU_DRAM_GB to 0 either way. Generated matrix is byte-identical to before: one cell, kimik3_tp8_conc1_kvnone_spec-mtp. kimik3_fp4_mi355x.sh: 1064 -> 438 lines. Removes the KV-offload backend case block (LMCache in four profiles, SimpleCPUOffloadConnector, OffloadingConnector, the Mooncake rejection), the LMCache server lifecycle (LMCACHE_LOG, LMCACHE_PID, wait_for_lmcache_ready, the cleanup hook), and OFFLOAD_ARGS / LMCACHE_MAMBA_CACHE_MODE_ARGS from the serve command -- none of it reachable from a kv-offloading: none arm. A matrix row that asks for a host tier now fails loudly instead of being silently ignored. The gpu-memory-utilization rationale is condensed to the run IDs and the conclusion, since the DSpark wrapper pins 0.95 and those defaults are now a stand-alone fallback. Kept: the launcher's Kimi-K3 NFS hub-cache mount and salloc denylist, and the e2e-tests.yml agentic-eval passthrough -- without the spec-decoding half the DSpark eval job resolves to the non-DSpark script name. Co-Authored-By: Claude Opus 5 (1M context) --- .../single_node/agentic/kimik3_fp4_mi355x.sh | 726 ++---------------- .../agentic/kimik3_fp4_mi355x_mtp.sh | 4 +- configs/amd-master.yaml | 429 +---------- perf-changelog.yaml | 6 +- .../matrix_logic/test_kimik3_mi355x_dspark.py | 2 +- 5 files changed, 71 insertions(+), 1096 deletions(-) diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh index 6b91ddf50d..15f3235820 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh @@ -17,9 +17,6 @@ set -x # with env VLLM_ROCM_USE_AITER=1 SAFETENSORS_FAST_GPU=1 AITER_SITUV2_A8W4=1 # AITER_BF16_FP8_MOE_BOUND=0 VLLM_USE_BREAKABLE_CUDAGRAPH=0. # -# The DRAM-offload arm adds LMCache's LMCacheMPConnector against a local -# `lmcache server`, again matching the reference command. -# # K3 is a 2.8T-parameter natively-multimodal MoE (896 routed experts, 16/token # plus shared) on Kimi Delta Attention, gated MLA and Attention Residuals, with # a 1M-token native context. @@ -33,8 +30,10 @@ set -x # MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR, DURATION, # EP_SIZE # -# KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=lmcache. Mooncake is -# deliberately unsupported: the upstream recipe marks both +# GPU-only. The single shipped key +# (kimik3-fp4-mi355x-vllm-agentic-dspark) sets kv-offloading: none, and this +# recipe rejects a row that asks for a host tier rather than silently ignoring +# it. Mooncake in particular stays unsupported: the upstream recipe marks both # kv_store_distributed_mooncake and kv_store_centralized_mooncake as # `unsupported` on every hardware target for this model. # @@ -125,650 +124,50 @@ export AIPERF_HTTP_TCP_USER_TIMEOUT=900000 # ---- Server config ---------------------------------------------------------- SERVER_LOG="$RESULT_DIR/server.log" -LMCACHE_LOG="$RESULT_DIR/lmcache_server.log" mkdir -p "$RESULT_DIR" SERVER_PID="" -LMCACHE_PID="" cleanup_agentic_services() { local exit_code=$? trap - EXIT INT TERM set +e stop_background_process_tree "$SERVER_PID" "vLLM server" 60 - stop_background_process_tree "$LMCACHE_PID" "LMCache server" exit "$exit_code" } trap cleanup_agentic_services EXIT trap 'exit 130' INT trap 'exit 143' TERM -wait_for_lmcache_ready() { - { set +x; } 2>/dev/null - # Generous by default. With memory pinning unavailable on ROCm, LMCache - # disables its LazyMemoryAllocator and allocates the whole L1 pool UP FRONT, - # so "ready" can be minutes away and scales with --l1-size-gb. A 180s budget - # timed out at exactly 178s on a 2249 GB pool. - local attempts="${LMCACHE_READY_ATTEMPTS:-1800}" - # The health route has moved across LMCache versions, and a wrong path is - # indistinguishable from a slow start: both just never return 200. Probe the - # known spellings and report which one answered. - local paths=(/healthcheck /health /v1/health /status /) - local i p - for ((i = 1; i <= attempts; i++)); do - for p in "${paths[@]}"; do - if curl --output /dev/null --silent --fail \ - "http://127.0.0.1:${LMCACHE_HTTP_PORT}${p}"; then - echo "LMCache server healthy after ${i}s (endpoint ${p})" - set -x - return 0 - fi - done - if [[ -n "$LMCACHE_PID" ]] && ! kill -0 "$LMCACHE_PID" 2>/dev/null; then - echo "LMCache server died before becoming healthy. Log follows:" >&2 - cat "$LMCACHE_LOG" >&2 || true - exit 1 - fi - # Heartbeat so a slow eager allocation is visibly distinct from a hang. - if (( i % 60 == 0 )); then - echo " ... still waiting for LMCache (${i}s / ${attempts}s), L1=${LMCACHE_L1_SIZE_GB:-?}GB" - fi - sleep 1 - done - echo "Timed out after ${attempts}s waiting for LMCache healthcheck." >&2 - echo "Tried endpoints: ${paths[*]} on port ${LMCACHE_HTTP_PORT}. Log follows:" >&2 - cat "$LMCACHE_LOG" >&2 || true +# ---- KV cache --------------------------------------------------------------- +# This recipe ships a single GPU-only arm (kimik3-fp4-mi355x-vllm-agentic-dspark, +# kv-offloading: none), so there is no host-DRAM tier here. The LMCache / +# SimpleCPUOffload / OffloadingConnector plumbing that the bring-up sweeps used +# lives in this branch's history; re-add it with the arm that needs it rather +# than carrying dead connectors. Fail loudly instead of silently ignoring a +# matrix row that asks for one. +if agentic_kv_offload_enabled; then + echo "Error: KV offloading is not wired up for Kimi-K3 on MI355X." >&2 + echo " This recipe ships the GPU-only DSpark arm; got" >&2 + echo " KV_OFFLOADING='${KV_OFFLOADING:-}' KV_OFFLOAD_BACKEND='${KV_OFFLOAD_BACKEND:-}'." >&2 exit 1 -} - -# ---- KV offload ------------------------------------------------------------- -# TOTAL_CPU_DRAM_GB is the aggregate host-DRAM budget the matrix generator -# derives from dram-utilization and the runner's available-cpu-dram-mib, capped -# at the 2,861,022 MiB (3 TB decimal) agentic limit. Per -# benchmarks/single_node/agentic/README.md it must be consumed as given and -# never replaced with a model-specific constant. -OFFLOAD_ARGS=() -# Only the LMCache arms set this (to `--mamba-cache-mode align`); every other -# arm leaves vLLM to pick the mode. -LMCACHE_MAMBA_CACHE_MODE_ARGS=() +fi -# fp8 KV is the DEFAULT for every arm on this model. +# fp8 KV is the DEFAULT for this model. # # It is the first thing that made the trace run well: run 30442578333 (c8, # kvnone, 3600s) scored 32.59 output tok/s / 696 total tok/s/GPU at TTFT 2.6s, -# against 19.17 / 535 / 69.1s for the bf16 offload cell in the same window. -# K3 costs ~217 KiB KV/token -- 3x K2.7 -- so halving the KV element size is -# the single largest lever available, exactly as in -# [[fp8-kv-cache-mandatory]] for kimik2.7. +# against 19.17 / 535 / 69.1s for the bf16 cell in the same window. K3 costs +# ~217 KiB KV/token -- 3x K2.7 -- so halving the KV element size is the single +# largest lever available, exactly as in [[fp8-kv-cache-mandatory]] for +# kimik2.7. # -# This is set here, before the KV_OFFLOAD_BACKEND case below, for two reasons: -# the agentic matrix has no per-cell env channel (so a kvnone cell has no other -# way to ask for fp8), and the mla_gluon patch further down gates on -# KV_CACHE_DTYPE=fp8 and must see it already set. -# -# Set KV_CACHE_DTYPE=auto for a bf16 A/B. +# Set here, ahead of the serve command, because the mla_gluon patch further +# down gates on KV_CACHE_DTYPE=fp8 and must see it already set. The DSpark +# wrapper pins KV_CACHE_DTYPE=auto to match the upstream AMD reproducer, so +# this default only applies to stand-alone runs. KV_CACHE_DTYPE="${KV_CACHE_DTYPE:-fp8}" -if agentic_kv_offload_enabled; then -case "${KV_OFFLOAD_BACKEND:-}" in - lmcache|lmcache-k27|lmcache-budget|lmcache-k3) - require_agentic_kv_offload_backend "$KV_OFFLOAD_BACKEND" - # lmcache-k3: LMCache's OWN validated K3 server command - # (docs/source/recipes/kimi_k3.rst, PR #4277). Deliberately minimal -- - # the `reference` profile's flags (--max-gpu-workers, --l1-align-bytes, - # --eviction-trigger-watermark, --supported-transfer-mode) are not part of - # what upstream validated on K3, and an unrecognised flag is a silent - # server-start failure. Start from the known-good command; add tuning - # flags only once this arm produces a number. - if [ "$KV_OFFLOAD_BACKEND" = "lmcache-k3" ]; then - LMCACHE_PROFILE=k3upstream - fi - # The server profile has to be selectable from config: the agentic matrix - # has no per-cell env channel, so the backend NAME carries it. - # lmcache -> AMD K3 reference server command - # lmcache-k27 -> kimik2.7_fp4_mi355x.sh's server command - if [ "$KV_OFFLOAD_BACKEND" = "lmcache-k27" ]; then - LMCACHE_PROFILE=k2.7 - fi - # lmcache-budget: same reference server, but L1 sized from the agentic DRAM - # budget (shm-capped) instead of the reference's flat 512 GB -- i.e. what - # kimik2.7 does, which ran a 1199 GB pool successfully on this fleet. The - # agentic README also asks recipes to consume TOTAL_CPU_DRAM_GB rather than - # a constant. Kept as a separate backend name because the matrix has no - # per-cell env channel to vary it from config. - if [ "$KV_OFFLOAD_BACKEND" = "lmcache-budget" ]; then - LMCACHE_L1_SIZE_GB="${LMCACHE_L1_SIZE_GB:-$TOTAL_CPU_DRAM_GB}" - fi - - # LMCache on K3 REQUIRES prefix caching. K3's Kimi Delta Attention layers - # are Mamba KV-cache groups, and vLLM only selects mamba_cache_mode='align' - # -- the mode that keeps reusable state snapshots -- when prefix caching is - # enabled. Without it the connector refuses to initialise at KV-transfer - # setup with: - # ValueError: LMCache cannot serve this model's KV cache groups: - # group N: MambaSpec with mamba_cache_mode='none' (only 'align' keeps - # reusable state snapshots) - # (lmcache/integration/vllm/kv_cache_group_edits.py:137). Measured on - # gfx950 with LMCache v0.5.3.dev47. This is a hard dependency, not a tuning - # choice, so force it here rather than leaving it to the caller -- the - # agentic matrix has no per-cell env channel to set it from config. - if [ "${PREFIX_CACHING:-}" = "false" ]; then - echo "Error: PREFIX_CACHING=false is incompatible with KV_OFFLOAD_BACKEND=lmcache." >&2 - echo " LMCache needs mamba_cache_mode='align', which vLLM only selects when" >&2 - echo " prefix caching is enabled. Use KV_OFFLOADING=none to measure without it." >&2 - exit 1 - fi - PREFIX_CACHING=true - - # --max-num-batched-tokens and --chunk-size stay at the AMD reference - # values (4096 / 1024). They were briefly pinned to 768 because LMCache - # dev HEAD (v0.5.3.dev47) rejects the reference values on this hybrid model: - # ValueError: Mamba-hybrid models with LMCache require - # block_size <= max_num_batched_tokens < 2 * block_size ... block_size=768 - # AssertionError: LMCache chunk size should be a multiple of vLLM block size - # CONFIRMED on 0.5.1 as well (run 30348060242, g09): the pinned release - # raises the same ValueError, so these are properties of the LMCache/vLLM - # Mamba-hybrid integration, not of dev HEAD. Only the LazyMemoryAllocator - # stall was dev-specific. Both stay pinned to 768 for the LMCache arms. - # - # Worth keeping in view either way: at ~106k-token average ISL, 768 would - # mean ~138 chunked-prefill steps per turn versus ~26 at 4096, so the two - # settings are not performance-equivalent. - # Both values derive from vLLM's UNIFIED block size N, which is NOT a - # constant: it depends on TP and on the KV dtype. Upstream's recipe quotes - # N=768 for K3 at TP8, but that is the bf16 figure, and it explicitly says - # to "re-read N from vLLM's `Setting attention block size to N tokens` - # startup log line" after changing anything. Measured on this fleet with - # --kv-cache-dtype fp8 at TP8, three independent runs (30453589555, - # 30521327099, 30525206671) all log: - # Setting attention block size to 1536 tokens - # fp8 halves the element size, so twice as many tokens fit one page. Since - # fp8 is this recipe's default, hardcoding 768 would violate BOTH LMCache - # constraints on every default run. - # - # Constraints (lmcache/integration/vllm, and upstream's recipe): - # chunk_size % N == 0 -> chunk_size = N - # N <= max_num_batched_tokens < 2 * N -> sit just under 2N - # `align` snapshots the KDA state only on a block boundary at the end of a - # scheduler step, so the per-step budget must cover at least one full block - # but stay below two. Upstream validated 1500 against N=768 (just under - # 2N); 3000 against N=1536 is the same ratio. - if [ "${KV_CACHE_DTYPE:-}" = "fp8" ]; then - LMCACHE_UNIFIED_BLOCK="${LMCACHE_UNIFIED_BLOCK:-1536}" - else - LMCACHE_UNIFIED_BLOCK="${LMCACHE_UNIFIED_BLOCK:-768}" - fi - LMCACHE_K3_CHUNK_SIZE="${LMCACHE_CHUNK_SIZE_OVERRIDE:-$LMCACHE_UNIFIED_BLOCK}" - MAX_NUM_BATCHED_TOKENS="${LMCACHE_MAX_NUM_BATCHED_TOKENS:-$(( LMCACHE_UNIFIED_BLOCK * 2 - 72 ))}" - LMCACHE_CHUNK_SIZE="$LMCACHE_K3_CHUNK_SIZE" - LMCACHE_CHUNK_SIZE_K27="$LMCACHE_K3_CHUNK_SIZE" - # Guard the derivation rather than trusting it: a wrong N is a 20-minute - # engine-init failure after a 1.56 TB weight load. - if [ $(( LMCACHE_K3_CHUNK_SIZE % LMCACHE_UNIFIED_BLOCK )) -ne 0 ]; then - echo "Error: LMCache chunk-size $LMCACHE_K3_CHUNK_SIZE is not a multiple of N=$LMCACHE_UNIFIED_BLOCK." >&2 - exit 1 - fi - if [ "$MAX_NUM_BATCHED_TOKENS" -lt "$LMCACHE_UNIFIED_BLOCK" ] \ - || [ "$MAX_NUM_BATCHED_TOKENS" -ge $(( LMCACHE_UNIFIED_BLOCK * 2 )) ]; then - echo "Error: --max-num-batched-tokens $MAX_NUM_BATCHED_TOKENS outside [N, 2N) for N=$LMCACHE_UNIFIED_BLOCK." >&2 - exit 1 - fi - echo "LMCache: N=$LMCACHE_UNIFIED_BLOCK (kv-cache-dtype=${KV_CACHE_DTYPE:-auto})" \ - "--max-num-batched-tokens=$MAX_NUM_BATCHED_TOKENS --chunk-size=$LMCACHE_K3_CHUNK_SIZE" - - # Required by upstream's K3 recipe, and by the KDA backend generally: - # `align` is the only Mamba cache mode the KDA backend supports, and it is - # the mode that keeps reusable state snapshots. vLLM picks it implicitly - # when prefix caching is on, but upstream passes it explicitly and so do - # we -- an implicit default that silently flips to 'none' is exactly how - # this arm failed before. - LMCACHE_MAMBA_CACHE_MODE_ARGS=(--mamba-cache-mode align) - - # Upstream's K3 recipe requires this on the LMCache path. It defaults to 0 - # and gates the fused latent-MoE tail in the K3 model definition. - export VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION="${VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION:-1}" - - # GPU-memory headroom for the LMCache arm specifically. - # - # Attempt 1 (run 30559676068) cleared every LMCache-specific hurdle -- the - # source build, both K3 edit classes, `align`, N=1536, a healthy server -- - # and then died on the SAME allocation wall the no-offload arm hits: - # HSA_STATUS_ERROR_OUT_OF_RESOURCES / "Available Free mem : 0 MB" on the - # workers. But it hit it at num_computed_tokens=184,320 with kv_cache_usage - # at 5.2%, against 360,960-373,248 at 10-21% without the connector. The - # threshold roughly HALVED, because LMCacheMPConnector holds GPU-side - # staging buffers on top of the transient chunked-prefill workspace. - # - # Two levers, both scoped to this arm so the no-offload curve is untouched: - # - # 1. max_num_seqs 128 -> 32. This is the structural one. The chunked MLA - # workspace is sized from max_num_seqs, and under fp8 it had already - # doubled (98,304 -> 196,608 rows) because the page went 768 -> 1536. - # 128 slots was never meaningful at concurrency 8 -- the widest tree - # fan-out observed in the trace is 13 live streams in one lane, so 32 - # leaves real headroom while cutting the workspace 4x. - # 2. gpu_memory_utilization 0.88 -> 0.85 (244.8 GiB, ~36 GiB free vs ~28). - # An arm with an extra GPU-side consumer needs more slack than one - # without; 0.88 is calibrated for the no-offload case. - # - # Both are overridable, and both should be walked back toward the - # no-offload values once this arm produces a number -- a like-for-like - # comparison against 682.63 tok/s/GPU eventually needs matching - # max_num_seqs. - MAX_NUM_SEQS="${MAX_NUM_SEQS:-32}" - GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.85}" - - # LMCache is NOT in the kimi-k3 image (verified: no `lmcache` module and no - # CLI), so build it against ROCm. Clone to a container-local dir, NOT the - # bind-mounted /workspace, so a later job's `clean: true` checkout does not - # trip over root-owned build artifacts. - # - # The matrix passes kv-offload-backend as JSON in KV_OFFLOAD_BACKEND_METADATA - # (e.g. {"name":"lmcache","version":""}). Honour its `version` as the - # build ref so a version A/B is a config change rather than a recipe edit; - # an explicit LMCACHE_GIT_REF still wins, and the pin below is the fallback. - LMCACHE_CFG_VERSION="" - if [ -n "${KV_OFFLOAD_BACKEND_METADATA:-}" ]; then - LMCACHE_CFG_VERSION=$(KV_META="$KV_OFFLOAD_BACKEND_METADATA" python3 -c ' -import json, os -try: - d = json.loads(os.environ["KV_META"]) - print(d.get("version", "") if isinstance(d, dict) else "") -except Exception: - print("") -' 2>/dev/null || true) - fi - - # LMCache K3 support is NOT in any PyPI release yet (latest is 0.5.2, - # 2026-07-22). Upstream's own recipe -- docs/source/recipes/kimi_k3.rst, - # added by LMCache PR #4277 -- says: "use a nightly build from 2026-07-27 - # or newer; stable LMCache releases include K3 support starting from - # 0.5.3". Two commits on dev are what make K3 work at all: - # #4206 (2026-07-23) _MambaUnifiedViewEdit -- re-views a Mamba/KDA - # UNIFIED state tensor as a single attention tensor. This is the - # fix for our long-standing blocker - # "ValueError: expected a Mamba [conv_state, ssm_state] tensor - # list, got Tensor" (run 30350911388): K3's Kimi Delta Attention - # supplies one fused state tensor, and before #4206 the adapter - # only understood the [conv_state, ssm_state] pair. - # #4278 (2026-07-28) _SubpagedMLAAttentionViewEdit -- re-views K3's - # kernel-paged MLA tensor [N*12, 64, 576] as logical pages - # [N, 768, 576]. - # 0.5.1 predates BOTH, which is why every LMCache attempt on this model - # has died at engine init. The historical "pin 0.5.1, never build dev" - # note in this file was correct for its date and is now obsolete. - # - # Nightly WHEELS are CUDA-only, so ROCm must build from source. Flags are - # upstream's (docs/source/getting_started/installation.rst): gfx950 is - # MI355X/MI350X. - # - # LMCACHE_GIT_REF pins the build. Default is the #4278 merge commit rather - # than dev HEAD so the build is reproducible and cannot drift under us. - # A config-supplied `version` that looks like a release (0.5.3+) still - # installs from PyPI once such a release exists. - LMCACHE_GIT_REF="${LMCACHE_GIT_REF:-1dfa07772b8b2ae79653c830d63e297da39c970f}" - LMCACHE_VERSION="${LMCACHE_VERSION:-${LMCACHE_CFG_VERSION:-git}}" - if ! python3 -c "import lmcache.integration.vllm.lmcache_mp_connector" >/dev/null 2>&1; then - if [ "$LMCACHE_VERSION" = "git" ]; then - # Clone to a container-local dir, NOT the bind-mounted /workspace: - # a later job's `clean: true` checkout trips over root-owned build - # artifacts left behind there. - LMCACHE_SRC="${LMCACHE_SRC:-/opt/lmcache-src}" - echo "Building lmcache from source at ref $LMCACHE_GIT_REF (ROCm/gfx950)" - rm -rf "$LMCACHE_SRC" - git clone --filter=blob:none https://github.com/LMCache/LMCache.git "$LMCACHE_SRC" - git -C "$LMCACHE_SRC" checkout --detach "$LMCACHE_GIT_REF" - git -C "$LMCACHE_SRC" --no-pager log -1 --oneline - ( - cd "$LMCACHE_SRC" - export PYTORCH_ROCM_ARCH="gfx942,gfx950" - export TORCH_DONT_CHECK_COMPILER_ABI=1 - export CXX=hipcc - export BUILD_WITH_HIP=1 - if command -v uv >/dev/null 2>&1; then - uv pip install --system -e . --no-build-isolation \ - || python3 -m pip install -e . --no-build-isolation - else - python3 -m pip install -e . --no-build-isolation - fi - ) - else - echo "Installing lmcache==$LMCACHE_VERSION" - if command -v uv >/dev/null 2>&1; then - uv pip install --system "lmcache==$LMCACHE_VERSION" \ - || agentic_pip_install --quiet "lmcache==$LMCACHE_VERSION" - else - agentic_pip_install --quiet "lmcache==$LMCACHE_VERSION" - fi - fi - # Fail here, loudly, rather than 20 minutes later inside vLLM: this - # runs before the engine loads 1.56 TB of weights. - python3 -c "import lmcache.integration.vllm.lmcache_mp_connector" >/dev/null - fi - python3 -c "import lmcache; print('lmcache', getattr(lmcache,'__version__','?'))" || true - # Assert the two K3 fixes are actually present in whatever got installed. - # A silently-old build would otherwise fail much later with the original - # Mamba ValueError and look like a fresh bug. - python3 - <<'PYEOF' -import sys -try: - from lmcache.integration.vllm import kv_cache_group_edits as e -except Exception as exc: - sys.exit(f"FATAL: cannot import kv_cache_group_edits: {exc}") -names = {type(x).__name__ for x in getattr(e, "_EDITS", ())} -missing = {"_MambaUnifiedViewEdit", "_SubpagedMLAAttentionViewEdit"} - names -if missing: - sys.exit( - f"FATAL: installed lmcache lacks K3 support: missing {sorted(missing)}. " - f"Registered edits: {sorted(names)}. Needs dev >= #4278 (2026-07-28)." - ) -print(f"lmcache K3 edits present: {sorted(names)}") -PYEOF - - LMCACHE_HOST="${LMCACHE_HOST:-127.0.0.1}" - LMCACHE_PORT="${LMCACHE_PORT:-5555}" - LMCACHE_HTTP_PORT="${LMCACHE_HTTP_PORT:-8080}" - - # L1 is SHM-backed: if it exceeds free /dev/shm, LMCache silently disables - # SHM and falls back to a pickle path that crashes at load. Cap at 90% of - # free /dev/shm so SHM stays enabled, and say so loudly -- the capped value - # is the number that actually backs the run. - if [ "${LMCACHE_PROFILE:-}" = "k3upstream" ]; then - # Upstream's recipe says --l1-size-gb 100, but that is a - # does-it-work value, not a capacity value. Attempt 2 (run - # 30563325963) ran the whole 3600 s green at 100 GB and the host tier - # did NOTHING: 122,770 chunks stored, 1,879 evictions, and an external - # prefix cache hit rate of 0.0%. A write-only cache. - # - # The arithmetic says it could not have worked. K3 costs ~217 KiB of - # KV per token at bf16, ~108 KiB at fp8, so 100 GB holds well under - # 1M tokens -- SMALLER than the 3,322,266-token GPU pool it is meant - # to back. An offload tier below the size of the GPU pool can only - # evict before reuse. Worse, it dragged the GPU tier down with it: - # GPU prefix hit fell 88.4% -> 33.6% and GPU KV usage 87.5% -> 52.7%, - # consistent with vLLM releasing blocks to an external tier that then - # threw them away. - # - # Attempt 3 (run 30575475359) then sized L1 from TOTAL_CPU_DRAM_GB, - # which the /dev/shm cap landed at 1360 GB, and that BROKE GPU IPC: - # Error handling request RequestType.REGISTER_KV_CACHE - # torch.UntypedStorage._new_shared_cuda(...) - # torch.AcceleratorError: CUDA error: invalid device pointer - # (hipErrorInvalidDevicePointer) - # once per TP rank, so all 8 vLLM workers then died on - # "LMCache server did not respond to register_kv_caches within 300.0s". - # The timeout was the symptom; the registration itself failed. - # - # Attempt 2 at 100 GB logged ZERO IPC errors and registered cleanly. - # L1 size was the only difference between the two runs, so pinning - # ~1.36 TB of host memory is what exhausts the driver's mapping - # resources and makes the subsequent GPU IPC import invalid. Note the - # lazy allocator was NOT the problem here: 134 expansions completed in - # 3m37s and the pool was fully allocated long before vLLM finished - # loading weights. - # - # So L1 has a working range: above the GPU tier (GPU KV is ~52 GiB/rank - # at GMU 0.85, ~416 GB across TP8) and below the IPC ceiling somewhere - # under 1360 GB. 512 GB registered cleanly (run 30579846678) and lifted - # external hits 0.00% -> 0.82%, but throughput did not move. - # - # LMCACHE_L1_SHM_FRACTION sizes L1 as a fraction of FREE /dev/shm, - # which is the resource that actually binds (shm was 1512 GB free on - # this fleet, so 0.60 -> ~907 GB). Sizing off shm rather than - # TOTAL_CPU_DRAM_GB is deliberate: the DRAM budget is 2399 GB, so any - # fraction of it lands past the shm cap and straight back on the 1360 - # GB value that broke IPC. - # - # 0.60 bisects the known range -- 1.8x the 512 GB that works, 0.67x - # the 1360 GB that fails -- so this run locates the IPC ceiling as - # well as testing whether more cache changes the throughput picture. - # If it fails to register, the ceiling is between 512 and 907 and the - # next step is 0.45 (~680 GB), not a return to 512. - LMCACHE_L1_SHM_FRACTION="${LMCACHE_L1_SHM_FRACTION:-0.60}" - _shm_free_gb=$(df -BG --output=avail /dev/shm 2>/dev/null | tail -1 | tr -dc '0-9') - if [ -n "$_shm_free_gb" ] && [ "${_shm_free_gb:-0}" -gt 0 ]; then - _l1_from_shm=$(awk -v f="$LMCACHE_L1_SHM_FRACTION" -v s="$_shm_free_gb" \ - 'BEGIN { printf "%d", f * s }') - LMCACHE_L1_SIZE_GB="${LMCACHE_L1_SIZE_GB:-$_l1_from_shm}" - echo "LMCache L1 sized from /dev/shm: ${LMCACHE_L1_SHM_FRACTION} x ${_shm_free_gb}G free = ${LMCACHE_L1_SIZE_GB}G" - else - # df failed -- fall back to the value already proven to register. - LMCACHE_L1_SIZE_GB="${LMCACHE_L1_SIZE_GB:-512}" - echo "WARNING: could not read free /dev/shm; falling back to L1=${LMCACHE_L1_SIZE_GB}G" >&2 - fi - else - LMCACHE_L1_SIZE_GB="${LMCACHE_L1_SIZE_GB:-512}" # reference value - fi - - # Optional ceiling on the eager allocation. Memory pinning is unavailable in - # the ROCm container ("CudaPinMemoryBackend: neither torch cudart nor - # libcudart is available"), so LMCache disables its LazyMemoryAllocator and - # allocates the ENTIRE L1 pool before serving a request. - # - # Default is NO ceiling, matching kimik2.7_fp4_mi355x.sh, which sizes L1 to - # TOTAL_CPU_DRAM_GB capped only by /dev/shm and successfully ran a 1199 GB - # pool on this same fleet. The agentic README also requires consuming - # TOTAL_CPU_DRAM_GB rather than substituting a model-specific constant. Set - # LMCACHE_L1_MAX_GB to impose one (the AMD reference command uses 512). - LMCACHE_L1_MAX_GB="${LMCACHE_L1_MAX_GB:-0}" - if [ "$LMCACHE_L1_MAX_GB" -gt 0 ] && [ "$LMCACHE_L1_SIZE_GB" -gt "$LMCACHE_L1_MAX_GB" ]; then - echo "WARNING: capping LMCACHE_L1_SIZE_GB ${LMCACHE_L1_SIZE_GB} -> ${LMCACHE_L1_MAX_GB}" \ - "(eager allocation ceiling). The offload pool for this cell is" \ - "${LMCACHE_L1_MAX_GB}G, NOT the ${TOTAL_CPU_DRAM_GB}G the matrix budgeted." - LMCACHE_L1_SIZE_GB="$LMCACHE_L1_MAX_GB" - fi - - SHM_FREE_GB=$(df -BG --output=avail /dev/shm 2>/dev/null | tail -1 | tr -dc '0-9') - if [ -n "$SHM_FREE_GB" ] && [ "$SHM_FREE_GB" -gt 0 ]; then - SHM_CAP_GB=$(( SHM_FREE_GB * 90 / 100 )) - if [ "$LMCACHE_L1_SIZE_GB" -gt "$SHM_CAP_GB" ]; then - echo "WARNING: capping LMCACHE_L1_SIZE_GB ${LMCACHE_L1_SIZE_GB} -> ${SHM_CAP_GB}" \ - "to fit /dev/shm (${SHM_FREE_GB}G free). The offload pool for this" \ - "cell is ${SHM_CAP_GB}G, not the ${TOTAL_CPU_DRAM_GB}G the matrix budgeted." - LMCACHE_L1_SIZE_GB="$SHM_CAP_GB" - fi - fi - - LMCACHE_L1_INIT_SIZE_GB="${LMCACHE_L1_INIT_SIZE_GB:-20}" - # --max-gpu-workers 1 avoids concurrent-GPU-transfer stalls under heavy - # async-load pressure; CPU-side workers stay at 8. - LMCACHE_MAX_GPU_WORKERS="${LMCACHE_MAX_GPU_WORKERS:-1}" - LMCACHE_MAX_CPU_WORKERS="${LMCACHE_MAX_CPU_WORKERS:-8}" - LMCACHE_CHUNK_SIZE="${LMCACHE_CHUNK_SIZE:-1024}" - LMCACHE_L1_ALIGN_BYTES="${LMCACHE_L1_ALIGN_BYTES:-16384}" - LMCACHE_EVICTION_WATERMARK="${LMCACHE_EVICTION_WATERMARK:-0.85}" - LMCACHE_EVICTION_RATIO="${LMCACHE_EVICTION_RATIO:-0.10}" - LMCACHE_MQ_TIMEOUT="${LMCACHE_MQ_TIMEOUT:-300}" - # Identical prefixes must hash to identical block keys across ranks. - export PYTHONHASHSEED="${PYTHONHASHSEED:-0}" - - # Two server profiles. `reference` is the AMD K3 reference command. - # `k2.7` reproduces kimik2.7_fp4_mi355x.sh's server exactly -- the one - # configuration known to have served this agentic trace on this fleet - # (1199 GB L1, TP4) -- so the two can be A/B'd without a recipe edit. - # The K2.7 flags all still exist on LMCache dev: --max-workers lives in - # lmcache/v1/multiprocess/config.py alongside the newer split - # --max-gpu-workers/--max-cpu-workers, and --l1-read-ttl-seconds in - # lmcache/v1/distributed/config.py. - LMCACHE_PROFILE="${LMCACHE_PROFILE:-reference}" - echo "Starting LMCache MP server (profile=$LMCACHE_PROFILE, L1=${LMCACHE_L1_SIZE_GB}GB)..." - case "$LMCACHE_PROFILE" in - k2.7) - export LMCACHE_BLOCKING_TIMEOUT_SECS="${LMCACHE_BLOCKING_TIMEOUT_SECS:-60}" - LMCACHE_CMD=( - lmcache server - --host "$LMCACHE_HOST" - --port "$LMCACHE_PORT" - --http-host "$LMCACHE_HOST" - --http-port "$LMCACHE_HTTP_PORT" - --l1-size-gb "$LMCACHE_L1_SIZE_GB" - --l1-init-size-gb "$LMCACHE_L1_INIT_SIZE_GB" - --l1-read-ttl-seconds "${LMCACHE_L1_READ_TTL_SECONDS:-7200}" - --chunk-size "${LMCACHE_CHUNK_SIZE_K27:-256}" - --max-workers "${LMCACHE_MAX_WORKERS:-$((TP * 2))}" - --eviction-policy LRU - ) - ;; - k3upstream) - # Upstream's validated K3 command, plus the worker and eviction tuning - # needed by this TP8 agentic workload. The pinned LMCache revision - # supports these flags, and four workers serialize pairs of TP ranks. - # Upstream's original command was: - # lmcache server --port 6555 --chunk-size 768 --max-workers 4 \ - # --l1-size-gb 100 --eviction-policy LRU - # --chunk-size comes from the N derivation above (1536 under fp8, not - # upstream's bf16 768). L1 defaults to upstream's 100 GB here rather - # than the reference 512 GB: memory pinning is unavailable in the ROCm - # container, so LMCache allocates the WHOLE L1 pool before serving, - # and a smaller pool means a much shorter and more predictable - # bring-up. Raise it via LMCACHE_L1_SIZE_GB once the arm is green. - LMCACHE_CMD=( - lmcache server - --host "$LMCACHE_HOST" - --port "$LMCACHE_PORT" - --http-host "$LMCACHE_HOST" - --http-port "$LMCACHE_HTTP_PORT" - --chunk-size "$LMCACHE_CHUNK_SIZE" - --max-workers "${LMCACHE_MAX_WORKERS:-$TP}" - --l1-size-gb "$LMCACHE_L1_SIZE_GB" - --eviction-trigger-watermark "$LMCACHE_EVICTION_WATERMARK" - --eviction-ratio "$LMCACHE_EVICTION_RATIO" - --eviction-policy LRU - ) - ;; - reference) - LMCACHE_CMD=( - lmcache server - --host "$LMCACHE_HOST" - --port "$LMCACHE_PORT" - --http-host "$LMCACHE_HOST" - --http-port "$LMCACHE_HTTP_PORT" - --l1-size-gb "$LMCACHE_L1_SIZE_GB" - --l1-init-size-gb "$LMCACHE_L1_INIT_SIZE_GB" - --max-gpu-workers "$LMCACHE_MAX_GPU_WORKERS" - --max-cpu-workers "$LMCACHE_MAX_CPU_WORKERS" - --chunk-size "$LMCACHE_CHUNK_SIZE" - --l1-align-bytes "$LMCACHE_L1_ALIGN_BYTES" - --eviction-trigger-watermark "$LMCACHE_EVICTION_WATERMARK" - --eviction-ratio "$LMCACHE_EVICTION_RATIO" - --eviction-policy LRU - --supported-transfer-mode lmcache_driven - ) - ;; - *) - echo "Error: unsupported LMCACHE_PROFILE '$LMCACHE_PROFILE' (expected: reference, k2.7, k3upstream)" >&2 - exit 1 - ;; - esac - printf '%q ' "${LMCACHE_CMD[@]}" > "$RESULT_DIR/lmcache_command.txt" - printf '\n' >> "$RESULT_DIR/lmcache_command.txt" - "${LMCACHE_CMD[@]}" > "$LMCACHE_LOG" 2>&1 & - LMCACHE_PID=$! - echo "LMCache server PID: $LMCACHE_PID" - wait_for_lmcache_ready - - # LMCacheMPConnector is registered in this image's vLLM (verified against - # KVConnectorFactory), so the reference profile needs no - # kv_connector_module_path. The k2.7 profile passes it (and the ZMQ-style - # lmcache.mp.host) exactly as kimik2.7_fp4_mi355x.sh does. - if [ "$LMCACHE_PROFILE" = "k2.7" ]; then - LMCACHE_CONNECT_HOST="${LMCACHE_CONNECT_HOST:-tcp://$LMCACHE_HOST}" - OFFLOAD_ARGS=( - --kv-transfer-config - "{\"kv_connector\":\"LMCacheMPConnector\",\"kv_connector_module_path\":\"lmcache.integration.vllm.lmcache_mp_connector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"lmcache.mp.host\":\"$LMCACHE_CONNECT_HOST\",\"lmcache.mp.port\":$LMCACHE_PORT}}" - ) - else - OFFLOAD_ARGS=( - --kv-transfer-config - "{\"kv_connector\":\"LMCacheMPConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"lmcache.mp.port\":$LMCACHE_PORT,\"lmcache.mp.mq_timeout\":$LMCACHE_MQ_TIMEOUT}}" - ) - fi - ;; - vllm-simple|vllm-simple-fp8|vllm-simple-fp8-lazy) - require_agentic_kv_offload_backend "$KV_OFFLOAD_BACKEND" - # vllm-simple-fp8-lazy is vllm-simple-fp8 with lazy offload. In eager mode - # the connector stores every completed block to CPU immediately; in lazy - # mode it only stores under GPU-block pressure (manager.py _lazy_mode / - # _estimate_lazy_target_blocks walk the GPU free queue). Every fp8+connector - # cell so far has died while the GPU pool was nearly EMPTY -- 11.6% usage - # with a 0.0% external hit rate in run 30505656455, i.e. mid store-storm - # with nothing yet to gain from the host tier -- so the store path is where - # to look, and lazy removes almost all of it in that regime. - # - # Also the mode Wenyao's kimik3_fp4_mi355x_mtp.sh pins (SIMPLE_LAZY_OFFLOAD - # =true) as part of the gfx950 bundle validated in PR #2367 against an - # eight-rank GPU memory access fault. - if [ "$KV_OFFLOAD_BACKEND" = "vllm-simple-fp8-lazy" ]; then - SIMPLE_LAZY_OFFLOAD="${SIMPLE_LAZY_OFFLOAD:-true}" - fi - # vllm-simple-fp8 is the same connector with an fp8 KV cache. fp8 halves - # bytes/token in the GPU pool, which on this KV-bound corpus moves the - # eviction wall itself rather than just adding headroom (measured: the pool - # peaks at 98.6-99.8% usage even at c8). ROCm maps fp8 -> fp8_e4m3. - # UNVALIDATED on K3's hybrid geometry: the pool spans Kimi Delta Attention - # state and gated-MLA latent, and fp8 support across both spec types is - # unconfirmed on this build. - case "$KV_OFFLOAD_BACKEND" in - vllm-simple-fp8|vllm-simple-fp8-lazy) - KV_CACHE_DTYPE="${KV_CACHE_DTYPE:-fp8}" - ;; - esac - # vLLM's own SimpleCPUOffloadConnector -- the AMD reference's native - # offload path. Unlike LMCache it does not go through the Mamba - # [conv_state, ssm_state] adapter that K3's Kimi Delta Attention breaks - # ("expected a Mamba [conv_state, ssm_state] tensor list, got Tensor", - # runs 30350911388), so it is the only offload tier that can currently - # initialise on this model. - # - # cpu_bytes_to_use is server-wide; cpu_bytes_to_use_per_rank overrides it - # per rank (simple_cpu_offload_connector.py:66). The agentic README requires - # consuming TOTAL_CPU_DRAM_GB, so derive it rather than hardcode the - # reference's 236223201280 (= 220 GiB/rank, 1760 GiB across 8 ranks). - SIMPLE_RANKS="${GPU_COUNT:-$TP}" - CPU_BYTES_PER_RANK="${SIMPLE_CPU_BYTES_PER_RANK:-$(( TOTAL_CPU_DRAM_GB * 1000 * 1000 * 1000 / SIMPLE_RANKS ))}" - # Identical prefixes must hash to identical block keys across ranks. - export PYTHONHASHSEED=42 - # lazy_offload MUST be a JSON boolean. The reference command passes the - # STRING "false", and the connector does - # lazy_offload = bool(extra_config.get("lazy_offload", False)) - # (simple_cpu_offload_connector.py:77) -- bool("false") is True in Python, - # so the string silently selects LAZY, the opposite of what it reads as. - # Default to real eager offload; SIMPLE_LAZY_OFFLOAD=true swaps it. - # The connector logs "lazy"/"eager" at line 95, so server.log confirms which - # mode actually engaged. - SIMPLE_LAZY_OFFLOAD="${SIMPLE_LAZY_OFFLOAD:-false}" - OFFLOAD_ARGS=( - --kv-transfer-config - "{\"kv_connector\":\"SimpleCPUOffloadConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"cpu_bytes_to_use_per_rank\":$CPU_BYTES_PER_RANK,\"lazy_offload\":$SIMPLE_LAZY_OFFLOAD}}" - ) - echo "SimpleCPUOffloadConnector: ${CPU_BYTES_PER_RANK} B/rank x ${SIMPLE_RANKS} ranks, lazy_offload=$SIMPLE_LAZY_OFFLOAD" - ;; - vllm-native) - require_agentic_kv_offload_backend vllm-native - # OffloadingConnector, vLLM's other native tier. --kv_offloading_size is - # GiB (vllm/config/vllm.py multiplies by 1<<30) while TOTAL_CPU_DRAM_GB is - # decimal GB, so convert or we over-request by ~7.4%. - unset VLLM_USE_SIMPLE_KV_OFFLOAD - KV_OFFLOAD_GIB=$(( TOTAL_CPU_DRAM_GB * 1000000000 / 1073741824 )) - OFFLOAD_ARGS=( - --kv_offloading_backend native - --kv_offloading_size "$KV_OFFLOAD_GIB" - ) - echo "OffloadingConnector: ${KV_OFFLOAD_GIB} GiB" - ;; - mooncake) - echo "Error: Mooncake is unsupported for Kimi-K3. The upstream recipe marks" >&2 - echo " kv_store_{distributed,centralized}_mooncake as 'unsupported' on" >&2 - echo " every hardware target for this model." >&2 - exit 1 - ;; - *) - echo "Error: unsupported KV_OFFLOAD_BACKEND '${KV_OFFLOAD_BACKEND:-}' (expected: lmcache, lmcache-k27)" >&2 - exit 1 - ;; -esac -fi - # ---- Parallelism ------------------------------------------------------------ # TP8 or TEP8. No DP-attention arm: upstream strategy_min_gpus.multi_node_dep is # 16, so DEP is not a single-node strategy for this model. @@ -848,14 +247,13 @@ fi # and hash sizes only align with prefix caching on -- an omission has been # reported to trip "tokens_per_block not divisible by tokens_per_hash" at load. # Set PREFIX_CACHING=true/false to force it either way. -# ON by default for non-DSpark arms. This trace is built around large shared -# prefixes -- theoretical prefix-cache hit is 98.1%, and a live kvnone cell +# ON by default for a stand-alone run. This trace is built around large shared +# prefixes -- theoretical prefix-cache hit is 98.1%, and a live GPU-only cell # measured 92.8% server-side -- so a run with reuse disabled is not measuring # the workload. Reuse also costs essentially no KV (1,414,660 vs 1,420,824 -# tokens) and improves ITL (484 vs 577 ms). The offload arms additionally -# require it: LMCache needs mamba_cache_mode='align', which vLLM only selects -# when prefix caching is on. Turning it off for kvnone alone would also make -# the kvnone-vs-offload comparison at matched concurrency meaningless. +# tokens) and improves ITL (484 vs 577 ms). The DSpark wrapper overrides this +# to PREFIX_CACHING=auto, which passes neither flag and matches the upstream +# AMD reproducer exactly. # # It was briefly defaulted off for kvnone after two cells (c2/g19, c4/g17, # run 30412966635) died in warmup with it on, one dump naming the Mamba @@ -943,49 +341,27 @@ if [ "${EVAL_ONLY:-false}" = "true" ]; then MAX_NUM_SEQS="$EVAL_MAX_NUM_SEQS" fi -# 0.88, not the reference's 0.95. Measured on cluster:mi355x-amds: 0.95 asks for -# 273.59 of 287.98 GiB and cleared only 2 of 9 cells. Observed free memory at -# startup across seven nodes -- g09 281, g11 275/212/208, g14 256, g16 271/262, -# g15 21, g18 22 -- so even nominally clean nodes sit below 273.59 once driver -# overhead and transient co-tenancy are counted. This is not a denylist problem: -# g11 and g16 each measured both above and below the line hours apart. 0.88 -# (253.4 GiB) clears every observation except the two genuinely occupied nodes. -# 2026-07-30: 0.90 (259.19 GiB) was tried and REVERTED. It does not fail at -# engine init -- it comes up clean with a 4,302,357-token pool -- and then dies -# mid-prefill. Run 30521327099 lost c4, c8 and c16 on three separate nodes with -# an identical signature: HSA_STATUS_ERROR_OUT_OF_RESOURCES / "Available Free -# mem : 0 MB" on 4 of 8 ranks, surfacing as hipErrorUnknown at -# gpu_model_runner.py:314. Every death was at num_computed_tokens 360,960 to -# 364,032 with num_running_reqs=1 and kv_cache_usage only ~21%, so it is not -# concurrency and not KV exhaustion: the transient chunked-prefill workspace -# scales with context and needs more than the ~22 GiB of unreserved headroom -# 0.90 leaves. 0.88 leaves ~28 GiB and served 987 requests at ISL p90 374,550 / -# p95 511,146 with a 0/165 error rate (run 30453589555, byte-identical serve -# command apart from this one flag). -# -# The extra 443K tokens of pool 0.90 buys are unusable anyway -- peak usage at -# death was 21%. Raising this again requires bounding the prefill workspace -# first (the build_mla_chunked_context_metadata aggregate-chunk cap that only -# exists in hyukjleeamd/kimi-k3-mla-b128:fp8-kv-fixed, not in the mla_gluon -# gist patched below, which is a decode-path fix only). -# c1 is the exception, and it gets 0.85. It is the ONE cell with no result at -# any memory fraction: it died at num_computed_tokens 373,248 at BOTH 0.88 -# (runs 30453589555, 30525206671) and 0.90 (run 30521327099), always with -# num_running_reqs=1. A single lane replays the deepest trajectories with -# nothing to interleave, so it reaches the fatal band early and reliably -- -# c1 is the worst case for this failure, not the mildest. +# Stand-alone fallback only: the shipped DSpark arm pins GPU_MEM_UTIL=0.95 in +# kimik3_fp4_mi355x_mtp.sh to match the upstream AMD reproducer exactly, and +# that wrapper's value wins over these defaults. # -# Trading pool for headroom is close to free at c1 specifically. One -# trajectory cannot fill the pool: the bf16 c1 cell sustained 13 requests at -# ~291K average ISL against a 1.96M-token pool, and every fp8 death happened -# at 10-21% KV usage. So the ~530K tokens that 0.85 gives up were never going -# to be used, while the ~13 GiB/rank of extra headroom goes straight to the -# transient chunked-prefill allocation that is actually killing the cell. +# 0.88, not the reference's 0.95, is what the GPU-only bring-up cells actually +# survived on cluster:mi355x-amds. 0.95 asks for 273.59 of 287.98 GiB and +# cleared only 2 of 9 cells; observed free memory at startup varies 208-281 GiB +# across nodes, hours apart on the same node. 0.90 (259.19 GiB) comes up clean +# with a 4,302,357-token pool and then dies mid-prefill -- +# HSA_STATUS_ERROR_OUT_OF_RESOURCES on 4 of 8 ranks at num_computed_tokens +# ~362K with kv_cache_usage only ~21% (run 30521327099) -- because the +# transient chunked-prefill workspace scales with context and needs more than +# the ~22 GiB of unreserved headroom it leaves. 0.88 leaves ~28 GiB and served +# 987 requests at ISL p90 374,550 with a 0/165 error rate (run 30453589555). # -# This makes c1 not strictly like-for-like with the rest of the curve. That is -# the right trade while c1 has no data at all; revisit once the -# build_mla_chunked_context_metadata aggregate cap lands and the whole curve -# can return to one fraction. +# c1 gets 0.85: it died at num_computed_tokens 373,248 at both 0.88 (runs +# 30453589555, 30525206671) and 0.90 (run 30521327099), always with +# num_running_reqs=1 -- a single lane replays the deepest trajectories with +# nothing to interleave, so it reaches the fatal band early and reliably. One +# trajectory cannot fill the pool anyway, so the ~530K tokens 0.85 gives up buy +# ~13 GiB/rank of headroom for free. if [ "${CONC:-}" = "1" ]; then GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.85}" else @@ -997,7 +373,7 @@ MAX_NUM_BATCHED_TOKENS="${MAX_NUM_BATCHED_TOKENS:-4096}" echo "Starting vllm server..." export PYTHONNOUSERSITE=1 -# Long-context forward passes (~370K tokens with fp8 KV + DRAM offload) can exceed +# Long-context forward passes (~370K tokens under fp8 KV) can exceed # vLLM's default 300s worker RPC timeout, killing the engine with # "RPC call to sample_tokens timed out". Widen it. export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS="${VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS:-1200}" @@ -1040,13 +416,11 @@ VLLM_CMD=( --reasoning-parser kimi_k3 "${MAX_MODEL_LEN_ARGS[@]}" "${PREFIX_CACHE_ARGS[@]}" - "${LMCACHE_MAMBA_CACHE_MODE_ARGS[@]}" "${KV_CACHE_DTYPE_ARGS[@]}" "${KV_BLOCK_SIZE_ARGS[@]}" "${EAGER_ARGS[@]}" "${SPEC_ARGS[@]}" "${EVAL_SERVE_ARGS[@]}" - "${OFFLOAD_ARGS[@]}" ) printf '%q ' "${VLLM_CMD[@]}" | tee "$RESULT_DIR/vllm_command.txt" printf '\n' | tee -a "$RESULT_DIR/vllm_command.txt" diff --git a/benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh b/benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh index 6e92c322a7..4dad02c02a 100755 --- a/benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh +++ b/benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh @@ -3,11 +3,11 @@ set -euo pipefail # DSpark variant of kimik3_fp4_mi355x.sh. The MI355X launcher routes # spec-decoding=mtp rows to this suffix, while the shared base recipe owns the -# model, KV-offload, AgentX replay, and eval plumbing. +# model, AgentX replay, and eval plumbing. # # Keep this wrapper aligned with the upstream AMD Kimi-K3 DSpark reproducer. # The first AgentX validation is deliberately GPU-only at c1 so a server or -# kernel failure cannot be attributed to a KV-offload connector. +# kernel failure has nowhere else to hide. export SPEC_DECODE=true export KV_CACHE_DTYPE="${KV_CACHE_DTYPE:-auto}" export GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.95}" diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index e5eed5f1eb..7b1ad24a09 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -616,43 +616,17 @@ kimik2.5-fp4-mi355x-atom-disagg: additional-settings: - "DECODE_NODES=2" -kimik3-fp4-mi355x-vllm-agentic: - image: vllm/vllm-openai-rocm:kimi-k3 - model: moonshotai/Kimi-K3 - model-prefix: kimik3 - runner: cluster:mi355x-amds - precision: fp4 - framework: vllm - multinode: false - scenarios: - agentic-coding: - # 0.80 of the runner's available-cpu-dram-mib (3_095_781), capped by the - # matrix generator at the 2,861,022 MiB / 3 TB-decimal agentic limit -> - # 2399 GB aggregate at TP8, which SimpleCPUOffloadConnector divides into - # 279.28 GB per rank. - - dram-utilization: 0.80 - search-space: - # TP=8 only: the MXFP4 checkpoint is 1.561 TB decimal (1.420 TiB, 96 - # safetensors), ~195 GB/GPU of the 288 GB part, so TP=4 cannot load. - # gpu-memory-utilization is 0.88 in the recipe, not the reference's 0.95: - # 0.95 asks for 273.59 of 287.98 GiB and cleared only 2 of 9 cells on this - # fleet, because most nodes have more than 14.4 GiB of overhead. - # - # GPU-only, low concurrency. Measured at c8 the pool already peaks at - # 98.6% with a 13.8% prefix hit rate, so c8 is past the eviction crossing; - # c1/c2/c4 trace the region where the 1,957,290-token pool still holds the - # working set and no host tier is needed. - - { tp: 8, kv-offloading: none, conc-list: [1, 2, 4] } - # SimpleCPUOffloadConnector. LMCache cannot serve this model (its Mamba - # path expects [conv_state, ssm_state] and Kimi Delta Attention supplies - # one fused tensor), so this is the only working offload tier. Measured at - # c8 it lifts the prefix hit rate 13.8% -> 80.7% and drops TTFT from 193 s - # to 4.8 s for +70% output throughput, on a heavier ISL sample. c4 probes - # the crossover below which the GPU pool alone suffices; c16/c32 probe how - # far the 2234 GiB host tier extends the concurrency ceiling. - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [4, 8, 16, 32] } - -kimik3-fp4-mi355x-vllm-agentic-mtp: +# Kimi-K3 MXFP4 MI355X AgentX, DSpark only. Kimi-K3 agentic is onboarded +# spec-decode-only from day 0 -- the non-DSpark arm is neither run nor +# published (MODELS.md) -- so this is the single shipped key for this +# model/SKU, and it is the AMD sibling of kimik3-fp4-b300-vllm-agentic-dspark +# in configs/nvidia-master.yaml. +# +# TP=8 only: the MXFP4 checkpoint is 1.561 TB decimal (1.420 TiB, 96 +# safetensors), ~195 GB/GPU of the 288 GB part, so TP=4 cannot load. Upstream +# strategy_min_gpus agrees (single_node_tp and multi_node_tep both 8, DEP 16+), +# which is why there is no DP-attention arm either. +kimik3-fp4-mi355x-vllm-agentic-dspark: image: vllm/vllm-openai-rocm:kimi-k3 model: moonshotai/Kimi-K3 model-prefix: kimik3 @@ -662,389 +636,14 @@ kimik3-fp4-mi355x-vllm-agentic-mtp: multinode: false scenarios: agentic-coding: - - dram-utilization: 0.80 - search-space: + # No dram-utilization: this arm is GPU-only, so the generator resolves + # TOTAL_CPU_DRAM_GB to 0 either way and the field would be inert. + - search-space: # Exact upstream AMD DSpark reproducer: GPU-only TP8, seven draft tokens, # probabilistic draft sampling, TRITON_MLA, and real block rejection. # Start at c1; expand only after the AgentX-fast preflight is healthy. - { tp: 8, kv-offloading: none, spec-decoding: mtp, conc-list: [1] } -kimik3-fp4-mi355x-vllm-agentic-fp8kvtest: - image: vllm/vllm-openai-rocm:kimi-k3 - model: moonshotai/Kimi-K3 - model-prefix: kimik3 - runner: cluster:mi355x-amds - precision: fp4 - framework: vllm - multinode: false - scenarios: - agentic-coding: - # 0.80 of the runner's available-cpu-dram-mib (3_095_781), capped by the - # matrix generator at the 2,861,022 MiB / 3 TB-decimal agentic limit -> - # 2399 GB aggregate at TP8, which SimpleCPUOffloadConnector divides into - # 279.28 GB per rank. - - dram-utilization: 0.80 - search-space: - # kv-offloading: none, NOT `kv-offload-backend: none` -- YAML parses a - # bare `none` as the string 'none', which fails KVOffloadBackendMetadata - # validation and takes the whole master config down with it. - - { tp: 8, kv-offloading: none, conc-list: [8] } - -# LMCache server-profile A/B for Kimi-K3. NOT shipped -- delete once one profile -# wins. Both cells are conc 8 with prefix caching forced on, --max-num-batched-tokens -# 768 and --chunk-size 768 (all three are hard Mamba-hybrid requirements the recipe -# applies automatically), so the ONLY difference is the lmcache server command: -# lmcache -> AMD K3 reference: --max-gpu-workers 1 --max-cpu-workers 8, -# --l1-align-bytes 16384, --eviction-trigger-watermark 0.85, -# --eviction-ratio 0.10, --supported-transfer-mode lmcache_driven -# lmcache-k27 -> kimik2.7 recipe: --max-workers TP*2, --l1-read-ttl-seconds 7200, -# kv_connector_module_path + ZMQ-style lmcache.mp.host -kimik3-fp4-mi355x-vllm-agentic-lmprofile: - image: vllm/vllm-openai-rocm:kimi-k3 - model: moonshotai/Kimi-K3 - model-prefix: kimik3 - runner: cluster:mi355x-amds - precision: fp4 - framework: vllm - multinode: false - scenarios: - agentic-coding: - - dram-utilization: 0.80 - search-space: - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache-k27, version: "0.5.1" }, conc-list: [8] } - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.1" }, conc-list: [8] } - -# Retry-only key: single cell, re-dispatched when a bringup cell lands on a -# node carrying another tenant's allocation. Delete once it has a result. -# fp8 KV-cache probe: identical to the winning vllm-simple c8 cell except for -# --kv-cache-dtype fp8, which should roughly double the 1,957,290-token GPU pool. -kimik3-fp4-mi355x-vllm-agentic-fp8sweep: - # Full fp8-KV sweep. Supersedes the bf16 numbers: fp8 is now the recipe - # default (see kimik3_fp4_mi355x.sh), so every cell here runs - # --kv-cache-dtype fp8 with prefix caching on, and the mla_gluon kernel is - # patched from the pinned gist revision that run 30442578333 used. - # - # Stock image on purpose -- 30442578333 proved the patched stock image runs - # fp8 clean; the custom hyukjleeamd/kimi-k3-mla-b128 image is the fallback. - image: vllm/vllm-openai-rocm:kimi-k3 - model: moonshotai/Kimi-K3 - model-prefix: kimik3 - runner: cluster:mi355x-amds - precision: fp4 - framework: vllm - multinode: false - scenarios: - agentic-coding: - - dram-utilization: 0.80 - search-space: - # No-offload arm. c8 repeats 30442578333 (696 tok/s/GPU) as a control; - # c1/c4 replace the bf16 points (159 / 421 tok/s/GPU) from run - # 30430688459 so the whole curve is one KV dtype. - - { tp: 8, kv-offloading: none, conc-list: [1, 4, 8] } - # Offload arm at the same dtype -- isolates what - # SimpleCPUOffloadConnector costs or buys under fp8. c16 probes whether - # fp8 pushes the concurrency ceiling past the bf16 collapse at c16. - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple-fp8 }, conc-list: [8, 16] } - -kimik3-fp4-mi355x-vllm-agentic-fp8retry: - # Retry of the two cells that died in run 30453589555, plus a new none c16. - # - # Both deaths were GPU faults (hipErrorUnknown), not timeouts -- no guard was - # anywhere near firing (VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1200 vs a ~180s - # stall; warmup-grace-period 1800 vs 420s elapsed). GPU faults have hit every - # arm on this cluster today (bf16 kvnone on g11/g17/g18/g19, fp8 kvnone on - # g10, fp8+connector on g11/g19), so these are retried unchanged on the - # assumption they are the cluster's background fault rate rather than config. - # - # none c16 is new: the no-offload arm has never been measured above c8, and - # fp8 roughly halves KV per token, so the concurrency ceiling should have - # moved up. - image: vllm/vllm-openai-rocm:kimi-k3 - model: moonshotai/Kimi-K3 - model-prefix: kimik3 - runner: cluster:mi355x-amds - precision: fp4 - framework: vllm - multinode: false - scenarios: - agentic-coding: - - dram-utilization: 0.80 - search-space: - - { tp: 8, kv-offloading: none, conc-list: [1, 16] } - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple-fp8 }, conc-list: [8] } - -kimik3-fp4-mi355x-vllm-agentic-fp8eval: - # lm-eval on the one config that is actually proven on this stack: fp8 KV, - # no offload, c8 -- the control cell from run 30442578333 (696 tok/s/GPU), - # repeated clean in 30453589555. The offload arm is excluded on purpose: it - # has failed 3/3 under fp8 (see the fp8b64 key below). - # - # The recipe pins EVAL_FRAMEWORK=lm-eval, so this runs lm-eval rather than - # run_eval's swebench default for agentic scenarios. Dispatch with - # --evals-only; mark_eval_entries flips single-node agentic rows to - # run-eval=True (generate_sweep_configs.py:360). - image: vllm/vllm-openai-rocm:kimi-k3 - model: moonshotai/Kimi-K3 - model-prefix: kimik3 - runner: cluster:mi355x-amds - precision: fp4 - framework: vllm - multinode: false - scenarios: - agentic-coding: - - dram-utilization: 0.80 - search-space: - - { tp: 8, kv-offloading: none, conc-list: [8] } - -kimik3-fp4-mi355x-vllm-agentic-fp8none: - # GPU-only (no offload) fp8-KV concurrency curve at GMU 0.88. - # - # One key, one curve: c1/c4/c8/c16 with --kv-cache-dtype fp8 (the recipe - # default) and no KV connector, so the whole curve shares a KV dtype, a - # memory fraction and a warmup. The offload arm is deliberately absent -- - # it has failed 4/4 under fp8 (see the fp8lazy key below), and mixing it in - # would put failure-prone cells in the same dispatch as the control curve. - # - # GMU comes from the recipe default (GPU_MEM_UTIL in kimik3_fp4_mi355x.sh), - # because the agentic matrix has no per-cell env channel. The first dispatch - # of this key ran at 0.90 and lost c4/c8/c16 to a prefill-workspace OOM at - # ~362K context on three separate nodes; see the comment on GPU_MEM_UTIL for - # the evidence. 0.88 is the proven setting. - # - # c16 is the open question. The bf16 no-offload arm was never measured above - # c8, and the bf16 vllm-simple curve collapsed at c16 (TTFT 295 s). fp8 - # roughly halves KV per token, so the ceiling should have moved up; this - # cell is what says whether it did. - image: vllm/vllm-openai-rocm:kimi-k3 - model: moonshotai/Kimi-K3 - model-prefix: kimik3 - runner: cluster:mi355x-amds - precision: fp4 - framework: vllm - multinode: false - scenarios: - agentic-coding: - - dram-utilization: 0.80 - search-space: - - { tp: 8, kv-offloading: none, conc-list: [1, 4, 8, 16] } - -kimik3-fp4-mi355x-vllm-agentic-fp8none-c1: - # The one missing point on the fp8 no-offload curve. - # - # c1 has NO result at any memory fraction. It died at num_computed_tokens - # 373,248 at 0.88 (runs 30453589555 and 30525206671) and at 0.90 (run - # 30521327099), every time with num_running_reqs=1 and KV usage 10-21% -- - # the probabilistic chunked-prefill workspace OOM, not KV exhaustion. - # c1 is the WORST case for it: one lane replays the deepest trajectories - # with nothing to interleave, so it reaches the fatal band early and every - # time. - # - # The recipe now drops GPU_MEM_UTIL to 0.85 for CONC=1 only. At c1 the pool - # is hugely oversized anyway (13 requests sustained against a 1.96M-token - # pool in the bf16 run), so the ~530K tokens given up were never usable, - # while the extra ~13 GiB/rank of headroom goes to the allocation that is - # actually killing the cell. - # - # If this still dies, the memory fraction is not the lever and the - # aggregate-chunk cap in mla_attention.py is required -- do not keep - # shaving GMU. - image: vllm/vllm-openai-rocm:kimi-k3 - model: moonshotai/Kimi-K3 - model-prefix: kimik3 - runner: cluster:mi355x-amds - precision: fp4 - framework: vllm - multinode: false - scenarios: - agentic-coding: - - dram-utilization: 0.80 - search-space: - - { tp: 8, kv-offloading: none, conc-list: [1] } - -kimik3-fp4-mi355x-vllm-agentic-lmcachek3: - # LMCache on Kimi-K3, fp8 KV, c8 -- the first attempt with an LMCache build - # that actually supports this model. - # - # Every prior LMCache attempt died at engine init with - # ValueError: expected a Mamba [conv_state, ssm_state] tensor list, got Tensor - # because K3's Kimi Delta Attention supplies ONE fused state tensor. That is - # now fixed upstream: LMCache #4206 (2026-07-23) added _MambaUnifiedViewEdit - # for unified Mamba state, and #4278 (2026-07-28) added - # _SubpagedMLAAttentionViewEdit for K3's kernel-paged MLA tensor. LMCache - # also shipped its own validated K3 recipe (#4277, - # docs/source/recipes/kimi_k3.rst). None of that is in 0.5.1, which is what - # this recipe pinned, and none of it is in any PyPI release (latest 0.5.2). - # - # version: "git" makes the recipe build from source at the #4278 merge - # commit with ROCm flags, since nightly wheels are CUDA-only. The recipe - # asserts both edit classes are registered before loading 1.56 TB of - # weights, so a stale build fails in seconds rather than 20 minutes. - # - # backend name lmcache-k3 selects upstream's minimal validated server - # command instead of this recipe's `reference` flag set -- those extra flags - # were never validated on K3 and an unrecognised flag is a silent - # server-start failure. - # - # c8, c16, and c32. - # - # c8 is the peak of the no-offload fp8 curve (682.63 tok/s/GPU, run - # 30525206671) and is now measured with LMCache: 601.64 tok/s/GPU, -11.9% - # (run 30579846678). The reason it loses is that there is no work for a - # host tier at this concurrency -- the LMCache server logged 67,827 stores - # against 80 retrievals with only 217 evictions, because the GPU pool never - # fills (52.4% usage) so vLLM never has to spill. LMCache still pays to - # store every block and drags the GPU tier down with it (prefix hit - # 88.4% -> 58.3%). - # - # c16 is the case that should invert that. Without offload, c16 is where - # the no-offload arm collapses: 412.21 tok/s/GPU, GPU prefix hit 3.9%, KV - # pinned at 100.0%, TTFT mean 323 s (run 30525206671). The working set no - # longer fits, so the GPU tier thrashes -- which is exactly the regime a - # host tier exists for. If LMCache cannot win at c16 it has no case on this - # model. - # - # c32 extends the capacity-pressure curve after LMCache recovered c16 from - # 441.99 to 607.45 tok/s/GPU when L1 grew from 512 to 907 GB. This tests - # whether the larger host tier can keep helping once the live working set - # doubles again. - # - # max_num_seqs stays 32. aiperf bounds in-flight requests by --concurrency, - # so it provides 2x headroom at c16 and exactly covers c32. Raising it would - # re-inflate the chunked MLA workspace that caused the attempt-1 OOM. - # - # c8 repeats deliberately: it gives a same-run, same-code pair against c16 - # rather than a cross-run comparison. - image: vllm/vllm-openai-rocm:kimi-k3 - model: moonshotai/Kimi-K3 - model-prefix: kimik3 - runner: cluster:mi355x-amds - precision: fp4 - framework: vllm - multinode: false - scenarios: - agentic-coding: - - dram-utilization: 0.80 - search-space: - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache-k3, version: "git" }, conc-list: [8, 16, 32] } - -kimik3-fp4-mi355x-vllm-agentic-fp8lazy: - # fp8 KV + SimpleCPUOffloadConnector in LAZY mode. One variable off the cell - # that has now failed 4/4 (g17/g19/g12/g12), and the block-size attempt in run - # 30505656455 is void: vLLM overrode --block-size 64 to 1536 because K3's - # attention page must be >= its mamba/KDA page (interface.py:911), so that - # cell was byte-identical to the eager failures. - # - # Why lazy. Eager stores every completed block to CPU immediately. Every - # fp8+connector death has happened while the GPU pool was nearly empty -- - # 11.6% usage, 0.0% external hit rate in 30505656455 -- i.e. deep in a store - # storm with nothing yet to gain from the host tier, and the last two deaths - # were a 600 s RCCL _ALLGATHER_BASE hang rather than an OOM. Lazy only stores - # under GPU-block pressure, so it removes nearly all connector traffic in - # exactly the regime that is killing the cell. - # - # It is also the mode kimik3_fp4_mi355x_mtp.sh already pins for gfx950 - # (PR #2367) against an eight-rank GPU memory access fault. - # - # Control: bf16 + the same connector in EAGER mode ran to 99.9% pool usage - # with zero faults (run 30431115226), and fp8 WITHOUT the connector peaks at - # 754 tok/s/GPU at c8 (run 30453589555). - image: vllm/vllm-openai-rocm:kimi-k3 - model: moonshotai/Kimi-K3 - model-prefix: kimik3 - runner: cluster:mi355x-amds - precision: fp4 - framework: vllm - multinode: false - scenarios: - agentic-coding: - - dram-utilization: 0.80 - search-space: - # PAIRED. Run 30511545046 tried lazy alone and is uninterpretable: it - # landed after #2415 swapped the agentic warmup from - # --agentic-cache-warmup-duration 600 to --warmup-requests-per-lane 10 - # (benchmark_lib.sh:1793, new utils/aiperf pin), so it moved two variables - # at once and died inside the new warmup -- its "0 successful, OSL=1" is - # that warmup's one-token primers, not a lazy symptom. Every earlier - # failure used the old time-based warmup, so none of them is a baseline - # any more. Run the eager control on the SAME warmup as lazy. - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple-fp8 }, conc-list: [8] } - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple-fp8-lazy }, conc-list: [8] } - -kimik3-fp4-mi355x-vllm-agentic-fp8kv: - # fp8 KV on the OFFLOAD arm. This is the fp8kvtest arm (run 30442578333: - # 696 total tok/s/GPU, TTFT 2.6s, the first clean fp8 result) plus the - # SimpleCPUOffloadConnector, so it isolates what the connector costs under - # fp8. - # - # Stock image on purpose. The stock image routes fp8 KV into AITER's - # mla_gluon path and asserts `mla_gluon[bh16bn128] requires batch_size=1, - # got 128` (run 30411594835) -- but the recipe now patches that kernel from - # a pinned gist revision whenever KV_CACHE_DTYPE=fp8, which is exactly how - # 30442578333 passed. Using stock keeps this cell one variable away from - # that proven run. - # - # hyukjleeamd/kimi-k3-mla-b128:fp8-kv-fixed bakes the same kernel in and is - # the alternative, but its only trial (run 30434731504) cleared init and the - # full 600s warmup and then died at the profiling transition with an async - # hipErrorIllegalAddress on g14 -- unattributed, possibly the node. Swap the - # image back to compare once this cell has a number. - image: vllm/vllm-openai-rocm:kimi-k3 - model: moonshotai/Kimi-K3 - model-prefix: kimik3 - runner: cluster:mi355x-amds - precision: fp4 - framework: vllm - multinode: false - scenarios: - agentic-coding: - - dram-utilization: 0.80 - search-space: - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple-fp8 }, conc-list: [8] } - -kimik3-fp4-mi355x-vllm-agentic-retry: - image: vllm/vllm-openai-rocm:kimi-k3 - model: moonshotai/Kimi-K3 - model-prefix: kimik3 - runner: cluster:mi355x-amds - precision: fp4 - framework: vllm - multinode: false - scenarios: - agentic-coding: - - dram-utilization: 0.80 - search-space: - - { tp: 8, kv-offloading: none, conc-list: [1, 2, 4] } - -kimik3-fp4-mi355x-vllm-agentic-native: - image: vllm/vllm-openai-rocm:kimi-k3 - model: moonshotai/Kimi-K3 - model-prefix: kimik3 - runner: cluster:mi355x-amds - precision: fp4 - framework: vllm - multinode: false - scenarios: - agentic-coding: - - dram-utilization: 0.80 - search-space: - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [8] } - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-native }, conc-list: [8] } - -kimik3-fp4-mi355x-vllm-agentic-bringup: - image: vllm/vllm-openai-rocm:kimi-k3 - model: moonshotai/Kimi-K3 - model-prefix: kimik3 - runner: cluster:mi355x-amds - precision: fp4 - framework: vllm - multinode: false - scenarios: - agentic-coding: - - dram-utilization: 0.80 - search-space: - - { tp: 8, kv-offloading: none, conc-list: [8] } - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.1" }, conc-list: [8] } - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache-k27, version: "0.5.1" }, conc-list: [8] } dsr1-fp8-mi355x-atom: image: rocm/atom:rocm7.2.3_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom20260511 diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 85c8fd16b6..fc850013db 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -5341,11 +5341,13 @@ pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2312 - config-keys: - - kimik3-fp4-mi355x-vllm-agentic-mtp + - kimik3-fp4-mi355x-vllm-agentic-dspark description: - - "Add the Kimi-K3 MXFP4 MI355X AgentX DSpark variant with Inferact/Kimi-K3-DSpark, seven draft tokens, probabilistic draft sampling, and TRITON_MLA on ROCm." + - "Add the Kimi-K3 MXFP4 MI355X AgentX DSpark recipe with Inferact/Kimi-K3-DSpark, seven draft tokens, probabilistic draft sampling, and TRITON_MLA on ROCm. This is the only key for this model/SKU: Kimi-K3 agentic is onboarded spec-decode-only from day 0, so the non-DSpark arm is neither run nor published (MODELS.md), and the key name matches the NVIDIA sibling kimik3-fp4-b300-vllm-agentic-dspark." - "Align the first AgentX validation with the upstream AMD reproducer: GPU-only TP8 at concurrency 1, real block rejection, --gpu-memory-utilization 0.95, --max-num-batched-tokens 4096, --max-num-seqs 16, multimodal encoder data parallelism, and no --enforce-eager or explicit prefix-cache override." - "Use VLLM_ROCM_USE_AITER=1, SAFETENSORS_FAST_GPU=1, AITER_SITUV2_A8W4=1, AITER_BF16_FP8_MOE_BOUND=0, and VLLM_USE_BREAKABLE_CUDAGRAPH=0 with the stock vllm/vllm-openai-rocm:kimi-k3 image." + - "The recipe is GPU-only (kv-offloading: none) and rejects a matrix row that asks for a host tier rather than silently ignoring it; the LMCache / SimpleCPUOffload / OffloadingConnector plumbing the bring-up sweeps used is not carried, and returns with the arm that needs it." + - "Passes spec-decoding and the kv-offload-backend metadata through to the agentic-eval job in e2e-tests.yml, which previously hardcoded spec-decoding: none and rendered an object-valued kv-offload-backend as a broken string. Without the spec-decoding passthrough the DSpark eval job would resolve to the non-DSpark script name." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2403 - config-keys: diff --git a/utils/matrix_logic/test_kimik3_mi355x_dspark.py b/utils/matrix_logic/test_kimik3_mi355x_dspark.py index be8477e2cc..fcd9b8a13c 100644 --- a/utils/matrix_logic/test_kimik3_mi355x_dspark.py +++ b/utils/matrix_logic/test_kimik3_mi355x_dspark.py @@ -56,7 +56,7 @@ def test_reference_prefix_cache_mode_emits_no_override() -> None: def test_dspark_diagnostic_matrix_is_gpu_only_c1() -> None: master = yaml.safe_load(MASTER_CONFIG.read_text()) - search_space = master["kimik3-fp4-mi355x-vllm-agentic-mtp"]["scenarios"][ + search_space = master["kimik3-fp4-mi355x-vllm-agentic-dspark"]["scenarios"][ "agentic-coding" ][0]["search-space"]