cookbook: humans& NVFP4 4/6 recipe on Qwen3-30B-A3B (miles disagg, stitch-v2)#22
Open
jvmncs wants to merge 18 commits into
Open
cookbook: humans& NVFP4 4/6 recipe on Qwen3-30B-A3B (miles disagg, stitch-v2)#22jvmncs wants to merge 18 commits into
jvmncs wants to merge 18 commits into
Conversation
The trainer can OOM-kill on host RAM (the publish/update_weights full-model gather is the peak consumer at ~1.69 TiB on the ~1.95 TiB B200:8 node), and Megatron only reports GPU memory. helpers.start_host_mem_monitor() logs a per-node [hostmem] trace from a daemon thread (called in Trainer.start_ray on every node) so 'modal app logs -f' shows the phase/peak that blows the budget.
The overlap-drain race was never reproduced and may have been theoretical; stop claiming in_place requires a special SGLang build with that fix.
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(sync): compose multi-version catch-up into one engine reload _sync_once applied + reloaded the engine once per intermediate version on a multi-version catch-up (N pauses, N full update_weights_from_disk). The host-side apply (apply_deltas) already chain-replays the whole tail in a single call, so the per-version loop only multiplied the expensive engine reload. Verify chain contiguity up front, then drive a single commit that applies the target manifest once and advances current_version to latest -- one pause, one host-apply, one reload regardless of tail depth. Co-Authored-By: jason.mancuso@modal.com <jvmncs@gmail.com> * docs(protocol): document apply_manifest chain-replay requirement The sync manager now composes a multi-version catch-up into a single apply_manifest call, so the EngineAdapter contract requires it to replay every intermediate version from the applied one to the target, not apply manifest as a single delta off the previous version. Make that explicit.
Delete the throwaway modal_probes/ directory and cookbook/miles_disagg/ modal_probes.py (validation/profiling scratch), and root out everything stapled into the examples that existed only to serve them: - modal_train.py: drop the env-gated R3_PROBE all-to-all logging injection (the functional R3 dispatch + GLM4Moe bias bakes are unchanged). - kimi_k2_6_nvfp4_disagg: drop ANCHOR_MODEL. Its only real consumers were the deleted probes; prepare_checkpoints merely snapshot_download'd it and never diffed anything, so the 'design anchor / diff against our base' lifecycle comments described validation that never ran. Remove the dead fetch and rewrite the docstring/README/last-layer-carveout comment to stand on their own. - slime moonlight_disagg: drop the 'verified by modal_probes/verify_mla_routing' doc reference. No change to any train/serve/sync code path. Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Explicit pool claim/reset: one trainer call owns one pool epoch * Align reconcile tests with #4 composed-reload semantics After rebasing onto main (which includes #4, composing a multi-version catch-up into a single engine reload), two reconcile tests still asserted the pre-#4 per-version apply accounting: - local_disagg/harness_test.py (added by this PR): a cold latecomer catching up v1..v2 now records a single composed apply [2], not [1, 2]. - standalone_rollouts/provider_test.py (pre-existing; #4 changed the behavior but did not update this cookbook test): same one-apply tail. The served version / run_id invariants are unchanged; only the per-apply accounting moved. No production code change.
…ine (#9) * Consolidate cookbook trainers onto a shared sidecar/serving/launch spine Collapse the accidentally-forked cookbook scaffolding onto one trainer-agnostic spine, leaving each example to encode only its intended axes (slime vs miles quant; disagg trainer-writer vs standalone frontdoor-writer): - cookbook/sidecar.py: unified sidecar (arg parsing, bulletin board + Volume reloader, concurrent stale-aware base materialization). slime/miles sidecars become thin adapters naming only their host-side delta decoder. - cookbook/serving.py: one B200 SGLang image builder; build_int4/build_nvfp4 remain as thin wrappers. - cookbook/trainer_helpers.py: shared config-prep, train-command, host-RAM monitor, and a smoke check with a wake_on_demand flag. slime/miles helpers re-export with their per-trainer params. Fixes a latent missing socket import in the miles host-mem monitor. - cookbook/rollout_control.py: shared trainer-side primitives (distributed_rank, read_setting, session affinity) for both control planes; collapses the standalone slime hook fork onto them. - standalone announce_claim: claim-at-launch parity with bulletin claim_pool, wired into the standalone trainer launch. - Tests for the new shared modules + announce_claim. Co-Authored-By: jason.mancuso@modal.com <jvmncs@gmail.com> * Fix consolidation runtime gaps flagged in review - serving image: mount the whole cookbook package at /root/cookbook (not just the per-trainer subdir) so the sidecar subprocess (python3 -m cookbook.<t>.sidecar), which is never imported at deploy time and so is invisible to Modal's import-time automounting, can resolve the shared cookbook.sidecar spine it delegates to. - modal_probes: import parallel_init_local_checkpoint from cookbook.sidecar (the function moved there from cookbook.miles_disagg.sidecar and gained a required disk_delta_module arg). Co-Authored-By: jason.mancuso@modal.com <jvmncs@gmail.com> * Mount the whole cookbook package in the trainer images The consolidation moved the shared spine (helpers/hooks/sidecar/ trainer_helpers/ray_cluster/sidecar_process) up to the cookbook package root, so cookbook.<trainer>.helpers now imports cookbook.* root modules and the per-trainer sidecar is a thin adapter over cookbook.sidecar. The trainer images still mounted only their per-trainer subdir (/root/cookbook/<trainer>) with include_source=False, so those root modules were absent at runtime: the trainer's helpers import and the 'python3 -m cookbook.<trainer>.sidecar' subprocess (the server image for the small trainer-image-reuse experiments, e.g. qwen) would ImportError. Mount the whole cookbook package, matching cookbook/serving.py and the standalone modal_train/modal_serve images (which were already widened). Co-Authored-By: jason.mancuso@modal.com <jvmncs@gmail.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Remove dead and speculative code from the core library - generate_rollout compatibility wrapper and the unreachable per-request version-pinning branch (the live hook contract never supplies rollout_id); the request hook now does only what runs: retry budget + session affinity - test-only helpers: transition_artifact_paths, parse_extra_key_version, validate_policy (redundant with request_context) - register_routes extension point nothing passes - version error helpers inlined into evaluate_version_policy - unused top-level re-export facade in stitch/__init__.py * Remove dangling cookbook artifacts - unreachable per-request version-pin branch in the standalone slime hook (the hook contract never supplies rollout_id; announce_and_wait already gates staleness) + its helper, config knob, and tests - megatron_r3_num_out_tokens.patch: never applied, the fix is baked inline * Dedup the slime/miles twin adapter modules - bulletin_hooks reads generic DELTA_APP_NAME / DELTA_SERVER_CLS_NAME env fallbacks; per-recipe hooks.py become pure re-export shims kept only because configs reference them by dotted string - per-recipe helpers.py shrink to the real per-recipe axes (sidecar module, model-script attr, wake mode) + one-line wrappers; miles-only materialize_node_local_yaml and start_host_mem_monitor move out of the shared trainer_helpers into miles_disagg/helpers - serving wrappers take the shared trainer_repo_* kwarg names; the shared builder computes the cookbook mount dir itself * De-noise prose: drop provenance labels, war stories, and docstring essays Keep the behavioral fact, drop the provenance: external PR/issue labels (slime PR #5, radixark/miles#1261, P0.1), incident anecdotes, milestone roadmap prose (M1/M2/M3), and layout/design essays compressed to what the code needs the reader to know. Pinned SHAs stay; their changelog trailers go. * Trim READMEs and packaging - root README: document all four cookbook recipes; fix the nonexistent slime extra claim - miles_disagg README: compress the fork changelog and bring-up checklist, fold the GLM-4.5-Air section into a variant note, point at the fork for the convert tools, document _spawn_into_deployed - standalone_rollouts README: compress the compatibility spec-dump and the SLIME-harness tail - slime_disagg README: merge the redundant Protocol notes into the run section - pyproject: drop uvicorn from the sglang extra (packaged code never imports it; Modal images install it explicitly)
The routing-replay (R3) fix for the baked radixark/Megatron-LM was an inline build-time string bake. Move it to the patches/ + git-apply flow introduced for SGLang so all source patches share one mechanism: - add patches/megatron-r3-dispatch.patch (with the R3 rationale header) - extract shared helpers._apply_git_patches core; add apply_megatron_runtime_patches (repo_dir=/root/Megatron-LM) - apply MEGATRON_RUNTIME_PATCHES in Trainer.start_ray (SPMD, every node, before Ray start) - set MEGATRON_RUNTIME_PATCHES on all five configs (the old bake was unconditional) - drop the inline _R3_DISPATCH_* constants, the run_commands bake, and the now-unused shlex import Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The weight-sync checksum probe caught the patch corrupting fused w8a8 weights (qkv_proj/gate_up_proj) on every update_weights_from_disk: its transposed_weight_loader staged each shard in a fresh UNINITIALIZED buffer and copied the whole buffer over the fused weight, clobbering sibling shards with allocator leftovers (nondeterministic under num_threads=8 loading); its _sglang_fp8_reload_ready early-return also skipped requantize_with_max_scale on reload, so reloaded raw per-shard bytes served against the boot-collapsed max scale. Rewritten architecture: the registered checkpoint-layout Parameters are never replaced (stock shard-aware loaders keep working; the wrapper machinery is deleted), postprocess re-runs after every load doing the shard->max-scale requant IN PLACE, and kernels read derived *_serving attributes (transposed views / collapsed scalars) that refresh pointer-stably so captured CUDA graphs survive reloads. Applies cleanly on both dafb2b3 (current pin) and weight-sync-upstream. Verified on GLM-4.5-Air-FP8 H200:4 (nan-dev): record/replay/fast and partial-fallback/full checksums all match per tensor per rank; before the fix all three comparisons failed on exactly the fused weights.
…load metrics (#17) - VersionManifest.transition_weights: publishers list the touched tensor names per version; EngineAdapter docs updated (protocol stays engine-neutral). - WeightSyncManager: stage_manifest (host delta apply) runs OUTSIDE the commit gate, commit_manifest (engine reload) inside — the pause now covers only the reload. All-empty tails skip the reload entirely (skipped_noop_reload). The tail's touched-name union is passed to commit_manifest(weight_names=...) for engines that can partially reload; a post-commit board re-check keeps late wakes from being dropped (kept from #12). - SGLang adapter: stage/commit split with weight_names payload (STITCH_PARTIAL_RELOAD=0 kill switch), reload-timing parse from the fork's '[reload timing]' message into server_info metrics; the legacy apply_manifest keeps #12's empty-delta skip. Measured (with the pinned fork): K2.6 NVFP4 steady-state pause 280s -> ~10s on real June deltas (688 touched names), checksum-verified; GLM bf16 218s -> ~56s via plan replay; Moonlight empty deltas -> 0 pause.
…ry knob (#18) - serving.py: pin modal-projects/sglang weight-sync-upstream@7fd4288bc (record/replay load plans, O(delta) partial reloads with pre-flight quant-support checks, reload phase timing; 7 reviewable commits + CPU unit tests) and enable SGLANG_ENABLE_RELOAD_LOAD_PLAN pool-wide. - sidecar.py: fsync each copied shard during base materialization so the first reload doesn't eat the writeback backlog. - slime_disagg/standalone pins bumped to the instrumented slime fork. - configs: rollout_memory_mib knob (page-cache residency for reloads), moonlight fp4_param_gather=False (miles' NVFP4 export rejects True).
Move the Modal probes out of cookbook/miles_disagg into a top-level profiling/ package: they are cross-experiment measurement tools (any EXPERIMENT_CONFIG), not part of the miles example. profile_reload brings up one rollout engine identical to the Server cls and times every weight-update phase (cold boot, materialize, record/replay reloads, host delta apply, partial reload) with per-tensor checksum gates between reload strategies; it applies the experiment's SGLANG_RUNTIME_PATCHES exactly like the Server cls, so it measures the code production serves. disk_survey samples fresh-container ephemeral disk. max_inputs -> single_use_containers (deprecation); app renamed weight-sync-probes. delta_audit is the CPU-only leading indicator for O(delta) partial reload cost: from each published version's index alone it recomputes the closure the load plan applies (expert + fused-sibling rules, validated against the engine's logged plan_closure), checks upload completeness, and fingerprints touched expert slots across runs (identical sets => structural, not learning). First use showed every published K2.6 delta touches only routed expert 0 per layer across all 8 runs — the 0.44% density behind the ~10s partial reloads is smoke-test-scale training plus NVFP4 quantization masking sub-quantum updates; audit any new run's bulletin before extrapolating.
jvmncs
force-pushed
the
nvfp4-46-recipe
branch
2 times, most recently
from
July 12, 2026 02:58
6915dad to
72c31d8
Compare
jvmncs
force-pushed
the
nvfp4-46-recipe
branch
2 times, most recently
from
July 15, 2026 03:04
4eb965f to
4e9553a
Compare
jvmncs
changed the title
| # flashinfer 0.6.13: the 4/6 error-domain fix (FI#3448). The three packages move in | ||
| # lockstep — sglang hard-fails on a version mismatch, correctly, since a stale jit-cache | ||
| # would silently serve kernels from the old version. | ||
| SERVING_IMAGE_EXTRA_COMMANDS = ( |
Contributor
There was a problem hiding this comment.
might not needed when miles merge NVFP4 pr and have a new docker
they will merge the pr very soon
Contributor
Author
There was a problem hiding this comment.
will hold off on merging until then. validation run looked good though
Contributor
Author
stitch-v2 validation results (jason-dev, ≤11 concurrent B200)Short tuned async run (4 steps,
🤖 Generated with Claude Code |
* Repoint to the /pull_weights pins: engine-side delta apply, no trainer package in the pool
The pinned engine (modal-projects/sglang@weight-sync-miles, c3cd6404, over an
lmsysorg/sglang:v0.5.14 env image — docker supplies the environment, the pin's
python tree is the runtime code) carries /pull_weights + the hardened
local_checkpoint receiver and the quantized-reload restore protocol, and the
pinned trainer (modal-projects/miles@disagg-publish, b44be620) publishes plain
HF-layout deltas from current miles main. Both oracle-validated
(GLM-4.5-Air-FP8 H200, Kimi-K2.6-NVFP4 B200).
- SGLangDiskDeltaAdapter: stage_manifest POSTs /pull_weights (engine
materializes the local checkpoint: seeded from its own model_path,
chain-replayed, checksum-verified, reseed-on-corruption); commit_manifest
stays the plain disk reload. Host-side apply_deltas/init_local_checkpoint
and the base-fingerprint materializer are gone — the engine owns all of it,
including the stale-base case (checksum mismatch -> auto reseed + replay).
- The pool image installs no trainer package; the sidecar imports neither
slime nor miles. The engine refreshes the Modal Volume itself via
--custom-pull-weights-pre-read-hook (stitch.providers.modal, reads
DELTA_VOLUME_NAME from the serving image env).
- STITCH_PARTIAL_RELOAD now defaults to 0: the pin has no partial reload
(full reloads re-derive quantized kernel state; partial needs a per-module
restore design first). Load-plan record/replay stays enabled — the full-
reload speedup rides it.
- miles_disagg configs move to the pin's arg surface (transfer-mode
disk-delta, custom_update_weight_post_write_path) and drop the
sglang-fp8-reload-attrs runtime patch (superseded by the restore protocol).
- standalone_rollouts still pins a pre-/pull_weights slime image; marked TODO
for the slime phase.
* adapter: surface the engine's error body on failed pull/reload
sglang returns pull/reload failures as HTTP 400 with the traceback in the
JSON body; raise_for_status() before reading it reduced every failure to a
bare status code. Read the body first and put the engine's message in the
RuntimeError.
* trainer image: pin the dated tag, not latest (Modal caches mutable tags)
modal.Image.from_registry caches per tag string and never re-pulls a moved
mutable tag, so radixark/miles:latest was silently serving a stale pull whose
baked sglang predates encoding_dsv4 (on sglang-miles since 2026-06-01) —
crashing today's miles main at import. Pinning the dated tag makes the image
an explicit, immutable input like every other pin here.
* sglang pin: 964e6e24 (receiver fails fast on a not-yet-visible source version)
The e2e surfaced a transient Modal Volume cross-host visibility race: the
`latest` pointer / pool wake can beat the version dir becoming visible after
reload, so a pull for weight_vN hits a missing source dir. That is not local
corruption — reseeding can't conjure the bytes and for Air's multi-GB base the
wasted full copy is expensive — so the receiver now re-raises FileNotFoundError
immediately and the sidecar's reconcile retries once the source is visible.
Also refresh the serving.py docstring for the no-trainer-package design.
* self-review cleanup: drop dead plumbing, consolidate the sidecar entrypoint
Adapter and cookbook wiring cleanups from a pass over the /pull_weights repoint:
- Remove the dead base_checkpoint_dir field + its whole plumbing chain
(adapter, build_manager, --base-checkpoint-dir arg, sidecar_process, the
recipe call sites). The engine seeds from its own --model-path; the sidecar
never used the value, yet required it — a misleading required arg.
- Drop the vestigial apply_manifest + _transition_files from the adapter: the
sync manager always uses the staged stage_manifest/commit_manifest split for
this adapter, so the combined path was unreachable. sync.py keeps the generic
apply_manifest fallback for hypothetical non-staged adapters.
- Consolidate the two byte-identical per-trainer sidecar entrypoints into one
(added __main__); both recipes point
SIDECAR_MODULE at it. Deletes cookbook/{miles,slime}_disagg/sidecar.py.
- Make delta_volume_name a required builder arg (was defaulted "") so a
missing cross-host-visibility wire is a construction error, not silent stale
reads.
- Add the /pull_weights pre-read hook to the slime recipe's server args (it
was only on miles) and fix comments across sync.py/serving.py/modal_train
that still described the old host-side-apply model.
- Bump the pins to the self-reviewed sglang (725ad453) and miles (bdaf8d04).
* disagg weight sync: pre-read hook waits for full materialization
The Modal-volume pre-read hook now reloads AND verifies the target version is
fully present on this host — dir + index + every referenced blob — retrying
with backoff until materialized or a deadline, before it returns. A single
reload fired on the publisher's wake routinely raced ahead of the object
store's read-after-write propagation, so the engine saw the version dir (or
its multi-GB blobs) as missing and failed the pull; a partial blob set then
looked like a checksum mismatch and triggered a needless reseed loop. Blocking
until complete is the readiness half of the fix. The engine-side half —
reclassify an incomplete source as not-ready rather than corruption — rides in
the sglang pin bump to 7091d83.
Also carries the current disagg run config: load-plan record/replay stays
disabled (its record path hangs on GLM-Air fused MoE) and the rollout pool
scales 2->4 for the elastic-join test.
* pin sglang f3a39d4 (stage-verify-apply + one reload/host + reseed-from-base)
Carries the disk-delta receiver robustness fixes: the engine stages each delta
to local disk and size-verifies it before the in-place apply, only the per-host
lock winner reloads the volume, and a reseed resets to the pristine boot
checkpoint instead of the mutated model_path.
* disagg hook: single reload, not a reload loop (fixes slow pulls + watchdog kills)
The pre-read hook reloaded the Modal volume in a tight loop waiting for the
target version to "materialize". Repeated reload() calls thrash the v2 mount and
stall the very materialization they poll for: delta pulls went from ~3s to
100-500s, and a 494s pull tripped sglang's 300s watchdog, killing a replica
mid-run. The caller only pulls after seeing the version at the bulletin pointer,
so ONE reload surfaces it; completeness is handled downstream by the engine's
stage + size-verify (a short/absent blob fails fast and the sidecar retries with
a fresh reload). Drops the now-unused _version_materialized helper.
* pin sglang cbfe88f (in-memory stage-verify-apply + whole-file seed)
Bumps to the refined receiver: the delta apply reads each blob whole-file into
memory and size-verifies before the XOR (no redundant on-disk staging copy), and
the base seed / reseed reads whole-file too, dropping the chunked shutil.copy2
that made pulls ~40x slower on the volume mount. Correctness unchanged: one
reload per host, reseed from the pristine base, per-blob size + per-tensor
checksum verification.
* finalize pins: sglang 2347c32, miles bdaf8d04 (profiling reverted)
Drops the temporary pipeline-profiling commits from all three repos and pins the
finalized weight-sync work: sglang 2347c32 (in-memory stage-verify-apply + one
reload per host + reseed-from-pristine-base; the [pull-profile] timing is
reverted, docstring corrected to match the in-memory apply) and miles bdaf8d04
(publish-profile reverted). num_rollout back to 10.
* docs: stitch-v2 design & rewrite plan
Consolidated first-principles design for the disaggregated-rollout library:
3 rings (core / default adapters / edge), 3 ports (Store/Engine/Platform),
full-vs-delta as the anchor/chain manifest field, stated correctness
invariants, and the phased extract-and-de-leak rewrite plan.
* stitch-v2 Phase 0: skeleton
Move profiling/ -> tools/profiling/; scaffold tests/ and examples/ with
role READMEs; add tests/ to pytest testpaths (src/cookbook kept during the
transition). No logic yet — establishes the target tree per DESIGN.md.
* stitch-v2 Phase 1: versions.py + Store/Engine/Pool ports
The pure keystone, split by role:
- versions.py — the domain vocabulary + pure rules (VersionRef, VersionManifest
{full anchor | delta}, VersionConstraint, SyncState, ReplicaState/PoolState,
decide_pointer_move/PointerRewind). No I/O / Modal / engine / framework.
- stores|engines|pools/base.py — the Store / Engine / Pool Protocols, each living
with its instances-to-be (matches "sglang is an instance of Engine").
Consistency: one internal name per concept — diff/compression/checksum,
min_version/exact_version, and server_info keys ready/applied/sync_state/reason.
External wire names (Cognition's *_format, a framework's delta_encoding) translate
at the boundary, never as a second name in core. Comments trimmed to extra-info-only.
Functional checks pass.
* stitch-v2 Phase 1: sync.py (Reconciler + AdmissionGate) + in-memory harness
Ports the reconcile brain + commit gate onto the new contract, retyped and trimmed:
- Reconciler: refresh -> read pointer -> run-switch(reset to base) -> stage the tail
(engine walks back to the nearest anchor) -> reload once -> flip applied under the
gate. Empty-delta advances the version with no reload. Non-cancelling wake; a 409
triggers catch-up.
- AdmissionGate: constraint check + serving-version capture under one lock; commit
holds the gate across apply AND the version flip (in_place flips before resume);
quiesce drains all + flushes, in_place drains only exact pins.
- Engine port gains flush/pause/resume for the commit modes.
Dropped vs the old: the RolloutSyncManager Protocol, the staged/non-staged dual
path (the port always stages+commits), the manager-side chain pre-verify (the engine
owns chain integrity during stage), and the heavy per-substage metrics (kept
target_version/stage_s/commit_s/skipped_reload).
tests/reconcile_test.py runs the real Reconciler+gate against fake Store/Engine
(no Modal/sglang/GPU) — 7/7 pass: fresh reconcile, catch-up, run-switch+reset,
empty-delta skip, admit ok/reject->wake, and version-flips-before-resume. This is
the Phase-1 gate: the core passes with fakes before any instance exists.
testpaths -> ["tests"] (old co-located tests target superseded code; removed in Phase 3).
* stitch-v2 Phase 1 complete: service.py + publish.py
- service.py: create_app (the versioned proxy — admit -> engine.stamp_request ->
forward -> engine.stamp_response, retryable 409, client-disconnect abort; now
engine-agnostic, no sglang import), serve() (sidecar entrypoint over injected
store+engine), readiness() (aggregate /server_info across replicas -> PoolState).
- publish.py: publish_version (from_hf_index -> decide_pointer_move -> store.publish
-> advance_pointer -> pool.wake), claim_run, constrain_request (version bound +
affinity). Composes the Store+Pool ports; no Modal.
- Engine port finalized: stamp_request(served) / stamp_response(served,current) put
version handling behind the port; generate_url -> upstream_url; +flush/pause/resume.
Phase-1 gate: 13/13 across tests/{reconcile,publish}_test.py, all against fakes
(no Modal/sglang/GPU). The whole core is engine/provider/framework-agnostic and proven.
* stitch-v2 Phase 2: concrete instances (ModalVolumeStore / SGLangEngine / ModalFlashPool)
Port the proven Modal/sglang code onto the three ports, de-leaking as we go:
- stores/modal_volume.py: ModalVolumeStore (was bulletin.py slime layout + the
Volume half of providers/modal.py). run_id-partitioned chains at root/<identity>,
a `latest` text pointer, commit/reload for durability/visibility. volume_name=None
is a plain local dir, so the store is exercisable without Modal.
- engines/sglang.py: SGLangEngine (was the disk-delta adapter + servers/sglang.py
stamping). stage=/pull_weights, commit=/update_weights_from_disk, flush/pause/
resume, reset=rmtree, stamp_request=extra_key namespacing, stamp_response=
meta_info vs OpenAI top-level. Dropped dead surface: applied_version (Reconciler
owns applied), parse_reload_timing (fork instrumentation), weight_names partial
reload (off, unsupported by the pinned engine).
- pools/modal_flash.py: ModalFlashPool (Flash-client half of providers/modal.py).
gateway_url / discover_replicas / wake (POST /wake fan-out) / scale.
Removed applied_version from the Engine port; fixed the stale FakeEngine method
names; deleted the superseded co-located engines/sglang_test.py.
Instances harness (tests/instances_test.py): port conformance via isinstance,
the store driven through the real publish_version flow (full + claim+delta chain +
external-dir copy), and engine request/response stamping. Full suite 19/19 green
with fakes/local dirs; HTTP + volume-backed paths validated e2e.
* stitch-v2 Phase 3a: glm45_air_fp8 example — stitch-facing assembly
The first example's core wiring onto the new ports (the Modal-app infra + MilesConfig
land next). Everything Modal-specific lives in the example; the library stays general.
- examples/glm45_air_fp8/hooks.py: the three framework hook shims on the core —
commit_and_wake -> publish_version (rank 0 publishes; other ranks flush their shard
via ModalVolumeStore.commit), claim_pool -> claim_run, gated_rollout_request_hook ->
constrain_request with a TTL-cached latest-pointer floor. Reads run coordinates off
the trainer args namespace; builds ModalVolumeStore + ModalFlashPool.
- examples/glm45_air_fp8/sidecar.py: Server-side entrypoint — ModalVolumeStore +
SGLangEngine -> serve().
- examples/glm45_air_fp8/config.py: the deployment identity constants both sides key off.
- examples/glm45_air_fp8/hooks_test.py: co-located harness (4/4), no Modal/torch.
Two core fixes the port surfaced:
- claim_run now runs decide_pointer_move first, restoring the reused-run_id rewind
guard the old board.claim had (the run_id is the per-launch fence token).
- publish_version/claim_run wake the pool best-effort (pointer already durable;
replicas self-sync), so a transient control-plane error can't fail a train step.
- ModalVolumeStore.commit(): concrete multi-rank shard-durability affordance (not on
the Store port).
Core suite 20/20; example harness 4/4.
* stitch-v2 Phase 3b: glm45_air_fp8 full Modal app
Completes the first example — the deployable Modal app assembled on the new core.
The old cookbook tree stays as reference until Phase 4 e2e proves this, then it goes.
- config.py: ModalConfig + MilesConfig (the cli_args reflection) + the concrete GLM
FP8 experiment. The three plug points point at examples.glm45_air_fp8.hooks.*.
- modal_app.py: the Modal app — trainer image (miles/Megatron/TE) + Server image
(weight-sync sglang, no trainer package), volumes, the Server cls (sglang +
stitch.service.serve sidecar), the Trainer cls (miles on Ray; train() resolves the
rollout endpoint via ModalFlashPool.gateway_url and claims the pool), prep
(checkpoints/torch_dist/dataset), and launch/smoke entrypoints.
- infra.py: the non-stitch glue (Ray bring-up, runtime git patches, host-RAM monitor,
config materialization, the miles train command, the sidecar subprocess, and the
Flash-pool smoke — the last adapted to the new /server_info shape via VersionRef).
- convert_hf_to_torch_dist_modal.py + patches/megatron-r3-dispatch.patch: example assets.
- stores/modal_volume.py: add pull_weights_pre_read_hook (the engine-side
--custom-pull-weights-pre-read-hook target: one Volume reload before a pull).
- pyproject: modal-extra comment now names the instances.
Core suite 20/20; example harness 4/4; all example modules import (config yields
116 miles CLI args). The Modal app is e2e-validated in Phase 4.
* stitch-v2: core naming pass + ports as inheritable base classes
Review feedback on the core: cleaner names + real inheritance.
Ports are now plain base classes (methods raise NotImplementedError; subclasses
override — a missing override surfaces at call time). ModalVolumeStore(Store),
SGLangEngine(Engine), ModalFlashPool(Pool) inherit. Pool.wake/scale are concrete
no-op defaults (the optional hooks). The test fakes subclass the bases too.
Renames:
- Engine.upstream_url() -> base_url() (httpx/OpenAI-client convention; the
engine isn't "upstream" of itself)
- Store.open_version() -> materialize() ("open" wrongly implied a handle; it
ensures local availability and returns the dir)
- VersionManifest.diff -> delta_encoding (parallels compression/checksum; the
index/wire key stays "diff")
- sync.PolicyViolation -> ConstraintUnmet (states the actual condition)
- modal_flash._https -> _normalize_url
DESIGN.md abstraction descriptions synced to the new names + the subclass model.
Core suite 20/20; example harness 4/4.
* stitch-v2: review feedback — cookbook move, in_place default, flush_on_commit, cleanups
1. examples/ -> cookbook/ (the repo's established name): the recipe now lives at
cookbook/glm45_air_fp8/; all dotted-path/mount references updated
(examples.glm45_air_fp8 -> cookbook.glm45_air_fp8). Removed the examples/ dir.
2. sglang.py: inlined the one-use _EXTRA_KEY_DELIM (";") into _extra_key.
3. commit mode + flush:
- default commit_mode is now "in_place" (AdmissionGate/Reconciler/serve); still
configurable per replica. The glm45 recipe explicitly pins "quiesce" (fp8 reload
is exact) with a comment noting the override; sidecar CLI default -> in_place.
- SGLangEngine.flush_on_commit (default False): flush_cache is no longer hardcoded.
Flushing is the reconciler's job (it flushes before a quiesce reload; in_place
keeps in-flight KV), so this is an opt-in escape hatch, not on VersionRef.
4. service.py: _DROP_HEADERS moved up beside BLOCKED_ROUTES/VERSIONED_ROUTES.
DESIGN.md layout/plan updated examples -> cookbook. Core suite 20/20 (added an
in_place-default assertion; test_fresh_reconcile now pins quiesce explicitly);
recipe hooks 4/4.
* stitch-v2: revert commit_mode default to quiesce (keep flush_on_commit)
Per feedback — the default is quiesce again; flush_on_commit stays as the opt-in.
* stitch-v2: restructure cookbook into common/ + miles/ + slime/
Per feedback — the cookbook holds many things, layered: shared code factored out, the
framework/use-case specifics in per-framework subdirs, an experiment config per run.
Replaces the flat glm45_air_fp8 recipe (which over-collapsed and dropped slime).
- common/ (framework-agnostic, shared by every recipe): config.py (ModalConfig + mount
paths), images.py (the weight-sync sglang SERVING image), server.py (register_server —
the shared Server cls, parameterized by image/gpu/volumes/config), sidecar.py (the one
serve() entrypoint), hooks.py (commit_and_wake/claim_pool/gated_rollout_request_hook
logic), ray.py, process.py, smoke.py. Co-located hooks_test 4/4.
- miles/ (self-contained config, own pins): pins.py (trainer image + MILES_* pins),
config.py (MilesConfig), app.py (EXPERIMENT_CONFIG -> common Server + miles Trainer +
prep + entrypoints), prep.py (bf16/fp8/nvfp4 + torch_dist), hooks.py (re-export common),
configs/glm45_air_fp8.py (migrated). torch-dist wrapper + megatron patch moved here.
- slime/ (symmetric): pins.py (slime image), config.py (SlimeConfig), app.py (download_model
prep; publish hook key custom_delta_pre_push_path), hooks.py, configs/kimi_k2_6_int4.py.
Each framework's config is fully separate (own cli_args reflection). The Server is shared;
the Trainer + prep + pins are per-framework. The old *_disagg tree stays as reference
until e2e. Core suite 20/20; common hooks 4/4; miles/slime configs import (116/132 args).
* stitch-v2 cookbook: drop overengineering + clearer file names
- Remove the per-framework hooks.py re-export shims (needless indirection): configs now
point custom_*_path straight at cookbook.common.hooks.*, and app.py imports the hooks
from common directly.
- Dedup the identical WARMUP_PAYLOAD into register_server (built from model_name); drop
the warmup_payload param and the per-app copies.
- Intuitive, symmetric file names:
common/ray.py -> common/ray_cluster.py (a file named ray.py importing ray reads badly)
common/images.py -> common/serving_image.py paired with
{miles,slime}/pins.py -> {miles,slime}/trainer_image.py
Core suite 20/20; common hooks 4/4; all cookbook modules import.
* stitch-v2 cookbook: port all remaining miles/slime experiment configs
Faithful ports (hyperparameters/arch/sglang-args untouched) of every old experiment:
miles/configs/: glm45_air_bf16, moonlight_nvfp4, kimi_k25_2layer_nvfp4, kimi_k2_6_nvfp4
(+ glm45_air_fp8 already migrated)
slime/configs/: qwen3_4b_delta_flash, qwen3_4b_delta_flash_hillclimb, moonlight,
moonlight_int4 (+ kimi_k2_6_int4 already migrated)
Per-config transforms only: imports -> common.config + <framework>.config; hook paths ->
cookbook.common.hooks.*; dropped the now-shared build_serving_image; added
LOCAL_CHECKPOINT_PATH; APP/VOL names -> stitch-<model>-<precision>. Post-port fixes:
- MEGATRON_RUNTIME_PATCHES + deploy-command docstrings -> the new miles/ path + app module
- qwen3 request hook was pointing at the dead stitch.trainers.slime -> common.hooks
(its mode=exact/lag=0/... knobs are exactly what that hook reads)
- dropped the orphan per-config contract test (only one config had one; not in testpaths)
All 10 configs import; hook paths -> common.hooks; core suite 20/20; common hooks 4/4.
* stitch-v2 cookbook: version=2 volumes, tests -> tests/, dedup launch helpers
- All Modal volumes now created with version=2 (the shared hf-cache / data / checkpoints
/ prep / sglang-cache were still on the default; the delta volume already was).
- Moved the co-located cookbook hook test to tests/cookbook_hooks_test.py (all tests
live under tests/ now; pytest collects it — 24/24).
- Factored the identical _build_train_cmd + _resolve_and_materialize out of miles/slime
app.py into common/launch.py (build_train_cmd(cfg, root, model_script_attr) +
resolve_config(cfg, tmpdir, yaml_fields)); dropped the now-unused shlex/Any imports.
miles keeps _materialize_node_local_yaml (te_precision_config_file is miles-only).
Core + cookbook suites 24/24.
* stitch-v2: refresh DESIGN.md to the built structure
Layout (versions.py + per-component base.py, not contract.py; cookbook common/miles/slime,
not a flat recipe), framework-integration (shared common/hooks referenced by config, no
per-framework shim), naming conventions (delta_encoding/base_url/materialize), the
what-moved table, and phase status (0-3 done, 4 pending).
* cookbook/common/server: serialized=True (register_server builds Server outside global scope)
Modal requires @app.cls classes to be global-scope unless serialized; the shared Server
factory builds it dynamically, so serialize it (cloudpickle) instead of import-by-reference.
* cookbook: module-level Server delegating to shared common.server logic
serialized=True can't cross the local(3.14)/image(3.12) Python boundary. Replace the
register_server factory with common.server.serve_startup/serve_stop (shared logic) and a
thin module-level Server class in each app.py (Modal imports @app.cls by reference at
global scope — no pickle, no version constraint). Honors 'share the Server' via the
shared startup/stop; the per-app class skeleton + decorators are inherently per-app.
* cookbook/app: single-source config access (no scattered globals/env-mix)
The experiment module exp is the single source of truth; app.py reads it uniformly
(modal_cfg.X / <framework>_cfg.Y / exp.CONST). Removed the value-renaming globals
(MODEL_NAME/APP_NAME/N_TRAIN_NODES); the only resolved values are MIN_CONTAINERS
(deploy-time pool-floor override) and ROLLOUT_CONCURRENCY (target_inputs-or-engine
fallback), both in a labeled deploy-env block. n_train_nodes is now a MilesConfig/
SlimeConfig property (single source on the config); dropped the redundant
ray_cluster.training_nodes.
Both apps import; configs resolve; suite 24/24.
* cookbook: general constants -> common/constants.py; drop MIN_CONTAINERS env-override; fix torch_dist orphan
- common/constants.py: the general deployment constants (container mount paths, ports,
timeouts) moved out of config.py + the app.py globals. app.py imports them.
- min_containers is now just modal_cfg.rollout_min_containers (the env-override was wrong:
config is the single source; deploy-time pool-down isn't a config concern).
- prepare_torch_dist: wrap the single name in place (the prior _prepare_torch_dist left an
unregistered clustered PartialFunction at module scope -> 'strange type' runner error).
- configs import DATA_PATH/PREP_PATH from common.constants.
Both apps import; 10 configs import; suite 24/24.
* sync/service: fix proxy 422 (Request injection) + offload blocking store I/O
service.py: drop `from __future__ import annotations`. The FastAPI route
handlers are introspected at runtime; `Request` is a create_app-local import,
so under stringized annotations FastAPI resolves it only against module globals,
fails, and demotes `request` to a required query param -> 422 on every proxied
call. Unit tests never hit the real HTTP route, so it slipped through.
sync.py: offload store.refresh()/materialize() (blocking I/O on the ModalVolume
instance) via asyncio.to_thread so a multi-second reload never stalls the async
serving loop; also clears the Modal AsyncUsageWarning. The Store port stays
uniformly synchronous — the async concern lives in the async consumer.
* cookbook: fix baseline publish crash + pre-claim smoke health gate
commit_and_wake: the framework fires custom_update_weight_post_write_path as a
"flush this dir to the volume" primitive at three boundaries (baseline clean-dir,
per-version files, post-barrier pointer), and writes its own shards+index+latest.
The v2 rewrite dropped the old dir-name guard and called publish_version (which
reads model.safetensors.index.json) unconditionally, so the baseline call on the
run-root dir crashed: FileNotFoundError model.safetensors.index.json. Restore the
guard: every rank flushes; rank 0 publishes only when handed a weight_vNNNNNN dir.
General across miles+slime (keys on the dir name, not a framework-specific marker).
smoke: a fresh pool serves the base but has no claimed run, so a run-scoped exact
version-0 pin 409s forever. Probe unversioned when /server_info run_id is null —
a real pre-claim health gate; keep the version-pinned check for a claimed pool.
* stores: add S3Store (S3 / S3-compatible bucket-backed Store)
Third Store instance behind the port (base.py invited "Add S3 / NFS as new
subclasses"). Versions live under <prefix>/<run_id>/weight_vNNNNNN/, pointer at
<prefix>/latest. S3 is strongly read-after-write consistent so refresh is a no-op
and a single latest PUT is the durable move; with no shared mount, materialize
downloads a version's chain into cache_dir (incremental — skips objects already
local at the same size). boto3 is imported lazily so the module loads without it.
Includes a pull_weights_pre_read_hook (DELTA_S3_URI) for multi-host engines, the
S3 analogue of the Modal-Volume reload. Fake-S3 harness: pointer round-trip +
publish->read_manifest->materialize (27 tests total).
* engine: prefetch base to NVMe at startup so first sync is delta-only
Adds Engine.prefetch() (optional, no-op default) and fires it as a background,
best-effort task in Reconciler.startup — overlapping the base copy with serving
+ the first rollout, so the first real stage() applies only the delta instead of
paying the full ~base-size copy on the trainer's critical path (the trainer is
blocked waiting for the pool to reach v1).
SGLangEngine.prefetch POSTs /pull_weights target_version=0: verified against the
fork receiver (weight_updater/local_checkpoint), for v0 it seeds from the engine's
own model_path (source_dir unused) and applies no deltas. Safe to overlap a
concurrent first stage(): the receiver holds a per-host fcntl flock and is
idempotent (applied>=target early-returns), so the v1 pull just applies the delta
onto the already-seeded base. Bonus: being the first pull, it captures the pristine
model_path as the reseed base before any commit mutates it.
* cleanup: delete v1 tree; rename miles/slime -> *_disagg; default reload load-plan on
- Delete dead v1 core: protocol.py, bulletin.py, sync_test.py (old API), and the
servers/ trainers/ providers/ packages. Clean surface = versions/sync/service/
publish + the three ports + their instances.
- Delete superseded cookbook: old miles_disagg/slime_disagg/local_disagg/
standalone_rollouts trees + loose v1 files (bulletin_hooks, serving, sidecar,
sidecar_process, trainer_helpers, ray_cluster, rollout_control).
- Rename the clean cookbook recipes miles -> miles_disagg, slime -> slime_disagg;
update every cookbook.{miles,slime} reference.
- serving_image: default SGLANG_ENABLE_RELOAD_LOAD_PLAN=1. The old "record path
hangs the GLM fused-MoE loader" no longer reproduces on the pinned fork; being
validated e2e (isolated tests show replay ~2x slower than native multithread
load for GLM-Air fp8, so this is under evaluation, not a proven speedup).
- README/DESIGN: refresh structure to *_disagg; document elastic join (scale the
pool mid-run, replicas self-sync to latest).
28 tests pass; miles_disagg + slime_disagg apps and all 10 configs import.
* glm45_air_fp8: disable reload load-plan (native multithread load makes replay ~2x slower)
Add a per-config SGLANG_ENV override (applied to the sglang process env at Server
startup, over the image's baked defaults) and set glm45_air_fp8 to
SGLANG_ENABLE_RELOAD_LOAD_PLAN=0.
Why per-config, not global: the load-plan reload optimization's win is relative to
the native load speed. It (a) overlaps H2D copies across a thread pool and (b) skips
the model's per-tensor name->param routing scan. GLM-4.5-Air-FP8's config already
sets --model-loader-extra-config enable_multithread_load (native ~15s), so the plan's
parallelism is redundant and its per-name dispatch overhead makes replay ~2x slower
(15s -> 34s, not CPU-bound). Kimi-K2.6-NVFP4 sets no multithread loader (single-
threaded native ~300s) and has a far larger tensor count, so the plan is a large win
there (~300s -> ~10s) and keeps the image default (flag on).
* engines: add VLLMEngine scaffold (TODO stubs, no impl yet)
Placeholder for a vLLM rollout engine behind the Engine port (sglang is the reference).
All weight-sync methods raise NotImplementedError with a note on the vLLM equivalent;
__init__/base_url are filled in, prefetch inherits the no-op default. Fill in when a
vLLM pool is brought up.
* cookbook: pin sglang stitch-sglang-v0.5.15 fork + enable checkpoint prefetch
- Bump the serving fork to modal-projects/sglang `stitch-sglang-v0.5.15`
(base tag v0.5.15 + the weight-sync patch stack) over lmsysorg/sglang:v0.5.15.
- Add cookbook/common/SGLANG_FORK.md documenting the patch stack, the upstreaming
PRs (#30367, #30761), and how to re-port onto a newer sglang release; link it
from the README.
- Enable --weight-loader-prefetch-checkpoints in every serving config: parallel
page-cache prefetch of shards cuts the cold-volume weight read ~3x on a cold
boot / elastic joiner (GLM-4.5-Air-FP8 cold start 716s -> 231s).
* cookbook: bump sglang fork pin to 43fccb0c (general NVFP4 partial + record-at-boot)
Re-pin to stitch-sglang-v0.5.15 @ 43fccb0c and update SGLANG_FORK.md:
- C5 (modelopt fp4 partial) is now general — cutlass AND trtllm per-expert
incremental post-loading — and restores the NVFP4 raw-compare in the weight
checker so a partial reload is byte-verifiable via /weights_checker.
- New commit: record the load plan during the initial load, so the first
update_weights_from_disk already goes partial (no full record-reload) — a win
for elastic joiners.
* cookbook: add glm5_2_nvfp4 config (GLM-5.2 744B-A40B NVFP4, disaggregated)
Modeled on kimi_k2_6_nvfp4. GLM-5.2 is glm_moe_dsa (MLA MoE, DeepSeek-V3-arch), so it
is configured like Kimi, not GLM-4.5:
- MLA serving (tokenspeed_mla + fp8 KV) + checkpoint prefetch.
- model_name=glm_moe_dsa -> convert_deepseekv3_to_hf export; megatron_to_hf raw;
miles_model_script scripts/models/glm5.2-744B-A40B.sh.
- first 3 layers dense/bf16 (first_k_dense_replace=3); all routed experts NVFP4.
- NVFP4-QAT recipe + xor disk-delta + GRPO shared with the Kimi recipe.
- Source zai-org/GLM-5.2 is bf16 (no dequant); served NVFP4 base via convert_hf_to_nvfp4.
Trainer parallelism carries Kimi's 128-GPU layout as a starting point; size + validate
the PP split for the 78-layer arch before a real run (flagged in-file). Parsers are the
closest available (glm45 reasoning / glm47 tool) pending a GLM-5.2-specific parser.
* stitch: reject corrupt weight-version pointers as control-state errors
VersionRef.parse now raises on non-empty unparseable pointer content instead of
silently coercing to (None, 0) -- a torn / garbage `latest` was indistinguishable from
"no pointer" and could serve or extend the wrong weights. Callers already treat a
missing/empty pointer as None before parsing, so a corrupt pointer surfaces as a
reconcile error instead. Ported to stitch-v2 from stitch#31.
* stitch: make run switches admission-atomic + reload base on reset
- AdmissionGate.commit closes admission BEFORE draining, so no request is admitted
mid-drain (an in_place non-exact request must not straddle a base reset).
- SGLangEngine.reset re-materializes base AND reloads it into the engine, so a run
switch that lands at v0 serves base on the GPU -- not the previous run's weights
under the new (run, 0) identity.
- Reconciler._switch_run also resets when a prior pass errored (dirty v0 staging), not
only when the applied version > 0.
Ported to stitch-v2 from stitch#32.
* stitch: fsync the latest pointer before the atomic rename
ModalVolumeStore._atomic_write flushes + fsyncs the temp file before os.replace, so the
pointer write is durable across a crash. Ported to stitch-v2 from stitch#30.
* cookbook: config-specific reload-plan flag + declarative Modal decorators
Behavior + cleanliness pass on the cookbook, no functional change to the sync path:
- SGLANG_ENABLE_RELOAD_LOAD_PLAN is opt-in per recipe, not a serving-image default:
the NVFP4 configs set SGLANG_ENV={"...":"1"} (single-threaded native load, so
replay wins big); glm45_air_fp8 leaves it off (its native load is already fast).
- Drop --model-loader-extra-config from the glm45 configs: the values passed
(enable_multithread_load=true, num_threads=8) equal sglang's own defaults, so the
flag was a no-op. Multithreaded load stays on; the prefetch flags (real overrides)
stay.
- Trainer and prepare_torch_dist use inline conditional decorators for the
multi-node RDMA/EFA case instead of building a kwargs dict and rebinding the
decorated object afterward. Removes the PartialFunction workaround comment.
- Drop the dead te_precision_config_file entry from YAML_CONFIG_FIELDS (app.py
materializes it to a node-local path before resolve_config runs, so the tmpdir
path never fired); simplify the over-guarded custom_config_path coercion; read the
source's quant scheme in prep instead of matching "int4" in the repo name.
- pytest: add pythonpath=["."] so `uv run pytest` resolves the top-level cookbook
package (the hook test) out of the box.
* docs: capability-first README, design-only DESIGN, de-attribute fork doc
- README leads with what the framework achieves (weight sync, version correctness,
elastic sync, coordination) and how a request flows, not a file-by-file tour;
the detailed architecture lives in DESIGN.md, linked once.
- DESIGN drops the completed rewrite-plan checklist and the old->new "what moved"
migration table (both historical now that the rewrite has landed) and keeps only
the design: purpose, ports, domain model, invariants, naming, testing.
- SGLANG_FORK.md: the patch tiers are described by what they do, not who wrote them
(git carries authorship); fix the reload-plan flag note to match the code (opt-in
per recipe, not on-by-default).
* Remove AGENTS.md/CLAUDE.md; drop DESIGN.md from the repo
AGENTS.md and CLAUDE.md were local agent scaffolding, not project docs. DESIGN.md
stays as a working design note on-disk but is no longer tracked. Drop the README
link to DESIGN.md (would 404 without it published) and the test docstring's
reference to AGENTS.md.
* sync: restore the background reconcile backstop
The rewrite left convergence purely event-driven — startup, an external /wake, or a
409 self-heal — and dropped the periodic pull the original sidecar ran. A wake is
best-effort and at-most-once (fanned to whoever is discoverable at publish time), so a
replica whose cold start raced the last publish/wake, or whose wake was simply lost,
would sit stale until the NEXT publish. Safety held (the gate 409s any pinned request it
can't serve), but independent liveness did not: a replica getting no qualifying traffic
never self-healed.
Reconciler now runs a periodic backstop (default 5s, --reconcile-interval /
STITCH_RECONCILE_INTERVAL, 0 to disable) that re-checks latest and nudges a reconcile.
It reuses wake() — idempotent and non-cancelling — so a caught-up tick is a cheap
pointer read and an in-flight reconcile is left to finish. Started after the initial
sync and cancelled in a new shutdown() the service lifespan calls. This also makes the
"self-sync on their next poll" comments in sync/publish/pools true again.
Tests: a replica behind after a lost wake converges on its own; interval=0 disables it.
* tests: co-locate as google-style *_test.py; tighten host-mem docstring
Move the suite back beside the code it tests (foo_test.py next to foo.py), the
convention v1 followed: reconcile_test -> sync_test, publish_test, s3_store_test ->
stores/s3_test, cookbook_hooks_test -> common/hooks_test, and split instances_test into
stores/modal_volume_test + engines/sglang_test + pools/modal_flash_test. pytest testpaths
now scans src/ and cookbook/; the wheel already excludes *_test.py.
Also reword start_host_mem_monitor's docstring to state the why without the war-story tone.
* test: convergence-liveness conformance tests (backstop fix verified) (#45)
* test: red conformance tests for wake-driven convergence liveness
Three strict-xfail tests pin the self-convergence property the wake-driven-only
Reconciler does not provide (transient-error recovery, lost publish wake, wake
delivered during a pass's caught-up recheck), plus three green companions pinning
the recovery channels that do exist (explicit wake, constraint-409 self-wake).
If a self-convergence mechanism lands, the reds XPASS as errors -> promote them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PCAdJizffy3fwe4sQZL7So
* test: promote liveness reds — backstop fix verified, boundary parametrized
2b3f2db's periodic backstop makes all three formerly-xfail scenarios converge,
so each now runs twice: [backstop] pins the fix, [wake-only] (interval=0) keeps
the wake-driven-only non-convergence boundary documented.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PCAdJizffy3fwe4sQZL7So
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* test: fold liveness tests into sync_test.py (google-style), trim to 3 (#46)
tests/reconcile_liveness_test.py landed (#45) just after the co-location move
(a4dee7f), leaving it uncollected (testpaths no longer scan tests/) and with a
broken reconcile_test import. Fold it into src/stitch/sync_test.py and cut the
verbiage: drop the lost-wake scenario (covered by the existing backstop pair),
merge each scenario's backstop/wake-only legs into one test, keep one 409-channel
companion. 3 tests instead of 9, argless for the direct-run harness.
Claude-Session: https://claude.ai/code/session_01PCAdJizffy3fwe4sQZL7So
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* cookbook configs: fp8 base-case ready + strip verbose comments
- glm45_air_fp8: drop the explicit SGLANG_ENV. sglang reads
SGLANG_ENABLE_RELOAD_LOAD_PLAN defaulting to "0", so unset already means no
load-plan / no partial reload — the plain fp8 base case needs no extra config.
- Strip verbose docstrings, decorative section headers, and restate-the-code
comments across all miles + slime configs; keep the non-obvious "why" notes and
the TODO(glm5.2) deploy checklist. Comments/docstrings only — zero config-value
changes (AST-verified).
* cookbook: EXPERIMENT_CONFIG required + baked into images (was silently defaulting)
The container re-imports app.py and read EXPERIMENT_CONFIG with a default
("glm45_air_fp8"), so a non-default deploy (e.g. kimi) got the right app name /
volumes from the deploy-time decorators but the WRONG model + delta volume at
runtime (container defaulted to GLM) — a silent mismatch that only surfaced because
GLM happened to be the default. Two-part fix:
- EXPERIMENT_CONFIG is now required (os.environ[...], no default) in both app.py, so a
missing value fails loud instead of quietly serving the wrong experiment.
- build_{trainer,serving}_image take `experiment` and bake EXPERIMENT_CONFIG into the
image env (inside the builder, before add_local_*), so the container's re-import
resolves the same experiment as the deploy.
Also rename the deprecated --cuda-graph-max-bs -> --cuda-graph-max-bs-decode.
* kimi_k2_6_nvfp4: 8-node trainer (TP8/PP2/CP4/EP32), 10 GRPO steps
Drop from 16 nodes (EP16) to the proven kimi_k25_lora_8nodes topology TP8/PP2/CP4/EP32
(64 GPUs) — EP32 keeps expert weights+grads ~63GB/rank, comfortable on B200. num_rollout
10 to publish enough deltas to verify real-training tensor-sparsity.
* sync: serialize base prefetch with stage + expose prefetch state
Robustness on the prefetch↔serve↔stage overlap:
- A stage() now awaits the in-flight base prefetch before writing the host-local
checkpoint (both write the same dir), so prefetch→stage is strictly ordered in the
reconciler instead of relying on the engine's file lock to serialize a concurrent
target=0 seed and target=N apply. Test: a stage blocks until the prefetch finishes.
- server_info exposes prefetch_done / prefetch_error (two-level readiness): `ready`
means serveable and does NOT wait on the prefetch — the replica serves while the base
seeds — and the prefetch state makes a slow/failed seed observable instead of silent.
A stage that had to seed the full base itself records metrics["paid_base_copy"].
* sync: drain_all on run switches — nothing crosses an incompatible transition (#47)
in_place lets rolling requests cross compatible weight commits, but _switch_run used
the same mode-dependent gate, so a rolling request could be admitted during — and
decode across — a base reset (review P0-B; in_place is the configured mode for the
NVFP4 configs). commit(drain_all=True) now gates and drains everything for a run
switch regardless of mode; compatible commits are unchanged.
* miles_disagg: buffered fp8 reload, drop region pins, right-size kimi pool
- glm45_air_fp8: swap the mmap-prefetch flags for --weight-loader-disable-mmap
+ multithread load. The dense fp8 delta reloads the full checkpoint every
step; mmap on the ephemeral disk is pathological (~0.3 GB/s), buffered ~8.
- Drop the region="us" pin on both configs so the scheduler places freely.
- kimi_k2_6_nvfp4: cap the rollout pool at 2x B200:4 so the 64-GPU trainer gang
co-schedules instead of the pool autoscaling up and re-competing for B200.
* miles_disagg: fp8 reload via fastsafetensors per-rank read (nogds)
The dense fp8 delta reloads the full checkpoint every step; under gVisor the read
is bound by the Sentry single-thread byte-copy (~5 GB/s/node), not the disk.
fastsafetensors splits the safetensors across TP ranks (each reads 1/N), cutting
the per-node read ~408->102 GB => GLM-4.5-Air-fp8 reload ~130s -> ~23s (measured).
- Pin the fork to 13479b5 (fastsafetensors: env-gated nogds; GDS/nvidia-fs is
absent under gVisor, so force the O_DIRECT+bounce path).
- serving_image: install fastsafetensors + SGLANG_FASTSAFETENSORS_NOGDS=1.
- glm45_air_fp8: --load-format fastsafetensors (replaces interim disable_mmap).
- SGLANG_FORK.md: document patch 7.
---------
Co-authored-by: Jason Mancuso <7891333+jvmncs@users.noreply.github.com>
jvmncs
marked this pull request as ready for review
July 15, 2026 08:47
…30b_a3b_nvfp4_46) The full 4-bitter-Lesson recipe (miles PR #1261) as a miles_disagg experiment: per-token row-scaled NVFP4, dequantized backward (TE 2.17 + TE#3141 fix), bit-exact four-over-six on weights+activations, last-15%-BF16 selective precision, on the blog's own testbed (Qwen3-30B-A3B, DAPO-Math-17k, GRPO). The experiment carries its stack deltas through new per-experiment hooks: trainer_image gains image_tag/miles_repo_ref/setup_commands overrides (TE 2.17 cu13 + the dequantized-backward fix), the shared serving image gains extra_commands/extra_env (flashinfer 0.6.13 lockstep + the FLASHINFER_* side of the 4/6 quantizer contract), and prep applies PREP_ENV so the served base is converted under the trainer's exact NVTE_* settings. miles is branch nvfp4-46-recipe-v2 of modal-projects/miles (the shared pin merged with PR #1261). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vcp3ejtJfewbt4jKwrxVWR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
qwen3_30b_a3b_nvfp4_46— the full humans& "4-bitter Lesson" NVFP4 RL recipe (radixark/miles#1261) as a miles_disagg experiment on the stitch-v2 cookbook, on the blog's own testbed (Qwen3-30B-A3B, DAPO-Math-17k, GRPO):NVTE_BACKWARD_OVERRIDE=dequantized; TE 2.17.0 cu13 + the TE#3141 save-original-input fix applied as a self-verifying edit)NVTE_*) and sampler (FLASHINFER_*) sides of the quantizer contract pinned to identical conventions (MAE, e4m3-max-256, fast-math off) in one place in the configInfrastructure changes (per-experiment stack hooks)
trainer_image.build_trainer_imagegainsimage_tag/miles_repo_ref/setup_commandsoverrides;common.serving_image.build_serving_imagegainsextra_commands/extra_env;prep.prepare_checkpointsappliesexp.PREP_ENV(the served base must be quantized under the trainer's exact NVTE_* settings or the delta baseline diverges from the export).app.pywires all of these from the experiment module.modal-projects/miles@nvfp4-46-recipe-v2= the shared pin (bdaf8d04) merged with radixark PR #1261 (clean merge, zero conflicts; the shared-experts convert exclusion and upstream disk-delta mode are preserved).stitch-sglang-v0.5.15fork image + flashinfer 0.6.13 python/cubin/jit-cache in lockstep (0.6.13 carries the 4/6 error-domain fix, flashinfer#3448).Validation (Modal, jason-dev, ≤11 concurrent B200)
Same bar as the pre-rebase PR (details being re-run on the v2 stack; results posted below as a comment):
PREP_ENV, pool smoke at weight_version 0train_rollout_logprob_abs_diff ≈ 0.042–0.045(PR #1261 reports 0.031), XOR delta publish → engine/pull_weightsapply → versioned rolloutsattention_backend=flash(BF16 attention, outside the quantization contract) — TE 2.17's cuDNN fused-attn graph rejects the base image's libcudnn 9.16; drop at cuDNN ≥ 9.19🤖 Generated with Claude Code
https://claude.ai/code/session_01Vcp3ejtJfewbt4jKwrxVWR