Refactor Jetson-PI provider onto model runtime v1#157
Draft
LiangSu8899 wants to merge 36 commits into
Draft
Conversation
Phase 0/1 checkpoint: verify the flashrt_cpp_llama_cpp_provider static library can link against Jetson-PI's llama/mtmd APIs, including the Pi0 whole-graph entry point mtmd_helper_eval_chunks_pi0. - FLASHRT_CPP_WITH_JETSON_PI + JETSON_PI_ROOT embed Jetson-PI via add_subdirectory(... EXCLUDE_FROM_ALL). Build options are FORCE-set so stale cache or conflicting -D cannot break the 'mtmd without the full tools tree' isolation (LLAMA_HTTPLIB/LAMA_CURL off, tools/server/examples off, JETSON_PI_BUILD_MTMD on). Requires BUILD_TESTING=ON (link-check is a test target); a FATAL_ERROR explains the constraint if violated. - jetson_pi_link.cpp references mtmd_default_marker, the default-param helpers, and &mtmd_helper_eval_chunks_pi0 so the link line exercises the Phase 1 Pi0 infer entry, not just generic symbols. It also asserts that frt_llama_cpp_pi0_runtime_create_with_engine rejects an empty engine. - test_llama_cpp_jetson_pi_link drives that symbol end-to-end. Depends on Jetson-PI commit 153730a (JETSON_PI_BUILD_MTMD). No GGML types enter FlashRT public headers (c_api.h stays clean); the Pi0 stage remains a provider-owned callback, not a CUDA Graph. Reviewed by two independent Claude Code subagents (high); their major findings addressed: (M1) added mtmd_helper_eval_chunks_pi0 reference; (M2) added FORCE to the LLAMA_* cache sets; (M3) added the WITH_JETSON_PI/!BUILD_TESTING FATAL_ERROR; (m1) disabled LLAMA_HTTPLIB. Co-Authored-By: Claude <noreply@anthropic.com>
FlashRT now loads a real Jetson-PI Pi0 provider through the unified
frt_model_runtime_v2 entry and runs one in-process whole-graph Pi0 infer.
The Pi0 glue (marker injection, tokenize, encode+decode, KV reset, state
padding) lives entirely in Jetson-PI's jetson_pi_pi0 library; the FlashRT
engine is a thin frt_image_view/prompt/state -> jetson_pi_pi0_infer mapping.
Provider (薄映射 engine):
- jetson_pi_engine.{h,cpp}: frt_llama_cpp_default_engine_factory() returns a
static factory whose create_pi0 opens jetson_pi_pi0, validates the model's
action_shape against the config, and fills frt_llama_cpp_engine_v1.
set_input(IMAGES) swizzles frt_image_view[] (RGB8/BGR8/RGBA8/BGRA8/GRAY8,
honors stride_bytes) into packed RGB and stashes per-view pointers;
set_input(PROMPT/STATE) stash; run_infer calls jetson_pi_pi0_infer;
get_output copies the action chunk out. atomic refcount + close on drop.
create_pi0 failures report through a thread_local sink (factory->last_error).
- c_api.h: dropped frt_llama_cpp_pi0_config.state_dim (ABI break — redundant
with action_dim; Jetson-PI pads state to action_dim internally). state port
shape is now {action_dim}. pi0_runtime.cpp + existing fake test updated.
- CMake: engine source decoupled from BUILD_TESTING (builds whenever
FLASHRT_CPP_WITH_JETSON_PI is on); provider links jetson_pi_pi0 (not mtmd/
llama directly), so the provider's only Jetson-PI dependency is the policy
API. FLASHRT_CPP_WITH_JETSON_PI=1 compile def gates the engine TU.
Test + fixture:
- test_llama_cpp_jetson_pi_engine.cpp: skip when FLASHRT_PI0_MODEL/MMPROJ/
FIXTURE_DIR unset; bogus-path contract sub-test; real-weight sub-test
(set_input images/prompt/state -> run_stage INFER -> get_output actions;
asserts shape, no NaN/Inf, non-zero). action_steps/dim read from env.
- prepare_pi0_fixture.py: converts a LIBERO .npz to image.png/wrist_image.png/
state.bin (zero-padded to action_dim)/prompt.txt.
Verified: ctest 4/4 pass (skip path). With real pi0_base weights + fixture,
engine test runs one Pi0 infer and produces a 10x32 action chunk in ~63s.
Reviewed by two independent Claude Code subagents (high); addressed:
view_to_rgb rejects unknown pixel formats (no silent RGB8 fallback);
actions_buf cleared on each set_input so get_output can't return stale data;
image_ptrs computed after rgb_scratch is fully grown (no dangling-pointer
risk); state.bin size checked against action_dim; dead image_channels field
removed.
Depends on Jetson-PI commit f0265b3 (jetson_pi_pi0 target).
Co-Authored-By: Claude <noreply@anthropic.com>
…ct docs Close out the Phase 1 review follow-ups (M7/M9 + header docs): - test_llama_cpp_jetson_pi_engine: add sub-test C (config action_dim mismatch vs model must fail open cleanly, engine freed) and sub-test D (multi-tick: second infer with fresh inputs must (1) return non-zero from get_output after set_input but before run_infer — staleness guard from the B1 fix — and (2) reproduce the first tick's actions, proving KV reset + no cross-tick leak). Update the file header to document A/B/C/D coverage and the new FLASHRT_PI0_ACTION_STEPS/DIM env overrides. - jetson_pi_engine.h: document factory lifetime, thread safety, and the engine error-code space (0 / -1 / -2 / -5 / -7 / -8) so hosts can branch on codes without reading the .cpp. Verified: ctest skip path PASS; real-weight path (pi0_base 10x32) PASS — all 23 checks green including the new multi-tick reproducibility assertion. Co-Authored-By: Claude <noreply@anthropic.com>
…ovider (Phase 2)
Phase 2 of the FlashRT<->Jetson-PI migration: the unified Python entry
flash_rt.load_model(...) can now select the Jetson-PI llama.cpp/GGML Pi0
provider and run one in-process whole-graph Pi0 infer end to end
(Python -> ctypes -> SHARED provider .so -> frt_model_runtime_v2 ->
jetson_pi_pi0). No torch/jax, no GPU arch detection.
- cpp/CMakeLists.txt: new SHARED target flashrt_cpp_llama_cpp_provider_c
(pi0_runtime.cpp + jetson_pi_engine.cpp, PIC ON, links flashrt_runtime +
jetson_pi_pi0). The STATIC flashrt_cpp_llama_cpp_provider is kept for the
C++ tests. The SHARED .so exports frt_llama_cpp_default_engine_factory and
frt_llama_cpp_pi0_runtime_open_with_engine_factory for ctypes dlopen.
- flash_rt/frontends/jetson_pi/pi0.py: Pi0JetsonPiFrontend dlopens the .so,
opens a Pi0 runtime via the default factory, and drives set_input(images/
prompt/state) -> run_stage(INFER) -> get_output(actions) through ctypes
mirrors of frt_image_view / frt_model_runtime_v2 / frt_llama_cpp_engine_
factory_v1. Returns the raw action_steps x action_dim action chunk
(no unnormalization; caller post-processes). ABI-gates on abi_version and
struct_size before touching the struct.
- flash_rt/api.py: load_model gains framework="jetson_pi" (early-return
branch before torch/jax dispatch) + keyword-only mmproj_path/backend/
action_steps/action_dim/lib_path. framework validation extended to
("torch","jax","jetson_pi").
- flash_rt/tests/test_jetson_pi_pi0_python.py: env-gated smoke test
(FLASHRT_PI0_MODEL/MMPROJ/FIXTURE_DIR/LIB) — skips when unset; otherwise
load_model -> predict -> assert shape (10,32), no NaN/Inf, non-zero.
- docs/jetson_pi_usage.md: build, usage, LD_LIBRARY_PATH, limitations.
Verified: smoke test skip path PASS; real-weight path (pi0_base 10x32)
PASS, actions (10,32) no NaN, non-zero. C++ ctest 4/4 regression PASS.
Reviewed by two independent Claude Code subagents (high); addressed:
removed dead n_threads param (M1); added abi_version/struct_size ABI gate
(M2); _find_lib now hard-errors on a missing explicit lib_path instead of
silently falling back (M3); json.dumps(ensure_ascii=False) so non-ASCII
paths survive the C JSON parser (m1); infer() guards closed state + accepts
bytes prompt (m2/m3/m6); _FactoryV1 hoisted to module scope (n2); dropped
unused Sequence import; unified float32 byte constant.
Co-Authored-By: Claude <noreply@anthropic.com>
Extend the Jetson-PI provider from Pi0-only to also serve generic GGUF LLMs
(text completion), reusing the existing provider lifecycle, staged ports,
and callback-stage mechanism. One raw prompt in -> one generated text blob
out; no streaming, no multi-turn state, no chat template (caller applies it).
C++ provider (append-only, Pi0 path untouched):
- c_api.h: FRT_LLAMA_CPP_MODEL_LLM=2, frt_llama_cpp_llm_config, LLM port/
stage enums, create_llm fn pointer on frt_llama_cpp_engine_factory_v1
(inserted between create_pi0 and last_error), frt_llama_cpp_llm_runtime_*
declarations.
- llm_runtime.cpp: v2 runtime wrapper, 2 STAGED ports (prompt TEXT in, text
TEXT out) + 1 callback 'infer' stage; strict JSON open (all sampler fields
required, no defaults). Mirrors pi0_runtime.cpp shape.
- jetson_pi_engine.cpp: LlmEngine + verbs (set_input PROMPT / run_infer calls
jetson_pi_llm_generate / get_output TEXT) + create_llm in the default
factory. set_input clears text_buf (staleness guard).
- CMake: STATIC + SHARED provider add llm_runtime.cpp; link jetson_pi_llm
alongside jetson_pi_pi0.
- test_llama_cpp_jetson_pi_llm.cpp: env-gated e2e (bogus path + real
qwen3-0.6b generate + printable check).
Python:
- frontends/jetson_pi/llm.py: LlmJetsonPiFrontend (dlopen same SHARED .so,
open LLM runtime, generate(prompt)->str, infer({prompt:...})->{text:...}).
Reuses pi0.py ctypes mirrors; _find_lib reads FLASHRT_LLM_LIB.
- api.py: load_model(framework=jetson_pi, config=llm, ...) returns the
LlmJetsonPiFrontend directly (NOT wrapped in VLAModel — LLMs are not VLA).
New keyword-only LLM kwargs (n_ctx/n_threads/temp/top_k/top_p/seed/
max_tokens).
- tests/test_jetson_pi_llm_python.py: env-gated smoke (qwen3-0.6b).
- docs/jetson_pi_usage.md: LLM chapter + raw-prompt limitation note.
Verified: C++ ctest 5/5; Python LLM smoke (qwen3-0.6b) generates readable
text; Pi0 C++ + Python paths unchanged.
Reviewed by two independent Claude Code subagents (high); addressed:
B1 — _FactoryV1 ctypes mirror now includes create_llm (was missing, causing
Phase 2 Pi0 + Phase 3 LLM bogus-path to read create_llm as last_error
and segfault); added factory struct_size ABI gate in both frontends;
M1 — _find_lib takes an env_var arg; LLM frontend reads FLASHRT_LLM_LIB;
M2 — jetson_pi_llm.h documents n_ctx=0->4096 / seed=0->LLAMA_DEFAULT_SEED /
max_tokens=0->512 zero-semantics;
M3 — LlmJetsonPiFrontend rejects max_tokens<=0 (C side still defaults, but
Python no longer disagrees);
m2 — removed duplicate jetson_pi config warning.
Depends on Jetson-PI commit (jetson_pi_llm target).
Co-Authored-By: Claude <noreply@anthropic.com>
Add MLLM (vision+text → text) support through the Jetson-PI provider: - c_api.h: MLLM config, port/stage enums, factory create_mllm - mllm_runtime.cpp: v2 runtime (3 STAGED ports + callback infer stage) - jetson_pi_engine.cpp: MllmEngine + create_mllm factory entry - pi0.py: _FactoryV1 ctypes mirror updated with create_mllm field - mllm.py: Python MllmJetsonPiFrontend (images + prompt → text) - api.py: config="mllm" branch returns MllmJetsonPiFrontend - C++ e2e + Python smoke tests (Qwen2.5-VL-3B verified) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…t conditions) Four-gate assessment (scheduling benefit / buffer contract / sync semantics / profiling evidence) of splitting the whole-graph Pi0 infer stage into finer context/encoder/diffusion/action stages. Findings (all verified against code by two independent reviews): - Pi0 emits the action chunk atomically after the full K=10 diffusion loop (action_ready only at llama-context.cpp:1389; early reads rejected :798). Time-to-first-action == time-to-full-chunk -> no streaming win. - The K denoise steps are a strict serial chain (step i+1 consumes step i's cross.action) -> no intra-tick K-parallelism. - The encoder already runs once outside the K-loop -> no redundant work to remove by splitting encoder/diffusion. - The real latency levers live INSIDE Jetson-PI: per-step graph rebuild (can_reuse=false for state/action/sinusoidal/cross_kv), per-step D2H + host Euler recurrence, cross-KV device->host->device round-trip. None are FlashRT stage decompositions. - FlashRT ABI can DECLARE finer stages but cannot host OBSERVABLE inter-stage buffer contracts today (no intermediate port direction, provider-owned path rejects buffers/SWAP, no per-step get_output) -> that is Phase 6 territory. Decision: DEFER Phase 5. The real optimization work is routed to Jetson-PI internals behind the existing single infer stage. Conditions for revisiting documented in section 7. Profiling plan (section 8) to be backfilled on the Jetson target. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Four-lens evaluation (exec/ CUDA coupling, provider+zero-copy needs, mllm reference, ABI/memory-domain constraints) of how to design FlashRT's backend-neutral exec/backend vtable + memory-domain contract. Findings (verified against code): - exec/ already has a backend-neutral SHAPE: exec/backend/backend.h is 13 free functions, CUDA isolated to one 107-line TU (cuda_backend.cpp), zero virtual, zero CUDA headers in src/include. A "full vtable" rewrite solves nothing the free-function seam doesn't already solve at link time. - The provider-owned callback-stage bypass ALREADY EXISTS and is in production use (llama_cpp provider never includes exec.h). CUDA-Graph vs provider-owned coexistence is already met. - GGML is self-contained; FlashRT executes no GGML ops, so a full backend vtable would have no consumer. Contradicts CLAUDE.md "Jetson-PI as model provider, not bare exec/ hardware backend". - The ONLY genuinely missing piece for the stated zero-copy goal is the memory-domain CONTRACT (Phase 5 Gap 2, deferred to Phase 6 at model_runtime_api.md:127-129). Decision: GO-MINIMAL-CONTRACT. Build an opaque frt_memory_token + copy_to_host/copy_from_host/sync/destroy + frt_rt_location_kind enum, attachable to a provider-owned port via an append-only port-desc-v2 (struct_size probe, same pattern as copy_verbs/copy_verbs_v2). Relax the provider-owned rejection for the token form ONLY; raw frt_buffer rejection stays. exec/ untouched. FlashRT NEVER dereferences the token — zero-copy is an advertised (location_kind=HOST_VISIBLE) capability, not an assumption. Satisfies Phase 5 revisit condition (e) only; conditions (a)-(d) are Jetson-PI-internal. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Phase 6 deliverable is the memory-domain CONTRACT, not a full backend-neutral exec/backend vtable. Decision rationale (four-lens eval, GO-MINIMAL-CONTRACT) is in docs/phase6_backend_vtable_eval.md (commit b9919be); exec/ already has a backend-neutral shape (13 free functions, CUDA isolated to one TU), the provider-owned callback-stage bypass already coexists with CUDA Graph, and GGML is self-contained so a full vtable would have no consumer. ## The contract - Opaque `frt_memory_token` + `frt_memory_token_verbs` (copy_to_host/copy_from_host/sync/destroy) + `frt_rt_location_kind` (HOST_VISIBLE/DEVICE_LOCAL), attached to a provider-owned port via `frt_runtime_builder_add_port_token`. FlashRT NEVER dereferences the token; zero-copy is an ADVERTISED capability (location_kind=HOST_VISIBLE), not an assumption. This is the ONLY buffer form permitted on provider-owned ports; the builder still rejects a raw `frt_buffer` there (which would imply same-backend device sharing with no contract). - exec/ is byte-identical except a doc-only split of backend.h into shared primitives (malloc/free/stream/event/memcpy/memset) vs CUDA-only graph-capture (capture_begin/end/graph_launch/graph_exec_destroy), plus a commented `frt::be::import` extension point (declared only, not implemented for non-CUDA). No vtable, no registry, no exec/src changes. - Two coexisting contracts, honest about each: (A) the CUDA-export SWAP path (frt_buffer + frt_buffer_dptr/wrap + frt_graph_bind + INPUT|OUTPUT role bitmask) unchanged; (B) the provider-owned token path above. See docs/exec_contract.md §7a and docs/model_runtime_api.md. ## Carrier (chosen mechanism, recorded honestly) Tokens ride a NEW trailing `port_tokens`/`n_port_tokens` array on `frt_model_runtime_v2`, index-parallel with `ports` (`n_port_tokens == n_ports` ALWAYS). This matches how v2 already carries parallel ports/stages/stages_v2 arrays and keeps `frt_runtime_port_desc` byte-identical. The eval's §8/§13 sketch named a `frt_runtime_port_desc_v2` tail-probe form; the shipped implementation uses the simpler parallel array instead (a revision note is appended to the eval so history does not contradict the code). `destroy` fires once when the holder refcount hits zero — on the provider-owned v2 path the holder IS the token's lifetime owner (there is no v2 wrap/override path, so the holder refcount is the single relevant count). ## Two-subagent review (claude-code internal, high) + second revision Before commit, two independent adversarial subagents reviewed the diff (correctness+ABI lens, contract-fidelity lens). Both flagged blockers, confirmed against source (one reproduced a defect with a standalone repro): 1. INDEX-ALIGNMENT BUG (confirmed by repro): the original diff claimed `port_tokens[i]` was parallel to `ports[i]`, but `add_port` (the normal path) pushed nothing into `port_tokens`, so a runtime that mixed `add_port` + `add_port_token` got `n_port_tokens != n_ports` with tokens column-shifted against the wrong ports. FIX: `add_port` now also pushes a null-handle token entry, making `n_port_tokens == n_ports` and the index-parity a literal invariant (the release-time destroy loop already no-ops on null handles). A new mixed-port test locks this in. 2. DEAD DECLARATION: `frt_runtime_port_desc_v2` was declared but never built or read by any consumer — ABI surface with no consumer. FIX: removed; comments now describe the actual `port_tokens` parallel-array carrier. 3. APPEND-ONLY GATE (pre-existing, made acute by v2 growth): the pybind readers `as_export`/`as_model`/`as_model_v2` used a strict `struct_size !=` comparison, so a producer built against the grown v2 (or a future larger v2) would be rejected — breaking the append-only contract the C ABI (`frt_model_runtime_wrap`, already `>=`) relies on. FIX: changed to `<` across all three pybind readers. 4. GRANULARITY WORDING: the contract said destroy fires at "port/export refcount" zero; the implementation ties it to the holder refcount. Recorded honestly as a narrower (and for this path equivalent) invariant; no per-port counter added (no consumer — over-design per CLAUDE.md). ## Verification - runtime/build: cmake build green; ctest green (model_runtime). - New mixed-port test (add_port + add_port_token on one provider-owned builder) passes, asserting n_port_tokens == n_ports, null handle on the non-token port, minted handle on the token port at the aligned index, and destroy-once-at-release. - Existing Phase 6 tests unchanged and green; raw frt_buffer on a provider-owned port still rejected; pybind still accepts the grown v2. - Syntax-checked the llama_cpp provider source against the changed header (gcc 11.2, conda): clean. In scope (minimal): token types, add_port_token, index-parallel port_tokens, append-only ABI evolution, backend.h doc split, smoke/mixed tests, docs. Out of scope: full exec/backend vtable, runtime-selectable backend registry, cross-backend zero-copy, Phase 5 ABI gaps (1/3/4/5). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The env-gated Pi0 end-to-end test hardcoded backend="cpu" in all three JSON literals, so FLASHRT_PI0_BACKEND could not actually drive the GPU path. Add an env override (default "cpu", byte-identical to the prior behavior) and wire it into the real-Pi0 JSON only; the bogus-path and action_dim-mismatch sub-tests keep "cpu" hardcoded so they stay cheap and deterministic (no wasteful 37-layer GPU offload on a path that only exercises the action_dim rejection). Verified on GPU2 (RTX 4090, sm_89) with pi0_base: offloaded 37/37 layers to GPU, two-tick inference + staleness guard, actions non-NaN and reproducible across ticks, == JETSON_PI ENGINE PASSED ==. Co-Authored-By: Claude <noreply@anthropic.com>
The Python smoke test hardcoded backend="cpu", so the Jetson-PI CUDA path through flash_rt.load_model(framework="jetson_pi", ...) was never exercised end-to-end from Python (docs/jetson_pi_usage.md flagged it as unverified). Add FLASHRT_PI0_BACKEND env override (default "cpu", byte-identical) and document the CUDA build/run recipe. Verified on GPU2 (RTX 4090, sm_89) with pi0_base, the CUDA-built libflashrt_cpp_llama_cpp_provider_c.so: offloaded 37/37 layers to GPU, actions shape (10,32), no NaN/Inf, non-zero, == JETSON_PI PYTHON PASSED ==. First real end-to-end run of the Python ctypes entry on the CUDA backend. Co-Authored-By: Claude <noreply@anthropic.com>
Follow-up to the CLAUDE.md refresh (which deleted the mllm/ repo and
corrected the stale pi0_libero_base weights path). Two reviewer-flagged
inconsistencies in docs CLAUDE.md points readers to:
- jetson_pi_usage.md: the action_steps comment named pi0_libero_base as a
live alternative ("pi0_libero_base is 50"). That GGUF fails this fork's
check_tensor_dims (embedding_length=1152 vs token_embd rows 2048) — see
CLAUDE.md weights note. Replace with a pointer to the note so a reader
landing in the usage doc doesn't try the broken pair.
- phase6_backend_vtable_eval.md §4 cites mllm file:line (X2XOp.cpp:13-15
etc.) as provenance for the distilled patterns. The mllm/ repo is now
deleted; add a NOTE block flagging the citations as historical and the
reclone path, so a future session doesn't chase a missing repo. The
distilled patterns are self-contained in the section.
No behavior change.
Co-Authored-By: Claude <noreply@anthropic.com>
Extends the Phase 2 CUDA-verification pattern (env-override backend, default "cpu" byte-identical) to the Phase 3 (LLM) and Phase 4 (MLLM) Python smoke tests, which were previously hardcoded backend="cpu". All three Jetson-PI providers now have a real-GPU end-to-end run from the Python unified entry. - test_jetson_pi_llm_python.py: add FLASHRT_LLM_BACKEND env (default "cpu"). - test_jetson_pi_mllm_python.py: add FLASHRT_MLLM_BACKEND env (default "cpu"). - docs/jetson_pi_usage.md: mark CUDA verified for all three providers with offload counts; add the env-override recipe to the LLM/MLLM sections; qualify the MLLM result as a GPU execution-path check (raw prompt → template-token noise; caption coherence needs a caller-applied chat template), distinct from Pi0/LLM whose output is coherent. Verified on GPU2 (RTX 4090, sm_89): - LLM qwen3-0.6b-q4_k_m offloaded 29/29 layers, sane text, PASSED (/tmp/llm_cuda_run2.log) - MLLM Qwen2.5-VL-3B-Instruct-q4_0 offloaded 37/37 layers (+ VIT/mmproj on CUDA0), template-token output, PASSED (/tmp/mllm_cuda_run2.log) GPU2 mem returns to 0 MiB after each run. 2 claude-code-internal subagent reviews (opus, high): APPROVE-WITH-NITS each; must-fix was the doc summary overstating MLLM as fully verified — fixed by qualifying it. Two "below"→"above" directionality nits — fixed. Co-Authored-By: Claude <noreply@anthropic.com>
Phase 6's memory-domain contract (frt_memory_token + verbs + parallel port_tokens array) shipped as 5d25672 but had zero providers calling frt_runtime_builder_add_port_token — paper-only. This closes that gap: the Pi0 provider mints a token on its actions OUT port, backed by the engine's live actions_buf host buffer, advertised HOST_VISIBLE. - jetson_pi_engine.cpp: file-scope token verbs (copy_to_host reads the LIVE actions_buf at call time, not a mint-time pointer — std::vector reallocates on clear/assign; -7 not-ready / -5 too-small mirror engine_get_output; copy_from_host returns -3 unsupported for an OUT port; sync is a HOST_VISIBLE no-op; destroy is null — the engine owns actions_buf and frees it in release, which fires after the holder's token-destroy loop). Process-global static const verbs table + frt_jetson_pi_actions_token_verbs() accessor. - jetson_pi_engine.h: declare the accessor; handle is engine.self, HOST_VISIBLE. No GGML/Engine type in the public header (handle stays opaque frt_memory_token). - pi0_runtime.cpp: actions port switches add_port -> add_port_token under #if FLASHRT_CPP_WITH_JETSON_PI (handle=engine.self, verbs accessor, bytes=action_steps*action_dim*sizeof(float)). The #else keeps the prior add_port so the always-built fake-engine test (test_llama_cpp_provider, no real engine to back a token) still links and passes. The port desc is byte-identical (STAGED, no raw buffer), so model identity/fingerprint and get_output are unchanged — the token is an additive parallel read path. - test_llama_cpp_jetson_pi_engine.cpp: new sub-test E — n_port_tokens==n_ports, only actions carries a non-null HOST_VISIBLE token, verbs complete, copy_to_host byte-identical to get_output, copy_from_host==-3, sync==0, and a staleness assertion (copy_to_host fails after set_input clears actions_buf) proving the token reads the live buffer, not a snapshot. Verified on GPU2 (RTX 4090, pi0_base, backend=cuda): offloaded 37/37 layers, all sub-tests A-E ok, == JETSON_PI ENGINE PASSED == (/tmp/pi0_token_run2.log). Python pi0 smoke (BACKEND=cuda) re-run unchanged: 37/37, PASSED — additive change didn't disturb the GPU path. Non-JETSON_PI build (build-provider) + fake-engine test: builds, == LLAMA_CPP PROVIDER PASSED ==. JETSON_PI=ON fake-engine test: also PASSED (token minted with FakeEngine* handle but never read; destroy null, no crash). 2 claude-code-internal subagent reviews (opus, high): reviewer 1 caught a must-fix (non-JETSON_PI link breakage — the token path referenced a JETSON_PI-only symbol unconditionally) -> fixed with the #if/#else guard; reviewer 2 confirmed the must-fix resolved and the fake-engine-under-ON type-confusion is inert (no consumer, null destroy). Both APPROVE. Two nits applied (comment wording; move action_window_bytes inside #if). Closes CLAUDE.md Current Initial Assessment gap (1): Phase 6 contract now has a real token consumer. Co-Authored-By: Claude <noreply@anthropic.com>
…(§14)
Closes jetsonpi迁移.txt §14 gap: previously FlashRT only verified
shape/non-NaN/reproducibility, never compared numbers to the native
Jetson-PI output. Adds C++ parity tests that drive the FlashRT
frt_model_runtime_v2 path AND a direct jetson_pi_pi0/jetson_pi_llm call
with identical inputs/backend in the same binary, and compare.
- test_llama_cpp_jetson_pi_parity.cpp (Pi0): FlashRT open+set_input+
run_stage+get_output vs direct jetson_pi_pi0_open/action_shape/infer/
close; asserts max_diff <= 1e-5. Fixtures loaded once, fed to both.
- test_llama_cpp_jetson_pi_llm_parity.cpp (LLM): FlashRT path vs direct
jetson_pi_llm_open/generate/close with greedy sampling; asserts EXACT
text equality (greedy=argmax, no RNG, deterministic).
- CMakeLists.txt: two new targets under if(FLASHRT_CPP_WITH_JETSON_PI),
mirroring the existing engine/llm targets.
Determinism foundation (verified against source): Pi0's noise latent is
create_normal_noise_cpp11(.., seed=41), reset per tick; KV cleared per
infer — deterministic given fixed model+inputs+backend+n_threads. LLM
greedy builds a greedy-only sampler chain (no dist/RNG) — deterministic.
FlashRT adds no numerical perturbation vs the direct call (image swizzle
is identity for RGB8 fixtures; prompt/state copied verbatim; marker
injection + zero-pad happen inside the shared library for both paths).
Verified on GPU2 (RTX 4090): Pi0 CPU + CUDA parity max_diff == 0,
PASSED; LLM CUDA parity exact text match, PASSED (/tmp/{pi0,llm}_
parity_*.log). GPU2 mem back to 0 after. (LLM CPU parity runs but is
slow — two model loads; CUDA is the practical path, documented.)
2 claude-code-internal subagent reviews (opus, high): both APPROVE-WITH-
NITS, no must-fix. Highest-priority concern (could two Pi0 engines
overlap on the GPU and make max_diff==0 a coincidence?) verified CLEAN —
the release chain (holder_release -> destroy_owner -> engine_release ->
jetson_pi_pi0_close -> llama_free) is fully synchronous; the first
engine's GPU resources are freed before the second opens. Nits applied:
fclose leak -> ifstream existence check; LLM CPU-cost note; action_steps
default caveat (pi0_base is 10, default 50 fails the open).
MLLM parity deferred — its smoke output is template-token noise, not a
stable parity target.
Co-Authored-By: Claude <noreply@anthropic.com>
…r current conditions) Resolves CLAUDE.md TODO-2 (§8.1 LLM prefill + host-repeatable decode + next_token/logits ports) by evaluation/DEFER, mirroring the Phase 5 precedent (8a4cb9d). No code changes — a decision document. Gate (verified against source): Jetson-PI narrow API jetson_pi_llm exposes only open/generate/close/open_error; generate hardcodes llama_memory_clear (:200) + llama_sampler_reset (:201) every call — KV and sampler state wiped per completion. No prefill/decode_step/reset/ get_logits symbol. FlashRT cannot fabricate the prefill/decode boundary from its side (the only seam is generate, opaque inside) — same gate Phase 5 hit. Repeated generate(max_tokens=1) is killed by the unconditional KV clear: re-prefill O(n^2), no KV reuse. Four lenses: lens 3 (sync semantics) PASSES — unlike Pi0 flow-matching (no usable intermediate state, TTFA==TTC), an LLM per-token decode IS the natural boundary, so TODO-2 is implementable not incoherent. But lens 1 (scheduling benefit) FAILS — no jetson_pi caller multiplexes LLM sessions or interleaves decode. The torch frontends qwen3_rtx/ qwen36_rtx do per-step decode, but on framework="torch"/NVFP4, a different provider not driven through jetson_pi; §8.1 is the jetson_pi spec, so the scoping is legitimate (stated explicitly in the eval). Lens 2 half-met (no logits consumer). Lens 4 real ABI cost (engine vtable run_infer -> run_stage, append-only + 2-repo refactor of the path TODO-1 0cee8c7 just proved byte-identical). Revisit (GO when ANY of a-d): a real jetson_pi caller needs host-driven token looping / speculative decode / multi-session KV reuse; b FlashRT gains a jetson_pi multi-session scheduler (caveat: doesn't exist today, near-redundant with a); c caller needs logits; d streaming UX requirement. Until then the one-shot generate (TODO-1 byte-parity verified) is the honest, sufficient LLM face. CLAUDE.md (unversioned, at migrate root): TODO-2 -> DEFER with eval-doc pointer + revisit a-d; TODO-3 (Vulkan/SYCL) advanced to current. 2 claude-code-internal subagent reviews (opus, high): both APPROVE-WITH- NITS, no must-fix. Nits applied: split the "all already linked and used piecewise" list (llama_get_logits_ith is linked-but-uncalled); add the torch-LLM-frontend scope note to lens 1 (forecloses the rationalization reading); flag revisit (b) as contingent on a jetson_pi serving path that doesn't exist today; fix the §5 test attribution (base llm test is Phase 3 083f6f8, parity test is TODO-1 0cee8c7). Co-Authored-By: Claude <noreply@anthropic.com>
Closes CLAUDE.md TODO-3 (jetsonpi迁移.txt §6/§14 Vulkan). Jetson-PI now
accepts "vulkan" (commit de66ccc on the Jetson-PI branch); FlashRT needs no
code change (backend forwarded verbatim). This adds the Vulkan build recipe
+ verified-per-model results to jetson_pi_usage.md.
- Verified bullet: CPU/CUDA/Vulkan backends; Vulkan verified for Pi0
(pi0_base, 36 layers + VIT on Vulkan0, parity max_diff=0) and LLM
(qwen3-0.6b, 28 layers on Vulkan0, greedy text byte-identical). MLLM-on-
Vulkan not verified this round (§6: each model×backend separately).
- Backend-mapping note: rewritten to state the truth — unknown backends are
REJECTED (INVALID), not a silent CPU fallback (honors the no-fallback
contract); the device is chosen by GGML's scheduler from compiled-in
backends, not pinned by the string; the recipes use single-backend builds
to avoid CUDA+Vulkan split.
- Vulkan build recipe: -DGGML_VULKAN=ON -DGGML_CUDA=OFF, conda compilers,
CC/CXX exported for the vulkan-shaders-gen C++17 host tool, GGML_VK_VISIBLE_DEVICES
pinning. Documents the Khronos-1.3.295-headers prerequisite (conda-forge
vulkan-headers 1.3.231 lacks the vk_video av1 codec headers + hpp_macros
that vulkan_core.h unconditionally #includes) and warns against in-place
conda-env overwrite (a later conda update clobbers it).
Verified on GPU2 (RTX 4090, GGML_VK_VISIBLE_DEVICES=2): LLM + Pi0 smoke +
parity PASSED; GPU2 mem clean after.
2 claude-code-internal subagent reviews (opus, high): both REQUEST-CHANGES
on the doc only (code approved). Must-fix: the original note's "silently
runs CPU" claim was factually wrong (code rejects unknowns) and contradicted
the no-fallback contract; the device-pinning ("cuda"->CUDA0) overstated
determinism (GGML scheduler picks, not the string); the Vulkan recipe hid
the Khronos-headers prerequisite. All three fixed. OpenCL/SYCL deferred
(no oneAPI/DPC++ compiler; OpenCL backend less complete for these models).
Co-Authored-By: Claude <noreply@anthropic.com>
- docs/jetson_pi_usage.md / validation_matrix.md: 把后端从私有 repo (diantoudedianshan/Jetson-PI, branch flashrt-migration-merge, pin 9c8e8b30) 更新为公开 repo PKU-SEC-Lab/Jetson-PI-Edge 的 Jetson-PI-flashrt 分支 (tip 5de3f9e, 2026-07-15)。Build 节补充 clone 公开 repo 的指引; 旧私有 sha 作为出处记录保留。关联 issue #143。 - cpp/CMakeLists.txt: test_pi05_runtime / test_device_staging 的 FLASHRT_CPP_WITH_EXEC guard 不是 spurious —— 这两个 test 经 flashrt_cpp_pi05 -> runtime.cpp 传递引用 frt_graph_replay (定义在 flashrt_exec), 关 EXEC 时无法链接。补注释说明 guard 必要性, 避免后续 被当无关改动删掉。同时统一 parity 测试块缩进, 并注释 STATIC provider PRIVATE vs SHARED _c PUBLIC 的 link 可见性差异 (SHARED 供 ctypes dlopen, NEEDED 必须显式带上 libjetson_pi_*.so)。
师兄要求: 在 docs/jetson_pi_usage.md 开头补一段, 介绍 Jetson-PI 推理引擎 的出处 (arXiv:2607.12659) 与定位 —— 面向 onboard 实时机器人控制, 通过 llama.cpp/GGML 跑 PI0/PI0.5, 多后端 (desktop NVIDIA GPU / Jetson Orin/Thor / CPU-only / NPU edge), 代码在 PKU-SEC-Lab/Jetson-PI-Edge; 并说明本 FlashRT provider 是让应用通过 FlashRT Python API 加载同一份 Jetson-PI GGUF 模型, 而非跑 Jetson-PI HTTP foreground server。
# Conflicts: # cpp/CMakeLists.txt # docs/model_runtime_api.md # flash_rt/runtime/export.py # runtime/bindings/runtime_pybind.cpp # runtime/include/flashrt/model_runtime.h # runtime/src/internal.h # runtime/src/model_runtime.cpp # runtime/src/runtime_export.cpp # runtime/tests/test_model_runtime.cpp # runtime/tests/test_model_runtime_py.py
This was referenced Jul 20, 2026
heiheiha798
added a commit
to heiheiha798/FlashRT
that referenced
this pull request
Jul 21, 2026
…it lands Rebases the Jetson-PI provider onto flashrt-project#157 (model-runtime v1 generic plan) and stops advertising a `context -> action` split that the backend does not honor. The Jetson-PI narrow C API `jetson_pi_pi0_context()` runs the full `mtmd_helper_eval_chunks_pi0` and caches the final action; `action()` only copies the cache (`pending_context` is never assigned, the prepare_chunks_pi0_for_model/decode_pi0 path is unwired). Advertising a 2-stage plan would misrepresent that cached handoff as an encode/decode boundary. PI0.5 proprioceptive state is also not yet serialized into the `Task: ..., State: ...;\nAction:` prompt through the narrow API — it still goes through the legacy `llama_set_pi0_state` tensor, which the PI0.5 graph never reads (zeros-vs-ones is bit-identical). - pi0_runtime.cpp: force `staged_context_action = false` and publish a single `infer` stage. LLM/MLLM `reset -> prefill -> decode` is untouched. - tests: demote the three context/action assertions (parity, provider, python) to single-infer expectations; the parity test no longer compares against a split path. - new test_llama_cpp_jetson_pi_state_parity.cpp: skip-gated PI0.5 reference-policy state-parity scaffold (boundary-region states x<-1 / (-1,0) / (0,1) / >=1, images/task fixed). Activates with FLASHRT_PI0_STATE_PARITY_READY=1 once the backend lands. - docs: drop the `context -> action for VLA` advertisement and staged example; soften the VLA selected-plan gate to single `infer`. Backend corrections (PI0.5 state serialization + real context/action split) are tracked in the linked Jetson-PI-Edge PR; the split plan, staged tests, and frontend helpers are re-enabled once that lands and state parity passes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This keeps the Jetson-PI Pi0/LLM/MLLM integration contributed in #148, but rebases its public boundary onto the existing
frt_model_runtime_v1capability model.The provider remains optional and owns all GGML/llama.cpp semantics. FlashRT runtime core, exec, shared kernels, and existing graph producers are unchanged.
Why this refactor
#148 originally introduced a parallel model-runtime v2 to represent provider-owned stages and memory. Since then, the v1 ABI gained an additive, size-probed generic stage-plan extension. Maintaining both runtime families would give producers and hosts two competing authorities for the same ports, stages, identity, and lifecycle.
This version therefore:
frt_model_runtime_v1from every provider path;query_extension;step()as whole-model compatibility sugar;get_outputinstead of a provider-specific memory token;runtime/,exec/, andcsrc/.Selected plans
infercontext -> actioninferreset -> prefill -> decodeExecutor references intentionally differ from stage indices and are validated by the provider contract tests.
Build and packaging
frt_model_runtime_open_v1.$ORIGINlookup within the deployment prefix.Validation
dlsymfrt_model_runtime_open_v1runtime/,exec/, andcsrc/have zero diff from current mainThe real-checkpoint run at
048cd21b36517c79cebaa93c5c46c90d60a5deeemeasured a 62.358 s open and a 266.857 ms steady two-view p50 for the converted F16 GGUF path. These are qualification measurements, not production performance claims.The run also found two model-level blockers that the wrapper/direct parity test cannot detect:
state[action_dim], but the tested PI0.5 policy has an eight-value proprioceptive state whose normalized and discretized values belong in the formatted prompt. Changing the raw staged state produced bit-exact actions, while correctly formatting zero versus one state changed actions (cos=0.990631708,max_abs=1.555626512).context()computes and caches the final action, whileaction()only copies it (264.385 msversus0.145 ms). Until the backend exposes a real pending-context/action boundary, the provider should publish oneinferstage rather than imply a computational split.The ABI/lifecycle refactor remains independently reviewable, but this draft is not yet a PI0.5 release candidate. Direct Jetson-PI versus FlashRT-wrapper parity proves that the adapter preserves the backend result; it does not prove equivalence to the PI0.5 reference policy.
Relationship to #148
This is a reviewable replacement built directly on the original contribution and preserves its commit authorship. I am leaving #148 open while we discuss the boundary with the contributor.