Skip to content

runtime_contract.md

Latest commit

 

History

History
207 lines (174 loc) · 11.1 KB

runtime_contract.md

File metadata and controls

207 lines (174 loc) · 11.1 KB
Raw

FlashRT Runtime Export (runtime/)

The hand-off surface between a FlashRT model runtime (producer) and a host/serving layer (consumer). One captured, replay-ready model is packaged as one POD struct — frt_runtime_export_v1 — and adopted by the consumer.

The exec contract (docs/exec_contract.md) fixes how to replay. The runtime export fixes what a deployed model IS: which streams, graphs, buffers, and restorable state regions exist, and the identity that stored state is bound to. Both layers are mechanism only. The base export does not publish scheduling policy. The model-runtime face may publish one selected stage DAG as execution mechanism; orchestration policy remains the consumer's job.

Structure

producer (owns model + capture)              consumer (owns loop + state policy)
─────────────────────────────────            ──────────────────────────────────
today:                                        e.g. FlashRT-Nexus capsule host,
  Python setup/capture                        a robot loop, a server shell
    flash_rt/runtime/export.py
      └─ _flashrt_runtime.Builder ──┐
later (same struct, host unchanged): │        adopt(export*)
  native model runtime .so           ├──►  frt_runtime_export_v1  ◄──┘
    frt_runtime_open_v1(config,&out)─┘        │ ctx, streams[], graphs[],
                                              │ buffers[], capsule_regions[],
                                              │ fingerprint/identity/manifest,
                                              │ owner + retain/release
                                              ▼
                                        replay / snapshot / restore
                                        via exec.h handles only

runtime/
  include/flashrt/runtime.h   the ABI (structs + builder). Consumers need ONLY
                              this header + exec.h — the struct is plain data.
  src/runtime_export.cpp      builder + export lifetime (no CUDA, no exec link)
  bindings/runtime_pybind.cpp `_flashrt_runtime` (setup/dev bridge)
  tests/                      model-free acceptance
flash_rt/runtime/export.py    Python producer: RuntimeExport / build_export()

The contract, in five rules

  1. One struct, two producers. Today Python fills it in-process; later a native model runtime .so exports frt_runtime_open_v1 (symbol name is in the header) and fills the same struct. Consumers never change.
  2. Consumers see handles, never internals. No Python, torch, model code, or kernel headers cross this boundary — only frt_* handles, POD descriptors, and strings owned by the export.
  3. Identity is split from discovery. identity is the canonical string (weights digest, quant, kernel version, arch — supplied by the producer — plus graph names and the full capsule-region layout, appended by the builder). fingerprint = FNV-1a 64 of identity, computed only by the builder: one implementation, one hashing rule. manifest_json is free-form discovery data; editing it never invalidates stored state.
  4. Region order is contractual. Restorable state regions are matched by position on restore, so their order/name/offset/bytes are all fingerprinted.
  5. Lifetime is explicit. The consumer calls retain(owner) on adopt and release(owner) when done — from any thread. The phase-1 Python producer handles GIL acquisition inside release. While a reference is held, every handle in the struct (including native_handle stream pointers) stays valid; the Python process stays resident as the setup host, because CUDA graph execs are process-local by construction.

Producing an export (phase 1, Python)

export = pipeline.export_runtime(identity={"weights_sha256": digest})
# hand export.ptr (an frt_runtime_export_v1*) to the native consumer

Pi05Pipeline.export_runtime() is the reference producer: streams = the capture stream, graphs = infer / decode_only, buffers = the pipeline IO surface, default capsule region = the rollout boundary (diffusion_noise, the region set validated by serving/robot_recap/verify_capsule.py).

The model runtime ABI (flashrt/model_runtime.h)

The export describes a captured model's static execution assets; it does not say how dynamic inputs enter the model each tick. That is the model runtime ABI — frt_model_runtime_v1 — the standard face of one deployed, tickable model:

                 ┌────────────────────────────────────────────┐
   host tick ──► │ frt_model_runtime_v1                       │
                 │   ports[]   modality/dtype/shape/update    │
                 │   stages[]  subgraph DAG (export graphs)   │
                 │   verbs     set_input · get_output ·       │
                 │             prepare(warm) · step(sugar)    │
                 │   exp ────► frt_runtime_export_v1          │
                 │             (state/replay kernel, frozen)  │
                 └────────────────────────────────────────────┘

The contract is data first, verbs as sugar. Ports carry the load-bearing update class — the two-speed hot path:

  • SWAP — the port is a device-buffer window; the host writes raw bytes directly (its own copy verb / cap_swap). Microsecond lane, zero model code in the loop. (observation tensors, noise seeds, numeric state)
  • STAGED — the runtime's set_input transforms host data (tokenize / resize / normalize / embed) into bound buffers. (prompt text, camera frames)
  • SETUP — legal only outside the tick.

Production contract for both hot classes: never recapture, never allocate, never rebind graph pointers — only buffer contents change. Replay graphs are fixed-shape or bucket-keyed; a bucket miss is handled by prepare in the warm phase, never inside a tick. step fires the declared stage order for simple hosts; scheduling hosts fire stages themselves (that is what the stage DAG is for).

Full structure map: cpp_runtime_design.md. Field-by-field interface reference: model_runtime_api.md.

Three construction paths share the contract: the export builder assembles export + ports + stages under ONE identity (frt_runtime_builder_finish_model — a port-schema change changes the fingerprint); frt_model_runtime_wrap wraps an existing export with ports/verbs (identity inherited); and frt_model_runtime_override_verbs retains an existing declaration while replacing only its verbs. Consumers retain/release only the model runtime; it holds the export/base reference internally.

Construction is fail-closed on the STAGED promise: a STAGED input requires set_input, and a STAGED output requires get_output, on integrated build, adapter wrap, and verb override alike. Validation happens before owner/export/ base retain; failed finish_model also leaves the builder available for a corrected retry. SWAP/SETUP-only declarations may still use unsupported stubs. The common Python producer checks the actual callback objects before _assemble(); the low-level pybind builder maps None to null verbs, so a trampoline does not count as an implementation unless its callback exists.

C++ model runtime layer

The runtime export is still only the hand-off surface. Model IO semantics live one layer above it in FlashRT's native C++ path:

  • cpp/runtime/ defines the non-frozen native runtime manager interfaces.
  • cpp/modalities/ contains reusable modality primitives: tensor views, vision preprocess, and action postprocess.
  • cpp/families/ contains model-family contracts such as VLA.
  • cpp/models/<model>/ contains thin model adapters that bind family + modality primitives to concrete buffer names, shapes, normalization, action schemas, and state regions.

Nexus should not implement or own these rules. It adopts frt_runtime_export_v1 and drives snapshot/restore/replay; FlashRT model runtimes prepare inputs and decode outputs.

Pi0.5 is the reference C++ model runtime under cpp/models/pi05/. The current implementation is the adopted-export path: setup/capture can still be produced by Python, while the native runtime owns vision prepare, graph replay dispatch, action decode, and export lifetime. It includes a CUDA vision resize/normalize/cast path for device buffers, with CPU reference tests, and a conservative D2H action staging path. The flashrt_cpp_pi05_c target exposes a small C host ABI around this path so C/Python/serving shells can drive the adopted export without including C++ classes. A future pure C++ checkpoint loader/tokenizer/capture path must produce the same frt_runtime_export_v1, so Nexus and serving hosts do not change.

Extending the ABI

Additive only after v1; never reorder or remove. The two public structs use their own version policy:

  • frt_runtime_export_v1: append fields with the export contract's FRT_RUNTIME_ABI_VERSION + struct_size policy.
  • frt_model_runtime_v1: the existing prefix ends at release and is measured by FRT_MODEL_RUNTIME_V1_BASE_SIZE. Consumers require only that prefix; future additive tails are individually size-probed while FRT_MODEL_RUNTIME_ABI_VERSION remains 1. The embedded verbs table is frozen and cannot grow because fields follow it.

The first model-runtime tail is query_extension. It is read only after the FRT_MODEL_RUNTIME_V1_QUERY_EXTENSION_SIZE probe. CORE assigns extension IDs; providers cannot add private IDs to the public namespace, attach tables after finish, or compute a parallel identity. The current generic stage-plan table describes one selected GRAPH/OPAQUE DAG and remains owned by the runtime.

A provider with no FlashRT graph resources may use the core metadata-only model builder. Its non-null zero-resource export is an identity/lifetime anchor, not an execution context and not a model-state snapshot promise. OPAQUE and step-only runtimes remain fail-closed for model-level state until a separately reviewed state capability exists.

An incompatible major type/symbol requires a separately reviewed ABI. An additive tail does not justify duplicating the full model-runtime interface.

Validation

PYTHONPATH=.:./exec/build:./runtime/build python runtime/tests/test_runtime_export.py
./runtime/build/test_model_runtime

The export test covers: ctypes-mirror layout check of every field, fingerprint determinism / identity sensitivity / region-order sensitivity / manifest insensitivity, retain-release lifetime against the Python anchor, and replay through exported handles. The model-runtime test covers: an independently compiled v1 ABI baseline, STAGED rejection/retry and no-retain failures, port/stage declaration, port schema in the identity fingerprint, verb dispatch, and lifetime on all three construction paths. The consumer side is validated in the FlashRT-Nexus repo (adopt + snapshot/restore through the capsule core).