Gap
The compiled engine (and therefore every embedded consumer — the Dynamo Mocker, build_aic_engine callers) models VL models as TEXT-ONLY: vision encoder ops are deliberately NOT emitted into the EngineSpec (sdk/engine.py::build_engine_spec_json, "Vision encoder ops are intentionally NOT emitted into the spec").
Root cause: the compile path threads no image configuration (num_images_per_request, image_height/image_width, num_image_tokens), so the compiled engine cannot reproduce BaseBackend._run_encoder's token-count math (eff_batch, eff_s, pre/post-merge counts) needed to query the vision ops with correct shapes. Emitting the encoder ops without that config would make the engine query them unconditionally with wrong shapes — worse than skipping.
The Python sweep path is NOT affected: _run_encoder evaluates the encoder op list through build_ops_json + EngineHandle.evaluate_ops_json, so the per-op VALUES are already engine-computed (single-oracle holds). This issue is purely about the compiled/embedded path's ability to see the encoder phase at all.
What landing this takes
- Thread runtime image config through
compile_engine / build_engine_spec_json (and the EngineBuildRequest on the Rust side) — likely as an optional EngineConfig extension (schema bump).
- Emit
model.encoder_ops into the spec (the op classes are already Rust-backed since the pyo3 op unification — _spec_json works today; vision ops live in operators/vision.rs).
- Port
_run_encoder's token-count composition (eff_batch/eff_s, pre/post-merge token counts, encoder-DP) into the Rust runtime, gated on the image config being present (absent config keeps today's text-only behavior, bit-for-bit).
- Parity pins for at least one VL model (qwen3vl) with image config set, plus the config-absent no-op case.
Trigger / priority
Needed only when an embedded consumer wants to model VL serving (e.g. Mocker simulating a VL deployment). Until then the Python path serves VL correctly. Filed so the deferral recorded in build_engine_spec_json's comment has a tracking home after the rust-migration ladder closed (#1357, PR #1566).
Gap
The compiled engine (and therefore every embedded consumer — the Dynamo Mocker,
build_aic_enginecallers) models VL models as TEXT-ONLY: vision encoder ops are deliberately NOT emitted into theEngineSpec(sdk/engine.py::build_engine_spec_json, "Vision encoder ops are intentionally NOT emitted into the spec").Root cause: the compile path threads no image configuration (
num_images_per_request,image_height/image_width,num_image_tokens), so the compiled engine cannot reproduceBaseBackend._run_encoder's token-count math (eff_batch,eff_s, pre/post-merge counts) needed to query the vision ops with correct shapes. Emitting the encoder ops without that config would make the engine query them unconditionally with wrong shapes — worse than skipping.The Python sweep path is NOT affected:
_run_encoderevaluates the encoder op list throughbuild_ops_json+EngineHandle.evaluate_ops_json, so the per-op VALUES are already engine-computed (single-oracle holds). This issue is purely about the compiled/embedded path's ability to see the encoder phase at all.What landing this takes
compile_engine/build_engine_spec_json(and theEngineBuildRequeston the Rust side) — likely as an optionalEngineConfigextension (schema bump).model.encoder_opsinto the spec (the op classes are already Rust-backed since the pyo3 op unification —_spec_jsonworks today; vision ops live inoperators/vision.rs)._run_encoder's token-count composition (eff_batch/eff_s, pre/post-merge token counts, encoder-DP) into the Rust runtime, gated on the image config being present (absent config keeps today's text-only behavior, bit-for-bit).Trigger / priority
Needed only when an embedded consumer wants to model VL serving (e.g. Mocker simulating a VL deployment). Until then the Python path serves VL correctly. Filed so the deferral recorded in
build_engine_spec_json's comment has a tracking home after the rust-migration ladder closed (#1357, PR #1566).