Skip to content

Compiled engine cannot model the VL encoder phase (image config not threaded through compile) #1567

Description

@tianhaox

Gap

The compiled engine (and therefore every embedded consumer — the Dynamo Mocker, build_aic_engine callers) models VL models as TEXT-ONLY: vision encoder ops are deliberately NOT emitted into the EngineSpec (sdk/engine.py::build_engine_spec_json, "Vision encoder ops are intentionally NOT emitted into the spec").

Root cause: the compile path threads no image configuration (num_images_per_request, image_height/image_width, num_image_tokens), so the compiled engine cannot reproduce BaseBackend._run_encoder's token-count math (eff_batch, eff_s, pre/post-merge counts) needed to query the vision ops with correct shapes. Emitting the encoder ops without that config would make the engine query them unconditionally with wrong shapes — worse than skipping.

The Python sweep path is NOT affected: _run_encoder evaluates the encoder op list through build_ops_json + EngineHandle.evaluate_ops_json, so the per-op VALUES are already engine-computed (single-oracle holds). This issue is purely about the compiled/embedded path's ability to see the encoder phase at all.

What landing this takes

  1. Thread runtime image config through compile_engine / build_engine_spec_json (and the EngineBuildRequest on the Rust side) — likely as an optional EngineConfig extension (schema bump).
  2. Emit model.encoder_ops into the spec (the op classes are already Rust-backed since the pyo3 op unification — _spec_json works today; vision ops live in operators/vision.rs).
  3. Port _run_encoder's token-count composition (eff_batch/eff_s, pre/post-merge token counts, encoder-DP) into the Rust runtime, gated on the image config being present (absent config keeps today's text-only behavior, bit-for-bit).
  4. Parity pins for at least one VL model (qwen3vl) with image config set, plus the config-absent no-op case.

Trigger / priority

Needed only when an embedded consumer wants to model VL serving (e.g. Mocker simulating a VL deployment). Until then the Python path serves VL correctly. Filed so the deferral recorded in build_engine_spec_json's comment has a tracking home after the rust-migration ladder closed (#1357, PR #1566).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions