feat(frontend): pass through vLLM cached image UUIDs

[furionw]

Based on frontend-crates#119.

Why

vLLM extends OpenAI-compatible chat image parts with opaque processor-cache UUIDs, including UUID-only cache hits. Dynamo needs to preserve that contract across Rust/Python preprocessing and distributed workers without treating client keys as router hashes.

What changed

• Preserve aligned URL-backed and UUID-only image slots through both chat processors and the vLLM worker.
• Prefer the canonical top-level image UUID while accepting the deprecated nested image UUID.
• Limit cache UUIDs to images and vLLM; audio/video UUIDs and non-vLLM backends fail fast. Ordinary audio/video inputs remain unchanged.
• Keep client UUIDs out of exact multimodal routing while retaining vLLM feature hashes and transfer metadata.
• Add documentation, focused unit coverage, and a maintained Gemma cache-fill/cache-hit E2E profile.

Dependency

Temporarily pins dynamo-protocols 2.0.2 to frontend-crates@522a09c pending the #119 release. This PR remains draft until that prerequisite lands.

Validation

• cargo test -p dynamo-llm cached_multimodal_uuid — 5 passed.
• cargo test -p dynamo-llm preprocessor::media::loader::tests — 15 passed.
• Focused Python frontend/image/vLLM suite — 113 passed; final worker suite — 40 passed.
• mm_agg_gemma4-e2b-it_uuid_passthrough-2 — passed with both the Rust and vLLM chat processors.
• Python/Rust bindings and KVBM built successfully.
• Pre-commit hooks, cargo fmt --all -- --check, and git diff --check passed.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Walkthrough

This PR implements vLLM's cached-MM UUID passthrough across the Dynamo stack, enabling multimodal requests to reuse cached processor results via optional UUID fields. Changes span wire types, frontend extraction/batching, backend preprocessing with slot-aligned UUID tracking, defensive media loading, and comprehensive test coverage.

Changes

Cached Multimodal UUID Passthrough

Layer / File(s)	Summary
Wire-Level Type Contracts lib/protocols/src/types/chat.rs, lib/protocols/src/types/impls.rs, lib/llm/src/protocols/openai/responses/mod.rs, lib/llm/src/protocols/anthropic/types.rs	Image, video, and audio URL fields become optional; UUIDs move from nested URL objects to top-level content-part fields. Conversions and trait impls updated to wrap optional URLs in Some(...) and omit old UUID-from-URL patterns.
Preprocessor Data Types lib/llm/src/protocols/common/preprocessor.rs	Add MultimodalData::UuidOnly(String) variant for cached placeholders and MultimodalUuidMap type for per-modality UUID lists. Add PreprocessedRequest.multi_modal_uuids field for forwarding to workers.
Frontend Multimodal Extraction components/src/dynamo/frontend/utils.py, components/src/dynamo/frontend/tests/test_multimodal_utils.py	extract_mm_urls now returns (mm_data, mm_uuids) tuple, handling both URL-present and uuid-only branches with aligned UUID tracking. Tests verify tuple return shape, URL/uuid-only variant selection, and alignment across modalities and edge cases.
Frontend Request Routing components/src/dynamo/frontend/sglang_processor.py, components/src/dynamo/frontend/vllm_processor.py	Update processors to unpack new tuple return. vLLM forwards image-modality UUIDs into extra_args["mm_hashes"] when present; SGLang discards UUIDs (vLLM-only).
Frontend Image Batching components/src/dynamo/common/multimodal/image_loader.py, components/src/dynamo/common/tests/multimodal/test_image_loader.py	Extend load_image_batch to recognize UuidOnly sentinels without HTTP fetch. Build parallel futures list and slot mapping; insert None for UUID-only entries and weave results back with correct alignment. Tests verify isolation, mixed batches, and no-HTTP all-uuid case.
Backend Preprocessing with Slot Alignment lib/llm/src/preprocessor.rs	Refactor gather_multi_modal_data to track uuid_map per modality and attach slot_idx to fetch tasks. Decode results replace placeholders at correct slot; UUID presence gates routing-side mm_hashes or padding-based forwarding to kv-router.
Defensive Media Loading lib/llm/src/preprocessor/media/loader.rs	Safe URL extraction via as_ref().and_then() for image/video parts. Return explicit error for uuid-only parts instead of unconditional borrow. Cache operations only occur when URL is present.
Backend Preprocessing Tests lib/llm/tests/preprocessor.rs	Add parameterized tests for URL+UUID combinations, uuid-only scenarios, and slot-alignment validation. Verify modality variants (Url/Decoded vs UuidOnly), multi_modal_uuids presence/alignment, and transport-layer mm_hashes forwarding. Include negative tests.
Integration & E2E Testing tests/utils/multimodal.py, tests/serve/multimodal_profiles/vllm.py, tests/e2e/test_uuid_passthrough_e2e.sh, tests/utils/client.py	Add UuidPassthroughChatPayload for two-request sequences (URL+UUID, then uuid-only cache hit). Extend multimodal profiles with uuid-passthrough test cases. Create e2e shell test exercising cache-fill, cache-hit, cache-miss, and multi-image scenarios. Defensively handle optional image_url in logging.
Documentation docs/features/multimodal/multimodal-vllm.md	Document cached media UUID passthrough with request patterns, optional UUID semantics, failure modes, and --mm-processor-cache-gb worker configuration guidance.

---

🎯 4 (Complex) | ⏱️ ~60 minutes

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check	⚠️ Warning	It provides useful context, but it does not follow the required template and is missing Overview, Details, reviewer start, and Related Issues.	Rewrite the PR description using the repo template, adding Overview, Details, Where should reviewer start?, and the required Related Issues section.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title is concise and accurately captures the main change: passing through cached image UUIDs.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 9

🧹 Nitpick comments (1)

lib/llm/tests/preprocessor.rs (1)

886-910: ⚡ Quick win

Avoid unconditional extra_args unwrap in the no-UUID branch.

Line 886 unconditionally unwraps preprocessed.extra_args; this makes the expected_mm_hashes = None case brittle if extra_args is absent. Use optional-safe access for the absence assertion.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/llm/tests/preprocessor.rs` around lines 886 - 910, The test
unconditionally calls preprocessed.extra_args.as_ref().unwrap(), which causes a
panic in the None branch; change the None branch to access extra_args safely
(e.g., use preprocessed.extra_args.as_ref().and_then(|extra|
extra.get("mm_hashes"))) and assert that that result is None instead of
unwrapping; keep the existing Some(...) branch that inspects extra intact and
only use safe option/borrow access for extra_args in the expected_mm_hashes ==
None path.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@components/src/dynamo/common/multimodal/image_loader.py`:
- Around line 228-267: The reconstruction can silently drop trailing inputs
because unsupported item shapes don't append to slot_to_future_idx and
zip(image_mm_items, slot_to_future_idx) isn't strict; update the loop over
image_mm_items so that every branch (including an explicit else fallback)
appends a sentinel (e.g., None) to slot_to_future_idx and logs/raises as
appropriate, and change the pairing to use zip(..., strict=True) when iterating
for results to surface any length mismatches; reference the variables
image_mm_items, slot_to_future_idx, image_futures and the handlers load_image
and _read_and_convert_nixl_image and the keys URL_VARIANT_KEY,
DECODED_VARIANT_KEY, UUID_ONLY_VARIANT_KEY to locate the code to modify.

In `@components/src/dynamo/common/tests/multimodal/test_image_loader.py`:
- Around line 264-334: The tests call an undefined helper _mock_http_client
causing F821; add a local definition or import the existing helper into this
test module so _mock_http_client is available to async tests like
test_load_image_batch_five_images_four_overlapping and
test_load_image_batch_all_uuid_only_no_http_fetch; ensure the helper returns the
same mock object used in the with patch(..., return_value=mock_client) blocks
and exposes awaited methods used in assertions (e.g., get, send) so
mock_client.get.assert_not_awaited() and mock_client.send.assert_not_awaited()
work as expected.

In `@docs/features/multimodal/multimodal-vllm.md`:
- Around line 74-75: Add a single clarifying sentence to the paragraph that
begins "The UUID is optional." stating that any media part missing both `url`
and `uuid` is treated as invalid input and will be rejected; reference the media
part wire contract and contrast this with cache-miss failures so readers know
malformed payloads (neither `url` nor `uuid`) are refused while cache-only
requests fail when the UUID is unknown.

In `@lib/llm/src/protocols/anthropic/types.rs`:
- Around line 1660-1665: The code currently dereferences img.image_url.url
directly but img.image_url is now Option<ImageUrl> and ImageUrl.url is
Option<Url>, causing E0609; update the access in types.rs to first unwrap or map
img.image_url (e.g., use img.image_url.as_ref() or match on img.image_url) and
then unwrap the inner url (e.g., .url.as_ref() or match) before calling
to_string(), and update the expect message to reflect both missing layers (e.g.,
"image_url and image_url.url required for anthropic image conversion") so you
don't index through a None.

In `@lib/protocols/src/types/chat.rs`:
- Around line 263-267: The media-part structs (the fields image_url:
Option<ImageUrl> and uuid: Option<String>, and the analogous video/audio fields)
must reject instances that have neither a URL nor a UUID; add a validation step
in the builder or deserializer for the ImagePart/VideoPart/AudioPart types that
returns an error if both image_url (or video_url/audio_url) and uuid are None.
Locate the builder/build function for these structs (the
derive_builder-generated build path for ImagePart/VideoPart/AudioPart) or
implement a custom Deserialize/try_from for the structs and enforce "if
self.image_url.is_none() && self.uuid.is_none() { return Err(...); }" (and the
same for video_url/audio_url) so malformed payloads fail at schema boundary.

In `@tests/e2e/test_uuid_passthrough_e2e.sh`:
- Around line 51-56: The cleanup() function currently uses unsafe pattern pkill
-f which can kill unrelated processes; update cleanup() to only kill the
captured PIDs (use WORKER_PID and FRONTEND_PID) by testing if each variable is
set and the PID exists, then send a TERM (kill -TERM "$WORKER_PID") and
wait/kill -KILL if necessary, ignore errors (|| true), and clear the variables
afterward; apply the same PID-targeted replacement for the other teardown block
around lines 68-75 that also uses pkill -f.
- Around line 77-104: The readiness polling loops for the frontend (/health
probe using PORT) and worker registration (WARMUP POST to /v1/chat/completions)
lack explicit timeout handling; add timeout failure branches after each for-loop
that print a clear error (e.g., "frontend not ready after timeout" / "worker
registration timed out") and exit non-zero (exit 1) so the script fails fast
when retries are exhausted. Ensure the messages reference the probe being used
(health or chat/completions) and keep behavior otherwise unchanged.
- Around line 95-98: The test currently writes warmup responses to a fixed
/tmp/_warmup.json which can collide in concurrent runs; change the curl call in
test_uuid_passthrough_e2e.sh to write to a unique temp file (e.g., create one
with mktemp or use the test framework's tmp_path/tempfile fixture), use that
temp filename in the -o argument for the curl that posts $WARMUP to
"http://localhost:$PORT/v1/chat/completions", and ensure the temp file is
removed after the test (or use a fixture-managed temp directory) so concurrent
runs don’t clash.

In `@tests/utils/client.py`:
- Around line 42-44: The truncation call can raise TypeError when image_url is a
dict like {"url": None}; update the guard around the _truncate_base64_url call
in the code handling part.get("image_url") so it only calls _truncate_base64_url
when image_url is a dict and image_url.get("url") is not None (i.e., explicitly
check for None before passing to _truncate_base64_url), leaving image_url
unchanged when the URL is null.

---

Nitpick comments:
In `@lib/llm/tests/preprocessor.rs`:
- Around line 886-910: The test unconditionally calls
preprocessed.extra_args.as_ref().unwrap(), which causes a panic in the None
branch; change the None branch to access extra_args safely (e.g., use
preprocessed.extra_args.as_ref().and_then(|extra| extra.get("mm_hashes"))) and
assert that that result is None instead of unwrapping; keep the existing
Some(...) branch that inspects extra intact and only use safe option/borrow
access for extra_args in the expected_mm_hashes == None path.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: e9576acb-543b-4403-950d-5bcb3f5d4b0d

📥 Commits

Reviewing files that changed from the base of the PR and between 7eb03b9 and 2c7cacf.

📒 Files selected for processing (19)

• components/src/dynamo/common/multimodal/image_loader.py
• components/src/dynamo/common/tests/multimodal/test_image_loader.py
• components/src/dynamo/frontend/sglang_processor.py
• components/src/dynamo/frontend/tests/test_multimodal_utils.py
• components/src/dynamo/frontend/utils.py
• components/src/dynamo/frontend/vllm_processor.py
• docs/features/multimodal/multimodal-vllm.md
• lib/llm/src/preprocessor.rs
• lib/llm/src/preprocessor/media/loader.rs
• lib/llm/src/protocols/anthropic/types.rs
• lib/llm/src/protocols/common/preprocessor.rs
• lib/llm/src/protocols/openai/responses/mod.rs
• lib/llm/tests/preprocessor.rs
• lib/protocols/src/types/chat.rs
• lib/protocols/src/types/impls.rs
• tests/e2e/test_uuid_passthrough_e2e.sh
• tests/serve/multimodal_profiles/vllm.py
• tests/utils/client.py
• tests/utils/multimodal.py

[devin-ai-integration]

Devin Review found 1 new potential issue.

[rmccorm4]

/ok to test 62ac636

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-019f6c5a-e14a-7101-b61f-38b3620eae73.docs.buildwithfern.com/dynamo/dev

[Chokoyo]

Hi @furionw, I’ve made some updates to this PR. Within the intended aggregated serving scope, vLLM uuid passthrough now should works E2E, including url+uuid, uuid-only, embedding cache, and text-prefix routing fallback. MM-aware routing and cross-replica cache affinity remain out of scope.