fix(worker): register a blob for every asset the stub produces - #1022
Open
georgi wants to merge 1 commit into
Open
fix(worker): register a blob for every asset the stub produces#1022georgi wants to merge 1 commit into
georgi wants to merge 1 commit into
Conversation
The stub context covered a ref type per entry point, not per type. Any factory it did not override fell through to ProcessingContext, which returns a ref with inline data and no uri; the executor extracts blobs only from a blob:// uri and the serializer strips raw data at any depth, so the bytes were discarded while the node reported success. Override the remaining producers: image_from_io (which image_from_url funnels through), model3d_from_io, and audio_from_segment's no-name path, which returned a memory:// ref pointing at a store that lives and dies inside the worker. Also carries the video_from_io and audio_from_io overrides, since this branch is cut from main. Add a coverage test with two halves: it drives every *_from_io funnel with a real buffer and follows the bytes through the executor, and it walks the source to require every producer to be overridden or to delegate to one — so the next factory added to ProcessingContext cannot ship as a silent data-loss bug.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the class behind #1020 (video) and #1021 (audio).
This is one defect class, not four bugs
WorkerContextcovers a ref type per entry point, not per type. Any factory it does not override falls through toProcessingContext, which with nonamereturns a ref carrying inlinedataand an empty uri.executor._extract_outputsextracts blobs only for ablob://uri, and_serialize_asset_refstrips every raw-bytesdatafield at any depth. The bytes are discarded and the node reports success — 114.6 s of A40 time for an empty mp4, in the case that started this.So every
*_from_*method added toProcessingContextfrom now on is a silent data-loss bug in the worker until someone remembers to override it. The coverage test below is what stops that.Changes
image_from_ioImageRef(data=…, uri="")blob://image_from_urlimage_from_iomodel3d_from_ioModel3DRef(data=…, uri="")blob://audio_from_segment(no name)AudioRef(uri="memory://<uuid>", data=…)blob://video_from_io,audio_from_ioBranch base. Cut from
mainas asked, so it also carries thevideo_from_ioandaudio_from_iooverrides — without them the coverage test cannot be green here. If #1020 and #1021 merge first, rebasing shrinks this diff to image / model3d / segment plus the tests; the conflicts are adjacent method insertions in one file. Their test files are not duplicated here — the coverage test drives those funnels behaviorally.audio_from_segmentkeeps the base implementation's WAV encoding and metadata; the override only re-homes the bytes when the base took itsmemory://shortcut. That shortcut exists so a later in-process read can get theAudioSegmentback without decoding. A worker output crosses a process boundary, so the host can never follow it — the fast path is not available to a worker at all, and the choice is between blob bytes and no bytes.Finding 1 —
image_from_urlVerified, not assumed. Its body is one line:
It downloads and funnels through
image_from_io, so overridingimage_from_iocovers it. No separate override.tests/worker/test_context_stub_remaining_blob_gaps.py::test_image_from_url_funnels_through_image_from_iopins that with a stubbeddownload_file— no network.Finding 2 —
asset_to_dataA reader, not a producer. Not this bug, not fixed. It takes an existing
AssetRef, and when that ref is amemory://handle with nodatait resolves the handle out of the in-process store and returns a copy withdatafilled in. It creates no new asset and no new bytes; it materializes bytes that already exist so a caller can read them.The coverage test excludes it structurally rather than by name: a method with an
AssetRef-typed parameter converts a ref, it does not produce output. Nothing is allowlisted.The coverage test
tests/worker/test_context_stub_blob_coverage.py, two halves, because neither is enough alone:*_from_ioproducer, drive each with a realBytesIO, and assert it returns ablob://uri and that the bytes come back out ofexecutor._extract_outputs. A guard test asserts the enumeration actually found a funnel for every type inASSET_REF_TYPES, so it cannot pass by covering nothing.ProcessingContext— returns a type inASSET_REF_TYPES, takes no ref parameter — must be overridden byWorkerContextor delegate (transitively, read off the source) to one that is. This is what catches a factory added tomorrow.The structural half alone would have been worse than nothing: it follows delegation without seeing branches, so it marked
audio_from_segmentcovered while its no-name branch dropped the audio. That is why the behavioral half exists, and the test's docstring says so.Red / green
Before (stub reverted to main, tests present):
Every red is behavioral. The
memory://line is theaudio_from_segmenthole the structural check could not see.After:
Whole worker suite:
225 passed in 9.25s. No network, no GPU, no model downloads — the one download is monkeypatched, and theAudioSegmentis 20 ms of generated silence.Nothing is left failing: every entry point the enumeration finds passes.