interop_bench — client-side MoQ publisher/subscriber matrix

Benchmarks and interop-tests client implementations of the moq-lite/hang protocol stack — the things that publish media into a relay (sinks) and subscribe to it (sources) — by running every publisher × subscriber pair against a local relay on the same real media clip and measuring what comes out.

NOTE: this project is LLM-generated.

fixture.h264 ──▶ publisher adapter ──▶ moq-relay ──▶ subscriber adapter ──▶ received.media
                      │                                     │
                events_pub.csv                        events_sub.csv
                (frame/chunk send log)              (frame/chunk arrival log)

How this differs from the neighbours:

• englishm/moq-interop-runner tests protocol conformance of IETF-MoQT implementations (session/announce/ subscribe control flow), Docker matrix, mostly relay-role — no media-path quality metrics, and a different protocol lineage than this plugin.
• .reference/moq/rs/moq-bench is a synthetic load generator for stressing a relay with many connections — the clients themselves aren't the subject.
• ../bench/ characterises the buffer-flow timing of this plugin's Source only (regression traces); no cross-implementation comparison.

interop_bench fills the gap: same clip, same relay, swap the clients, compare delivery correctness and media-path performance.

Quick start

nix develop --accept-flake-config # optional: shell with every dependency below
./make_fixture.sh                 # once: build fixtures/ (H.264 video + ADTS AAC audio)
./run.py --list                   # probe which adapters can run on this machine
./run.py                          # run all available pairs, print report
./run.py --media aac              # same matrix over the AAC fixture
./run.py --media h265             # ... the H.265 fixture
./run.py --media h264-bframes     # ... H.264 with a B-frame pyramid (PTS != DTS)
./run.py --pairs membrane:moq-cli,moq-cli:membrane

Needs: python3, ffmpeg, moq-relay + moq-cli, and Elixir (plus a Rust toolchain for the NIF) for the membrane adapters. The flake provides all of it, including a gst-launch wrapped with the moq-gst plugin; outside nix, binaries resolve via $MOQ_RELAY / $MOQ_CLI, then $PATH, then the debug build of a .reference/moq clone in the parent repo. A relay is auto-started with the in-tree relay.toml unless one is already running (QUIC is UDP — if you run this under a network sandbox, socket binds will fail).

The bench is standalone: the only tie to membrane_moq_plugin is the plugin checkout the membrane adapters compile, $MEMBRANE_MOQ_PLUGIN (default: the directory this bench sits in). make_fixture.sh takes any ~20 s clip as its source (arg or $BENCH_SRC; audio fallback $BENCH_AAC_SRC), defaulting to the parent repo's sample media.

Results land in results/<timestamp>/, one directory per pair (adapter logs, event CSVs, received media, metrics.json) plus a run-level report.md.

Implementations

adapter	publish	subscribe	media	event granularity	status
membrane (this plugin, MoQ.Sink/MoQ.Source)	✅	✅	h264, h264-bframes, h265, aac	frame	working (a membrane-annexb publisher variant exists behind the parked kidq330/sink-annexb branch, which teaches the Sink to accept Annex B input)
moq-cli (ffmpeg piped into moq-cli publish / subscribe)	✅	✅	h264, h264-bframes, h265, aac	chunk	working; B-frame and H.265 fixtures ride MPEG-TS (publish avc3 is raw H.264 only)
moq-gst (gst-launch with moqsink/moqsrc)	✅	✅	h264	none	working; the flake's moq-gst bundle provides a wrapped gst-launch, or set GST_PLUGIN_PATH at a cargo build -p moq-gst (or GST_MOQ_PREFIX="nix shell github:moq-dev/moq#moq-gst --command"). Audio would need AAC gst pipelines in its adapter scripts

Candidates worth adding (all speak the same moq-lite/hang stack): the JS @kixelated/hang client under bun (.reference/moq/js/), the Python and Go bindings (.reference/moq/py/, go/), and moq-clock as a media-free control-plane latency baseline. moq-cli --client-version moq-transport-XX also opens the door to benching against IETF-lineage clients (moxygen etc.) through a relay that speaks both.

How it works

Fixture. make_fixture.sh encodes a 20 s / 25 fps / 500-frame Annex B H.264 stream that every publisher stream-copies onto the wire (nobody re-encodes). Encoding choices are load-bearing: zerolatency (no B-frames → decode order = presentation order = fixture order), one slice per frame (sliced-threads off — the analyzer identifies a frame by the SHA-256 of its slice NAL, and all 500 hashes are unique), fixed 1 s GOP, SPS/PPS repeated before every IDR (subscribers join mid-stream).

For --media aac it also produces a 20 s ADTS AAC-LC stream (real Big Buck Bunny audio — every frame payload hashes uniquely, which a synthetic tone's would not). The analyzer identifies an AAC frame by the SHA-256 of its raw payload with the ADTS header stripped, so hashes survive both ADTS carriage and fmp4 re-extraction. The audio frame rate (~43.07 fps at 44.1 kHz) is derived from the ADTS header, not --fps.

Two adversarial video fixtures target paths the plain one can't reach:

• --media h264-bframes — the same clip with a fixed 3-frame B pyramid, so PTS != DTS and consumers see non-monotonic presentation timestamps in decode order. It rides real container timestamps end to end (MP4-first encode; moq-cli carries it in MPEG-TS instead of raw avc3). A frame counter is burned in because static-scene B-frames otherwise encode to identical skip slices and break the unique-hash invariant. Selecting it needs --media (the fixture shares the .h264 suffix).
• --media h265 — H.265, exercising the hvcC/hvc1 path when membrane publishes and the description-less in-band hev1 shape when moq-cli publishes via TS (upstream's only H.265 import form — the H.265 twin of the avc3 framing convention).

Adapter contract. An adapter is any executable obeying the env contract, declared in adapters/registry.json:

variable	role	meaning
MOQ_URL, MOQ_BROADCAST	both	relay URL (incl. auth path) and broadcast name
BENCH_MEDIA	both	fixture format: h264, h264-bframes, h265, or aac; adapters declare support via the registry's media list (video/audio are accepted as aliases for h264/aac)
BENCH_INPUT, BENCH_FPS	publish	fixture path and its (rounded) frame rate
BENCH_OUTPUT	subscribe	where to write the received stream (output field: es = raw elementary stream, or fmp4)
BENCH_EVENTS	both	CSV to append seq,wall_ns,pts_ns,bytes,keyframe rows to
BENCH_TIMEOUT_MS, BENCH_SUB_LATENCY_MS	both / subscribe	safety timeout; jitter-buffer knob

Publishers pace the fixture at realtime (Realtimer / ffmpeg -re), publish it completely, linger ~2 s so the tail flushes, and exit 0. Subscribers write what they receive and ideally exit when the broadcast ends; ones that follow forever are SIGINT'd by the harness after the publisher finished (reported as (stopped), not a failure). Event logs use the OS realtime clock so publisher and subscriber rows are joinable across processes on one host. CLI-pipe adapters get their event log from shim.py, a pass-through that timestamps pipe chunks. Every adapter runs under wrap.py, a SIGINT-proof time -l replacement that reports the process tree's CPU time and peak RSS (this macOS gates ps/top memory columns behind an entitlement).

Analysis (analyze.py). Received media is normalized to the fixture's elementary form (ffmpeg re-extraction for container outputs: -bsf h264_mp4toannexb for video, -f adts for audio), split into frames (slice NALs / ADTS payloads), hashed, and greedy-monotone matched against the fixture's hashes. That yields exact, implementation-independent delivery accounting:

• prefix_skipped — frames published before the subscriber joined (expected),
• mid_lost — frames skipped mid-stream (e.g. latency-cap group drops),
• tail_missing — frames cut by publisher shutdown,
• corrupt — received frames matching no fixture frame (bit-rot anywhere in the path).

On top of the matches: startup (subscriber spawn → first logged event), per-frame end-to-end latency (publisher send wall-clock → subscriber arrival wall-clock, joined per frame; frame-level logs on both sides required — chunk-level publishers get send times by mapping fixture byte offsets to pipe progress), and arrival cadence (spacing stats of the subscriber's frame log).

Pass/fail judges the media path: publisher exited 0, ≥1 frame delivered, zero corrupt frames, ≤1 s of tail missing. Subscriber exit behaviour is reported (⚠ = nonzero exit) but only fails a pair indirectly, via the tail bound, when it died mid-stream.

Baseline (2026-07-15, Apple Silicon, local relay, run 20260715_003426)

publisher	subscriber	status	frames	prefix/mid/tail	startup	e2e p50/p95	sub exit
membrane	membrane	✅	450/500	50/0/0	902 ms	1.5/3.0 ms	0
membrane	moq-cli	✅	474/500	25/0/1	335 ms	—	1 ⚠
membrane	moq-gst	✅	475/500	25/0/0	—	—	1 ⚠
moq-cli	membrane	✅	423/500	75/0/2	866 ms	41.8/46.2 ms	0
moq-cli	moq-cli	✅	447/500	50/0/3	242 ms	—	130 (stopped)
moq-cli	moq-gst	✅	447/500	50/0/3	—	—	-9 ⚠
moq-gst	membrane	✅	450/500	50/0/0	904 ms	—	0
moq-gst	moq-cli	✅	449/500	50/0/1	320 ms	—	1 ⚠
moq-gst	moq-gst	✅	450/500	50/0/0	—	—	1 ⚠

The matrix caught a real interop bug on its first run: moq-cli→membrane crashed the membrane subscriber's parser with 0 frames delivered. Upstream hang/moq-mux keys H.264 payload framing on catalog description presence (WebCodecs semantics — moq-mux/src/codec/h264/export.rs): description present ⇒ AVCC length-prefixed samples, absent ⇒ Annex B with in-band parameter sets. hang-native publishers (moq-cli publish avc3, moq-gst) use the no-description/Annex B form, while MoQ.Source assumed every in-band track was length-prefixed and synthesized an avcC. Fixed since: the Source maps description-less renditions to :annexb. A companion Sink change (accepting Annex B input, publishing upstream-style in-band tracks) is parked on the kidq330/sink-annexb branch together with its membrane-annexb publisher rows.

Audio baseline (2026-07-15, Apple Silicon, local relay, run 20260715_153159)

publisher	subscriber	status	frames	prefix/mid/tail	startup	e2e p50/p95	sub exit
membrane	membrane	✅	826/862	36/0/0	5682 ms	1.7/2.4 ms	0
membrane	moq-cli	✅	790/862	70/0/2	317 ms	—	1 ⚠
moq-cli	membrane	✅	617/862	134/111/0	728 ms	—	0 ⚠
moq-cli	moq-cli	✅	755/862	106/0/1	218 ms	—	130 (stopped)

The membrane→moq-cli pair doubles as a regression test for the Sink's AAC catalog description (AudioSpecificConfig): moq-cli's fmp4 export hard-fails with MissingAudioDescription on an AAC rendition without one, so this pair delivering frames at all proves the ASC reaches the catalog (Sink fix, 2026-07-15). moq-cli→membrane's mid-stream loss tracks the TS muxer's bursty output (≈60 ms arrival std) against the Source's 500 ms jitter budget; membrane→membrane delivers with zero mid-stream loss.

H.265 and B-frame baselines (2026-07-15, runs 20260715_232847 / 20260715_233045)

All eight membrane/moq-cli pairs pass with zero corrupt and zero mid-stream loss. Two results worth keeping:

• H.265, moq-cli→membrane (424/500): TS carriage publishes upstream's only H.265 shape — in-band hev1, catalog description absent — and the Source's description-less→:annexb mapping (the avc3-bug fix) covers it. membrane→moq-cli (474/500) proves the hvcC/hvc1 direction through fmp4 export.
• B-frames, both directions: PTS survives bit-exact. The membrane publisher's parser reconstructs the non-monotonic presentation order from the bitstream (0, 160, 80, 40, 120 ms…), membrane→membrane returns the identical PTS sequence on the subscriber's pads, and moq-cli's TS timestamps arrive equally intact (~48% of arrival steps non-monotonic, as encoded). Delivery accounting is decode-order hashing, so it is immune to reordering — the PTS check above was done against the event CSVs (events_pub.csv vs events_sub.csv pts columns).

Known limitations

• moq-cli event logs are chunk-granularity, so pairs with a moq-cli subscriber have no per-frame e2e latency (extension path: parse the fmp4 trun boxes to map bytes → samples). Its subscribe adapter also forces one-frame-period --fragment-duration; arrival times still include the muxer's fragment buffering.
• The moq-cli audio publisher wraps the fixture in MPEG-TS (moq-cli has no raw-ADTS input), so its pipe bytes no longer equal the fixture and chunk-progress send times are unavailable — no e2e latency for moq-cli-published audio pairs either.
• Latency knobs aren't equivalent across implementations (MoQ.Source latency is a jitter-buffer target, moq-cli --max-latency a group-skip threshold); both default to 500 ms here, but treat cross-implementation latency comparisons as indicative.
• Resource stats: CPU is the tree's mean utilisation over its lifetime; RSS is the peak of the largest single process in the tree (rusage semantics), not a sum. Membrane numbers include BEAM startup and Mix.install checks.
• One pair at a time, one host, localhost relay: this measures client-side costs and path correctness, not network behaviour under loss/congestion.
• moq-gst has no event log (events: none), so no startup/e2e/cadence numbers for its pairs, and its publisher must be fed from the fixture's MP4 sibling: h264parse on a raw bytestream produces DTS but no reliable PTS, and moqsink drops PTS-less buffers ("buffer outside the segment"). moqsrc also doesn't exit on broadcast end (the harness SIGINT/SIGKILLs it, shown as -9 ⚠ with no resource stats).