Skip to content

feat(VideoDecoderIssueDetector): measure local decode demand, not fps volatility - #49

Open
jenspalmqvist wants to merge 4 commits into
VLprojects:masterfrom
jenspalmqvist:execplan-decoder-cpu-throttling-signal
Open

feat(VideoDecoderIssueDetector): measure local decode demand, not fps volatility#49
jenspalmqvist wants to merge 4 commits into
VLprojects:masterfrom
jenspalmqvist:execplan-decoder-cpu-throttling-signal

Conversation

@jenspalmqvist

Copy link
Copy Markdown

Closes #48

Purpose

VideoDecoderIssueDetector emits decoder-cpu-throttling when the device that runs the library cannot decode the incoming video fast enough. Today it measures only how much the frame rate of each inbound stream wobbles over the last five samples. A remote participant whose encoder gives up frame rate because of a bad uplink or a slow CPU produces exactly that wobble on every receiver, so every receiver reports decoder-cpu-throttling about its own device. With three publishers, one wobbling sender is 33 percent of the inbound streams, above the default 30 percent threshold.

After this change the issue fires only when, per peer connection, frames that arrived on this device were not decoded, enough inbound streams show that shortfall, the incoming video together demands a large share of decoder time, and at least one affected stream is itself costly to decode. Streams with packet loss, high RTP jitter, hardware decoding, missing counters, no decoded frames, or a counter reset in the window are excluded. A sender that sends fewer frames no longer triggers the issue.

What changed

  • src/detectors/VideoDecoderIssueDetector.ts: volatility signal replaced by decode demand (totalDecodeTime per decoded frame times received frames per second), frame shortfall (framesReceived - framesDecoded), an affected-streams gate, and a per-stream corroboration. New params decodeDemandThreshold, affectedStreamDemandThreshold, frameShortfallPctThreshold, minFramesReceived, minWindowMs, maxPacketLossPct, maxJitterMs. affectedStreamsPercentThreshold keeps working. volatilityThreshold is accepted and ignored (deprecated).
  • Payload: adds decodeDemand, frameShortfallPct, windowMs, evaluatedStreams, throttledStreams. Keeps affectedStreamsPercent and the misspelled throtthedStreams (deprecated, same array as throttledStreams, still only the affected streams), with allFps and volatility kept per entry as deprecated informational fields.
  • src/types.ts: optional powerEfficientDecoder on ParsedInboundVideoStreamStats.
  • test/detectors/VideoDecoderIssueDetector.spec.ts: first spec for this detector, 22 tests with a cumulative-counter fixture builder. The test does not report a sender whose frame rate wobbles fails on the current code and passes here.
  • README.md: section rewritten; the old example showed a field the code never emitted.
  • docs/plans/EXECPLAN_DECODER_CPU_THROTTLING_SIGNAL.md: the design record, six review rounds, decision log, and test evidence.

Known limitation

Browsers drop a frame before decode only when a later frame is decodable without it, which needs temporal layers (simulcast or SVC). On a single-layer stream (peer-to-peer VP8 without simulcast, H.264) an overloaded receiver decodes every frame late and shows no shortfall, so this detector stays silent there. The README says so.

Compatibility notes (minor release)

Earliest emission moves from the sixth to the fifth poll. affectedStreamsPercent is rounded to three decimals. allFps holds one value per stored sample (4 by default) instead of exactly 5. A very high volatilityThreshold no longer silences the detector. The detector needs id, timestamp, framesReceived, framesDecoded, totalDecodeTime, packetsReceived, packetsLost, and jitter in inbound video stats and is silent without them.

Verification

  • npm test on Node 24: 58 passing, 0 failing (36 existing plus 22 new), run on this exact commit.
  • npm run lint and npm run lint:tests: clean on this exact commit.
  • Milestone checkpoint: with the previous detector and only the wobble test present, the suite reported that test failing at the sixth call with fps series [30, 30, 15, 30, 15], volatility 30, 33.3 percent affected.
  • Not performed: a browser run. The manual check in the plan (SFU call with simulcast, uplink throttle on one participant, CPU burner on another, unattached track) is still open and should be done in a staging call before relying on the release.
  • Reviews: three Codex adversarial reviews and two independent Claude reviews of the plan, one Codex adversarial review of this diff. All accepted findings are recorded in the plan.

🤖 Generated with Claude Code

https://claude.ai/code/session_01V1KrAS941bzfk9EuBPG33k

Jens Palmqvist and others added 4 commits September 4, 2026 12:38
… volatility

decoder-cpu-throttling fired on every receiver of a sender that lowered its
frame rate, because the detector only looked at inbound fps volatility. It
now requires, per peer connection, a local frame shortfall (received frames
not decoded) on more than affectedStreamsPercentThreshold of the inbound
streams over a window of at least minWindowMs, a summed decode demand
(totalDecodeTime per decoded frame times received frames per second) above
decodeDemandThreshold, and at least one affected stream whose own demand is
above affectedStreamDemandThreshold. Streams with packet loss above
maxPacketLossPct, with mean RTP jitter above maxJitterMs, decoded in
hardware (powerEfficientDecoder), without the required counters, with no
decoded frames, or with a counter reset inside the window are excluded.
framesDropped is no longer read.

Browsers drop a frame before decode only when a later frame is decodable
without it, so the detector covers streams with temporal layers (simulcast,
SVC) and is silent on single-layer streams, which decode every frame late
instead of dropping.

statsSample adds decodeDemand, frameShortfallPct, windowMs, evaluatedStreams
and throttledStreams. affectedStreamsPercent keeps its meaning (share of
inbound streams that show the problem) and is now rounded to three
decimals. throtthedStreams stays, deprecated, and still lists only the
affected streams; its entries keep allFps and volatility, deprecated, next
to the new numbers; allFps holds one value per stored sample (4 by default)
instead of exactly 5. The earliest emission moves from the sixth to the
fifth poll. affectedStreamsPercentThreshold keeps working.
volatilityThreshold is accepted and ignored; a high value no longer
silences the detector. New params: decodeDemandThreshold,
affectedStreamDemandThreshold, frameShortfallPctThreshold,
minFramesReceived, minWindowMs, maxPacketLossPct, maxJitterMs. The detector
now needs id, timestamp, framesReceived, framesDecoded, totalDecodeTime,
packetsReceived, packetsLost and jitter in the inbound video stats and is
silent without them. ParsedInboundVideoStreamStats gains an optional
powerEfficientDecoder field.

The ExecPlan in docs/plans/EXECPLAN_DECODER_CPU_THROTTLING_SIGNAL.md records
the design, six review rounds, and the test evidence.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1KrAS941bzfk9EuBPG33k
… stack

Seven runs of this branch's build inside a consumer web app's development container,
against its signalling backend and a host mediasoup SFU with simulcast. The sender-cap
run reproduces the old false positive's input (a 3 to 9 fps wobble on every
receiver) and no receiver reports. A stats shim on one receiver reports alone,
on Chrome and on Firefox. Leave and rejoin resets the history. Machine-wide CPU
overload fires the other detectors and not this one.

Two follow-ups recorded: mediasoup's probe stream (ssrc 1234) is counted in the
affectedStreamsPercent denominator, and Firefox reports no powerEfficientDecoder,
so the hardware-decode skip never applies there. No code changed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1KrAS941bzfk9EuBPG33k
…irefox

Firefox exposes every inbound-rtp field VideoDecoderIssueDetector reads but
not powerEfficientDecoder, so a Firefox receiver that decodes in hardware is
never excluded and can reach the demand thresholds on pipeline latency alone.
Observed with Firefox 146 in a local call on 2026-09-04.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1KrAS941bzfk9EuBPG33k
@jenspalmqvist
jenspalmqvist force-pushed the execplan-decoder-cpu-throttling-signal branch from af6c962 to 90c00a0 Compare September 5, 2026 09:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

VideoDecoderIssueDetector reports decoder-cpu-throttling on every receiver of one struggling publisher

1 participant