Spec does not define whether empty / no-data frames (e.g. padding) are delivered to a transform or dropped

[Drekabi]

Summary

The spec does not say whether encoded frames with no payload data (zero-length / null data) are exposed to an RTCRtpScriptTransformer, nor what a transform is expected to do with them. In practice the RTP stack produces such frames — e.g. padding-derived frames — and it is currently unspecified whether these reach transformer.[[readable]] at all.

Spec references

• §2.1.1 readEncodedData says, for every frame, to enqueue frame into transformer.[[readable]], with no carve-out for frames that carry no data.
• §2 Specification states the UA MUST invoke [[transformFrameAlgorithm]] "whenever the encoder outputs an encoded frame" / on every received frame — again with no exclusion for empty frames.
• §4.3 RTCEncodedVideoFrame and §4.5 RTCEncodedAudioFrame define .data as an ArrayBuffer, but say nothing about whether it may be zero-length, and padding is not mentioned anywhere.

So the algorithms imply "all frames are delivered" but never define the boundary case of a frame whose data is empty (or whether padding counts as "an encoded frame" at all).

Why this matters (interop + robustness)

1. Interop divergence: because delivery of empty/padding frames is unspecified, UAs are free to (a) deliver them, (b) silently drop them, or (c) coalesce them — and a transform that works in one browser can misbehave in another.
2. Author hazard / DoS-like footgun: the receive transform is in the audio critical path. A transform that assumes a non-empty payload and stalls/throws on a 0-byte frame can halt all playback for the stream, not just that frame. Conversely, dropping these frames is observably harmless to media (they carry no decodable payload and RTP loss accounting happens upstream of the transform).
3. Implementation evidence: we hit this in Gecko — no-data, padding-derived frames reached the receiver transform and caused the RTCEncodedAudioFrame-audiolevel.html WPT to fail. Whether a no-data frame should have been delivered to the transform in the first place is exactly the unspecified behavior at issue.

Questions to resolve

1. Are empty / no-data frames (e.g. padding) intended to be enqueued into the transformer's readable, or filtered out by the UA before §2.1.1 readEncodedData?
2. If they are delivered, must data be a (valid) zero-length ArrayBuffer, and what type/metadata should they carry so authors can recognize and pass them through?
3. Is RTP padding considered "an encoded frame" for the purposes of §2 at all?

Possible resolutions

• Filter at the UA: normatively state that the UA does not enqueue frames that carry no codec payload (e.g. padding), so transforms only ever see real encoded frames.
• Deliver but specify: require data to be a zero-length ArrayBuffer (never null), define the type/metadata for these frames, and add author guidance that transforms must forward frames they don't understand unchanged.

[Pehrsons]

Note RTCEncodedAudioFrameMetadata exposes sequenceNumber.

I think the contention here is whether to allow empty buffers for audio to keep the exposed seqno series as intact as possible; or to not expose empty audio buffers and to also disallow applications from knowing why the seqno series had that gap -- was it padding or network?

As mentioned, the audio buffer could be empty for a seqno if the packet contained only padding.

[youennf]

Do we have evidence this is happening in Chrome and/or Safari?
If so, I would go with an empty array buffer.

Otherwise, I would tend to filter them, as we can always decide to expose them later on.

[jan-ivar]

Motivation for the audio sequence number expressed in #154 (comment) seems to include packet loss detection.

Might a third option be to filter and deduct from the sequence number (any reason it needs to match the physical RTP one)?

[fippo]

libWebRTC uses padding-only for probing so this can lead to empty payload for audio packets. This is generally not considered part of the RTP payload so one would end up an empty array for the payload.

Note that this is generally inserted by the pacer so after the encode-side transform.

Sadly https://jsfiddle.net/fippo/2y7nrkcd/1/ doesn't seem to work for replacing the payload with an empty one at the sender so there is no easy way to test. A custom Chromium build with this patch confirms this is passed through to the receive transform:

index 2f892c429c..f00e32247c 100644
--- a/audio/channel_receive.cc
+++ b/audio/channel_receive.cc
@@ -746,6 +746,11 @@ void ChannelReceive::ReceivePacket(const uint8_t* packet,

   size_t payload_data_length = payload_length - header.paddingLength;

+  // Force an empty payload for RTP payload type 111 (Opus).
+  if (header.payloadType == 111) {
+    payload_data_length = 0;
+  }
+

Video has a similar problem, see https://issues.chromium.org/issues/40682672

Might a third option be to filter and deduct from the sequence number (any reason it needs to match the physical RTP one)?

No. Ask your favorite NetEq expert why.

[Drekabi]

Do we have evidence this is happening in Chrome and/or Safari? If so, I would go with an empty array buffer.

Otherwise, I would tend to filter them, as we can always decide to expose them later on.

As mentioned by fippo this is easiest exposed with padding. The test I mentioned fails for Chrome as well if you modify the test signaling to force Chrome to use stereo. At that point the packet size increases kicking off the BWE probbing and padding is sent.