Real-time communications and streaming technologies are converging within ultra low latency use cases such as augmented reality/virtual reality, game streaming or live streaming with fanout. These use cases may involve peer-to-peer interactions traversing Network Address Translators (NATs) so that they cannot be addressed solely with client/server APIs. For example, streaming a game from a console to a mobile device or a live streaming application supporting peer-to-peer fanout to improve scalability.
For these applications, today's WebRTC APIs may not be sufficient, due to:
-
Lack of support for custom metadata. In AR/VR applications, the metadata can be large enough to require custom packetization and rate control.
-
Lack of codec support. For music, the AAC codec is popular, but it is not supported in WebRTC implementations. However, AAC is supported in WebCodecs.
-
Lack of custom rate control. While WebRTC's built-in rate control is general purpose, it does not allow for rapid response to changes in bandwidth, as is possible with per-frame QP rate control in WebCodecs.
-
Inability to support custom RTCP messages. WebRTC implementations today do not support feedback messages such as LRR, RPSI or SLI, or extended statistics as provided by RTCP-XR.
Native applications can use raw UDP sockets, but those are not available on the web because they lack encryption, congestion control, and a mechanism for consent to send (to prevent DDoS attacks).
To enable new use cases, this document proposes an API to send and receive packets with encryption and customizable congestion control.
The RTCTransport API enables web applications to support:
- Custom payloads (such as ML-based audio codecs)
- Custom packetization
- Custom FEC
- Custom RTX
- Custom jitter buffers
- Custom bandwidth estimation
- Custom rate control
- Custom bitrate allocation
- Packet forwarding
This is not UDP Socket API. We must have consent, encryption, and congestion control.
RTCTransport enables these use cases by enabling applications to:
- Encode with a custom (WASM) codec or WebCodecs, and then packetize and send
- Receive packets, sends custom NACKs, receive those, and send custom retransmissions (RTX)
- Receive packets, put them in a custom jitter buffer, and then decode them using a custom codec (WASM) or WebCodecs
- Receive packets, send custom feedback, receive custom feedback, be notified of built-in feedback, use that information to calculate a bandwidth estimate, and use that estimate to set bitrates of encoders
- Forward packets from one RTCTransport to another, with full control over the entire packet (modulo encryption/CC exceptions)
- Improve connectivity for users by using knowledge of the exact usage scenario to better evaluate trade-offs and manage the network connection actively.
This enables the following WebRTC Extended Use Cases:
- Section 2.3: Video Conferencing with a Central Server
- Section 3.2.1: Game streaming
- Section 3.2.2: Low latency Broadcast with Fanout
- Section 3.5: Virtual Reality Gaming
Enable applications to:
- Establish encrypted peer-to-peer connections
- Send and receive packets over those connections
- Efficient control of when packets are sent and injection of additonal padding packets, in order to do custom pacing and probing.
- Have as much information about packets as possible (times, sizes, ECN bits, etc) to do custom congestion control.
- Do batch processing to run much less often than per-packet, to reduce overheads in high bandwidth situations, where packets are sent and received thousands of times per second.
Complexities of sending and receiving packets other than these requirements are still handled by the User Agent, such as encryption.
// TODO// TODO// TODO// TODO// TODO// TODO// TODO