Twitch Dual Format: detect & forward Enhanced Broadcasting dual canvas (vertical)

[Soulhackzlol]

Summary

Twitch Dual Format (vertical video) lets a broadcaster send a horizontal (16:9) and a vertical (9:16) version of the same stream simultaneously; Twitch serves each viewer the right one for their device. In OBS this rides the Enhanced Broadcasting (Multitrack Streaming) path as an additional canvas; it is not a second RTMP output.

InstantClone already does Enhanced Broadcasting passthrough, but it is built around a single canvas and would currently drop the vertical one. This issue scopes adding detect-and-forward support: when we detect a second (vertical) canvas in the Enhanced Broadcasting negotiation/stream, pass it through bit-faithfully to Twitch's IVS endpoint.

Status (June 2026): Twitch Dual Format is generally available to all streamers, no longer Alpha-gated. Twitch also added server-side transcoders: for eligible partners/affiliates the GPU only encodes the top horizontal + top vertical rungs, and Twitch's server transcoders fill the remaining lower resolutions. Eligibility (incl. 2K dual format) is shown in Creator Dashboard → Stream settings → advanced stream settings. This removes the prior "can't E2E-test without Alpha access" blocker; captures can now be taken from a normal account.

How OBS/Twitch deliver it (mechanism)

1. OBS 31.1+ has a native multi-canvas system. Aitum Vertical (v1.6.0+) registers as OBS's "Additional Canvas."
2. The user enables Settings → Stream → Enhanced Broadcasting (Multitrack Streaming) and selects Aitum Vertical as the additional canvas.
3. At go-live, GetClientConfiguration returns encoder configs for both canvases (a horizontal ladder + a vertical ladder). Everything is multiplexed over the single Enhanced Broadcasting RTMP connection to the session-allocated IVS endpoint.
4. Tracks are distinguished by TrackId; canvases are grouped by canvas_index (horizontal = 0, vertical = 1).

Requirements (for the streamer, not us): OBS 31.1.2+ (32+ recommended), Enhanced Broadcasting on, strong GPU (NVIDIA RTX 3070+ for 1080p), ~13.5 Mbps upstream. Now GA for all streamers. Server-side transcoders (eligible partners/affiliates) reduce GPU load by having Twitch fill the lower rungs, so the multitrack stream may carry only the top rung per canvas rather than a full local ladder. We must not assume a fixed 3-rung ladder per canvas.

Why it doesn't survive the proxy today

1. GetClientConfiguration advertises a single canvas. Our synthetic config hard-codes canvas_index:0 on every encoder_configurations entry and never emits a canvas_index:1 group, so a second canvas is never offered/relayed. See src/web.rs:1282 (and the 720p/480p siblings at :1289, :1297).
2. Multi-track flatten assumes one canvas. flatten_multitrack_video() takes the first track only (primary), and the per-destination policy pass_through_multitrack keeps all tracks for Twitch but drops the rest elsewhere. None of this is canvas-aware: vertical tracks would be treated as ladder noise. See src/h264.rs:181-301.
3. Primary-track IDR gate drops TrackId != 0 for non-Twitch destinations (README "What's solid", EB section, README.md:294). Vertical canvas tracks would be discarded by the same logic if not explicitly recognised.

Proposed approach (detect, then forward)

• Detect the additional canvas in the EB negotiation. Two config paths exist and both need canvas awareness:
  • the proxied Twitch GetClientConfiguration (real Dual Format): preserve canvas_index:1 encoder configs instead of assuming a single canvas;
  • the synthetic config in src/web.rs: emit canvas_index:1 entries when a vertical canvas is present.
• Forward canvas_index:1 track groups bit-faithfully to the IVS endpoint alongside canvas_index:0 (the Twitch pass_through_multitrack = true path should carry them once they're no longer dropped).
• Delay/cut correctness (the hard part): the arm/activate/cut machine must keep IDR alignment across both canvases so a cut doesn't glitch one. The existing primary-track IDR gate is single-canvas; cuts need a per-canvas anchor (or a documented "primary canvas drives cut timing" rule).
• Non-Twitch destinations: decide policy. Most likely forward canvas_index:0 only (flatten as today), since arbitrary RTMP endpoints don't understand a second canvas.

How can this be tested

Unit / integration (no Twitch account needed, fits the existing cargo test --release suite):

• Canvas-aware multi-track parsing. Mirror the existing EB multi-track tests in src/h264.rs: add fixtures carrying both a canvas_index:0 and a canvas_index:1 track group, and assert (a) Twitch passthrough keeps both canvases, (b) non-Twitch flatten keeps only canvas_index:0, (c) the vertical TrackId != 0 tags are no longer silently dropped for Twitch.
• Variable rung count. Feed a multitrack stream with only the top rung per canvas (server-transcoder-eligible shape) and assert bitrate_for_track / the tracks logic in src/web.rs doesn't assume 3 rungs or panic on fewer.
• Synthetic GetClientConfiguration. Assert the generated config emits a canvas_index:1 group when a vertical canvas is present, and that JSON shape round-trips.
• Delay/cut across canvases. Drive the arm/activate/cut state machine over a two-canvas fixture and assert each canvas cuts on its own IDR (no cross-canvas anchor leakage), reusing the timestamp-rewrite assertions already covering single-canvas cuts.

Capture-based (now possible since GA, any account):

• Point OBS (Aitum Vertical + Enhanced Broadcasting) at the proxy, go live, and capture the GetClientConfiguration request/response plus the multiplexed RTMP tags. Inspect canvas_index and TrackId on the wire to confirm our parser matches reality. Capture once on a server-transcoder-eligible account and once on an ineligible one to compare rung counts. Save both as test fixtures.

End-to-end (real Twitch, manual):

• Stream Dual Format through the proxy to a real Twitch channel. Verify the channel shows the horizontal feed on desktop and the vertical feed via the mobile/vertical toggle, both live simultaneously.
• While live, exercise Cut delay and confirm neither canvas glitches (no decoder error, no frozen frame on either format) and both resume monotonically.
• Add a non-Twitch destination alongside Twitch and confirm it receives a clean single (horizontal) canvas while Twitch gets both.

Open questions

• Wire format: how is canvas_index carried at the tag level vs. only in the GetClientConfiguration JSON? Confirm whether the data-plane multi-track header exposes canvas, or whether canvas/TrackId mapping must be derived from the config response. (Capturable now from any account; GA.)
• With server-side transcoders, how many rungs per canvas actually hit the wire? Confirm whether eligible accounts send only the top horizontal + top vertical, and how encoder_configurations advertises that the server fills the rest. Our tracks/ladder assumptions (bitrate_for_track, fixed 1080/720/480) must not break when fewer rungs arrive.
• Does the delay buffer/ring need per-canvas IDR indexing, or can a single index tag both canvases?
• How to gracefully degrade when only some destinations support dual canvas.

Docs needed

• A captured GetClientConfiguration response from a real Dual Format session (encoder_configurations with canvas_index:1), now capturable from any account since Dual Format is GA. Capture both a server-transcoder-eligible account (fewer rungs) and an ineligible one (full local ladder) to see how rung count differs.
• A capture of the multiplexed multi-track/multi-canvas RTMP tags (TrackId to canvas mapping on the wire).
• Aitum Vertical 1.6.0 release notes + OBS 31.1 multi-canvas/Enhanced Broadcasting behaviour.
• README "Status" update once supported, plus a new test mirroring the existing EB multi-track coverage but for a second canvas.

References

• Twitch: Dual Format vertical video
• Twitch: Enhanced Broadcasting
• Aitum Vertical 1.6.0 (additional canvas for Twitch Dual Format)
• Aitum 1.6.0 announcement
• StreamScheme setup guide (2026)

[Soulhackzlol]

Closing as already supported. Investigation + production testing showed Dual Format needs no functional changes: canvas_index is control-plane only, and the vertical canvas is just extra TrackIds that the existing pass_through_multitrack path already forwards bit-faithfully to Twitch (and drops for non-Twitch, which is correct).

Verified live against real Twitch — landscape ×4 + portrait ×3, HEVC vertical primary, delay cuts stable, dashboard "Excellent" — and confirmed it composes with VOD audio + EB.

The only deliverable was locking the behavior in with regression tests (forwarding + cut-anchor invariant), done in #28. The canvas-aware cut-hardening idea is not tracked further: it guards against a canvas IDR misalignment we have no evidence ever happens (OBS aligns both canvases' keyframes), so per simplicity-first it stays unbuilt until something actually breaks.