Plain-URL live engines cap at two concurrent per origin: the static persistentSession's 2-connections-per-host pool starves the third pump

[cmcpherson274]

Running more than two plain-URL live engines against one origin, the third engine's pump never receives a byte. Measured on the tvOS simulator and an Apple TV 4K (3rd gen, tvOS 26.6), engine 6.55.0, anchors re-verified unchanged at 6.56.3.

The shape

Host context: we now run the engine on N small live surfaces (a picture-in-guide preview tile, and an experimental multiview grid of up to four tiles), each its own engine instance per the "one engine per playback surface" doc. A forced 4-tile grid against one Jellyfin origin settles the same way on sim and device:

• Tiles 1–2 tune normally (started (route loopback, 4.5s/4.6s) on the sim run).
• Tiles 3 and 4: startup 0/8 dispatched → pump conn start gen=1 offset=0 open-ended → zero bytes → startup 1/8 sourceOpened only at +15.0 s (the reader's open deadline arc, both tiles, both platforms) → start timeout.

The origin is not the wall: three concurrent raw curl pulls of the same /Videos/<id>/stream.ts?Static=true endpoints (three channels, same client IP, no engine involved) all flowed at full rate for the 12 s probe window.

The anchor

AVIOReader.persistentSession is a private static let — one URLSession for the whole process — and makeSessionConfig sets httpMaximumConnectionsPerHost = 2 (Demuxer/AVIOReader.swift:54 and :3411 at 6.56.3). Two open-ended live stream.ts pulls occupy both connections to the origin for their whole lifetime, so a third pump's task queues in URLSession indefinitely. OriginRequestBudget is not involved: it is count-only with no host-declared limit and no 429/503/509 was ever answered.

Custom-source loads are unaffected (the host feeds those), which is why a full-screen tune on our host-owned spool coexists with one plain-URL tile — the ceiling only shows up at the second plain-URL live engine plus one.

The ask

Whichever fits the design: a per-instance session for the open-ended live pump, a LoadOptions knob for connections-per-host, or — if two per origin is the intended contract for N instances — a line of documentation saying so, so a host can plan surfaces around it. No urgency on our side: we measured it behind a disabled-by-default experiment and are shipping the multiview surface on AVFoundation regardless; the preview tile (one engine) is inside the ceiling.

[superuser404notfound]

Anchor confirmed, and it is the whole cause. I reproduced the transport half in isolation before touching the reader: three open-ended bytes=0- requests on a session configured exactly like makeSessionConfig, against a local origin that logs every request it receives. The origin sees two. The third is not refused and not slowed, it never arrives.

Two things that were not visible from outside, and both make the shape worse than "two per surface".

The pool is a static let, so those two connections are the process's allowance, not a surface's. Every reader draws on the same pair: the pump, the subtitle side reader (subtitleSideDemuxer inherits avioPrefetch: true, so it is on the persistent path too), the subtitle forward prefetcher, and one more pump per playback surface. The ceiling is not four tiles, it is however many long-lived reads the process happens to want, and on a source with a side reader running, one surface can already hold both.

The 15.0 s is engine-side and it is exact. awaitFirstPersistentData (Demuxer/AVIOReader.swift:1018 at 6.56.3) waits a flat 15 s for the first byte, then open() returns having received nothing, which is the startup 1/8 sourceOpened in your trace. The reason there is no line in between is structural: the only witness for a connection that delivers nothing is armed at connStallTimeout, 20 s, which is longer than the arc that has already given up. The case could not print anything about itself.

What is shipping. I took none of the three options literally.

A per-instance session gives up connection reuse across a reader's own ranges. The persistent path ends a connection every 32 MB, and a session per connection makes each of those a fresh TLS handshake, which is why the pool is shared at all (AE#220, AE#243). It also adds a lifecycle to something that currently never needs invalidating.

A LoadOptions knob for connections-per-host cannot work as an option: the session is process-wide and built once, so the first load to touch it would fix the value for every load after it.

The knob is LoadOptions.maxConcurrentSourceRequests, and it already exists (AE#377). It counts requests instead of connections, which is what an origin meters and the only thing that means anything over h2, it waits with a budget, it reports that it waited, and it lowers itself when the origin answers 429/503/509. The defect is that a second ceiling sat underneath it in silence: nil there is documented as "counts but does not cap", and that was untrue from the third concurrent open-ended read on. So the long-lived pool stops capping connections, and the request budget is the only ceiling. The bounded pool keeps its cap of two, because every request on it ends, so a request that waits there waits for one that is finishing.

Second half, independent of the first: a generation that has been on the link with nothing delivered is now reported at a quarter of the stall threshold, capped at 5 s, so it lands inside the arcs that give up on a source rather than after them. The line carries the two facts that separate the causes:

[AVIOReader] pump gen=1 no first byte after 5.0s at offset 0; 3 request(s) open to this origin, transport pool allows 64 connections per host

A request parked in the transport and an origin sitting on a request are identical from every vantage point downstream, and they take different fixes. It reports, it never acts, so nothing about when a connection ends has moved.

d89527b8 on fix/ae450-transport-pool-is-not-the-ceiling. The tests encode the case rather than the constant: a negative control that a cap of two parks the third open-ended request, the same three against the shipped long-lived config, and three live readers against one throttled origin. With the cap put back to two, the reader case fails after 15.030 s, which is the arc in your trace.

One thing worth checking on your side, because it would mean this diagnosis is wrong for your origin. httpMaximumConnectionsPerHost bounds nothing over HTTP/2, since URLSession multiplexes a session's requests onto a single connection. The engine records the negotiated protocol once per origin:

grep 'origin transport' <log>

If your Jellyfin answered h2 and the tiles still starved, then the cap was not the wall and I would want to see that trace. On http/1.1 it binds exactly, which is what the isolated reproduction measured.

Thanks for the shape of this report. The three-curl control is what made the origin falsifiable as the cause, and measuring it behind a disabled-by-default experiment on both the simulator and a device is why there was nothing left to guess at.

[cmcpherson274]

Checked, and the diagnosis holds for this origin: the 4-tile device leg that starved tiles 3 and 4 recorded

origin transport: http/1.1 at http://10.10.2.57:8096 (one request per connection, so a per-session connection cap bounds requests)

on the same device a few minutes before the 4-tile leg — correction: in the immediately-prior app process, not the tile leg's own (a fresh launch banner at 23:53:10 separates them, and the tile process itself happened not to log an origin-transport line). Every process that DID record the line against this origin — 6.55.0 and 6.56.3, sim and device — shows the same http/1.1, and the protocol is negotiated per origin, so the conclusion carries to the tile process. So the cap binds exactly here, as your isolated reproduction measured.

The side-reader detail is good to have on file for our side too: our full-screen live arm runs a custom-source spool (pool-exempt), but VOD on this rung is plain-URL — an external-subtitle side reader plus the pump would already hold the pair, which explains a class of "slow second surface" we had filed mentally under server load. We'll re-measure the grid against fix/ae450-transport-pool-is-not-the-ceiling when it rides a tag — the 5 s no-first-byte report with the request-vs-connection split is exactly what our silent arcs (15.006 s and 15.001 s measured; your harness's 15.030 s is the same shape) were missing.

[Edited same morning, review round: the transport line's process attribution corrected (prior app process on the same device, not the tile leg's own — the tile process logged no transport line; conclusion unchanged since the protocol is per-origin), and the 15.030 s figure returned to your harness (ours measured 15.006/15.001 s).]

[superuser404notfound]

It rides a tag now: 6.56.4 (31f3a5a0), with the fix at d89527b8 and the merge at 0d45011d.

Thanks for the transport-line check, and for correcting its process attribution rather than letting it stand. The conclusion carries either way, since the protocol is negotiated per origin and every process that recorded a line against that origin recorded the same http/1.1, but a reader of this thread now knows which process the quoted line came from, which is the difference between a measurement and a recollection.

Two things to expect on the grid re-measure. The four tiles should now be bounded by OriginRequestBudget alone, which counts requests rather than connections and says when it waits, so a tile that is genuinely waiting for the origin looks different from one that is not there. And on the silent arcs you measured at 15.006 s and 15.001 s, the 5 s report should now land inside them:

[AVIOReader] pump gen=1 no first byte after 5.0s at offset 0; N request(s) open to this
             origin, transport pool allows 64 connections per host

If a tile still starves with that line reporting a small request count, the cause is on the origin's side of the link rather than in the pool, and that is the trace worth having.

Your side-reader note is the more useful half of the last comment for us. An external-subtitle side reader plus the pump holding the pair on a plain-URL VOD rung is exactly the shape that would read as server load from outside, and it was a process-wide ceiling the whole time.