Split out of #65, which fixed the round-trip half of install/OTA speed and left this half untouched.
With -b 1M shipped (ca49712), a full install streams in 14m34s. Of that, only ~107 s is
round-trip time. ~702 s is bytes moving at the origin's rate, and no bundle-format change can
reach it. The OTA origin is now the ceiling.
The origin is not fast, and not steady
Measured 2026-08-26 from the device and from a workstation, each leg run alone (two contended
runs produced garbage and were discarded -- do not run two throughput tests over one link):
laptop + device -> Cloudflare edge, 50 MB 10.8 MB/s the line is NOT the constraint
LAN nginx -> device 70.3 MB/s nor wifi, squashfs, or decompression
Oracle, first 50 MB cold 9.6 MB/s burst allowance
Oracle, same 50 MB after pulling 300 MB 3.6 MB/s burst SPENT
Oracle, sustained across a 4 GB install 4.12 MB/s what an install actually gets
Burst-then-throttle. The identical request is 2.7x apart depending only on what preceded it. A
50 MB curl therefore flatters this server by 2.7x and must never be used to size it -- a trap this
issue exists partly to record.
It also drops long connections
A single 300 MB range died with Connection reset by peer 236 s in, on an otherwise idle link.
That is the same failure shape as the five Connection timed out after 20002 ms (retrying N/5) seen
in the final seven minutes of an early #65 install run. Three observations, two contexts.
Installs survive this only because rauc's requests are short and it retries five times. It is a
reliability property we are relying on by accident, and it argues against ever chasing "fewer, larger
requests" beyond the 1 MiB squashfs maximum.
What it would buy
Same bundle, same client, only the origin changed:
| origin |
TTFB |
bytes |
stream |
Oracle today (-b 1M) |
~107 s |
~702 s |
14m34s |
| something serving 10.8 MB/s |
~107 s |
~368 s |
~8 min |
Roughly another 45% off, and it compounds with what #65 already landed rather than overlapping it:
block size owns the round trips, the origin owns the bytes.
Why it is not already done
R2 mirroring is out on the free plan -- 10 GB total against ~4 GB per bundle, and the media
already occupy most of it. The media do publish to R2 with edge presence today; it is the bundle,
the single largest artifact, that comes from one box in one region.
Worth noting
ota/nginx-novadeck-ota.conf justifies HTTP/2 with "many concurrent Range requests". They are
sequential -- ~19/s with one outstanding at 128 KiB, fewer now. HTTP/2 silenced a real rauc warning
and should stay, but it was never going to help here.
Not yet established
- Whether the throttle is an Oracle egress allowance, a shaper, or something in nginx. No
limit_rate is set in our config, so it is not ours by configuration.
- Whether the resets share a cause with the throttle.
- What a second region or a CDN in front of the existing box would cost, versus a paid R2 tier.
Split out of #65, which fixed the round-trip half of install/OTA speed and left this half untouched.
With
-b 1Mshipped (ca49712), a full install streams in 14m34s. Of that, only ~107 s isround-trip time. ~702 s is bytes moving at the origin's rate, and no bundle-format change can
reach it. The OTA origin is now the ceiling.
The origin is not fast, and not steady
Measured 2026-08-26 from the device and from a workstation, each leg run alone (two contended
runs produced garbage and were discarded -- do not run two throughput tests over one link):
Burst-then-throttle. The identical request is 2.7x apart depending only on what preceded it. A
50 MB curl therefore flatters this server by 2.7x and must never be used to size it -- a trap this
issue exists partly to record.
It also drops long connections
A single 300 MB range died with
Connection reset by peer236 s in, on an otherwise idle link.That is the same failure shape as the five
Connection timed out after 20002 ms (retrying N/5)seenin the final seven minutes of an early #65 install run. Three observations, two contexts.
Installs survive this only because rauc's requests are short and it retries five times. It is a
reliability property we are relying on by accident, and it argues against ever chasing "fewer, larger
requests" beyond the 1 MiB squashfs maximum.
What it would buy
Same bundle, same client, only the origin changed:
-b 1M)Roughly another 45% off, and it compounds with what #65 already landed rather than overlapping it:
block size owns the round trips, the origin owns the bytes.
Why it is not already done
R2 mirroring is out on the free plan -- 10 GB total against ~4 GB per bundle, and the media
already occupy most of it. The media do publish to R2 with edge presence today; it is the bundle,
the single largest artifact, that comes from one box in one region.
Worth noting
ota/nginx-novadeck-ota.confjustifies HTTP/2 with "many concurrent Range requests". They aresequential -- ~19/s with one outstanding at 128 KiB, fewer now. HTTP/2 silenced a real rauc warning
and should stay, but it was never going to help here.
Not yet established
limit_rateis set in our config, so it is not ours by configuration.