Skip to content

OTA origin is the remaining ceiling: burst-throttled to ~4 MB/s and resets long transfers #70

Description

@loki666

Split out of #65, which fixed the round-trip half of install/OTA speed and left this half untouched.

With -b 1M shipped (ca49712), a full install streams in 14m34s. Of that, only ~107 s is
round-trip time. ~702 s is bytes moving at the origin's rate, and no bundle-format change can
reach it. The OTA origin is now the ceiling.

The origin is not fast, and not steady

Measured 2026-08-26 from the device and from a workstation, each leg run alone (two contended
runs produced garbage and were discarded -- do not run two throughput tests over one link):

laptop + device -> Cloudflare edge, 50 MB    10.8 MB/s    the line is NOT the constraint
LAN nginx -> device                          70.3 MB/s    nor wifi, squashfs, or decompression
Oracle, first 50 MB cold                      9.6 MB/s    burst allowance
Oracle, same 50 MB after pulling 300 MB       3.6 MB/s    burst SPENT
Oracle, sustained across a 4 GB install       4.12 MB/s   what an install actually gets

Burst-then-throttle. The identical request is 2.7x apart depending only on what preceded it. A
50 MB curl therefore flatters this server by 2.7x and must never be used to size it -- a trap this
issue exists partly to record.

It also drops long connections

A single 300 MB range died with Connection reset by peer 236 s in, on an otherwise idle link.
That is the same failure shape as the five Connection timed out after 20002 ms (retrying N/5) seen
in the final seven minutes of an early #65 install run. Three observations, two contexts.

Installs survive this only because rauc's requests are short and it retries five times. It is a
reliability property we are relying on by accident, and it argues against ever chasing "fewer, larger
requests" beyond the 1 MiB squashfs maximum.

What it would buy

Same bundle, same client, only the origin changed:

origin TTFB bytes stream
Oracle today (-b 1M) ~107 s ~702 s 14m34s
something serving 10.8 MB/s ~107 s ~368 s ~8 min

Roughly another 45% off, and it compounds with what #65 already landed rather than overlapping it:
block size owns the round trips, the origin owns the bytes.

Why it is not already done

R2 mirroring is out on the free plan -- 10 GB total against ~4 GB per bundle, and the media
already occupy most of it. The media do publish to R2 with edge presence today; it is the bundle,
the single largest artifact, that comes from one box in one region.

Worth noting

ota/nginx-novadeck-ota.conf justifies HTTP/2 with "many concurrent Range requests". They are
sequential -- ~19/s with one outstanding at 128 KiB, fewer now. HTTP/2 silenced a real rauc warning
and should stay, but it was never going to help here.

Not yet established

  • Whether the throttle is an Oracle egress allowance, a shaper, or something in nginx. No
    limit_rate is set in our config, so it is not ours by configuration.
  • Whether the resets share a cause with the throttle.
  • What a second region or a CDN in front of the existing box would cost, versus a paid R2 tier.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions