From 63da3bb56aa6d264561010b1bf73da807c6aa7d2 Mon Sep 17 00:00:00 2001 From: Adrian Hilber Date: Fri, 6 Nov 2026 20:56:20 +0100 Subject: [PATCH 1/3] Keep the interaction preview current on a slow path, also behind a proxy. A reverse proxy between the function and the browser reads the previews at once and holds them in its own buffers, so TCP_NOTSENT_LOWAT on our socket does not help. Lab (50 KB previews, 25 fps asked, 40 ms round trip, a buffering userspace proxy with a shaped link to the client): 1.3.4 delivered 10-11 s old previews at 2 Mbit/s; at 500 kbit/s the delay grew to 14 s and the session was closed as stuck (outbox_age) after 20 s, ending control. - Preview flow control: a client that answers each preview with preview_ack has at most 2 previews unanswered; the server skips it until an answer comes (counted as dropped) and then sends the newest picture. No answer for 5 s closes it as preview_not_taken. Clients that never ack are unchanged. The web UI and the widgets ack every frame. Acks are not counted against the input limit. - TCP_NOTSENT_LOWAT 32 KiB instead of 64 KiB: 16/32/64/128 KiB at 2 Mbit/s 40 ms gave 0.65/0.64/0.80/1.07 s median age; no fps loss at 6 and 16 Mbit/s. - websocket_stuck_close logs the interaction session and oldest_ms. - BROWSER_PREVIEW_QUALITY (75, 10-100, restart) for the preview JPEG. Median preview age after the change: behind the proxy 0.22 s at 2 Mbit/s and 1.1 s at 500 kbit/s (no close, control kept for 60 s); direct at 2 Mbit/s 0.36 s (1.3.4: 0.79 s). Cost: 17 instead of 22 fps on a 16 Mbit/s path with 40 ms round trip; none on a LAN. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_017ybuzgBaGxuB5UiaE8dRa4 --- CHANGELOG.md | 10 ++++++ README.md | 3 ++ SPEC.md | 12 ++++--- docs/api.md | 4 +-- docs/interact.schema.json | 12 +++++++ src/app/api.cpp | 3 +- src/app/application.cpp | 34 ++++++++++++++++--- src/app/application.hpp | 2 +- src/config/config.cpp | 2 ++ src/config/config.hpp | 1 + src/ops/httpserver.cpp | 16 ++++++++- src/ops/httpserver.hpp | 13 +++++-- src/ops/interact.cpp | 42 +++++++++++++++++++++-- src/ops/interact.hpp | 11 ++++++ tests/integration/interact_test.py | 33 +++++++++++++++--- tests/unit/test_config.cpp | 8 +++++ tests/unit/test_httpserver.cpp | 19 ++++++++++- tests/unit/test_interact.cpp | 54 ++++++++++++++++++++++++++++++ tests/unit/web/interact.test.mjs | 19 +++++++++++ web/src/interact.js | 3 ++ 20 files changed, 277 insertions(+), 24 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d9669cb..7965641 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,15 @@ # Changelog +## Unreleased + +- **Interaction preview stays current on a slow path, also behind a proxy** (SPEC §7.2, §7.3). 1.3.4 limits what the kernel holds unsent, but a reverse proxy between the function and the browser reads everything at once and holds the previews in its own buffers: in the lab (2 Mbit/s behind a buffering proxy, 50 KB previews) they arrived 10–11 s old; at 500 kbit/s the delay grew to 14 s and the session was closed as stuck after 20 s, which ended its control. + - **Preview flow control:** a client that answers each preview frame with `{"type":"preview_ack"}` has at most 2 previews unanswered; the server skips it until an answer comes (counted in `preview_frames_dropped_total`) and then sends the newest picture. A session that answers nothing for 5 s is closed as `preview_not_taken`. Clients that never send `preview_ack` work as before. The web UI and the widgets answer every frame. `preview_ack` is not counted against the input limit. + - **Kernel limit 32 KiB** instead of 64 KiB (`TCP_NOTSENT_LOWAT`): measured with 16, 32, 64 and 128 KiB at 2 Mbit/s and 40 ms; 16 and 32 KiB were equal and best, with no frame-rate loss at 6 and 16 Mbit/s. + - Lab, preview age at the client (median), 25 fps asked, 50 KB previews, 40 ms round trip: direct at 2 Mbit/s 2.8 s (1.3.3), 0.79 s (1.3.4), 0.36 s now; behind the proxy at 2 Mbit/s 10.1 s (1.3.4), 0.22 s now; behind the proxy at 500 kbit/s 14 s and closed (1.3.4), 1.1 s and control kept for 60 s now. Cost: on a 16 Mbit/s path with 40 ms round trip the preview runs at 17 instead of 22 fps (and is 54 instead of 80 ms old); on a LAN nothing changes. + - **Stuck-close log line:** `websocket_stuck_close` now also has the interaction `session` (0 on `/api/v1/events`) and `oldest_ms`, the age of the oldest waiting message. The counter `websocket_stuck_closes_total{reason}` is unchanged. + - Tests: unit tests for the window (per session, acks before the next send, a frame that was not sent, acks past the input limit) and for a held-back preview counting towards the stuck close; `tests/unit/web/interact.test.mjs` (the UI answers every frame; `preview_ack` in the schema); `tests/integration/interact_test.py` (2 previews unanswered, 1 more per answer, closed and counted 5 s after the last). +- **`BROWSER_PREVIEW_QUALITY`** (default 75, 10–100, restart): the JPEG quality of the previews (`preview.jpg` and the interaction preview). Lower makes each picture smaller, so a slow link carries more of them. + ## 1.3.4 (2026-11-05) - **Interaction preview shows every new picture** (SPEC §7.3). The WebSocket preview waited only 20 ms for a new preview and otherwise sent the previous JPEG again: on a busy page (YouTube on test-all-gfx) about 2 of 9 frames per second were new, so the preview stood still for seconds. It now waits for the preview like `preview.jpg` (up to 100 ms), and a session never gets the same picture twice. diff --git a/README.md b/README.md index 706c234..a29ba72 100644 --- a/README.md +++ b/README.md @@ -47,6 +47,7 @@ The most used settings (all of them, with defaults and "restart required" marker | `BROWSER_FRAME_CLOCK_PHASE_MS` | `0` | internal frame clock: its BeginFrames this many ms after each grain start | | `BROWSER_REALTIME` | `true` | tick and converter threads `SCHED_FIFO` when the container allows it: give it `--ulimit rtprio=10` (Compose `ulimits: {rtprio: 10}`); `--cap-add SYS_NICE` alone does not help a non-root process (SPEC §2.2) | | `BROWSER_CONVERT_THREADS` | `4` | threads that convert one frame (1–16) | +| `BROWSER_PREVIEW_QUALITY` | `75` | JPEG quality of the previews (10–100); lower makes each picture smaller, so a slow link carries more of them | | `BROWSER_AUDIO_BUFFER_MAX_MS` | `300` | adaptive audio buffer: when Chromium delivers audio late, the 60 ms target (`BROWSER_AUDIO_BUFFER_MS`) rises by the silence played, up to this, and the video delay follows; it falls back at 1 ms/s after 10 s without an underrun. At or below `BROWSER_AUDIO_BUFFER_MS` the target is fixed (SPEC §6) | | `BROWSER_OGRAF_DIR` | `/ograf` | the OGraf graphics library (on the `/config` volume) | | `BROWSER_OGRAF_UPLOAD_MAX_MB` | `256` | largest graphics ZIP (unpacked at most 4×) | @@ -58,6 +59,8 @@ The page to show, its CSS and JavaScript, zoom, background and presets are the s The preview on the Source page is the interaction window: back, forward and reload, and with "Interact" on, mouse (drag to select text, double and triple click), keys, text and IME go to the page and the preview runs at 25 fps instead of `BROWSER_PREVIEW_FPS`. +On a slow link the preview stays current instead of falling behind: the server always sends the newest picture and skips the ones the link cannot carry. Two things keep old pictures from piling up on the way. The kernel holds at most 32 KiB of a WebSocket unsent (`TCP_NOTSENT_LOWAT`), and the UI answers each picture (`preview_ack`), so at most 2 are on the way at once, also through a proxy. A client that stops reading or answering for 5 s is closed, which ends its control; the log line `websocket_stuck_close` names the client, the session and the reason, and `websocket_stuck_closes_total{reason}` counts it (`outbox_size`, `outbox_age`, `preview_not_taken`). + Operator screens can frame the interaction window as a widget: `GET /widgets` lists it (`interaction`, no parameters, `min_size` 640×400) and `GET /widget/interaction[?theme=dark|light|transparent]` is the page. It talks only to this instance's own origin (REST and both WebSockets on `WEB_PORT`), asks for `BROWSER_API_TOKEN` when one is set, and posts `widget-ready` and `widget-size` to its parent. Only the pages that `WIDGET_FRAME_ANCESTORS` names may frame it; on the platform set it to `'self' https://designer.`. **Sign-in pages (Zscaler, Microsoft Entra ID).** Windowless CEF cannot show Chromium's WebAuthn dialog (passkeys, security keys, the QR code for a phone), so a page that asks for a passkey would hang. WebAuthn is therefore hidden from pages (`BROWSER_WEBAUTHN=false`), and Entra offers Authenticator push, a code or a password instead. Sign in once in the interaction window; with `BROWSER_PROFILE=persistent` the session survives restarts until the identity provider ends it. A tenant that allows passkeys only cannot sign in here: ask for another method, or for the graphics hosts to bypass the proxy sign-in. diff --git a/SPEC.md b/SPEC.md index d318244..bd04a36 100644 --- a/SPEC.md +++ b/SPEC.md @@ -362,7 +362,9 @@ Client → server (JSON Schema, draft 2020-12; the same schema as a file: `docs/ {"type": "object", "required": ["type"], "properties": { "type": {"const": "preview"}, "fps": {"type": "integer", "minimum": 1, "maximum": 25, "description": "omitted: back to BROWSER_PREVIEW_FPS"}, - "width": {"type": "integer", "minimum": 160, "maximum": 1920}}} + "width": {"type": "integer", "minimum": 160, "maximum": 1920}}}, + {"type": "object", "required": ["type"], "description": "one per preview frame received, never more; after the first, the server keeps at most 2 previews unacknowledged", "properties": { + "type": {"const": "preview_ack"}}} ], "$defs": {"modifiers": {"type": "array", "items": {"enum": ["shift", "ctrl", "alt", "meta", "capslock"]}, "uniqueItems": true}} } @@ -380,7 +382,7 @@ Server → client: | `error` | `code`, `message` | rejected input, policy violations | | binary | 8-byte little-endian grain index, then a JPEG | preview frames | -Limits: 200 input messages per second per session, 64 KiB per message; beyond that the server drops messages and counts them. Malformed messages get an `error` and are ignored. +Limits: 200 input messages per second per session, 64 KiB per message; beyond that the server drops messages and counts them (`preview_ack` does not count and is never dropped). Malformed messages get an `error` and are ignored. ### 7.3 Preview @@ -388,7 +390,8 @@ The siblings preview with low-rate JPEG (multiviewer `/preview.jpg` at 5 fps and - `GET /api/v1/preview.jpg`: the newest frame, converted after the request arrives (each request gets a ticket and is answered with a preview made for it or a later one; 1.1.0 answered with the previous request's picture), encoded once per preview. - Preview frames over the interaction WebSocket (each a preview made after it was due, waited for up to 100 ms like `preview.jpg`; a session never gets the same picture twice, 1.3.4) at `BROWSER_PREVIEW_FPS` (default 10, up to 25 while the session controls the page) and `BROWSER_PREVIEW_WIDTH` (default 960). A session asks for its rate with `preview` (a watcher can only lower it); the UI asks for 25 when its session gets control and for the default (`preview` without `fps`) when it loses it. The server also drops a session to at most `BROWSER_PREVIEW_FPS` when it loses control (toggle off, take-over, timeout), so only the controller can have more. -- A client that stops reading never slows the others. Preview frames are sent without waiting: when a client's socket is full, its frame is dropped (the next one is newer; `preview_frames_dropped_total`), and a frame that has started goes out whole before anything else. Text messages (state, acks, errors, events) keep their order and are never dropped: what does not fit waits in a per-client outbox that the connection's own thread drains, so no sender ever waits for a client (state, cursor and console broadcasts come from the CEF UI thread, where a wait could trip the hang timeout). A client whose outbox holds more than 4 MiB or whose oldest message has waited 5 s, or that takes no preview frame for 5 s, is closed, which ends its control; such a close is logged (`websocket_stuck_close` with the client and reason) and counted (`websocket_stuck_closes_total{reason}`). The kernel keeps at most 64 KiB of a WebSocket unsent (`TCP_NOTSENT_LOWAT`), so on a slow path previews are dropped early instead of waiting seconds in the send buffer. This holds for `/api/v1/events` too. Polling a JPEG adds a round trip per frame; a push keeps the click-to-picture time short. WebRTC is not used: full-rate monitoring is mxl-webrtc-monitor's job (route the video flow to it). +- A client that stops reading never slows the others. Preview frames are sent without waiting: when a client's socket is full, its frame is dropped (the next one is newer; `preview_frames_dropped_total`), and a frame that has started goes out whole before anything else. Text messages (state, acks, errors, events) keep their order and are never dropped: what does not fit waits in a per-client outbox that the connection's own thread drains, so no sender ever waits for a client (state, cursor and console broadcasts come from the CEF UI thread, where a wait could trip the hang timeout). A client whose outbox holds more than 4 MiB or whose oldest message has waited 5 s, or that takes no preview frame for 5 s, is closed, which ends its control; such a close is logged (`websocket_stuck_close` with the client, the interaction session (0 on `/api/v1/events`), the reason, the outbox bytes and the age of the oldest waiting message, `oldest_ms`) and counted (`websocket_stuck_closes_total{reason}`). The kernel keeps at most 32 KiB of a WebSocket unsent (`TCP_NOTSENT_LOWAT`; 64 KiB in 1.3.4), so on a slow path previews are dropped early instead of waiting seconds in the send buffer. This holds for `/api/v1/events` too. +- Preview flow control (after 1.3.4): the kernel limit does not help when a proxy sits on the path, because the proxy reads at once and holds the previews in its own buffers (lab, 2 Mbit/s behind a buffering proxy: 10 s old previews with 1.3.4). A client that answers each preview frame with `preview_ack` therefore has at most 2 previews unanswered; the server skips a session whose 2 are unanswered (counted in `preview_frames_dropped_total`), so the newest picture goes out as soon as an answer comes back. A session that answers nothing for 5 s is closed as `preview_not_taken`. A client that never sends `preview_ack` is not limited (as before). The web UI and the widgets answer every frame as it arrives. An ack sent for no frame would let one more preview through for the rest of the session (lab: an extra ack before the first frame raised the latency at 2 Mbit/s from 230 to 390 ms). Polling a JPEG adds a round trip per frame; a push keeps the click-to-picture time short. WebRTC is not used: full-rate monitoring is mxl-webrtc-monitor's job (route the video flow to it). - The preview is made from the BGRA frame (downscaled box filter, composited over a checkerboard in the UI when transparent), so it costs nothing when no client watches. ## 8. NMOS @@ -616,6 +619,7 @@ The file is one JSON document, written atomically by the UI and the API (`tmp` + | `BROWSER_CONFIRM_DIALOGS` | `cancel` | no | `cancel`, `accept` | | `BROWSER_URL_ALLOW` / `_DENY` | empty / §14.3 | no | URL policy lists | | `BROWSER_PREVIEW_FPS` / `_WIDTH` | 10 / 960 | no | preview | +| `BROWSER_PREVIEW_QUALITY` | 75 | yes | JPEG quality of the previews (10–100); lower makes them smaller on a slow path | | `BROWSER_INTERACT_TIMEOUT_S` | 120 | no | control ends after this idle time | | `BROWSER_DEVTOOLS` | `false` | yes | DevTools proxy | | `BROWSER_DEVTOOLS_PORT` | 9222 | yes | internal DevTools port (loopback) | @@ -688,7 +692,7 @@ Prefix `mxl_browser_source_`. Durations in seconds with millisecond buckets. | `interact_sessions` | gauge | – (controlling sessions, 0 or 1) | | `interaction_events_total` | counter | `type`, `result` (`ok`, `rejected`, `dropped`) | | `preview_frames_total` | counter | – (preview frames sent; until 1.3.1 also those a blocked client never took) | -| `preview_frames_dropped_total` | counter | – (preview frames dropped because the client's socket was full, §7.3) | +| `preview_frames_dropped_total` | counter | – (preview frames dropped because the client's socket was full or, with `preview_ack`, 2 previews were unanswered, §7.3) | | `websocket_stuck_closes_total` | counter | `reason` (`outbox_size`, `outbox_age`, `preview_not_taken`; WebSockets closed because the client did not read, §7.3; 1.3.4) | | `devtools_sessions` | gauge | – | | `sender_enabled` | gauge | `sender` | diff --git a/docs/api.md b/docs/api.md index 3731e02..1103e0f 100644 --- a/docs/api.md +++ b/docs/api.md @@ -121,7 +121,7 @@ Our own routes: ## Preview -`GET /api/v1/preview.jpg` → `image/jpeg`, the newest frame at `BROWSER_PREVIEW_WIDTH`, composited over a mid-grey checkerboard where transparent. The preview is made after the request arrives (up to 100 ms); only when none comes in time is the last one sent (503 before the first paint). +`GET /api/v1/preview.jpg` → `image/jpeg`, the newest frame at `BROWSER_PREVIEW_WIDTH` and JPEG quality `BROWSER_PREVIEW_QUALITY` (75; also for the WebSocket preview), composited over a mid-grey checkerboard where transparent. The preview is made after the request arrives (up to 100 ms); only when none comes in time is the last one sent (503 before the first paint). ## Widgets @@ -194,7 +194,7 @@ Server → client: {"type":"error","code":"not_controller","message":"interaction is off for this session"} ``` -Binary frames: 8-byte little-endian grain index, then a JPEG (the preview), at `BROWSER_PREVIEW_FPS`. `preview` sets this session's rate: up to 25 while it controls the page, else up to `BROWSER_PREVIEW_FPS`; without `fps` it goes back to `BROWSER_PREVIEW_FPS`. Losing control drops the rate to at most `BROWSER_PREVIEW_FPS`. The UI asks for 25 while it controls the page. A client that does not keep up misses preview frames (never text messages); one that reads nothing for 5 s is closed. Schema: `docs/interact.schema.json`. `controller` is `self`, `other` or `none`. Input without control answers `error` `not_controller`. Coordinates are 0…1 of the picture. Wheel deltas are pixels with the DOM sign (positive `dy` scrolls down). A `down` focuses the page; the buttons that are down ride on every `move` until their `up` (a drag selects text), and `leave` or the end of control sends the `up` of a button still down. +Binary frames: 8-byte little-endian grain index, then a JPEG (the preview), at `BROWSER_PREVIEW_FPS`. `preview` sets this session's rate: up to 25 while it controls the page, else up to `BROWSER_PREVIEW_FPS`; without `fps` it goes back to `BROWSER_PREVIEW_FPS`. Losing control drops the rate to at most `BROWSER_PREVIEW_FPS`. The UI asks for 25 while it controls the page. A client that does not keep up misses preview frames (never text messages); one that reads nothing for 5 s is closed. Flow control: a client that sends `{"type":"preview_ack"}` for each binary frame it receives has at most 2 previews unanswered, so a proxy on the path cannot hold seconds of them (the UI does this); one that then answers nothing for 5 s is closed. `preview_ack` is not counted against the input limit. Schema: `docs/interact.schema.json`. `controller` is `self`, `other` or `none`. Input without control answers `error` `not_controller`. Coordinates are 0…1 of the picture. Wheel deltas are pixels with the DOM sign (positive `dy` scrolls down). A `down` focuses the page; the buttons that are down ride on every `move` until their `up` (a drag selects text), and `leave` or the end of control sends the `up` of a button still down. ## DevTools (`BROWSER_DEVTOOLS=true` only) diff --git a/docs/interact.schema.json b/docs/interact.schema.json index a460a60..941d7b7 100644 --- a/docs/interact.schema.json +++ b/docs/interact.schema.json @@ -240,6 +240,18 @@ "maximum": 1920 } } + }, + { + "type": "object", + "required": [ + "type" + ], + "description": "one per preview frame received, never more; after the first, the server keeps at most 2 previews unacknowledged", + "properties": { + "type": { + "const": "preview_ack" + } + } } ], "$defs": { diff --git a/src/app/api.cpp b/src/app/api.cpp index 0e035b2..3c5873c 100644 --- a/src/app/api.cpp +++ b/src/app/api.cpp @@ -426,6 +426,7 @@ namespace mbs::app return s && s->sendText(text); }, socket->peer()}); + socket->setSession(*session); std::lock_guard lock{_socketsMutex}; _interactSockets[*session] = socket; }; @@ -771,7 +772,7 @@ namespace mbs::app std::lock_guard lock{_jpegMutex}; if (!rgb.empty() && sequence != _jpegSequence) { - _jpeg = encodeJpeg(rgb, static_cast(_engine->previewWidth()), static_cast(_engine->previewHeight())); + _jpeg = encodeJpeg(rgb, static_cast(_engine->previewWidth()), static_cast(_engine->previewHeight()), _cfg.previewQuality); _jpegSequence = sequence; } jpeg = _jpeg; diff --git a/src/app/application.cpp b/src/app/application.cpp index 1c9b574..31d5040 100644 --- a/src/app/application.cpp +++ b/src/app/application.cpp @@ -1503,6 +1503,27 @@ namespace mbs::app } std::erase_if(due, [&](auto const& entry) { return rates.count(entry.first) == 0; }); std::erase_if(shown, [&](auto const& entry) { return rates.count(entry.first) == 0; }); + // A session that has not acknowledged its previews misses this one (SPEC §7.3). + std::erase_if(ready, [&](int id) { + if (_interact->previewWindowOpen(id)) + { + return false; + } + std::shared_ptr socket; + { + std::lock_guard lock{_socketsMutex}; + auto it = _interactSockets.find(id); + if (it != _interactSockets.end()) + { + socket = it->second; + } + } + if (socket && socket->skipBinary() == ops::WebSocket::Sent::Dropped) + { + _metrics.inc("preview_frames_dropped_total", {}, 1); + } + return true; + }); if (ready.empty()) { continue; @@ -1520,7 +1541,7 @@ namespace mbs::app std::lock_guard lock{_jpegMutex}; if (sequence != _jpegSequence && !rgb.empty()) { - _jpeg = encodeJpeg(rgb, static_cast(_engine->previewWidth()), static_cast(_engine->previewHeight())); + _jpeg = encodeJpeg(rgb, static_cast(_engine->previewWidth()), static_cast(_engine->previewHeight()), _cfg.previewQuality); _jpegSequence = sequence; _jpegGrain = _clock->indexAt(_clock->nowNs()); } @@ -1538,7 +1559,7 @@ namespace mbs::app frame[static_cast(b)] = static_cast((grain >> (8 * b)) & 0xFF); } frame += jpeg; - std::vector> targets; + std::vector>> targets; { std::lock_guard lock{_socketsMutex}; for (int id : ready) @@ -1546,7 +1567,7 @@ namespace mbs::app auto it = _interactSockets.find(id); if (it != _interactSockets.end() && shown[id] != jpegSequence) { - targets.push_back(it->second); + targets.emplace_back(id, it->second); shown[id] = jpegSequence; } } @@ -1554,9 +1575,14 @@ namespace mbs::app // Never blocks: a client whose socket is full misses this frame (SPEC §7.3). int sent = 0; int dropped = 0; - for (auto const& socket : targets) + for (auto const& [id, socket] : targets) { + _interact->previewSending(id); auto const result = socket->sendBinary(frame); + if (result != ops::WebSocket::Sent::Yes) + { + _interact->previewNotSent(id); + } sent += result == ops::WebSocket::Sent::Yes ? 1 : 0; dropped += result == ops::WebSocket::Sent::Dropped ? 1 : 0; } diff --git a/src/app/application.hpp b/src/app/application.hpp index 7f8854c..eaf421a 100644 --- a/src/app/application.hpp +++ b/src/app/application.hpp @@ -58,7 +58,7 @@ namespace mbs::app }; /// RGB24 → JPEG (stb), for the preview. - std::string encodeJpeg(std::vector const& rgb, int width, int height, int quality = 75); + std::string encodeJpeg(std::vector const& rgb, int width, int height, int quality); class Application final : public cef::PageObserver { diff --git a/src/config/config.cpp b/src/config/config.cpp index 09ae558..0f5f5a1 100644 --- a/src/config/config.cpp +++ b/src/config/config.cpp @@ -79,6 +79,7 @@ namespace mbs::config {"BROWSER_URL_DENY", "", false, false, "Comma list of host globs and CIDRs, added to the built-in deny list"}, {"BROWSER_PREVIEW_FPS", "10", false, false, "Preview frames per second (1-25)"}, {"BROWSER_PREVIEW_WIDTH", "960", false, false, "Preview width in pixels"}, + {"BROWSER_PREVIEW_QUALITY", "75", true, false, "JPEG quality of the previews (10-100); lower is smaller on a slow path"}, {"BROWSER_INTERACT_TIMEOUT_S", "120", false, false, "Interaction ends after this idle time"}, {"BROWSER_DEVTOOLS", "false", true, false, "DevTools proxy under /devtools/"}, {"BROWSER_DEVTOOLS_PORT", "9222", true, false, "Internal DevTools port (loopback only)"}, @@ -462,6 +463,7 @@ namespace mbs::config c.urlDeny = parseList(get("BROWSER_URL_DENY")); c.previewFps = parseInt("BROWSER_PREVIEW_FPS", get("BROWSER_PREVIEW_FPS"), 1, 25); c.previewWidth = parseInt("BROWSER_PREVIEW_WIDTH", get("BROWSER_PREVIEW_WIDTH"), 160, 1920); + c.previewQuality = parseInt("BROWSER_PREVIEW_QUALITY", get("BROWSER_PREVIEW_QUALITY"), 10, 100); c.interactTimeoutS = parseInt("BROWSER_INTERACT_TIMEOUT_S", get("BROWSER_INTERACT_TIMEOUT_S"), 5, 3600); c.devtools = parseBool("BROWSER_DEVTOOLS", get("BROWSER_DEVTOOLS")); c.devtoolsPort = parseInt("BROWSER_DEVTOOLS_PORT", get("BROWSER_DEVTOOLS_PORT"), 1, 65535); diff --git a/src/config/config.hpp b/src/config/config.hpp index 5f313b1..bb6ebc0 100644 --- a/src/config/config.hpp +++ b/src/config/config.hpp @@ -128,6 +128,7 @@ namespace mbs::config std::vector urlDeny; int previewFps = 10; int previewWidth = 960; + int previewQuality = 75; int interactTimeoutS = 120; bool devtools = false; int devtoolsPort = 9222; diff --git a/src/ops/httpserver.cpp b/src/ops/httpserver.cpp index 321ebe8..40731e1 100644 --- a/src/ops/httpserver.cpp +++ b/src/ops/httpserver.cpp @@ -276,7 +276,9 @@ namespace mbs::ops void WebSocket::stuck(char const* reason) { - log::warn("websocket_stuck_close", {{"client", _peer}, {"reason", reason}, {"outbox_bytes", static_cast(_outboxBytes)}}); + auto const oldest = _outbox.empty() ? Clock::duration{} : Clock::now() - _outbox.front().queued; + log::warn("websocket_stuck_close", {{"client", _peer}, {"session", _session}, {"reason", reason}, {"outbox_bytes", static_cast(_outboxBytes)}, + {"oldest_ms", static_cast(std::chrono::duration_cast(oldest).count())}}); if (_onStuck) { _onStuck(reason); @@ -367,6 +369,18 @@ namespace mbs::ops return Sent::Yes; } + WebSocket::Sent WebSocket::skipBinary() + { + std::lock_guard lock{_sendMutex}; + return _open.load() ? dropped(Clock::now()) : Sent::Closed; + } + + void WebSocket::setSession(int session) + { + std::lock_guard lock{_sendMutex}; + _session = session; + } + void WebSocket::close(int code) { std::lock_guard lock{_sendMutex}; diff --git a/src/ops/httpserver.hpp b/src/ops/httpserver.hpp index 6d630cf..0b2db2e 100644 --- a/src/ops/httpserver.hpp +++ b/src/ops/httpserver.hpp @@ -52,8 +52,9 @@ namespace mbs::ops /// one is newer); one that has started is finished first. A client that takes no binary /// frame for kStuckTimeout is closed too. Such a "stuck" close is logged and reported to /// `onStuck` with its reason (outbox_size, outbox_age, preview_not_taken). - /// The kernel keeps at most kUnsentBytes unsent (TCP_NOTSENT_LOWAT), so a slow path drops - /// previews early instead of holding seconds of them in the send buffer. + /// The kernel keeps at most kUnsentBytes unsent (TCP_NOTSENT_LOWAT, about two thirds of a + /// 960 px preview), so a slow path drops previews early instead of holding seconds of them + /// in the send buffer. class WebSocket { public: @@ -64,7 +65,7 @@ namespace mbs::ops static constexpr std::chrono::milliseconds kStuckTimeout{5000}; static constexpr std::size_t kOutboxBytes = 4 * 1024 * 1024; - static constexpr int kUnsentBytes = 64 * 1024; + static constexpr int kUnsentBytes = 32 * 1024; enum class Sent { @@ -76,6 +77,11 @@ namespace mbs::ops /// Queues the message; false when the socket is (or got) closed. bool sendText(std::string_view text); Sent sendBinary(std::string_view data); + /// A binary frame the caller held back because the client has not taken the previous + /// ones (preview flow control): counts towards kStuckTimeout like a full socket. + Sent skipBinary(); + /// The interaction session on this socket, for the stuck-close log line (0: none, events). + void setSession(int session); /// Sends a close frame and shuts the socket down; the reader thread ends. void close(int code = 1000); [[nodiscard]] bool open() const { return _open.load(); } @@ -104,6 +110,7 @@ namespace mbs::ops std::deque _outbox; std::size_t _outboxBytes = 0; bool _dropping = false; + int _session = 0; Clock::time_point _droppingSince; std::string _peer; std::function _onStuck; diff --git a/src/ops/interact.cpp b/src/ops/interact.cpp index e47e15f..4d5aab8 100644 --- a/src/ops/interact.cpp +++ b/src/ops/interact.cpp @@ -262,6 +262,33 @@ namespace mbs::ops return rates; } + bool InteractHub::previewWindowOpen(int id) const + { + std::lock_guard lock{_mutex}; + auto const found = _sessions.find(id); + return found == _sessions.end() || !found->second.previewAcks || found->second.previewUnacked < kPreviewWindow; + } + + void InteractHub::previewSending(int id) + { + std::lock_guard lock{_mutex}; + auto const found = _sessions.find(id); + if (found != _sessions.end()) + { + ++found->second.previewUnacked; // before it goes out, so its ack cannot come first + } + } + + void InteractHub::previewNotSent(int id) + { + std::lock_guard lock{_mutex}; + auto const found = _sessions.find(id); + if (found != _sessions.end() && found->second.previewUnacked > 0) + { + --found->second.previewUnacked; + } + } + int InteractHub::sessions() const { std::lock_guard lock{_mutex}; @@ -298,6 +325,17 @@ namespace mbs::ops out.emplace_back(session.peer.sendText, errorJson(code, why)); }; + picojson::value doc; + std::string const parseError = picojson::parse(doc, raw); + bool const valid = parseError.empty() && doc.is(); + // Preview acks are flow control, not input: the input limit does not drop them. + if (valid && text(doc.get(), "type") == "preview_ack") + { + session.previewAcks = true; + session.previewUnacked = std::max(0, session.previewUnacked - 1); + return; + } + // Input limit per session: beyond it, messages are dropped (one error per second). if (now - session.windowStart >= std::chrono::seconds(1)) { @@ -314,9 +352,7 @@ namespace mbs::ops } else { - picojson::value doc; - std::string const parseError = picojson::parse(doc, raw); - if (!parseError.empty() || !doc.is()) + if (!valid) { reject("invalid", "bad_message", "not a JSON object"); } diff --git a/src/ops/interact.hpp b/src/ops/interact.hpp index d6244b1..84da155 100644 --- a/src/ops/interact.hpp +++ b/src/ops/interact.hpp @@ -87,6 +87,15 @@ namespace mbs::ops /// Preview rate each session asked for (0: none); the controller gets up to maxPreviewFps. [[nodiscard]] std::map previewRates() const; + /// Preview flow control (SPEC §7.3): a session that answers previews with `preview_ack` + /// has at most kPreviewWindow of them unanswered, so a buffering proxy on its path cannot + /// hold seconds of them. False while it has that many. A session that never sent an ack + /// is always open. + [[nodiscard]] bool previewWindowOpen(int session) const; + /// Call before a preview goes out to `session`, and previewNotSent when it did not. + void previewSending(int session); + void previewNotSent(int session); + static constexpr int kPreviewWindow = 2; [[nodiscard]] int sessions() const; [[nodiscard]] bool controlled() const; [[nodiscard]] InteractCounters counters() const; @@ -96,6 +105,8 @@ namespace mbs::ops { Peer peer; int previewFps = 0; + bool previewAcks = false; // the client sent a preview_ack + int previewUnacked = 0; Clock::time_point windowStart{}; int windowCount = 0; }; diff --git a/tests/integration/interact_test.py b/tests/integration/interact_test.py index 823d9d2..c4bc4ea 100644 --- a/tests/integration/interact_test.py +++ b/tests/integration/interact_test.py @@ -2,10 +2,11 @@ # SPDX-License-Identifier: MIT """Interaction test (SPEC §17): drives interact.html through /api/v1/interact and checks the page's reports on /api/v1/events: clicks, keys, text, IME, wheel, then text selection (drag, Ctrl+C, -double and triple click), then a client that stops reading (it must not slow the controller's preview) -and the preview rates (watcher, controller, after take-over and release). With a container and its -domain directory it also reads the selection highlight from the newest video grain (`docker exec … -mxl-bs-grain-reader`, `DOCKER='sudo docker'` if needed). Standard library only. +double and triple click), then a client that stops reading (it must not slow the controller's preview), +the preview rates (watcher, controller, after take-over and release) and the preview flow control. +With a container and its domain directory it also reads the selection highlight from the newest +video grain (`docker exec … mxl-bs-grain-reader`, `DOCKER='sudo docker'` if needed). Standard +library only. interact_test.py [host:port] [container domain-dir] """ @@ -257,6 +258,29 @@ def stuck_client(host, port, observer): c.close() +def preview_flow_control(host, port): + """SPEC §7.3: a session that answers previews with `preview_ack` has at most 2 unanswered (so a + buffering proxy cannot hold seconds of them); one that stops answering is closed after 5 s and + counted as `preview_not_taken`.""" + reason = 'websocket_stuck_closes_total{reason="preview_not_taken"' + before = metric(host, port, reason) + sock = ws_connect(host, port, "/api/v1/interact") + for _ in range(ws_read(sock, 1)[1]): # answers what came so far + ws_send(sock, {"type": "preview_ack"}) + held = ws_read(sock, 2)[1] + assert held == 2, f"{held} previews unanswered" + ws_send(sock, {"type": "preview_ack"}) + more = ws_read(sock, 1)[1] + assert more == 1, f"{more} previews after one ack" + end = time.time() + 10 + while time.time() < end and get_json(host, port, "/api/v1/status")["interact"]["sessions"] > 0: + ws_read(sock, 0.5) + closes = metric(host, port, reason) - before + assert closes == 1, f"{closes} preview_not_taken closes" + print(f"OK: preview flow control: {held} previews unanswered, 1 more per ack; closed 5 s after the last ack and counted") + sock.close() + + def main(): host, port = (sys.argv[1] if len(sys.argv) > 1 else "127.0.0.1:8160").split(":") port = int(port) @@ -311,6 +335,7 @@ def main(): stuck_client(host, port, observer) preview_rates(host, port) preview_new_pictures(host, port) + preview_flow_control(host, port) if __name__ == "__main__": diff --git a/tests/unit/test_config.cpp b/tests/unit/test_config.cpp index 268d505..a14c522 100644 --- a/tests/unit/test_config.cpp +++ b/tests/unit/test_config.cpp @@ -91,6 +91,14 @@ TEST_CASE("service workers are off unless enabled") CHECK(errorKey({{"BROWSER_SERVICE_WORKERS", "sometimes"}}) == "BROWSER_SERVICE_WORKERS"); } +TEST_CASE("preview JPEG quality: 75 by default, 10 to 100") +{ + CHECK(load({}, {}, "host").config.previewQuality == 75); + CHECK(load({{"BROWSER_PREVIEW_QUALITY", "50"}}, {}, "host").config.previewQuality == 50); + CHECK(errorKey({{"BROWSER_PREVIEW_QUALITY", "9"}}) == "BROWSER_PREVIEW_QUALITY"); + CHECK(errorKey({{"BROWSER_PREVIEW_QUALITY", "101"}}) == "BROWSER_PREVIEW_QUALITY"); +} + TEST_CASE("frame-loop settings: pipeline, real-time threads, conversion threads, WebGPU") { auto const d = load({}, {}, "host").config; diff --git a/tests/unit/test_httpserver.cpp b/tests/unit/test_httpserver.cpp index 79e090a..353909e 100644 --- a/tests/unit/test_httpserver.cpp +++ b/tests/unit/test_httpserver.cpp @@ -425,6 +425,23 @@ TEST_CASE("WebSocket: a client that stops reading never blocks a send; text stay CHECK(!bigWs->sendText(std::string(WebSocket::kOutboxBytes + 1, 'x'))); CHECK(!bigWs->open()); ::close(bigFd); + + // Previews held back by flow control (skipBinary) count like a full socket: closed after + // kStuckTimeout without a sent preview; a sent one starts the time again. + auto [skipFd, skipWs] = connectSmall(); + CHECK(skipWs->skipBinary() == WebSocket::Sent::Dropped); + std::this_thread::sleep_for(std::chrono::milliseconds(500)); + CHECK(skipWs->sendBinary("p") == WebSocket::Sent::Yes); + auto const skipStart = std::chrono::steady_clock::now(); + auto skipped = WebSocket::Sent::Dropped; + while (skipped == WebSocket::Sent::Dropped && std::chrono::steady_clock::now() - skipStart < WebSocket::kStuckTimeout * 2) + { + skipped = skipWs->skipBinary(); + std::this_thread::sleep_for(std::chrono::milliseconds(40)); + } + CHECK(skipped == WebSocket::Sent::Closed); + CHECK(std::chrono::steady_clock::now() - skipStart >= WebSocket::kStuckTimeout); + ::close(skipFd); server.stop(); // Each stuck close was reported once, with its reason (the first client recovered). The // preview client's started frame waits in the outbox, so its age closes it. @@ -435,5 +452,5 @@ TEST_CASE("WebSocket: a client that stops reading never blocks a send; text stay reported += r + " "; } INFO(reported); - CHECK(stuckReasons == std::vector{"outbox_age", "outbox_age", "outbox_size"}); + CHECK(stuckReasons == std::vector{"outbox_age", "outbox_age", "outbox_size", "preview_not_taken"}); } diff --git a/tests/unit/test_interact.cpp b/tests/unit/test_interact.cpp index 805ce19..ed6d1fa 100644 --- a/tests/unit/test_interact.cpp +++ b/tests/unit/test_interact.cpp @@ -248,3 +248,57 @@ TEST_CASE("preview rates: losing control drops the controller's rate to the defa CHECK(!hub.controlled()); CHECK(hub.previewRates().at(idb) == 10); } + +TEST_CASE("preview flow control: a session that acks has at most two previews unanswered") +{ + FakePage page; + InteractHub hub(settings(), page); + Client a; + Client b; + int const ida = hub.open(a.peer()); + int const idb = hub.open(b.peer()); + for (int i = 0; i < 5; ++i) // no ack yet: no limit + { + CHECK(hub.previewWindowOpen(ida)); + hub.previewSending(ida); + } + for (int i = 0; i < 5; ++i) + { + hub.message(ida, R"({"type":"preview_ack"})"); + } + CHECK(hub.previewWindowOpen(ida)); + hub.previewSending(ida); + hub.previewSending(ida); + CHECK(!hub.previewWindowOpen(ida)); + CHECK(hub.previewWindowOpen(idb)); // per session + hub.message(ida, R"({"type":"preview_ack"})"); + CHECK(hub.previewWindowOpen(ida)); + hub.previewSending(ida); + CHECK(!hub.previewWindowOpen(ida)); + hub.previewNotSent(ida); // the socket was full: it does not count + CHECK(hub.previewWindowOpen(ida)); + hub.previewSending(ida); + hub.message(ida, R"({"type":"preview_ack"})"); // an ack can come before the next send + hub.message(ida, R"({"type":"preview_ack"})"); + CHECK(hub.previewWindowOpen(ida)); + CHECK(a.received.size() == 1); // only the state on open: acks get no answer +} + +TEST_CASE("preview flow control: the input limit does not drop acks") +{ + FakePage page; + InteractHub hub(settings(), page); + Client a; + int const id = hub.open(a.peer()); + auto const t0 = InteractHub::Clock::now(); + hub.message(id, R"({"type":"preview_ack"})", t0); + for (int i = 0; i < 250; ++i) + { + hub.message(id, R"({"type":"pointer","action":"move","x":0.5,"y":0.5})", t0); + } + hub.previewSending(id); + hub.previewSending(id); + CHECK(!hub.previewWindowOpen(id)); + hub.message(id, R"({"type":"preview_ack"})", t0); + CHECK(hub.previewWindowOpen(id)); +} diff --git a/tests/unit/web/interact.test.mjs b/tests/unit/web/interact.test.mjs index d5fd346..8b6913a 100644 --- a/tests/unit/web/interact.test.mjs +++ b/tests/unit/web/interact.test.mjs @@ -89,3 +89,22 @@ test("the preview messages are in docs/interact.schema.json", () => { assert.deepEqual(preview.required, ["type"], "fps must be optional (omitted: the default)"); assert.ok(preview.properties.fps.maximum >= 25); }); + +test("acknowledges every preview frame, before it is shown", () => { + const frames = []; + const client = new InteractClient({ frame: (grain) => frames.push(grain) }); + client.connect(); + const ws = FakeSocket.last; + ws.open(); + for (const grain of [7, 8]) { + const data = new ArrayBuffer(12); + new DataView(data).setUint32(0, grain, true); + ws.onmessage({ data }); + } + ws.onmessage({ data: new ArrayBuffer(8) }); // no picture: no ack + assert.deepEqual(frames, [7, 8]); + assert.deepEqual(ws.sent.filter((m) => m.type === "preview_ack"), [{ type: "preview_ack" }, { type: "preview_ack" }]); + const schema = JSON.parse(readFileSync(new URL("../../../docs/interact.schema.json", import.meta.url), "utf8")); + assert.ok(schema.oneOf.some((s) => s.properties.type.const === "preview_ack"), "no preview_ack in the schema"); + client.close(); +}); diff --git a/web/src/interact.js b/web/src/interact.js index 8a6082e..7ef9d2a 100644 --- a/web/src/interact.js +++ b/web/src/interact.js @@ -121,6 +121,9 @@ export class InteractClient { onMessage(data) { if (data instanceof ArrayBuffer) { if (data.byteLength <= 8) return; + // Flow control (SPEC §7.3): the server sends the next preview only when we have taken + // this one, so a buffering proxy on the path cannot hold seconds of them. + this.send({ type: "preview_ack" }); const v = new DataView(data); const grain = v.getUint32(0, true) + v.getUint32(4, true) * 2 ** 32; this.h.frame?.(grain, new Blob([new Uint8Array(data, 8)], { type: "image/jpeg" })); From af1c8dcae28897c25477fdec9845660bc423575d Mon Sep 17 00:00:00 2001 From: Adrian Hilber Date: Fri, 6 Nov 2026 21:21:34 +0100 Subject: [PATCH 2/3] Count a preview answer as reading and skip only new pictures. Review of the preview flow control: - An ack now resets the stuck timer (WebSocket::binaryTaken). Before, only a sent frame did, so a client that answered every frame while its window was shut by a backlog from before its first ack could be closed as preview_not_taken. The counter keeps counting every send from the first frame on, so that backlog is drained before new previews go out (clamping it on the first ack would leave the backlog in flight for good: each answer for a buffered frame would let a new one in). - A shut window skips a session (and counts a drop towards the stuck close) only when there is a new picture; a static page closes no one. - preview_ack is recognised with a size and substring check before the input limit; other messages are parsed after it, as before. - SPEC: an extra ack is ignored when nothing is unanswered. Lab, 500 kbit/s behind the buffering proxy, 40 ms, fresh connect: median age 1.17 s, 60 s without a close, control kept; two reconnects on the same instance: no close (the first seconds up to 3.9 and 7.6 s while the backlog drained). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_017ybuzgBaGxuB5UiaE8dRa4 --- CHANGELOG.md | 4 ++-- SPEC.md | 2 +- src/app/api.cpp | 6 +++++- src/app/application.cpp | 33 +++++++++++++++-------------- src/ops/httpserver.cpp | 6 ++++++ src/ops/httpserver.hpp | 3 +++ src/ops/interact.cpp | 20 +++++++++++++----- src/ops/interact.hpp | 5 ++++- tests/unit/test_httpserver.cpp | 6 +++++- tests/unit/test_interact.cpp | 38 ++++++++++++++++++++++++++++++++++ 10 files changed, 97 insertions(+), 26 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7965641..a9c266a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,11 +3,11 @@ ## Unreleased - **Interaction preview stays current on a slow path, also behind a proxy** (SPEC §7.2, §7.3). 1.3.4 limits what the kernel holds unsent, but a reverse proxy between the function and the browser reads everything at once and holds the previews in its own buffers: in the lab (2 Mbit/s behind a buffering proxy, 50 KB previews) they arrived 10–11 s old; at 500 kbit/s the delay grew to 14 s and the session was closed as stuck after 20 s, which ended its control. - - **Preview flow control:** a client that answers each preview frame with `{"type":"preview_ack"}` has at most 2 previews unanswered; the server skips it until an answer comes (counted in `preview_frames_dropped_total`) and then sends the newest picture. A session that answers nothing for 5 s is closed as `preview_not_taken`. Clients that never send `preview_ack` work as before. The web UI and the widgets answer every frame. `preview_ack` is not counted against the input limit. + - **Preview flow control:** a client that answers each preview frame with `{"type":"preview_ack"}` has at most 2 previews unanswered; a new picture skips it until an answer comes (counted in `preview_frames_dropped_total`) and then the newest picture goes out. Previews sent before the first answer count too, so what a proxy buffered then is drained first. A session is closed as `preview_not_taken` only when new pictures were skipped for 5 s without an answer (a static page closes no one). Clients that never send `preview_ack` work as before. The web UI and the widgets answer every frame. `preview_ack` is not counted against the input limit. - **Kernel limit 32 KiB** instead of 64 KiB (`TCP_NOTSENT_LOWAT`): measured with 16, 32, 64 and 128 KiB at 2 Mbit/s and 40 ms; 16 and 32 KiB were equal and best, with no frame-rate loss at 6 and 16 Mbit/s. - Lab, preview age at the client (median), 25 fps asked, 50 KB previews, 40 ms round trip: direct at 2 Mbit/s 2.8 s (1.3.3), 0.79 s (1.3.4), 0.36 s now; behind the proxy at 2 Mbit/s 10.1 s (1.3.4), 0.22 s now; behind the proxy at 500 kbit/s 14 s and closed (1.3.4), 1.1 s and control kept for 60 s now. Cost: on a 16 Mbit/s path with 40 ms round trip the preview runs at 17 instead of 22 fps (and is 54 instead of 80 ms old); on a LAN nothing changes. - **Stuck-close log line:** `websocket_stuck_close` now also has the interaction `session` (0 on `/api/v1/events`) and `oldest_ms`, the age of the oldest waiting message. The counter `websocket_stuck_closes_total{reason}` is unchanged. - - Tests: unit tests for the window (per session, acks before the next send, a frame that was not sent, acks past the input limit) and for a held-back preview counting towards the stuck close; `tests/unit/web/interact.test.mjs` (the UI answers every frame; `preview_ack` in the schema); `tests/integration/interact_test.py` (2 previews unanswered, 1 more per answer, closed and counted 5 s after the last). + - Tests: unit tests for the window (per session, acks before the next send, a frame that was not sent, acks past the input limit, a backlog before the first ack, the cheap `preview_ack` check) and for a held-back preview counting towards the stuck close and an answer resetting it; `tests/unit/web/interact.test.mjs` (the UI answers every frame; `preview_ack` in the schema); `tests/integration/interact_test.py` (2 previews unanswered, 1 more per answer, closed and counted 5 s after the last). - **`BROWSER_PREVIEW_QUALITY`** (default 75, 10–100, restart): the JPEG quality of the previews (`preview.jpg` and the interaction preview). Lower makes each picture smaller, so a slow link carries more of them. ## 1.3.4 (2026-11-05) diff --git a/SPEC.md b/SPEC.md index bd04a36..7fa1564 100644 --- a/SPEC.md +++ b/SPEC.md @@ -391,7 +391,7 @@ The siblings preview with low-rate JPEG (multiviewer `/preview.jpg` at 5 fps and - `GET /api/v1/preview.jpg`: the newest frame, converted after the request arrives (each request gets a ticket and is answered with a preview made for it or a later one; 1.1.0 answered with the previous request's picture), encoded once per preview. - Preview frames over the interaction WebSocket (each a preview made after it was due, waited for up to 100 ms like `preview.jpg`; a session never gets the same picture twice, 1.3.4) at `BROWSER_PREVIEW_FPS` (default 10, up to 25 while the session controls the page) and `BROWSER_PREVIEW_WIDTH` (default 960). A session asks for its rate with `preview` (a watcher can only lower it); the UI asks for 25 when its session gets control and for the default (`preview` without `fps`) when it loses it. The server also drops a session to at most `BROWSER_PREVIEW_FPS` when it loses control (toggle off, take-over, timeout), so only the controller can have more. - A client that stops reading never slows the others. Preview frames are sent without waiting: when a client's socket is full, its frame is dropped (the next one is newer; `preview_frames_dropped_total`), and a frame that has started goes out whole before anything else. Text messages (state, acks, errors, events) keep their order and are never dropped: what does not fit waits in a per-client outbox that the connection's own thread drains, so no sender ever waits for a client (state, cursor and console broadcasts come from the CEF UI thread, where a wait could trip the hang timeout). A client whose outbox holds more than 4 MiB or whose oldest message has waited 5 s, or that takes no preview frame for 5 s, is closed, which ends its control; such a close is logged (`websocket_stuck_close` with the client, the interaction session (0 on `/api/v1/events`), the reason, the outbox bytes and the age of the oldest waiting message, `oldest_ms`) and counted (`websocket_stuck_closes_total{reason}`). The kernel keeps at most 32 KiB of a WebSocket unsent (`TCP_NOTSENT_LOWAT`; 64 KiB in 1.3.4), so on a slow path previews are dropped early instead of waiting seconds in the send buffer. This holds for `/api/v1/events` too. -- Preview flow control (after 1.3.4): the kernel limit does not help when a proxy sits on the path, because the proxy reads at once and holds the previews in its own buffers (lab, 2 Mbit/s behind a buffering proxy: 10 s old previews with 1.3.4). A client that answers each preview frame with `preview_ack` therefore has at most 2 previews unanswered; the server skips a session whose 2 are unanswered (counted in `preview_frames_dropped_total`), so the newest picture goes out as soon as an answer comes back. A session that answers nothing for 5 s is closed as `preview_not_taken`. A client that never sends `preview_ack` is not limited (as before). The web UI and the widgets answer every frame as it arrives. An ack sent for no frame would let one more preview through for the rest of the session (lab: an extra ack before the first frame raised the latency at 2 Mbit/s from 230 to 390 ms). Polling a JPEG adds a round trip per frame; a push keeps the click-to-picture time short. WebRTC is not used: full-rate monitoring is mxl-webrtc-monitor's job (route the video flow to it). +- Preview flow control (after 1.3.4): the kernel limit does not help when a proxy sits on the path, because the proxy reads at once and holds the previews in its own buffers (lab, 2 Mbit/s behind a buffering proxy: 10 s old previews with 1.3.4). A client that answers each preview frame with `preview_ack` therefore has at most 2 previews unanswered; while it has 2, a new picture skips it (counted in `preview_frames_dropped_total`; an unchanged picture is not counted), so the newest picture goes out as soon as an answer comes back. Every preview counts from the first one on, also those sent before the first `preview_ack`: what a proxy buffered then is answered (drained) before new ones go out. An answer counts as reading: a session is closed as `preview_not_taken` only when new pictures were skipped for 5 s without an answer. A client that never sends `preview_ack` is not limited (as before). The web UI and the widgets answer every frame as it arrives. An extra `preview_ack` (for no frame) is ignored when no preview is unanswered; otherwise it is taken for a preview still on the way and lets one more through until the client has caught up (lab: an extra ack before the first frame, while a preview was already on the way, raised the age at 2 Mbit/s from 230 to 390 ms). Polling a JPEG adds a round trip per frame; a push keeps the click-to-picture time short. WebRTC is not used: full-rate monitoring is mxl-webrtc-monitor's job (route the video flow to it). - The preview is made from the BGRA frame (downscaled box filter, composited over a checkerboard in the UI when transparent), so it costs nothing when no client watches. ## 8. NMOS diff --git a/src/app/api.cpp b/src/app/api.cpp index 3c5873c..2aff76f 100644 --- a/src/app/api.cpp +++ b/src/app/api.cpp @@ -430,9 +430,13 @@ namespace mbs::app std::lock_guard lock{_socketsMutex}; _interactSockets[*session] = socket; }; - h.message = [this, session](std::shared_ptr const&, std::string const& message, bool binary) { + h.message = [this, session](std::shared_ptr const& socket, std::string const& message, bool binary) { if (!binary) { + if (ops::InteractHub::isPreviewAck(message)) + { + socket->binaryTaken(); // it reads, also while its window is shut + } _interact->message(*session, message); } }; diff --git a/src/app/application.cpp b/src/app/application.cpp index 31d5040..5aedbc3 100644 --- a/src/app/application.cpp +++ b/src/app/application.cpp @@ -1503,28 +1503,17 @@ namespace mbs::app } std::erase_if(due, [&](auto const& entry) { return rates.count(entry.first) == 0; }); std::erase_if(shown, [&](auto const& entry) { return rates.count(entry.first) == 0; }); - // A session that has not acknowledged its previews misses this one (SPEC §7.3). + // A session that has not acknowledged its previews misses a new picture (SPEC §7.3). + std::vector held; std::erase_if(ready, [&](int id) { if (_interact->previewWindowOpen(id)) { return false; } - std::shared_ptr socket; - { - std::lock_guard lock{_socketsMutex}; - auto it = _interactSockets.find(id); - if (it != _interactSockets.end()) - { - socket = it->second; - } - } - if (socket && socket->skipBinary() == ops::WebSocket::Sent::Dropped) - { - _metrics.inc("preview_frames_dropped_total", {}, 1); - } + held.push_back(id); return true; }); - if (ready.empty()) + if (ready.empty() && held.empty()) { continue; } @@ -1560,8 +1549,18 @@ namespace mbs::app } frame += jpeg; std::vector>> targets; + std::vector> skipped; { std::lock_guard lock{_socketsMutex}; + for (int id : held) + { + // Only a picture it has not had counts as missed: a static page closes no one. + auto it = _interactSockets.find(id); + if (it != _interactSockets.end() && shown[id] != jpegSequence) + { + skipped.push_back(it->second); + } + } for (int id : ready) { auto it = _interactSockets.find(id); @@ -1575,6 +1574,10 @@ namespace mbs::app // Never blocks: a client whose socket is full misses this frame (SPEC §7.3). int sent = 0; int dropped = 0; + for (auto const& socket : skipped) + { + dropped += socket->skipBinary() == ops::WebSocket::Sent::Dropped ? 1 : 0; + } for (auto const& [id, socket] : targets) { _interact->previewSending(id); diff --git a/src/ops/httpserver.cpp b/src/ops/httpserver.cpp index 40731e1..6d52ef0 100644 --- a/src/ops/httpserver.cpp +++ b/src/ops/httpserver.cpp @@ -375,6 +375,12 @@ namespace mbs::ops return _open.load() ? dropped(Clock::now()) : Sent::Closed; } + void WebSocket::binaryTaken() + { + std::lock_guard lock{_sendMutex}; + _dropping = false; + } + void WebSocket::setSession(int session) { std::lock_guard lock{_sendMutex}; diff --git a/src/ops/httpserver.hpp b/src/ops/httpserver.hpp index 0b2db2e..294fe17 100644 --- a/src/ops/httpserver.hpp +++ b/src/ops/httpserver.hpp @@ -80,6 +80,9 @@ namespace mbs::ops /// A binary frame the caller held back because the client has not taken the previous /// ones (preview flow control): counts towards kStuckTimeout like a full socket. Sent skipBinary(); + /// The client answered a binary frame (preview flow control): it reads, so the time + /// towards kStuckTimeout starts again, as when a frame goes out. + void binaryTaken(); /// The interaction session on this socket, for the stuck-close log line (0: none, events). void setSession(int session); /// Sends a close frame and shuts the socket down; the reader thread ends. diff --git a/src/ops/interact.cpp b/src/ops/interact.cpp index 4d5aab8..4b725fc 100644 --- a/src/ops/interact.cpp +++ b/src/ops/interact.cpp @@ -262,6 +262,17 @@ namespace mbs::ops return rates; } + bool InteractHub::isPreviewAck(std::string const& raw) + { + // Cheap: a size and substring check; only such a short message is parsed. + if (raw.size() > 64 || raw.find("preview_ack") == std::string::npos) + { + return false; + } + picojson::value doc; + return picojson::parse(doc, raw).empty() && doc.is() && text(doc.get(), "type") == "preview_ack"; + } + bool InteractHub::previewWindowOpen(int id) const { std::lock_guard lock{_mutex}; @@ -325,11 +336,8 @@ namespace mbs::ops out.emplace_back(session.peer.sendText, errorJson(code, why)); }; - picojson::value doc; - std::string const parseError = picojson::parse(doc, raw); - bool const valid = parseError.empty() && doc.is(); // Preview acks are flow control, not input: the input limit does not drop them. - if (valid && text(doc.get(), "type") == "preview_ack") + if (isPreviewAck(raw)) { session.previewAcks = true; session.previewUnacked = std::max(0, session.previewUnacked - 1); @@ -352,7 +360,9 @@ namespace mbs::ops } else { - if (!valid) + picojson::value doc; + std::string const parseError = picojson::parse(doc, raw); + if (!parseError.empty() || !doc.is()) { reject("invalid", "bad_message", "not a JSON object"); } diff --git a/src/ops/interact.hpp b/src/ops/interact.hpp index 84da155..feead0a 100644 --- a/src/ops/interact.hpp +++ b/src/ops/interact.hpp @@ -90,12 +90,15 @@ namespace mbs::ops /// Preview flow control (SPEC §7.3): a session that answers previews with `preview_ack` /// has at most kPreviewWindow of them unanswered, so a buffering proxy on its path cannot /// hold seconds of them. False while it has that many. A session that never sent an ack - /// is always open. + /// is always open. Every send counts from the first frame on, so the previews a proxy + /// buffered before the first ack are answered (drained) before new ones go out. [[nodiscard]] bool previewWindowOpen(int session) const; /// Call before a preview goes out to `session`, and previewNotSent when it did not. void previewSending(int session); void previewNotSent(int session); static constexpr int kPreviewWindow = 2; + /// A `preview_ack` message (a size and substring check first, so other input is not parsed twice). + [[nodiscard]] static bool isPreviewAck(std::string const& raw); [[nodiscard]] int sessions() const; [[nodiscard]] bool controlled() const; [[nodiscard]] InteractCounters counters() const; diff --git a/tests/unit/test_httpserver.cpp b/tests/unit/test_httpserver.cpp index 353909e..53fa491 100644 --- a/tests/unit/test_httpserver.cpp +++ b/tests/unit/test_httpserver.cpp @@ -427,11 +427,15 @@ TEST_CASE("WebSocket: a client that stops reading never blocks a send; text stay ::close(bigFd); // Previews held back by flow control (skipBinary) count like a full socket: closed after - // kStuckTimeout without a sent preview; a sent one starts the time again. + // kStuckTimeout without a sent or answered preview; a sent one or an answer (binaryTaken) + // starts the time again. auto [skipFd, skipWs] = connectSmall(); CHECK(skipWs->skipBinary() == WebSocket::Sent::Dropped); std::this_thread::sleep_for(std::chrono::milliseconds(500)); CHECK(skipWs->sendBinary("p") == WebSocket::Sent::Yes); + CHECK(skipWs->skipBinary() == WebSocket::Sent::Dropped); + std::this_thread::sleep_for(std::chrono::milliseconds(500)); + skipWs->binaryTaken(); auto const skipStart = std::chrono::steady_clock::now(); auto skipped = WebSocket::Sent::Dropped; while (skipped == WebSocket::Sent::Dropped && std::chrono::steady_clock::now() - skipStart < WebSocket::kStuckTimeout * 2) diff --git a/tests/unit/test_interact.cpp b/tests/unit/test_interact.cpp index ed6d1fa..71b7e21 100644 --- a/tests/unit/test_interact.cpp +++ b/tests/unit/test_interact.cpp @@ -284,6 +284,44 @@ TEST_CASE("preview flow control: a session that acks has at most two previews un CHECK(a.received.size() == 1); // only the state on open: acks get no answer } +TEST_CASE("preview flow control: previews sent before the first ack are drained first") +{ + // A proxy buffered 9 previews before the client's first ack came back: the window stays + // shut until the client has answered all but one of them, then it holds 2 again. + FakePage page; + InteractHub hub(settings(), page); + Client a; + int const id = hub.open(a.peer()); + for (int i = 0; i < 9; ++i) + { + hub.previewSending(id); + } + hub.message(id, R"({"type":"preview_ack"})"); + for (int i = 0; i < 7; ++i) + { + CHECK(!hub.previewWindowOpen(id)); + hub.message(id, R"({"type":"preview_ack"})"); + } + CHECK(hub.previewWindowOpen(id)); + hub.previewSending(id); + CHECK(!hub.previewWindowOpen(id)); + hub.message(id, R"({"type":"preview_ack"})"); + hub.message(id, R"({"type":"preview_ack"})"); + hub.message(id, R"({"type":"preview_ack"})"); // one too many: nothing unanswered, ignored + hub.previewSending(id); + hub.previewSending(id); + CHECK(!hub.previewWindowOpen(id)); +} + +TEST_CASE("preview flow control: only a short preview_ack message is one") +{ + CHECK(InteractHub::isPreviewAck(R"({"type":"preview_ack"})")); + CHECK_FALSE(InteractHub::isPreviewAck(R"({"type":"text","text":"preview_ack"})")); + CHECK_FALSE(InteractHub::isPreviewAck(R"({"type":"preview"})")); + CHECK_FALSE(InteractHub::isPreviewAck(R"({"type":"preview_ack")")); + CHECK_FALSE(InteractHub::isPreviewAck(R"({"type":"preview_ack","pad":")" + std::string(100, 'x') + R"("})")); +} + TEST_CASE("preview flow control: the input limit does not drop acks") { FakePage page; From 829ee81eabea0564b124516e2c63507904477404 Mon Sep 17 00:00:00 2001 From: Adrian Hilber Date: Fri, 6 Nov 2026 21:34:50 +0100 Subject: [PATCH 3/3] Release 1.4.0: version, image tags, CHANGELOG Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_017ybuzgBaGxuB5UiaE8dRa4 --- .github/workflows/container-codecs.yaml | 2 +- CHANGELOG.md | 2 +- CMakeLists.txt | 2 +- deploy/mxl-browser-source-gpu.yaml | 2 +- deploy/mxl-browser-source.yaml | 2 +- docker/docker-compose.demo.yaml | 2 +- docker/docker-compose.host.yaml | 2 +- docs/api.md | 2 +- 8 files changed, 8 insertions(+), 8 deletions(-) diff --git a/.github/workflows/container-codecs.yaml b/.github/workflows/container-codecs.yaml index 04c65b8..17292fb 100644 --- a/.github/workflows/container-codecs.yaml +++ b/.github/workflows/container-codecs.yaml @@ -1,7 +1,7 @@ name: Container (H.264/H.265) # The image variant with a CEF build that plays H.264 and H.265 (docs/proprietary-codecs.md). -# A tag codecs- (for example codecs-1.3.4) builds that commit and pushes +# A tag codecs- (for example codecs-1.4.0) builds that commit and pushes # -codecs. Normal releases (v*.*.*) are not touched. on: push: diff --git a/CHANGELOG.md b/CHANGELOG.md index a9c266a..80e5746 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog -## Unreleased +## 1.4.0 (2026-11-06) - **Interaction preview stays current on a slow path, also behind a proxy** (SPEC §7.2, §7.3). 1.3.4 limits what the kernel holds unsent, but a reverse proxy between the function and the browser reads everything at once and holds the previews in its own buffers: in the lab (2 Mbit/s behind a buffering proxy, 50 KB previews) they arrived 10–11 s old; at 500 kbit/s the delay grew to 14 s and the session was closed as stuck after 20 s, which ended its control. - **Preview flow control:** a client that answers each preview frame with `{"type":"preview_ack"}` has at most 2 previews unanswered; a new picture skips it until an answer comes (counted in `preview_frames_dropped_total`) and then the newest picture goes out. Previews sent before the first answer count too, so what a proxy buffered then is drained first. A session is closed as `preview_not_taken` only when new pictures were skipped for 5 s without an answer (a static page closes no one). Clients that never send `preview_ack` work as before. The web UI and the widgets answer every frame. `preview_ack` is not counted against the input limit. diff --git a/CMakeLists.txt b/CMakeLists.txt index 9b5b204..6378364 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,7 +1,7 @@ cmake_minimum_required(VERSION 3.24 FATAL_ERROR) project(mxl-browser-source - VERSION 1.3.4 + VERSION 1.4.0 DESCRIPTION "MXL media function: a web page rendered offscreen with CEF" LANGUAGES CXX C ) diff --git a/deploy/mxl-browser-source-gpu.yaml b/deploy/mxl-browser-source-gpu.yaml index 7dbe970..3cc9e59 100644 --- a/deploy/mxl-browser-source-gpu.yaml +++ b/deploy/mxl-browser-source-gpu.yaml @@ -60,7 +60,7 @@ spec: type: RuntimeDefault containers: - name: browser-source - image: ghcr.io/leeo86/mxl-browser-source:1.3.4 + image: ghcr.io/leeo86/mxl-browser-source:1.4.0 env: - name: NMOS_HOST_ADDRESS valueFrom: diff --git a/deploy/mxl-browser-source.yaml b/deploy/mxl-browser-source.yaml index 4ba72d2..bc0ce92 100644 --- a/deploy/mxl-browser-source.yaml +++ b/deploy/mxl-browser-source.yaml @@ -53,7 +53,7 @@ spec: type: RuntimeDefault containers: - name: browser-source - image: ghcr.io/leeo86/mxl-browser-source:1.3.4 + image: ghcr.io/leeo86/mxl-browser-source:1.4.0 env: - name: NMOS_HOST_ADDRESS valueFrom: diff --git a/docker/docker-compose.demo.yaml b/docker/docker-compose.demo.yaml index 389f20a..a3c3bda 100644 --- a/docker/docker-compose.demo.yaml +++ b/docker/docker-compose.demo.yaml @@ -21,7 +21,7 @@ services: - ./registry.json:/home/registry.json:ro browser-source: - image: ${MBS_IMAGE:-ghcr.io/leeo86/mxl-browser-source:1.3.4} + image: ${MBS_IMAGE:-ghcr.io/leeo86/mxl-browser-source:1.4.0} network_mode: host user: "1000:1000" shm_size: 1gb diff --git a/docker/docker-compose.host.yaml b/docker/docker-compose.host.yaml index 4deb461..97bd4f0 100644 --- a/docker/docker-compose.host.yaml +++ b/docker/docker-compose.host.yaml @@ -4,7 +4,7 @@ # GPU: add -f docker/docker-compose.gpu.yaml. services: browser-source: - image: ${MBS_IMAGE:-ghcr.io/leeo86/mxl-browser-source:1.3.4} + image: ${MBS_IMAGE:-ghcr.io/leeo86/mxl-browser-source:1.4.0} network_mode: host user: "1000:1000" shm_size: 1gb diff --git a/docs/api.md b/docs/api.md index 1103e0f..9e5c32d 100644 --- a/docs/api.md +++ b/docs/api.md @@ -127,7 +127,7 @@ Our own routes: No token (the page asks for it). SPEC §10.6. -`GET /widgets` → `[{"id":"interaction","title":"Browser interaction","params":{"type":"object","properties":{}},"min_size":{"w":640,"h":400},"version":"1.3.4"},{"id":"ograf","title":"OGraf graphics","params":{"type":"object","properties":{"layers":{"type":"string",…}}},"min_size":{"w":640,"h":480},"version":"1.3.4"}]`, with `Access-Control-Allow-Origin: ` when `WIDGET_FRAME_ANCESTORS` lists that origin (or `*`) and always `Vary: Origin`. Other methods: 405. +`GET /widgets` → `[{"id":"interaction","title":"Browser interaction","params":{"type":"object","properties":{}},"min_size":{"w":640,"h":400},"version":"1.4.0"},{"id":"ograf","title":"OGraf graphics","params":{"type":"object","properties":{"layers":{"type":"string",…}}},"min_size":{"w":640,"h":480},"version":"1.4.0"}]`, with `Access-Control-Allow-Origin: ` when `WIDGET_FRAME_ANCESTORS` lists that origin (or `*`) and always `Vary: Origin`. Other methods: 405. `GET /widget/interaction[?theme=dark|light|transparent]` and `GET /widget/ograf[?theme=…&layers=1,2]` → the UI page with only the interaction window, or the Graphics tab's preview, layers and form (only the listed layers; no library, upload or delete; `docs/ograf.md` §8), `Content-Security-Policy: frame-ancestors `, no `X-Frame-Options`. A bad `theme`: 400; another id: 404. Other query parameters are ignored.