Outbox dispatch head-of-line-blocks on one long in-flight activity; plain activities run serially (idle worker pool unused)

[JoshuaBearup]

Summary

Two related dispatch/throughput defects, both measured on a running server while dogfooding a real fan-out ingestion pipeline:

1. (severe) Global head-of-line block. A single long-running in-flight activity stalls all outbox dispatch on the node — across every workflow run, not just its own. While one ~12-minute activity was in-flight on one worker, 32 instant activities belonging to a different run sat undispatched on idle workers. This is an availability problem: one big activity freezes the whole node.

2. (scaling) Plain activities execute serially; a worker pool does not help. Plain (non-agent) activities run inline on a single current-thread serve loop in the worker SDK, and the server does not fan a fork across idle workers. Standing up a 12-connection worker pool produced zero parallelism — the giant activities still ran one at a time. Wall-time ≈ sum of activity durations regardless of how many workers are connected.

The two compound: a serial worker plus a head-of-line-blocking dispatcher means one large activity freezes the pipeline, and adding workers cannot recover it.

Context

Dogfooding a document-ingestion workflow authored in AWL: fan out over 32 council-meeting PDFs (fork … join), drive each through fetch → extract → embed → store, fold a manifest back. Several inputs are large scanned "attachment books" (up to ~593 pages); their extract activity is a genuine ~12-minute CPU job. Server aion 0.9.1 (haematite store 0.5.0, liminal 0.3.0), single-node, outbox liminal transport, default namespace.

Evidence (measured, not inferred)

Finding 1 — head-of-line block. With one worker busy on a ~12-min extract and 11 additional idle workers registered on the same (namespace, task_queue, node), a freshly-started run could not dispatch any of its 32 fetch steps (each is a sub-millisecond path+sha256):

09:54:59  fetch=0/32   cpu=98.9%  active=1  deadletters=0
09:55:59  fetch=0/32   cpu=99.2%  active=1  deadletters=0
09:56:39  fetch=0/32   cpu=99.5%  active=1  deadletters=0      <- orig worker (the one giant) KILLED here
09:56:59  fetch=20/32  cpu=0.2%   active=0  deadletters=0
09:57:20  fetch=32/32  cpu=1.7%   active=0  deadletters=0

deadletters=0 throughout — the dispatcher was not retrying/dead-lettering, it was genuinely blocked on acceptance by the one busy worker. The instant that worker was killed, all 32 fetches (on the idle pool) completed in seconds. One long in-flight activity had been starving unrelated work on idle workers.

Finding 2 — serial plain-activity execution. With the 12-worker pool healthy and dispatch unblocked, the giant extract activities still ran strictly one-at-a-time — one core busy, eleven idle:

09:57:40  extract=3/32   cpu=7.3%   active=0
09:58:20  extract=6/32   cpu=15.1%  active=0
09:59:00  extract=7/32   cpu=23.3%  active=1
10:01:01  extract=7/32   cpu=43.6%  active=1   <- one giant grinding on one worker; 11 idle

The fork over 32 extractions never fanned out. (active = worker processes above 20% CPU.)

Root cause

Worker SDK (certain, code-grounded). crates/aion-worker/src/runtime/liminal.rs:606-615: an agent dispatch is spawned ("MUST NOT block the serve loop"), but a plain activity runs inline — self.execute(&request).await directly in handle_pushed_frame, which the serve loop (serve_until_drop) awaits before receiving the next frame. That loop is block_on a current-thread runtime (runtime/liminal_serve.rs:104 new_current_thread, :142 block_on). Net: one plain activity per connection at a time. WorkerConfig::max_concurrency is set by the worker but is not consulted anywhere on the liminal plain-activity path — no spawn, no semaphore — so raising it changes nothing for plain activities (it is honoured for agent activities, which spawn).

Server dispatch (observed). The outbox claim/dispatch loop does not make progress on other pending rows / idle workers while a dispatch is awaiting acceptance by a busy worker — hence Finding 1. The cluster max_in_flight_activities ceiling is not the limiter here (≥32 activities flushed at once the moment the block cleared). I did not line-trace the server dispatch loop end-to-end, so treat the server half as a characterised symptom + hypothesis rather than a pinned line.

Proposed

• Worker SDK: run plain activities the way agent activities already run — spawn each under a max_concurrency semaphore instead of inline-awaiting on the serve loop — so a connection can execute up to its advertised concurrency and the serve loop stays free. (Alone, this fixes both findings for a single worker.)
• Server dispatcher: ensure one un-accepted / long-running dispatch cannot head-of-line-block dispatch of other rows to other (idle) workers — dispatch acceptance should not serialise the claim loop, and a fork should be free to fan across idle workers in the pool.

Severity

Finding 1 is severe for multi-tenant / mixed-workload use: any single long activity freezes dispatch for every run on the node. Finding 2 caps throughput at one activity's worth of work at a time regardless of worker count. Neither loses data — the run still completes durably and correctly, just serially — so not a correctness bug; an availability + scaling bug. Surfaced as a dogfooding finding; happy to share the full worker/server logs or the AWL pipeline if useful.

[JoshuaBearup]

Two more findings from the same pipeline, measured on a follow-up run (a scoped 18-doc batch), plus one positive:

Lost-worker-mid-activity is classified Terminal and fails the workflow (G-E) — should be retryable.
When a worker holding an in-flight activity dies, the engine records the failure with a retryable:-prefixed message but kind: Terminal, and fails the whole workflow — even though healthy workers are connected and could re-run it. Observed: the run's final aggregate activity landed on a worker that died mid-activity; the run went Failed at aggregate although all upstream store activities had already succeeded durably. The recorded failure:

{"kind":"Terminal","message":"retryable:worker WorkerId(4) lost before reporting activity result"}

followed by the workflow-level:

{"details":{"message":"activity failed","tag":"AwlActivityFailed"},"message":"workflow 4 returned error"}

A worker dying mid-activity is the canonical retryable condition (re-dispatch to another worker), not terminal. Suggest: classify lost-worker as retryable and re-dispatch to a healthy worker in the pool.

Positive: aion reopen recovers it cleanly. aion reopen <wf> re-drove the failed run and it reached Completed using the already-durable upstream results — a proven recovery path for this failure mode.

aion cancel leaves already-claimed outbox rows cycling as zombies.
Cancelling a workflow marks it cancelled, but its already-claimed outbox rows keep being retried by the dispatcher until they exhaust the attempt budget and dead-letter. A cancelled run's giant extract rows kept re-running (each a fresh long attempt) and — via the head-of-line block in the issue above — starved a different, live run's store stage until they finally dead-lettered minutes later:

WARN outbox dispatch failed; scheduling retry ... dispatch_key:"<cancelled-wf>:59" attempt:3
WARN outbox dispatch exhausted retry budget; dead-lettering row ... dispatch_key:"<cancelled-wf>:59" attempt:4

Suggest: cancel should promptly abandon a run's pending/claimed dispatch rather than leave zombie rows cycling (which, combined with the HOL block, starve live runs).

Happy to split the lost-worker item into its own issue if you'd prefer it tracked separately — grouping here since all three surfaced from the same dispatch/worker-lifecycle investigation.