fix: bound domain timeouts, wire limits, and RPC stream delivery

[smiggleworth]

Summary

This branch grew well past its original scope. It now contains five largely independent
units; the last section suggests how to review (or split) them.

1. Session isolation — a slow domain no longer closes the connection

• ingress classified DeliveryError::Timeout as a fatal route error and answered
  IngressDecision::Close, so one saturated Queue actor tore down the whole multiplexed
  WebSocket along with unrelated KV/Stream/Schedule/RPC traffic. A domain that does not
  answer in time now returns an error frame on that channel and the session survives.
  Verified for all seven domains.
• Queue, Stream and RPC now distinguish a busy-but-alive actor (Timeout) from a
  stopped one (ActorStopped). RPC previously collapsed both into ActorStopped.
• the three byte-identical copies of that mapping are consolidated into
  src/runtime/reply_wait.rs.
• read authenticated WebSocket state from the canonical ingress registry so the CONNECT
  deadline cannot close an already-authenticated session.
• session-cleanup tickets now give up on a bounded per-ticket age rather than an attempt
  count. The counter assumed compounding backoff, but the delay is worker-global and reset
  whenever any other ticket succeeded, so under normal churn a ticket was abandoned in
  ~80 ms instead of the documented ~2.3 s, leaking its subscriptions and inflight leases.
  New fitz_session_cleanup_permanent_failures_total.
• send_unit_actor_command returns its outcome; callers previously could not tell a
  completed command from one that timed out.

2. Wire limits — no response can panic the broker

• encode_single_tlv_frame no longer asserts. A TLV value carries a u16 length, and
  asserting on it turned any aggregate-overflow bug in any domain into a broker panic. It
  now returns DeliveryError::InvalidPayload.
• Schedule LIST was the live instance: limit = 0 means "all remaining" and nothing
  bounded bytes, so 250 schedules with 1 KiB payloads produced a 270 KB response and
  panicked the outbound sink. Both list_entries and list_entries_v2 are now bounded by
  a wire budget; list_entries_v2 capped entry count but never bytes.
• schedule definitions that could never appear in a LIST response are refused at create and
  batch-create. A CREATE arrives as one TLV value, so its payload can be ~140 bytes larger
  than the same definition costs as a list entry — accepting one left a schedule that fired
  normally but could never be listed.
• Queue reserve replies are bounded before messages transition to inflight, including
  wildcard reserves, and a message deliverable by a concrete reserve is no longer
  dead-lettered by a wildcard one (the budgets differed by the routing envelope).
• Stream READ budgets every item against its real per-item wire cost. A filter-excluded
  record is charged the cheap Filtered marker rather than its full Event cost, so a body
  the client never receives can no longer stop or fail a page. A single record that alone
  exceeds the ceiling still fails loudly with ERR_READ_RESPONSE_TOO_LARGE (2013) rather
  than being skipped — Stream guarantees exact replay, so silently dropping a committed
  event from a rebuilt aggregate is worse than a classifiable error.

3. RPC streaming — never drop a frame and continue the sequence

Under outbound backpressure the broker advanced the response sequence before delivery,
dropped the frame, and kept forwarding later ones, so callers saw sequence gaps
(124 != 123, 300 != 299) across a 100,000-frame / 6.1 GiB run while diagnostics
reported zero failures.

• pending_for_response_in_family no longer mutates on lookup — it neither advances
  next_expected_seq nor drops the request on stream_end.
• forwarding happens first, then commit-or-record-failure. A failed chunk leaves the cursor
  put so the worker can resend the same sequence; after MAX_RESPONSE_DELIVERY_ATTEMPTS
  the RPC terminates with an error to both caller and worker (the worker cancel is what
  stops late-response amplification).
• this also fixes the terminal-chunk case: a failed stream_end previously left the caller
  hanging until timeout, because the request was removed before the forward was attempted.
• outbound send retries were 100 yield_now calls — microseconds, not waiting. Now 8
  yields then bounded escalating sleeps.

4. Telemetry — health can no longer read green through a fault

• Queue had no transport failure counter; a response the actor produced but the
  transport could not deliver was logged and counted as a success. Added
  fitz_queue_response_route_failures_total.
• RPC wrote backpressure counters nothing read: the admin field reads one name, while the
  inline dispatch path and admission control incremented two others. Converged.
• transport_pressure was computed and used only in a hint string. It now drives a new
  TransportBackpressure diagnosis and counts toward failure_count.
• RPC diagnostics no longer describe ephemeral response loss as a durability gap.
• CONNECT-failure diagnostics are rate-limited: each failure ran a SHA-256, a base64 decode
  and a JSON parse of an attacker-controlled payload for an unauthenticated peer.

5. Schedule delivery distribution

round_robin_cursors is keyed per route, so a first-time route hit unwrap_or(0) and every
one-shot Single schedule chose the same subscriber, concentrating a fleet's load on one
client. Unseen routes now seed from the route hash.

Client-visible contract changes

• New error codes. schedule::ERR_BACKEND_ERROR (7010) — Schedule was the only domain
  without a generic backend code, and borrowing ERR_PARSE_ERROR would tell a client its
  cron was malformed when the broker was merely busy. schedule::ERR_TIMEOUT (7011) —
  deliberately not retryable; see below.
• Timeouts answer with non-retryable codes in every domain. A timed-out command was
  already enqueued and may still execute, and only queue ACK is deduplicated — so an
  automatic retry of a SEND double-enqueues. REQ-PROTO-012 classifies codes as retryable
  or fatal, with no third state for "outcome unknown", and REQ-ERR-006 pushes SDKs to a
  boolean IsRetryable helper that erases any prose caveat. Timeouts therefore use codes
  outside the retryable set: 1009 / 4007 / 3005 / 2012 / 5006 / 6010 / 7011. Notably RPC
  uses its backend code rather than ERR_RPC_TIMEOUT (6001), which is documented retryable.
• New DeliveryError::InvalidPayload variant. DeliveryError is pub and re-exported
  from crate::runtime, so this is technically breaking for external consumers.
• Behavior change: a domain timeout no longer closes the session. No spec currently
  states this, and clients will come to depend on it — it should be written down.

Dependency

cntryl-midge moved from 49442e6 to e04ecb4 (lockfile only; branch = "main"
unchanged). Of the code commits in that range, #255 adds bounded late-response
diagnostics — request kind, configured timeout, abandonment age, response variant, pending
depth — which is what makes unmatched-response warnings attributable at all, and #256
bounds synchronous runtime waits.

Two consequences absorbed here:

• MidgeError::Timeout now arrives where midge previously blocked indefinitely. The variant
  already existed, so nothing forced handling it; KV classified midge errors by message text
  and dropped it into a permanent BackendError. Typed variants are now matched first.
• midge's runtime_response_timeout (default 60 s, floored at storage_io_timeout + 30 s)
  is set explicitly at Engine open, so the relationship between the two budgets is visible
  in code. Fitz's 1 s domain deadlines sit far below it and always fire first — which is
  only safe because a timeout is now a retryable-free error frame rather than a session
  close.

Validation

• cargo fmt --all -- --check
• cargo test --workspace — 2,082 tests across 32 binaries, 0 failures
• cargo clippy --locked --workspace --all-targets --all-features -- -D warnings -D clippy::pedantic
• every fix added a test red-first: the panic reproduced at 270,260 bytes; the RPC gap
  reproduced as [0] where [0, 1] was expected; the cleanup ticket abandoned at 40 ms
  against a 2.3 s window; the session close reproduced as
  Close("route delivery failed: ... Delivery timed out") for all seven domains.
• the timeout-code guard asserts against the literal REQ-PROTO-012 retryable set rather
  than a hand-picked list, because an earlier hand-picked version missed both 6001 and 7010.

Known gaps

Stated plainly rather than left for a reviewer to find:

• No queue-only 8-client / 60 s session-survival test. This is the regression guard for
  the headline fix and it does not exist. benches/tier4_queue_concurrency.rs already runs
  8 clients over real transports but is iteration-driven and discards results.
• No end-to-end fault-injection test asserting admin failure/backpressure counters go
  nonzero — the exact regression that produced the false green.
• Actor reply timeouts are still a hardcoded 1 s (queue/actor/mod.rs,
  rpc/sink/state_model/constants.rs) rather than configurable, and are now load-bearing.
• Five constants are unmeasured judgment calls: the RPC delivery attempt budget, the
  outbound backoff schedule, STORAGE_RUNTIME_RESPONSE_TIMEOUT, and the two 1 s reply
  deadlines. Together they decide how long a saturated caller gets to drain.
• Enqueue has no idempotency. Non-retryable codes stop automatic retries; they do not
  make a retried SEND safe. Dedup or cancellation on the enqueue path is the real cure.
• ERR_QUEUE_FULL (4005) is still emitted by nothing while remaining documented as a queue
  code. Pre-existing; this branch touched the area and left it.
• fitz-ts has a matching client-side defect (a fixed 1,024-item handler queue, and a
  response writer that advances the sequence before encoding succeeds). Separate repo.

Review guidance

82 files, +4,012 / −340 is too large to review well as one unit. The five sections above are
close to independently reviewable, and two are low-risk enough to split off and land first:

• section 2 (the panic fix) is self-contained and unambiguous;
• the midge bump is lockfile-only and already green.

Section 1 deserves its own PR — it is the one with live protocol blast radius (a
behavior change plus the retry-taxonomy question above), and it is the one that should not
merge without the 60 s acceptance test.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.