Fire-and-forget messages poison the per-conversation reply FIFO → wrong-reply delivery on a later request

[ijnebzor]

Summary

Fire-and-forget conversation messages (sent without the reply-requested flag) leave their participant replies in the conversation's single shared reply FIFO. A subsequent reply-requested request on the same conversation then gets mis-correlated to that earlier fire-and-forget reply — the request returns the wrong payload, and its own true reply is silently discarded.

This is the exact "reply mis-correlated to a newer request" corruption the tombstone design (docs/READINESS-CONSUMER-DESIGN.md:107) sets out to prevent — but that guard only covers timed-out/late replies, not replies triggered by non-reply-requested messages.

Severity

High — silent semantic corruption, no error surfaced to either side, reachable through the ordinary public SDK on a single conversation handle. The mis-correlation is deterministic.

Root cause

The server's reply-correlation model assumes inbox replies are 1:1 with reply-requested pending entries. Any participant that emits a reply for a non-reply-requested message breaks that assumption, and match_reply correlates positionally (front of FIFO → oldest pending entry) with no ownership/op-id check.

• Fire-forget branch (crates/liminal-server/src/server/connection/apply.rs:571-580): forwards the message via the same services.conversation_message(...) path as a reply-requested frame, then returns NoResponse — admits no pending entry.
• Participant delivery (crates/liminal/src/conversation/participant.rs run_slice + crates/liminal/src/conversation/actor/core.rs:401 deliver_participant_reply): the participant runs behaviour.process(request) on every forwarded request and pushes any produced reply into the conversation's single shared inbox (VecDeque). behaviour.process sees only the Envelope — it has no access to the reply-requested flag, so a participant cannot self-gate its reply. EchoBehaviour replies to all.
• Drain (crates/liminal-server/src/server/connection/process.rs service_pending_replies): visits only conversations that have a pending entry, then drains the inbox FIFO, correlating each reply positionally via match_reply.
• Correlation (crates/liminal-server/src/server/connection/pending_reply.rs match_reply): binds the incoming reply to oldest_index_for(conversation_id) — the front-most pending entry — and never inspects the reply envelope for an op/correlation id.

Reproduction (single connection, single conversation C)

1. Open conversation C.
2. handle.send(C, "A") — no reply-requested flag. Server forwards A; admits no pending entry. Participant echoes → reply "A" pushed onto C's single reply FIFO. The drain only visits conversations with a pending entry, so "A" is never drained and sits in the queue.
3. handle.request(C, "B") — with the flag. Admits pending op_id=1; forwards B; echo → "B" appended after "A".
4. Drain now runs for C: pops "A" first (FIFO), match_reply binds it to the oldest pending entry (B's) → returns a reply carrying "A" on B's stream. request(reqB) unblocks with the wrong payload. Then pops "B", finds no pending entry, and discards the real reply.

Result: request(reqB) returns "A"; the correct reply to B is silently lost. Deterministic (the participant processes its inbox FIFO, so "A" is always ahead of "B").

Secondary corollary (same root cause)

Conversations that only ever receive fire-and-forget messages never get a pending entry, so their replies are never drained → unbounded inbox growth.

Suggested fix

Separate the reply-correlation channel from fire-and-forget processing: tag each forwarded request with its admitted op_id (or a "reply-expected" marker) and bind the incoming reply to that op_id rather than to FIFO position; replies to fire-and-forget messages must be discarded at source (or routed to a distinct sink), not fed into the request-reply inbox. This also resolves the unbounded-growth corollary.

Notes

Verified against ablative-io/liminal @ fe3f7ec. Not a duplicate (gh issue list --state all empty). Found during a review sweep; correlation logic independently traced end-to-end.