Skip to content

Wait in basic_publish while the connection is blocked - #76

Open
carlhoerberg wants to merge 2 commits into
mainfrom
wait-while-blocked
Open

carlhoerberg wants to merge 2 commits into
mainfrom
wait-while-blocked

Conversation

@carlhoerberg

Copy link
Copy Markdown
Member

Why

The client already handled connection.blocked: it logged it, set blocked? and called on_blocked. But basic_publish kept publishing.

  • RabbitMQ: a blocked server stops reading from publishing connections, so publishes stall on a full socket.
  • LavinMQ: it rejects publishes during flow control (low disk space, and memory pressure in Release memory under systemd memory pressure lavinmq#2307) with 406 PRECONDITION_FAILED, which closes the channel. In a test under a cgroup memory limit, lavinmqperf got connection.blocked and then had its channel closed about 20 ms later on the next publish.

What

basic_publish now waits while the connection is blocked:

  • On connection.blocked: the connection creates a channel that publishers wait on.
  • On connection.unblocked: it closes that channel, waking every waiting publisher at once.
  • On close: both close paths (the read loop ending, and Connection#close) also wake waiters, so a publisher never waits on a dead connection. They wake waiters last, after the channels are cleaned up. A woken publisher then behaves like any publish on a closed connection: after a server connection.close it raises, and after a transport failure the publish is dropped, as before.
  • When not blocked: it costs one ivar read per publish.
  • API: Connection#wait_while_blocked is public, for code that wants the same wait before doing other work.

The blocked/unblocked frame handling moved into process_blocked/process_unblocked, so read_loop isn't more complex than before.

Compatibility

Callers that published while blocked now wait instead. Against RabbitMQ that changes where they wait (in basic_publish instead of on the socket), not whether they wait.

Testing

spec/blocked_spec.cr uses a minimal fake AMQP server, since a real server can't be made to send connection.blocked on demand. It covers:

  • waiting until unblocked
  • not waiting when not blocked
  • stopping waiting when the connection closes while blocked

Full suite against a local LavinMQ: 94 examples, 1 failure. The failure is Error Handling Network errors wraps connection refused, which fails the same way on main with Crystal 1.21 ("Operation now in progress" instead of "Connection refused"), so it's unrelated.

🤖 Generated with Claude Code

https://claude.ai/code/session_011ob4Nr8aeNr1kj3PgMosUS

The client recorded connection.blocked and called on_blocked, but
basic_publish kept publishing. Against RabbitMQ that only stalls on a
full socket, but LavinMQ rejects publishes during flow control with a
channel error, so blocked publishers lost their channel.

basic_publish now waits until connection.unblocked, or until the
connection closes. Waiters are woken last on close, after the channels
are cleaned up, so they then behave like any publish on a closed
connection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011ob4Nr8aeNr1kj3PgMosUS
@carlhoerberg
carlhoerberg requested a review from a team as a code owner October 2, 2026 15:58
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011ob4Nr8aeNr1kj3PgMosUS
@carlhoerberg

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-06T18:30:53.730572Z a87592c Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a87592cb02

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Log.info { "Blocked by server, reason: #{f.reason}" }
@unblocked ||= ::Channel(Nil).new
@blocked = true
@on_blocked.try &.call(f.reason)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Invoke the blocked callback outside the read loop

When an on_blocked callback calls basic_publish (or the newly public wait_while_blocked), it waits on @unblocked from inside the connection's read_loop. That same fiber is therefore unable to read the subsequent connection.unblocked or close frame that would release the wait, permanently deadlocking the connection. Dispatch the callback outside the read-loop fiber or otherwise ensure callbacks cannot wait on an event only that loop can process.

Useful? React with 👍 / 👎.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant