Skip to content

Commit e85899d

Browse files
authored
fix(connection): detect SSL-layer EOF and match PG::ConnectionBad by class (mensfeld#99)
`connection_lost_error?` relied exclusively on message substrings, so a transient Postgres drop that surfaces at the SSL layer as `PQconsumeInput() SSL error: unexpected eof while reading` bypassed the single-retry path in `with_connection` and propagated as `PGMQ::Errors::ConnectionError` even though the connection would recover on the next checkout. Two complementary changes: 1. Match by class (`PG::ConnectionBad`, `PG::UnableToSend`) first. These are libpq's dedicated connection-failure classes; class-matching catches future OS/pooler/TLS variants without enumerating every message. 2. Add `"ssl error: unexpected eof"` and `"ssl syscall error"` to the substring list for the bare `PG::Error` cases. Observed in production behind a managed Postgres pooler: three unrelated controllers hit the same error in a six-minute window after an idle SSL teardown, then recovered. An earlier cluster with `PQsocket() can't get socket descriptor` (already in the list) was retried successfully, confirming the machinery works and only the pattern was missing. Refs mensfeld#98
1 parent b6922c5 commit e85899d

3 files changed

Lines changed: 76 additions & 18 deletions

File tree

CHANGELOG.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,11 @@
11
# Changelog
22

3+
## Unreleased
4+
5+
### Connection Management
6+
- **[Fix]** `PGMQ::Connection#connection_lost_error?` now detects SSL-layer teardown errors (`"PQconsumeInput() SSL error: unexpected eof while reading"`, `"SSL SYSCALL error: EOF detected"`). Observed in production behind a managed Postgres pooler: an idle connection torn down at the SSL layer caused the next enqueue to raise `PGMQ::Errors::ConnectionError` without triggering `with_connection`'s single-retry path, because the message didn't match any of the existing substrings.
7+
- **[Fix]** `PGMQ::Connection#connection_lost_error?` now also matches by class (`PG::ConnectionBad`, `PG::UnableToSend`) in addition to message substrings. These are dedicated connection-failure classes libpq raises when the socket is dead; class-matching catches future OS/pooler/TLS message variants without waiting for them to hit production.
8+
39
## 0.6.1 (2026-04-16)
410

511
### Connection Management

lib/pgmq/connection.rb

Lines changed: 32 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -107,28 +107,42 @@ def stats
107107

108108
private
109109

110-
# Checks if the error indicates a lost connection
110+
# Messages libpq raises when the server/pooler has already torn down the
111+
# socket. The list has grown organically with each pooler/TLS variant we
112+
# see in the wild; the class check below catches future variants that
113+
# libpq raises as `PG::ConnectionBad` or `PG::UnableToSend` without
114+
# waiting for a new message to hit production.
115+
LOST_CONNECTION_MESSAGES = [
116+
"server closed the connection",
117+
"connection not open",
118+
"connection is closed",
119+
"connection has been closed",
120+
"no connection to the server",
121+
"terminating connection",
122+
"connection to server was lost",
123+
"could not receive data from server",
124+
"pqsocket() can't get socket descriptor",
125+
"ssl error: unexpected eof",
126+
"ssl syscall error"
127+
].freeze
128+
private_constant :LOST_CONNECTION_MESSAGES
129+
130+
# Checks if the error indicates a lost connection.
131+
#
132+
# Matches in two steps: first by class (`PG::ConnectionBad` /
133+
# `PG::UnableToSend` are dedicated connection-failure classes libpq
134+
# raises regardless of message), then by message substring for the
135+
# bare `PG::Error` cases where libpq doesn't reach for the specific
136+
# subclass.
137+
#
111138
# @param error [PG::Error] the error to check
112-
# @return [Boolean] true if connection was lost
139+
# @return [Boolean] true if the connection was lost and a retry on a
140+
# fresh connection is appropriate
113141
def connection_lost_error?(error)
114-
# Common connection lost errors. Include the pg-gem C-extension message
115-
# ("PQsocket() can't get socket descriptor") that is raised when the
116-
# cached libpq socket descriptor is gone — e.g. after a server-side
117-
# close by a connection pooler such as PgBouncer.
118-
lost_connection_messages = [
119-
"server closed the connection",
120-
"connection not open",
121-
"connection is closed",
122-
"connection has been closed",
123-
"no connection to the server",
124-
"terminating connection",
125-
"connection to server was lost",
126-
"could not receive data from server",
127-
"pqsocket() can't get socket descriptor"
128-
]
142+
return true if error.is_a?(PG::ConnectionBad) || error.is_a?(PG::UnableToSend)
129143

130144
message = error.message.to_s.downcase
131-
lost_connection_messages.any? { |pattern| message.include?(pattern) }
145+
LOST_CONNECTION_MESSAGES.any? { |pattern| message.include?(pattern) }
132146
end
133147

134148
# Verifies a connection is alive and working.

test/lib/pgmq/connection_test.rb

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -282,6 +282,44 @@
282282

283283
refute connection.send(:connection_lost_error?, error)
284284
end
285+
286+
it "matches SSL-layer EOF raised when libpq's read returns 0 on a dead socket" do
287+
# Observed in production behind a managed Postgres pooler: an idle
288+
# connection is torn down at the SSL layer and the next I/O raises
289+
# this exact message. Semantically identical to "could not receive
290+
# data from server" (which is already covered), just one layer down.
291+
error = PG::ConnectionBad.new(
292+
"PQconsumeInput() SSL error: unexpected eof while reading"
293+
)
294+
295+
assert connection.send(:connection_lost_error?, error)
296+
end
297+
298+
it "matches SSL SYSCALL teardown errors" do
299+
error = PG::ConnectionBad.new(
300+
"SSL SYSCALL error: EOF detected"
301+
)
302+
303+
assert connection.send(:connection_lost_error?, error)
304+
end
305+
306+
it "matches PG::ConnectionBad by class, even when the message is opaque" do
307+
# libpq doesn't promise a stable message for every pooler/OS/TLS
308+
# combination, but it does raise a dedicated class for connection
309+
# failures. Class-matching catches future message variants without
310+
# growing the string list.
311+
error = PG::ConnectionBad.new("")
312+
313+
assert connection.send(:connection_lost_error?, error)
314+
end
315+
316+
it "matches PG::UnableToSend by class" do
317+
# Raised by libpq when the write side of the connection is gone —
318+
# another dedicated connection-failure class.
319+
error = PG::UnableToSend.new("")
320+
321+
assert connection.send(:connection_lost_error?, error)
322+
end
285323
end
286324

287325
describe "verify_connection! (private)" do

0 commit comments

Comments
 (0)