[olivermt]

Stacked on #14; watchdog-only diff.

Adds a configurable watchdog that detects silent replication connections and recovers from the durable checkpoint. Healthy idle streams, intentional pauses and snapshots are handled without false timeouts.

Validated with real connection-failure tests on PostgreSQL 15 and 18, the full PG18 suite, and static checks.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

CodeRabbit Verify

• Run CodeRabbit Verify to test the changes in this pull request.

[palermo-git]

Nice, the watchdog-only diff is smaller than I expected, and the way you handle pauses looks right to me. I've only read it so far, not run it, so tell me if I've got something wrong.

One thing worries me. When the watchdog fires and we reconnect, Postgres can still have the old walsender sitting on the slot. As far as I can tell, START_REPLICATION then fails because the slot's active, each failure burns max_command_retries, and Postgrex reconnects straight away. Five quick failures and you get a permanent {:command_error, :exhausted} halt, which is a rough ending for something meant to recover on its own.

It should only bite when wal_sender_timeout is longer than receive_timeout, or turned off. At the default 60s it's a race the server probably wins, which may be why you haven't seen it. The test wouldn't catch it either, since blackhole closes the server side and frees the slot right away. If you leave that socket open, with the wal_sender_timeout: "0" the test already sets, I'd expect it to show up.

There's a similar case on a healthy connection. In PG17's walsender, replies only get read every wal_sender_timeout / 2 while it's decoding a big transaction the publication filters out. With a 5 minute setting, that's 150s of silence on a perfectly fine stream, and then the reconnect hits the same active slot. I only looked at 17, so older versions may differ.

What I'd want:

• a test that leaves the server side open
• paced retries while the slot is still active, without spending the command budget
• a check of wal_sender_timeout on connect, so we warn or adjust if receive_timeout doesn't exceed it

I'm hesitant to default this on for everyone until that's sorted.

Couple of small things: the new README section ended up below the license line, and usage-rules.md and the Livebook need receive_timeout and the new events. On #14, the FlowControl moduledoc fixes the docs warning but makes an internal module public, so I'd rather fix that another way.

Also, did you see my last comment on #14? The stall signal and the subtransaction check are still open, and the #15 README says pause telemetry stays the stalled-sink alert, so I want to make sure we're on the same page. The test jobs all passed on e71ae1e, which was good to see.