Skip to content

postgres_cdc: add signalling support#4616

Open
josephwoodward wants to merge 5 commits into
mainfrom
jw/postgres_cdc_signal_support
Open

postgres_cdc: add signalling support#4616
josephwoodward wants to merge 5 commits into
mainfrom
jw/postgres_cdc_signal_support

Conversation

@josephwoodward

Copy link
Copy Markdown
Contributor

This change introduces signalling support into the PostgreSQL CDC connector allowing you to perform operations such as signal a new snapshot on an adhoc basis. It does this via a custom created signal table which is also being tracked by PostgreSQL's replication functionality, with control signals propagating from the WAL. When a signal is detected the signaller we determine the signal type then handle it accordingly. Currently only snapshots are supported but other signal types can added over time (such as logging).

Upon receiving a signal event, the process will:

  1. Received snapshot signal, setting the state to re-snapshot on restart then pauses streaming and performs snapshotting of the table(s) specified in the signal payload (it can be all tables, or just new tables).
  2. Once snapshotting is completed the connector resume streaming from its original cached checkpoint.

Demonstration

Given the following config:

input:
  postgres_cdc:
    dsn: ${PG_DSN:postgres://postgres:postgres@localhost:5432/testdb?sslmode=disable}
    stream_snapshot: true
    schema: public
    tables:
      - users
      - products
      - cart
    slot_name: bench_slot
    signal_table_name: rpcn_signal_table
    batching:
      period: 1s

output:
  file:
    path: "./benchmark_results.json"

In the screenshot below the process goes:

  1. Launch the component and stream an initial 100 records into benchmark_results.json
  2. Run task psql:signal:snapshot to publish a snapshot signal to the configured signalling table which the connector receives, pauses streaming then restarts to snapshot (now 201 records, 100x2 plus 1 signal event)
  3. Connector returns to streaming (301 records)
image

postgres_cdc: update tests to verify singal event

postgres_cdc: wait for messages to be acked before restarting
@josephwoodward
josephwoodward marked this pull request as ready for review July 21, 2026 17:26
Comment on lines +269 to +271
// confirmedLSNFromDB is unchanged from before the snapshot - the
// triggering signal was never acked, so resuming here redelivers
// it. The input layer recognizes and drops that redelivery.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment describes crash-recovery behavior that does not match the implementation, and contradicts the signal-handling path.

It states the signal "was never acked, so resuming here redelivers it. The input layer recognizes and drops that redelivery." But:

  1. No drop/dedup logic exists. postgresSignaller.Listen keeps no record of previously-processed signals, so a redelivered signal row would simply be re-detected and re-StoreSignaled, triggering another re-snapshot. There is nothing that "recognizes and drops that redelivery."
  2. It contradicts the ack path. The signal handler explicitly acks the signal's LSN before restart precisely so it is never redelivered — see input_pg_stream.go#L634-L639: "its LSN is durably confirmed to Postgres before we restart - it will never be redelivered, so there's no redelivery loop to guard against."

Please reconcile the two: if the signal is durably acked (as input_pg_stream.go does), then confirmedLSNFromDB here is not "unchanged" and this redelivery reasoning is misleading. If the signal can still redeliver (e.g. a crash before the standby status update reaches Postgres), the "recognizes and drops" mechanism the comment relies on needs to actually exist, otherwise the redelivered signal re-triggers a snapshot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant