A production-style backend that guarantees a webhook event is processed exactly once, even when the sender retries delivery, network calls fail halfway, or a worker crashes mid-processing.
Built to simulate a real-world scenario: a payment provider (Stripe-style)
sending payment_intent.succeeded webhooks, where duplicate delivery is
common and duplicate processing (double-charging a customer, creating
two orders) is not acceptable.
Webhook providers (Stripe, PayPal, GitHub, etc.) do not guarantee exactly-once delivery — they guarantee at-least-once. A slow response, a network blip, or a provider-side retry policy means the same event can legitimately arrive twice, three times, or more. A backend that isn't built to handle this will double-charge a customer, create duplicate orders, or send duplicate confirmation emails — a correctness bug that's invisible in a demo and expensive in production.
This system solves that at two levels: a fast in-memory check for the common case, and a hard database-level guarantee underneath it for the case that actually matters.
- Node.js / Express — webhook receiver
- RabbitMQ — durable queue between "received" and "processed", plus retry/dead-letter routing
- Redis — fast-path deduplication cache
- PostgreSQL — source of truth for deduplication and the actual business data
- Docker Compose — local infra for all three services
POST /webhooks/payment
|
v
┌───────────────────┐
│ Express receiver │ validates shape,
│ (webhook-server) │ publishes, returns fast
└─────────┬──────────┘
|
v
┌───────────────────┐
│ main_exchange │ (direct exchange)
└─────────┬──────────┘
| routing key: events
v
┌───────────────────┐
│ events.main │ durable queue
└─────────┬──────────┘
|
v
┌───────────────────┐
│ Worker │
└─────────┬──────────┘
|
┌─────────────┴─────────────┐
v v
┌─────────────────┐ ┌─────────────────────┐
│ Redis SET NX │ new? │ Postgres │
│ (fast dedupe) │ ──────> │ transaction: │
└─────────────────┘ │ - INSERT dedupe row │
│ - INSERT payment row│
│ (ON CONFLICT DO │
│ NOTHING on │
│ event_id) │
└──────────┬────────────┘
|
fails? ──┘
v
┌─────────────────────┐
│ events.retry │
│ (5s TTL, no │
│ consumer, DLX │
│ back to main) │
└──────────┬────────────┘
| after 3 attempts
v
┌─────────────────────┐
│ events.dlq │
│ (manual inspection) │
└─────────────────────┘
Redis alone isn't safe on its own — if Redis restarts, gets flushed, or
a key's TTL expires before a provider's retry arrives, a duplicate can
slip through undetected. Redis is a fast-path optimization: an
atomic SET key val NX EX ttl check that resolves in under a
millisecond for the overwhelming majority of requests.
Postgres is the actual source of truth. event_id is a PRIMARY KEY on the processed_events table, so a duplicate insert is rejected
by the database engine itself — not by application logic that could
have a bug. If Redis ever says "new" incorrectly (e.g. after a cache
flush), Postgres still catches it before any business side effect
happens.
Together: sub-millisecond checks for the common case, with a hard database-level guarantee behind it for the case that actually matters.
If "mark event as processed" and "create the payment record" were two separate writes, a crash between them creates a permanent inconsistency: either a payment with no dedupe record (so the next retry double-processes it), or a dedupe record with no payment (so a real payment silently disappears).
Both writes happen inside a single Postgres transaction
(BEGIN / COMMIT), keyed off the same event_id unique constraint.
Either both rows exist, or neither does. There's no window where the
system can be caught half-done.
Nacking a message straight back onto the same queue creates a tight loop that hammers a possibly-still-broken dependency (e.g. the DB is down) with zero backoff.
Instead, a failed message is republished to events.retry — a queue
with a fixed 5-second TTL and no consumer. Its only job is to hold
the message until it expires, at which point RabbitMQ's dead-letter
mechanism automatically routes it back into events.main for another
attempt. This gives delayed, broker-native retries without an external
scheduler. After 3 failed attempts, the message is routed to
events.dlq instead — a queue nothing auto-consumes, meant for manual
inspection rather than infinite silent retry.
The worker uses channel.prefetch(1) and manually acks every message
(channel.ack(msg)) only after it's genuinely done with it — success,
duplicate-skip, or handed off to retry/DLQ. Auto-ack would tell
RabbitMQ "delivered" the instant the message left the broker, before
processing even starts — meaning a worker crash mid-processing would
lose the message permanently instead of it being redelivered.
- Node.js 18+
- Docker Desktop (or Docker Engine + Compose)
git clone https://github.com/Sahoo999/idempotent-event-system.git
cd idempotent-event-system
docker-compose up -d # starts RabbitMQ, Redis, Postgres
npm installOpen two terminals:
# Terminal 1
node src/producer/webhook-server.js
# Terminal 2
node src/consumer/worker.jsSend a test event:
curl -X POST http://localhost:3000/webhooks/payment \
-H "Content-Type: application/json" \
-d '{"id":"evt_1","type":"payment_intent.succeeded","data":{"payment_intent":"pi_1","amount_cents":4999,"currency":"usd","status":"succeeded"}}'Watch the worker log the event being processed. Send the exact same
request again — the worker should log it as a duplicate, and no second
row should appear in the payments table.
RabbitMQ dashboard: http://localhost:15672 (guest / guest)
Health check: GET /health → 200 { status: "ok" }
-- should always return zero rows, by construction
SELECT event_id, COUNT(*) FROM processed_events GROUP BY event_id HAVING COUNT(*) > 1;- Webhook signature verification — real providers (Stripe, etc.) sign payloads with HMAC; the receiver should verify the signature before trusting anything in the body.
- Exponential backoff instead of a fixed 5s retry delay — avoids hammering a dependency that's down for longer than a few seconds.
- A DLQ alerting/replay tool — right now
events.dlqjust accumulates; a real system needs a job or admin UI that surfaces what landed there and lets someone replay or discard it. - Metrics — duplicate rate, DLQ depth, consumer lag exported to something like Prometheus, so failures are visible before they cause a support ticket.
- Horizontal scaling of the worker — safe to run N workers concurrently as-is, because the Redis check is atomic and the Postgres constraint is enforced at the database level, not in application memory.