Mailgun + SendGrid bounces and complains webhook

Author: Rajat

.env.example

@@ -0,0 +1,38 @@
+# Copy this file to .env before starting the stack. Do not commit .env.
+# Generate secrets with: openssl rand -base64 48
+POSTGRES_PASSWORD=
+BETTER_AUTH_SECRET=
+PIXEL_SIGNING_SECRET=
+# Generate this one with: openssl rand -base64 32
+ESP_CREDENTIALS_ENCRYPTION_KEY=
+
+# The initial account and its default team are created by the one-shot init
+# service. Its API key is shown exactly once in `docker compose logs init`.
+SUPER_ADMIN_EMAIL=admin@example.com
+
+# Public origins. For a local installation, keep these defaults. For a public
+# deployment, use the externally reachable HTTPS origins and set PROTOCOL=https.
+API_PUBLIC_URL=http://localhost:5000
+WEB_CLIENT=http://localhost:3000
+PROTOCOL=http
+DOMAIN=localhost:5000
+
+# Optional published host ports.
+API_PORT=5000
+WEB_PORT=3000
+
+# Optional platform SMTP, used for sign-in OTP emails. Team campaign delivery
+# credentials are configured in the SendLit dashboard instead.
+EMAIL_HOST=
+EMAIL_PORT=587
+EMAIL_USER=
+EMAIL_PASS=
+EMAIL_FROM=
+
+# Optional integrations and operational settings.
+AUTH_COOKIE_DOMAIN=
+ENABLE_TRUST_PROXY=false
+PROVISIONING_SECRET=
+MEDIALIT_APIKEY=
+MEDIALIT_SERVER=https://api.medialit.cloud
+MAX_UPLOAD_SIZE=10485760

.gitignore

@@ -1,5 +1,6 @@
 # Dependencies
 node_modules/
+.npmrc
 
 # Build
 dist/

AGENTS.md

@@ -2,7 +2,7 @@
 
 - Don't duplicate stuff over and over. Re-use existing code and libraries.
 - While making changes to the `apps/api` directory, make sure the REST API documentation and MCP server are updated as well.
-- For UI components, use shadcn/ui exclusively.
+- For UI components, use shadcn/ui exclusively. Always use Shadcn CLI for installing components. Never hand roll standard Shadcn components. Prefer shadcn/ui components over browser-native components.
 
 ## Architecture Tips
 

README.md

@@ -45,3 +45,29 @@ analytics, bounce handling and multi-user accounts are still on the roadmap.
 4. Build the two shared packages at least once so `apps/web` has something to
    import: `pnpm --filter @sendlit/email-editor build && pnpm --filter @sendlit/email-blocks build`
    (re-run, or use their `dev` scripts, after changing either package).
+
+## Self-hosting with Docker Compose
+
+The root Compose stack runs PostgreSQL, Redis, the API, and the web dashboard.
+It also uses a one-shot `init` service to apply database migrations and create
+the first account, its default team, and a team-scoped API key.
+
+```sh
+cp .env.example .env
+# Set the required secrets in .env (commands are included as comments there).
+docker compose up --build -d
+docker compose logs init
+```
+
+Set `SUPER_ADMIN_EMAIL` before the first start. The `init` logs contain the
+initial API key exactly once; save it in a password manager and use it as the
+`x-sendlit-apikey` header. If it is lost, create a replacement in the dashboard
+or through the authenticated API. Open the dashboard at `WEB_CLIENT` (by
+default, `http://localhost:3000`) and API documentation at `API_PUBLIC_URL/docs`.
+
+For an internet-facing deployment, set `API_PUBLIC_URL`, `WEB_CLIENT`,
+`PROTOCOL=https`, and `DOMAIN` to the public values before the first start.
+Put the API and web ports behind a TLS reverse proxy; set `ENABLE_TRUST_PROXY=true`
+when that proxy forwards client IPs. Back up the `postgres-data` volume and
+keep the `.env` secrets stable: changing the ESP encryption key makes stored
+team SMTP credentials unreadable.

apps/api/.env.example

@@ -5,9 +5,12 @@ DB_CONNECTION_STRING=postgres://sendlit:sendlit@localhost:5432/sendlit
 REDIS_HOST=localhost
 REDIS_PORT=6379
 
-PORT=4000
+PORT=5000
 NODE_ENV=development
 
+# Public URL Better Auth uses for OAuth callbacks, issuers and trusted origins.
+API_PUBLIC_URL=http://localhost:5000
+
 # OAuth2 JWT signing key — must be at least 32 bytes. Generate with:
 #   openssl rand -base64 48
 OAUTH_SIGNING_KEY=
@@ -37,6 +40,11 @@ ESP_CREDENTIALS_ENCRYPTION_KEY=
 PROTOCOL=https
 DOMAIN=
 
+# HMAC key for the suppression-list recipient fingerprint (never an unsalted
+# hash — see docs/bounces-and-complaints.md#8-suppression-model). At least
+# 32 bytes. Generate with: openssl rand -base64 32
+SUPPRESSION_HASH_KEY=
+
 # Platform mail transport for system email such as login OTPs. Campaign,
 # broadcast, sequence and ESP test mail require each team to configure its own
 # ESP in settings.
@@ -75,3 +83,7 @@ DEPLOY_ENV=
 # MediaLit integration for file uploads
 MEDIALIT_APIKEY=
 MEDIALIT_SERVER=https://api.medialit.cloud
+
+# Credentials for sign in with login
+GOOGLE_CLIENT_ID=
+GOOGLE_CLIENT_SECRET=

apps/api/README.md

@@ -12,6 +12,9 @@ email provider configuration.
   broadcasts.
 - Email delivery through BullMQ and nodemailer, using each team's configured
   ESP.
+- Bounce and complaint webhook ingestion (Resend, Postmark) with a durable
+  receipt inbox, canonical delivery events, and a per-workspace suppression
+  list enforced on every send path.
 - Open and click tracking, plus unsubscribe handling.
 - Better Auth session login, OAuth2 bearer-token authentication, and
   team-scoped API key authentication.
@@ -44,20 +47,24 @@ migrations before starting the API. The API image includes:
 - `apps/api/drizzle` — generated migration SQL and metadata;
 - `apps/api/dist/db/migrate.js` — a one-shot migration runner.
 
-The single-server deployment runs this through a Compose `migrate` service:
+The root self-hosted deployment runs this through a Compose `init` service:
 
 ```sh
-cd single-server-setup/sendlit
-docker compose up
+cp .env.example .env
+# Fill in the required secrets and SUPER_ADMIN_EMAIL.
+docker compose up --build -d
+docker compose logs init
 ```
 
 Startup order:
 
 1. Postgres starts and passes its health check.
-2. `sendlit-migrate` runs `node apps/api/dist/db/migrate.js`.
-3. The migration container exits successfully.
-4. `sendlit-api` starts.
-5. `sendlit-web` starts.
+2. Redis starts and passes its health check.
+3. `init` runs `node apps/api/dist/db/migrate.js`, then creates the configured
+   super-admin account, default team, and one-time API key.
+4. The init container exits successfully.
+5. `api` starts.
+6. `web` starts.
 
 This mirrors an init-container pattern. The API process itself does not apply
 migrations on every boot; it only checks that the database is reachable. For
@@ -119,7 +126,7 @@ pnpm --filter @sendlit/api run access-token email@address.com --team team_...
 The command prints only the access token. Use it in REST requests as:
 
 ```sh
-curl http://localhost:4000/contacts \
+curl http://localhost:5000/contacts \
   -H "Authorization: Bearer <token>" \
   -H "X-Sendlit-Team-Id: team_..."
 ```
@@ -188,12 +195,79 @@ Email automation runs through two loops in `src/automation/start.ts`:
 Tag and subscriber-added automations are event-driven through
 `automation/fire-event.ts`.
 
-Each team can configure its own SMTP-compatible ESP, such as SendGrid, Mailgun,
-Postmark, SES, Resend, or a custom SMTP server. Credentials are encrypted at
-rest with AES-256-GCM in `src/utils/secret-crypto.ts` and are never returned to
-clients. Campaign, broadcast, sequence, and ESP test mail require the team to
-have an ESP configured. Platform SMTP environment variables are reserved for
-system email such as login OTPs.
+Each team can configure one or more of its own SMTP-compatible ESPs (`esp_configs`),
+such as SendGrid, Mailgun, Postmark, SES, Resend, or a custom SMTP server, via
+`GET/POST /settings/esps` and `GET/PATCH/DELETE /settings/esps/:espId`. One
+user ESP is the team's default; a sequence, broadcast, or transactional send
+may pin a different one with `espId`. Credentials are encrypted at rest with
+AES-256-GCM in `src/utils/secret-crypto.ts` and are never returned to clients.
+`/settings/esp` (singular) remains as a backward-compatible alias over the
+team's default user ESP.
+
+Every sequence and transactional email persists an internal `deliveryRoute`
+(`custom` or `platform`) alongside the resolved user ESP's internal id
+(`outboxId`). `custom` is the only route currently reachable — it always
+pins a specific team-owned `esp_configs` row and is required before
+transactional mail can be queued or a sequence/broadcast can be marked
+active; `platform` is reserved for a future deployment-level SendLit-provided
+transport (resolved from deployment configuration, never from `esp_configs`)
+and isn't exposed by any endpoint yet. User-managed/custom ESP delivery never
+consults or increments the account's SendLit mail quota — those counters are
+reserved for the future platform route.
+
+## Bounce And Complaint Feedback
+
+Full design in [`docs/bounces-and-complaints.md`](./docs/bounces-and-complaints.md).
+Summary of what's implemented:
+
+- **Outbound ledger.** Every custom-route send (campaign or transactional)
+  gets an `outbound_messages` row before transport, carrying a generated RFC
+  `Message-ID`, the pinned ESP snapshot, and (once accepted) the transport's
+  response. `src/delivery-feedback/outbound-send.ts` /
+  `outbound-queries.ts`.
+- **Feedback connections.** Each user ESP with a reviewed provider adapter
+  (Resend, Postmark, SendGrid, and Mailgun — see `feedbackCapableProviders` in
+  `src/config/constants.ts`) can configure its own webhook connection under
+  `GET/PUT /settings/esps/:espId/feedback`, `POST .../feedback/rotate`,
+  `POST .../feedback/test`, `DELETE .../feedback`. Every configuration has
+  its own opaque `whc_...` connection id/URL, credential, and health —
+  switching the team's default ESP never moves or copies feedback config.
+  Changing an ESP's provider retires the old connection and a later `PUT`
+  creates a new one; deleting an eligible ESP retires its connection while
+  preserving historical events/suppressions.
+- **Public webhook route.** `POST /webhooks/esp/:provider/:connectionId`
+  (`src/delivery-feedback/webhook-route.ts`) is mounted before global JSON
+  parsing and before any session/API-key middleware — it authenticates
+  purely via the resolved connection's provider adapter (Svix signature for
+  Resend, a shared secret for Postmark, ECDSA Signed Event Webhook for
+  SendGrid, HMAC over timestamp+token for Mailgun), commits a durable,
+  encrypted receipt (`esp_webhook_receipts`) before returning `200`, and
+  never itself normalizes the payload inline.
+- **Async processing.** A BullMQ `esp-feedback` queue/worker
+  (`feedback-queue.ts`/`feedback-worker.ts`) plus a recovery poller
+  (`poller.ts`, 30s interval) drain receipts through
+  `process-receipt.ts`: provider adapter → canonical `email_delivery_events`
+  (idempotent per `(connectionId, providerEventKey)`) → correlate to an
+  `outbound_messages` row (`correlation.ts`) → delivery-state projection
+  (`projection.ts`) → suppression side effects. A committed receipt is
+  always the recovery source of truth if Redis is unavailable.
+- **Suppression.** `email_suppressions` is workspace-wide and
+  route-independent, keyed by an HMAC (`SUPPRESSION_HASH_KEY`) over the
+  normalized address — never derived from `contacts.subscribed`, so it
+  survives contact deletion/reimport. Checked before enqueue (transactional
+  `422 recipient_suppressed`; sequence sends skip the recipient without
+  counting it as sent) and again immediately before transport in the worker.
+  `GET/POST /suppressions*` exposes the list and owner release action;
+  complaint suppressions can only be released by a `sendlit_operator` actor.
+- **Retention.** `retention.ts` (hourly loop) purges raw receipt payloads
+  after 30 days and delivery events after 13 months, in bounded batches.
+- **Not yet implemented:** the Amazon SES adapter (its own PRD phase,
+  [`docs/aws-bounces-and-complaints.md`](./docs/aws-bounces-and-complaints.md) —
+  SNS signature/subscription handling is materially different from a direct
+  signed webhook); a provider without an adapter is never presented as
+  feedback-capable. Observability is limited
+  to the existing `captureEvent`/`captureError` posthog pattern — dedicated
+  dashboards/alerting and load/chaos testing are deployment-level follow-ups.
 
 ## MCP Server
 
@@ -215,7 +289,13 @@ OAuth client registration, authorization, token, introspection, revocation, and
 userinfo endpoints are served by Better Auth under `/api/auth/oauth2/*`.
 
 MCP tools live in `src/mcp/tools/*` and cover contacts, templates, sequences,
-ESP settings, teams, and API keys.
+ESP settings (both the default-ESP singleton tools and the multi-ESP
+collection tools — `list_esps`/`create_esp`/`get_esp`/`update_esp`/
+`delete_esp`/`test_esp`), teams, API keys, and bounce/complaint delivery
+feedback (`get_esp_feedback_connection`/`upsert_esp_feedback_connection`/
+`test_esp_feedback_connection`/`delete_esp_feedback_connection`,
+`list_delivery_events`/`get_delivery_event`,
+`list_suppressions`/`get_suppression`/`release_suppression`).
 
 ## More Context