Adopt patched Plunk as the multi-brand email platform

[vladbisceanu]

Current state

Production cutover completed on 2026-08-27. send.growthpath.systems serves the Plunk dashboard and api.send.growthpath.systems serves the split API; both are DNS-only Cloudflare CNAMEs to Railway production targets with valid TLS. Signups remain disabled.

The production gate and first-brand support patches are landed. #12 fixed the first-contact race at 576bc1408f06bcf662f7dfeb9fbfe778e21cf59c (upstream useplunk#469); #14 fixed Railway-domain session cookies at 380dcafb9021fc6c3f7b3efa790d95731f8988a4 (upstream useplunk#470); #17 added create-only subscription state at a09ac1c193122fe3a4216496bd090ce80c8382a9 (upstream useplunk#471); #19 fixed workflow email execution from the relational template binding at 62ee1a1480e61b8fb52f56617f555ba57f037d18, closing #18 (upstream useplunk#472).

The domain error-handling patch #25 landed at e6315a575e68867b47f0a1d116799ed8a297fe6d; unexpected provider failures now use the safe global 500 boundary and the dashboard understands legacy string and structured errors. Production IAM now grants Plunk's region-limited SES v1 identity lifecycle. The Bookling project contains bookling.dev; all three DKIM CNAMEs and the custom MAIL FROM MX and SPF TXT resolve publicly, while SES verification is still processing.

Current Railway deployments are API fe0d1bdb-bc54-401c-a155-24aaeb9f15cf, worker 59738f20-e9f9-4f94-af59-ec17fd27c1b2, and dashboard 9e3f7809-7c78-4469-9e2f-3daf24acf0d4. All three report SUCCESS, one running replica, and the exact immutable image ghcr.io/vladbisceanu/plunk@sha256:a2d0a15a705fb69dfa45b5cfd8249e4f8d3852d74e0a00406784b6b5b44cc14f, built from production merge e6315a575e68867b47f0a1d116799ed8a297fe6d. API health and the dashboard both return 200. PostgreSQL 18 and Redis 8.2 retain 50 GB volumes; Redis uses AOF with appendfsync everysec.

The bisceanu.com first-brand migration completed on 2026-08-28 in vladbisceanu/bisceanu-com#87 at site SHA cca5b29c25fbc9f53af20bd58e9e268b25d5c9a7. The site now sends repo-rendered inline HTML through /v1/send and starts double opt-in through one /v1/track call with subscribed:false and preserveExistingSubscription:true. Its post-retirement Vercel production redeploy is Ready and the named production proof succeeded again on that exact SHA.

The live Bisceanu confirmation workflow ec12c9ff-1f4e-4764-95b3-8a98b107277f and welcome workflow 7ce1660c-71ff-4fec-acc4-5ef30076b2d1 are enabled and unchanged. Production proof covered real English and Romanian form submissions, both owner notifications and localized confirmations, pending-before-confirm state, hosted confirmation, contact.subscribed welcome, RFC 8058 one-click headers, hosted unsubscribe, marketing rejection for a migrated-unsubscribed record, and SES-simulator bounce suppression. All synthetic proof contacts were deleted; no live workflow was edited.

Contact and suppression state were reconciled without resubscribing suppressed contacts. USESEND_API_KEY and USESEND_CONTACT_BOOK_ID were removed from Vercel Production and the bisceanu Infisical project only after received-mail proof; store readback confirms they are absent. The useSend Railway project was deleted on 2026-08-28 (#20); Vlad waived the final backup since its contact and suppression state were already reconciled into Plunk. The vladbisceanu/useSend fork repository remains as code history. Delayed Easy DKIM selector rotation and removal of the legacy usesend._domainkey TXT after its seven-day overlap are tracked in vladbisceanu/bisceanu-com#88.

Production backup protection remains active. PITR completed base backup set 20260827-202416F; manual backup 8a9fa50c-8dde-4c8c-a3c0-aeb907492c03 (pre-cutover-2026-08-27) is retained with daily and weekly schedules. An isolated point-in-time restore replayed WAL, reached its recovery target, completed archive recovery, and accepted connections without touching primary.

AWS remains healthy: SES production sending is enabled with HEALTHY enforcement; bisceanu.com and mail.bisceanu.com MAIL FROM report success. Configuration set growthpath-plunk-production publishes to the isolated SNS topic with exactly one confirmed HTTPS subscription at https://api.send.growthpath.systems/webhooks/sns.

Final verification: the Bisceanu repository pnpm verify passed lint, typecheck, build, 150 unit tests, and 18 browser tests. Platform build and functional suites passed for #19; its production behavior was then re-proved on the pinned image. The immutable image labels and all running service sources were independently read back.

Outcome

Promote the maintained Plunk fork from disposable evaluation to the shared multi-brand email platform. Plunk owns campaigns, contact state, templates, and nurture workflows. Product backends keep domain events and money, auth, and recovery decisions.

Scope

• Fix the first-contact /v1/track upsert race before production traffic.
• Keep main as an upstream mirror and production patches on the maintained fork branch. Track upstream response to fix: re-check subscription before SES submission useplunk/plunk#462 through fix(contacts): make concurrent first-contact upserts safe useplunk/plunk#469.
• Build production API, worker, and dashboard services from an immutable digest. Keep PostgreSQL and AOF Redis on persistent volumes. Do not use the all-in-one template.
• Production hostname is send.growthpath.systems, permanent from day one (tracking links and hosted unsubscribe pages embed it in delivered mail, so it can never move; decided 2026-08-27). Cutover order: provision and verify production Plunk on its Railway-generated domain first, then flip the DNS record from useSend to Plunk and migrate the bisceanu contact form in a separate follow-up. bisceanu.com is useSend's only live consumer and has negligible traffic, so temporary form downtime and broken useSend SNS callbacks during the flip are accepted. useSend stays reachable on its Railway-generated domain for data reference until retirement. mail.growthpath.systems stays reserved for SES custom MAIL FROM.
• Put agency customers behind a scoped ingest proxy. Never issue raw Plunk project keys.
• Enforce workflow changes by duplicate, drain, and switch. Never edit a live workflow.
• Migrate sending domains, templates, contacts, unsubscribe state, campaigns, and active nurture definitions with per-brand proof. bisceanu.com is the first migration (POC): contact form, double opt-in, SNS feedback, one-click unsubscribe.
• The GrowthPath migration is reshaped and sliced: growthpath-ai/growthpath#1218 is now an epic around Plunk workflows, a thin internal client and repo-owned templates, with four sub-issues. Two platform constraints it surfaced land here rather than there: per-brand hosted unsubscribe and preference URLs are instance-wide (DASHBOARD_URI / API_URI), so Heralded mail carries a growthpath.systems opt-out link; and a WEBHOOK step completes on a non-2xx response, so no workflow can treat an application 5xx as a failure.
• Retire useSend after its contact and suppression state are copied into Plunk and the bisceanu POC has production proof for send, engagement, unsubscribe, and bounce handling. Remaining brands migrate from Loops directly to Plunk.

Done when

• The first-contact race is fixed and the nine-check suite is repeatable against the production candidate.
• Each brand has exact production proof for transactional mail, marketing mail, suppression, and any migrated nurture flow.
• PostgreSQL backup and restore are proved before the first upgrade and documented for later upgrades.
• useSend is removed only after all consumers and suppression data have moved.
• The adoption ADR and all affected product issues point here as the owning program.

References

• Evaluation: vladbisceanu/setup-works#131
• Disposable deployment and evidence: vladbisceanu/setup-works#146
• Superseded platform decision: vladbisceanu/setup-works#49
• Adoption ADR: vladbisceanu/business-works#12
• GrowthPath migration: growthpath-ai/growthpath#1218

[vladbisceanu]

Spec edit (Vlad, 2026-08-27): send.growthpath.systems becomes Plunk's permanent hostname at cutover — provision and verify production Plunk on its Railway domain, flip DNS, migrate the bisceanu form immediately after. Brief bisceanu-form downtime and broken useSend SNS callbacks during the flip are accepted (bisceanu is useSend's only live consumer). useSend retires once its contact and suppression data are copied; other brands go Loops→Plunk directly. ADR updated to match (business-works#12).

[vladbisceanu]

Handover — 2026-08-27

• Production-gate PR: fix(contacts): make first-contact upserts race-safe #12 (fix/10-first-contact-race → growthpath-mail), closing Make first-contact tracking safe under concurrent requests #10 on merge.
• Portable upstream PR: fix(contacts): make concurrent first-contact upserts safe useplunk/plunk#469.
• Gate proof: eight forced parallel first-seen service upserts; twelve parallel /v1/track requests return twelve 200s, elect one normalized contact, persist twelve distinct events, and leave at most one active execution during the wait. Explicit concurrent opt-out and stale-observer event idempotency are covered.
• Verification: focused service + HTTP suites 76/76; API build passed; repository lint passed with existing warnings. The full suite passed 1,178/1,180, with only unrelated local performance misses (49,909 vs 50,000 renders/s and the existing 10k-contact benchmark timing out at 120s). The monorepo build reached and passed API, then stopped in unchanged landing with Could not parse output from TypeScript's --showConfig.
• Two local concurrency-review findings were fixed and given regressions: losing explicit opt-out ordering and duplicate subscription events from stale observers.
• Production remains unchanged at the digest recorded in Current state. No deploy or DNS cutover was attempted.

Next: review and merge #12, build the production candidate, rerun the nine-check suite, and verify the running digest before any cutover. Current state above has been updated.

[vladbisceanu]

Gate landed — 2026-08-27

• fix(contacts): make first-contact upserts race-safe #12 is merged into growthpath-mail at 576bc1408f06bcf662f7dfeb9fbfe778e21cf59c; Make first-contact tracking safe under concurrent requests #10 is closed.
• Focused production-gate evidence: 76/76 green, including twelve 200s, one contact, twelve durable events, and at most one active execution during the wait.
• The 10k-contact timeout reproduces on the untouched base; the unrelated rendering threshold passed there and its prior 0.18% miss was timing variance.
• Portable upstream PR fix(contacts): make concurrent first-contact upserts safe useplunk/plunk#469 remains open.
• Current production still runs a6a1b7deadecdc4cf6f0b74326d1e744c094b45f at the digest recorded above. No provider or DNS write was made.

Next session: build and pin a post-merge image; provision split API/worker/dashboard Railway services with volume-backed Postgres and AOF Redis; wire SES eu-west-1 and SNS for the bisceanu.com Plunk project; verify the Railway-generated domain and rerun all nine checks. Only then flip send.growthpath.systems to Plunk.

[vladbisceanu]

Adversarial provisioning critique — folded 2026-08-27

The critique changed the production design; provider mutations had not started. Dispositions:

1. Accepted: do not invoke Plunk domain creation until the current SES identity/MailFrom state is read. Production uses MAIL_FROM_SUBDOMAIN=mail, preserving the established mail.bisceanu.com contract.
2. Accepted: do not reuse the useSend/evaluation SNS topic. Create a Plunk-specific SES configuration set and SNS topic so unrelated message IDs cannot create cross-application 500/retry traffic.
3. Accepted: do not promote the disposable evaluation database/Redis state. Create a clean production Railway project; the evaluation project remains evidence/rollback reference.
4. Accepted: record and read back the split-service overrides: API start + /health + migration pre-deploy, worker start with no public domain, dashboard start + /; all use the exact same image digest.
5. Accepted: this app diff has no migration. On the clean database, run all migrations through API pre-deploy before starting worker, then dashboard. Rollback during candidate provisioning is image-only to the prior pinned digest.
6. Accepted: AOF is restart durability, not paired state backup. Candidate state is disposable until verification; production backup/restore must treat PostgreSQL and queued Redis state consistently before live traffic.
7. Accepted: build from a clean 576bc1408f06bcf662f7dfeb9fbfe778e21cf59c checkout, embed the OCI revision label, record the GHCR digest, and read back that digest from every running app deployment before pinning Current state.
8. Accepted: run the nine checks in a dedicated internal Plunk project. Cleanup is part of proof: queues resumed, rate/concurrency restored, injected DB objects and temporary subscriptions removed, and no unexpected pending/delayed jobs.
9. Accepted: Railway-generated-domain verification proves only the candidate. No production message may leave with Railway-hosted links; hosted unsubscribe/CORS/callback proof must be repeated after the later send.growthpath.systems DNS and runtime-URI cutover.

Contracts: DomainService.ts, SESService.ts, Webhooks.ts, docker-entrypoint-nginx.sh, QueueService.ts, the accepted ADR, and setup-works#146. DNS remains out of scope for this session.

[vladbisceanu]

Production provisioning handover — 2026-08-27

• Gate Make first-contact tracking safe under concurrent requests #10 is closed by fix(contacts): make first-contact upserts race-safe #12 (576bc1408f06bcf662f7dfeb9fbfe778e21cf59c); the twelve-request first-contact proof is green. Upstream: fix(contacts): make concurrent first-contact upserts safe useplunk/plunk#469.
• Provisioning exposed the invalid Railway-domain session cookie. fix(auth): scope session cookies to the API host #14 closed Make dashboard sessions safe across deployment domains #13 at 380dcafb9021fc6c3f7b3efa790d95731f8988a4; upstream: fix(auth): scope session cookies to the API host useplunk/plunk#470. Phone proof now shows login 200, authenticated user/project reads, and successful GrowthPath project creation.
• API, worker, and dashboard are all SUCCESS on the same pinned image: ghcr.io/vladbisceanu/plunk@sha256:e3168ecd57d3586cbb7e223c092ae77b6be75d3df89b0e02333dbb4304127840. Railway source readback matches on all three.
• Candidate hosts are active with valid certificates: plunk-api-candidate.growthpath.systems and plunk-candidate.growthpath.systems. Generated Railway fallback hosts remain healthy. Signups are disabled.
• AWS is complete: isolated SES configuration set growthpath-plunk-production, isolated SNS topic growthpath-plunk-production-ses-events, confirmed signed HTTPS subscription, SES/DKIM and mail.bisceanu.com MAIL FROM both Success.
• The sole GrowthPath tenant has one verified bisceanu.com domain. It was imported after read-only SES checks; the existing identity was not rewritten.
• SES simulator smoke passed end to end: API 200, durable DELIVERED, processed email.sent and email.delivery events. Temporary Railway SSH keys were removed after readback.
• No cutover: send.growthpath.systems still resolves to useSend (awpmu2uc.up.railway.app). The disposable evaluation stack is untouched.

Next: rerun the full nine-check matrix against this production candidate and clean up its injected state; prepare the bisceanu contact/suppression migration; then perform the accepted DNS/runtime-URI flip to send.growthpath.systems + api.send.growthpath.systems and repeat hosted-link, unsubscribe, callback, and form proof. Backup/restore proof is required before the first production database upgrade.

[vladbisceanu]

Adversarial cutover critique, folded 2026-08-27

The final cutover review found real blockers. DNS and production mutations had not started. All ten findings are accepted with these changes:

1. The current bisceanu Function is still a useSend client. Before DNS moves, deploy a dormant Plunk adapter with its candidate API URL, project-scoped credentials, templates, and workflow. Prove both transactional messages and double opt-in against the candidate. Activation happens only inside the cutover window.
2. Contact and suppression migration gets a quiescent boundary. Stop form ingress, take the final useSend delta, normalize email identity, import with "unsubscribed wins," reconcile counts and the exact suppressed set, then activate Plunk. Routine ingestion must never send subscribed:true for an existing contact.
3. A Plunk project does not isolate Redis queues, the worker, Postgres failure injection, or SES quota. Run the destructive parts of checks 2, 3, 5, 7, and 9 on the disposable mirror with the exact production digest. The production primary gets only project-scoped E2E, isolation, unsubscribe, and concurrency smoke tests.
4. Do not run the seeded upgrade probe on production. Enable a real production backup path first and restore the production backup into scratch. Run the old seeded-version upgrade in a separate scratch database.
5. DNS rollback is valid only until the first Plunk message leaves. Existing useSend hosted links keep the permanent hostname. Audit outstanding confirmations and hosted links, then either add compatibility routing or explicitly accept their loss. After the first Plunk send, rollback means switching the consumer back while keeping the Plunk hostname.
6. Pre-attach api.send.growthpath.systems; publish Railway verification TXT records before traffic moves. Stop ingress and sends, flip the dashboard CNAME, wait for Railway routing and TLS, apply final runtime URLs to all roles, then resume. Final auth proof starts with a fresh login because the host-only candidate cookie does not transfer.
7. Keep the confirmed candidate SNS subscription during cutover. Add and confirm api.send.growthpath.systems/webhooks/sns, prove feedback through it, then remove the candidate subscription after the overlap.
8. Before changing runtime URLs, prove no production PENDING email and no test-owned waiting, delayed, active, or retryable job. Test cleanup must remove injected objects and leave queue rates and concurrency at production values.
9. Runtime URL changes create new deployments. Re-read the immutable image digest and OCI revision from the final running API, worker, and dashboard deployments. Deployment success alone is not proof.
10. Public evidence contains redacted counts, hashes, timestamps, and request IDs only. Exact signed SNS envelopes stay encrypted and private for the replay check, then are destroyed.

The practical consequence is a hard no-go on a DNS-only flip. The form adapter, quiescent final migration, production backup/restore, isolated destructive matrix, empty-queue proof, final domains, and post-redeploy digest readback are now part of the gate.

[vladbisceanu]

Cutover handover — 2026-08-27

• DNS is live: send.growthpath.systems → Plunk dashboard and api.send.growthpath.systems → Plunk API. Both public endpoints return 200, Railway reports propagated records and valid TLS, and Cloudflare readback matches the Railway targets.
• Final runtime config advertises only the final URLs; signups are disabled; credentialed CORS accepts https://send.growthpath.systems.
• API/worker/dashboard deployments are bb402c33-706f-424a-a22e-755be2e59300, 516ee16d-3c17-4e07-866e-9bba6c0ae104, and 6466c121-8a5c-4c2e-8025-6b0c67fed136. All are SUCCESS on pinned digest sha256:e3168ecd57d3586cbb7e223c092ae77b6be75d3df89b0e02333dbb4304127840.
• PITR is enabled; manual backup 8a9fa50c-8dde-4c8c-a3c0-aeb907492c03 is retained. An isolated sibling restore reached the target and accepted connections without touching primary.
• SES remains production-enabled/healthy; domain, DKIM, and custom MAIL FROM are successful. SNS now has exactly one confirmed subscription at the final API hostname; its confirmation POST returned 200 and was logged as successful. No API/worker error-level entries appeared in the post-cutover window.
• Repository verification: 1,185/1,186 tests passed; the only miss is the known 10k-contact 120-second performance timeout. Lint: zero errors, existing warnings only. Evaluation signups were re-locked and the exact digest was re-read.
• Per Vlad’s cutover ruling, the dormant bisceanu form and its contact/suppression migration are deferred; temporary outage is accepted because it has no traffic. useSend stays intact on its generated Railway domain for later reference.