[UMBRELLA] Booking + maintenance productionization, wave 5 — verified residuals, HLD/LLD verdict, 11-PR train

[teetangh]

Why this issue exists

temp/booking1.txt and temp/booking2.txt were the original staff-engineer briefs for productionizing booking + maintenance. They have already driven four waves: #676/#677 + PR #873 (June), PRs #998–#1018 (2026-07-17), the #1169 train (PRs #1170–#1180, released 2026-08-15 via #1184), and the 2026-08-21..31 follow-ups (#1191, #1204, #1205, #1207–#1220, #1260, #1261, #1272, #1287). This issue is the reconciliation: what the briefs asked for that shipped, what is still open (verified against current dev at 4bce4bc, file:line), the HLD/LLD verdict, and the wave-5 PR train. It supersedes #676 and completes #1169; both are closed with a pointer here.

Full working document (decisions, specs for PRs 1–3, verification recipe): the session plan file, mirrored into docs/booking/ by PR 11.

1. What the briefs asked for that has shipped

• Locking / transactions / serializability: interval-atom locks that fail closed (ADR B11, fix(booking): interval-granular slot locks in one namespace; trials enter the exclusion net #1170), CAS state machines for all five lifecycles, GiST + CHECK sidecars with db:assert-sidecars (fix(db): every documented booking guarantee becomes real, mapped to one push #1176, fix(booking): sidecar automation + trial refund front door #1215), Serializable checkout with retry, race suite categories 10–11 (test(race): interval-lock semantics + load scenarios for the atom lock layer #1187).
• Multi-device / multi-tab / incognito: consultee-booking: lock, initialAllocation 409 + in-tx advisory lock (ADR B10), per-request idempotency keys with replay, open-order reuse (fix(checkout): reuse open PENDING order across remounts; CLASS lock TTL 300s→600s #1220), guest keeps slot across the auth wall (fix(booking-journey): a guest keeps their slot across the auth wall #1261).
• Heatmap freshness: ADR 16 (no Realtime), 60 s visible-tab poll + max-age=30, swr=60 (perf(scheduling): the grid stops being permanently stale, and the dead client allocator is deleted #1178). Server Actions: NO migration. Queues: ADR 13/14/22, QStash pre-approved (Install QStash for Class-A crons and Not-Before reminder delivery (per #866 / ADR 14) #1010).
• Reschedule + cancel across roles: respond/counter/withdraw loop (fix(booking): the reschedule loop closes; lifecycle authz, proration, and audit stop lying #1174/fix(booking-ui): the consultee can finally answer a reschedule proposal #1177), cancel preview + money truth (fix(booking-ux): every paid or destructive action shows its price first #1180), proration (Subscription refund proration for partially-consumed plans (audit C7/M6) #1006), decline/withdraw + allocation notifications (fix(booking): allocation + reschedule notifications, expiry money closure #1208, fix(booking): partial-reschedule parent status, candidate-walk truncation, decline/withdraw notifications #1214).
• Refund doctrine: two front doors, org/credit rails (fix(payments): freeze obeys the cancellation doctrine; every refund rail enters one front door #1173), group-event refunds (fix(booking): Pre-MVP algorithm integrity — #1071 #1012 #1005 #1091), reject auto-refund (Auto-refund on consultant REJECT of a paid request (audit M3) #1004), credit-funded cancel (fix(refunds): credit-funded booking cancellation restores credits, reverses utilization, nets earnings (#1161) #1218), trial refund front door (fix(booking): sidecar automation + trial refund front door #1215), no-show corroboration (fix(appointments): corroborate no-shows against Stream, and stop charging for sessions nobody attended #1287).
• Approval path: mint-after-commit (fix(booking): approval path stops inventing appointments, dropping org context, and calling gateways in-tx #1172), appointmentId threaded + frozen quote (fix(checkout): approval payments carry appointmentId; pending surfaces quote frozen amounts (#1181, #1182) #1217). Trials: in the GiST net, shared lock namespace, paid-trial expiry job, org-trials surface deleted (fix(dashboard): org and personal contexts stop bleeding into each other #1175).
• Events: capacity pre-check + bounded waits (fix(booking): flash-sale checkout — capacity pre-check, bounded lock waits, client busy-retry #1207), CLASS lock TTL 600 s (fix(checkout): reuse open PENDING order across remounts; CLASS lock TTL 300s→600s #1220), class duration truth + payment-guarded deletes (fix(booking): allocation audit — class duration truth, per-day cap parity, payment-guarded deletes, bounded occupancy reads #1191). Org scoping: lib/api/scope single source (fix(dashboard): org and personal contexts stop bleeding into each other #1175), ADR 18/19/20/21, org money rails (fix(launch): close the P0/P1 end-to-end audit blockers — booking money loss, org money rails, server-side join policy #1260).
• Maintenance: PENDING-payment appointments no longer hard-deleted by the freeze (fix(payments): freeze obeys the cancellation doctrine; every refund rail enters one front door #1173), DEGRADED write-block + cron guard + webhook DB-health + preflight + UI guard, cron fleet audited (fix(cron): every scheduled job locked, bounded, and able to notice its own death #1179), pool-budget guard (chore(ci): cron hygiene guard — pool-budget model #1204).

2. HLD / LLD verdict

The concurrency LLD is genuinely strong: interval-atom Redis locks that fail closed, WHERE-clause CAS state machines, the slot_no_confirmed_overlap GiST backstop, a Serializable recount for finite capacity, idempotency at three layers, a refund-policy snapshot per appointment. The HLD data model is the weak link and the reason every fix wave had to be applied four or five times: five parallel booking shapes with three status enums, a polymorphic Appointment with four nullable FKs, participation modelled as an implicit per-slot m:n join, isTentative overloaded to mean both "unpaid hold" and "being rescheduled", and two god modules (SlotAllocationService.ts 3,901 LOC, checkout.ts 3,666 LOC) each carrying ~40 issue-numbered patches. Correct by accumulated patch, not by construction.

Locking verdict (per-path census done): keep the hybrid. Redis = cheap serialization + gateway-order hygiene (a mutex loser mints zero Razorpay orders); Postgres (SSI, GiST, CAS-in-WHERE) = correctness. No row-version columns are needed on booking models. Removing the mutex would move the 1:1 slot decision to the capture webhook, i.e. both strangers pay and one is auto-refunded.

• KEEP: the lock module, CAS layer, GiST/Serializable backstops, heavy reads outside the write tx, ADR 16/17/21.
• CREATE: AppointmentParticipant (schema-only now, written from day one post-reset, readers flip later), BookingStatusHistory (same-tx append from the CAS helpers; no drainer until QStash), a k6 load harness.
• UPDATE: hold expiry as a read predicate on Payment.expiresAt (not a cron); retire the 204 s default lock retry from request paths; guard the appointment lock; one reschedule mechanism; availability read model for the grid; split the god modules by extraction (own epic, after the participant model lands).
• DELETE: the 2026-03-06 checklist and the dependency-graph doc, live DST paths (fold DST travel hazard: frozen utcOffsetMinutes silently shifts a traveling consultant published availability (#872 stub caveat) #1200 into DST / IANA-TZID timezone implementation — algorithms + UI (post-MVP) #872), the two unscheduled cleanup routes, confirmed dead code.

Decisions locked with the owner (2026-09-02): schema scope = participant model + status history + hygiene (no outbox dispatch, no holdExpiresAt column); #1206 = partial allocation with explicit consultant confirm + tracked remainder; "approved but never paid" = EXPIRED via CAS (not REJECTED, not revert-to-PENDING); 11 PRs in 4 waves from dev; #874 load gate executed in-train against a deploy preview; ADR ids A9 (participant), A11 (row-version, deferred), A12 (status history; B4 is taken).

3. Confirmed defects in current code (re-verified at line level for the money class)

1. scripts/payments/cleanup-abandoned-payments.ts:283-299 — the fix(trials): stop trial cancellation from deleting the payment #1074 class, still live: slotOfAppointment.deleteMany({ where: { appointmentId } }) with no isTentative filter, then consultation.delete / subscription.delete cascading Appointment → Payment. Fix: soft-cancel via CAS EXPIRED + deletedAt.
2. app/api/cleanup/approval-payments/route.ts:42-96 — read-then-update revert APPROVED_PENDING_PAYMENT → PENDING; a capture landing in between leaves a SUCCEEDED payment on a PENDING request. Route is also unscheduled (Vercel-Cron-shaped); delete it, one semantics = EXPIRED.
3. No CAS on group-event status: bookings/{webinars,classes}/crud-with-plan/route.ts write client-supplied status bare; auto-complete-appointments.ts:103,188 mark events COMPLETED bare — a CANCELLED event can be resurrected after refunds.
4. Bare terminal writes in sweeps: cleanup-invalid-appointments.ts:180,296,388,474 (+ four unguarded slot deleteMany), cleanup-abandoned-payments.ts:444 (→ REJECTED, semantically "consultant declined"), cleanup-stale-pending-consultations.ts:132.
5. SlotOfAppointment.completionStatus and TrialSession.status have no CAS helpers; lib/stream/session-handlers.ts:141,251, jobs/meetings/reconcile-orphaned-sessions.ts:142, actions/maintenance/drain-sessions.ts:253, auto-complete-appointments.ts:521 write them by id.
6. No rate limit and no lock on appointments/[id]/cancel and /reschedule; trial accept has locks but no limiter.
7. app/api/admin/refunds/route.ts:163 calls raw refundPayment, so an org_* intent dies on UNKNOWN_GATEWAY.
8. AE-2 collaborator availability enforced on one route only (webinars/crud-with-plan:808); no class route, no allocation mode; class POST is Read Committed and maps no 23P01.
9. db:assert-sidecars exists and nothing runs it; db:sidecars runs twice per push.
10. app/api/cleanup/auto-complete-trials is an unscheduled, buffer-less twin of the hourly job's trial arm.
    11–13. HTTP cleanup routes carry no maintenance guard (manual-trigger bypass; GA runs jobs/*); 13 billing/compliance/contract jobs lack abortIfMaintenance (incl. release-pending-trust-earnings, auto-renew-contracts); /api/organizations/* and request-for-approval + availability writes + reschedule/respond|withdraw are outside the DEGRADED write block.
11. MAX_CANDIDATE_STARTS_PER_ROW still exits silently when rowEndMs is omitted (row end is now the bound; MAX_CANDIDATE_STARTS_PER_ROW is a silent truncation risk in auto-allocate candidate walks (tracked undeveloped in #1169) #1194 closes with telemetry).
    15–18. Client engine relocated not deleted (docs overstate); no ETag on the polled grid; approval-payment.ts:344 sentinel; lock:approval_payment: is a second name for the approval atom.
12. DEFAULT_RETRY_CONFIG = 204.6 s worst case vs a 26 s function ceiling; used by request-for-approval, trial accept, all allocation modes and the approval routes → 504 not 409.
13. Approval lock TTL (30 s) equals the approval tx timeout.
    21–28. acquireLock single-shot in approval-payment; trial-request P2002 → 500; attendee remove read-then-act; 13 Redis round trips per consultation booking; three different terminal outcomes for "approved but never paid"; R7 trials skip published-window validation; R8 checkout lock renewal outside the P2034 retry loop (4 × 25 s > 60 s TTL); R5 10 unwrapped Serializable sites incl. both approval routes.

B2B (org-funded): cancel preview/dialog promise a card refund for wallet-paid bookings; earnings can be permanently unaccrued after the 30-day sync window; org context not re-validated inside the lock; no DB idempotency on utilization per (assignment, appointment); allocation-time assignment resolve omits status: ACTIVE; payer admins cannot see unallocated org-funded requests; five orgMember scope fall-throughs; program assignment accepts a non-ACTIVE membership.

Schema: 10 of 14 booking models lack deletedAt; 9 naive DateTime columns (all of TrialSession); 6 redundant prefix indexes; no usable index for Appointment (organizationId, deletedAt); phantom columns ConsultantReview.isAnonymous / AppointmentFeedback.slotOfAppointmentId (no code refs; capture then let the reset drop); STAGED sidecar block no longer at the bottom; A9/A11/B4 had no ADR home.

Docs / prompts: all 8 prompts/booking-algorithm-tests/ files cite /api/events/* (renamed 2026-06-12); .claude/skills/booking-doctrine rule 6 asserts the funded-elsewhere org arm removed by #1166 ORG-8; docs/booking/15-checklist.md (2026-03-06), 06-dependency-graphs.md, README.md, 01-architecture.md paths, docs/payments/checkout-flow/03|04|06 (Nov 2025), booking/17 + enterprise/10/05 line refs, ADR 16 wording; changelog has no 2026-09-01 entry.

4. Still open and NOT in this train (by band)

• P0 orthogonal: P0: production email delivery dead since 2026-06-18 — invalid RESEND_API_KEY hard-locks signup and password reset #1298 prod Resend key invalid (blocks checkout at the auth wall). What is left after the 2026-09-01 release: ops rollout, open PRs, DB and hygiene #1312 §DB phantom columns handled by PR 2's capture script.
• launch: scale / post-MVP: DST / IANA-TZID timezone implementation — algorithms + UI (post-MVP) #872 DST (fold DST travel hazard: frozen utcOffsetMinutes silently shifts a traveling consultant published availability (#872 stub caveat) #1200 in), [RESILIENCE] Session overrun detection and conflict prevention #472 overrun, Consultant planner has no archive path for booked plans #1058 archive, Draw the slot grid in schedulingTimezone so the columns match the caps being enforced #1168 grid tz, API layer: unify auth, response contract, validation & route structure across app/api/** #1203 API unification, U2 god-module split (own epic after PR 2), Install QStash for Class-A crons and Not-Before reminder delivery (per #866 / ADR 14) #1010/Cron architecture: measured GitHub Actions throttling breaks the sub-hourly fleet — move ~10 event-shaped jobs to QStash, keep business crons on GA #866 QStash execution, DB-layer correctness and observability: automate the sidecar apply, attribute constraint violations, and settle the one remaining trigger #1092 Sentry constraint attribution.

5. The train (11 PRs, 4 waves, each based on dev, serial merges, rebase --onto restacks)

#	Branch	Scope	Wave
1	fix/booking-cas-and-delete-sweep	defects 1–5, 7, 26, 27: soft-cancel instead of delete, CAS everywhere, transitionSlotCompletion + transitionTrialSession, admin refund front door, delete the unscheduled approval-payments route	W1
2	fix/booking-schema-finalization	AppointmentParticipant + writers, BookingStatusHistory + helper appends, hygiene, phantom-column capture, STAGED relocation + reset runbook, db:assert-sidecars chained + one shared parser, ADRs A9/A11/A12, seeds	W1
3	fix/booking-hold-expiry-and-locks	hold expiry as predicate, dead limiter, PING cache, appointment lock + limiters, one approval-lock name, retire DEFAULT retry, approval TTL, org context re-validated in-lock, R5/R7/R8, exhausted-P2034 → 409	W2
4	fix/allocation-collaborators-and-partial	AE-2 in every mode + class routes, class POST Serializable + 23P01, #1194 telemetry, assignment ACTIVE, utilization appointmentIds, engagement return on slot delete, #1206 partial allocation	W2
5	fix/maintenance-and-cron-coverage	delete auto-complete-trials route (close #1278), 13 jobs guarded, throwing guard for HTTP cleanup routes, DEGRADED list reconciled incl. /api/organizations/*, SystemJobExecution retention, cron.stale	W2
6	fix/booking-lifecycle-tail	#1199 runs grouping, preview timeout, R13/R17, sentinel, trial P2002 → 409, attendee remove guard, org pins → lib/api/scope + orgMember fall-throughs, org-paid cancel copy via rail, payer-admin requests view, assignment membership ACTIVE, org-admin arm on respond, TRIAL out of org zod	W2
7	fix/booking-money-parity-tests	price-parity imports the pure fn, refund-preview-vs-actual test, org receivable view, unbounded earnings: none healer + alert	W3
8	feat/booking-outbox-and-audit	status-history rows from every transition, RescheduleLog view, staff booking surface (metadata-only)	W3
9	perf/availability-read-model	ConsultantBusyInterval maintained in the write tx, grid reads it, ETag/304	W3
10	test/load-and-chaos-exit-gate	k6 harness for the five write paths, chaos 6/14c/17 against a deploy preview, numbers into #874	W3
11	docs/booking-wave-5	checklist/README/dependency-graphs/01 paths/ADR 16, payment docs, changelog, prompts corpus (/api/bookings/*, orgScope, cancel/preview, partial), doctrine rule 6, dead-code sweep, TODO retargeting, DST fold	W4

6. Pre-MVP reset checklist (owned by PR 2's runbook)

Capture phantom columns → snapshot → npm run db:push (push → sidecars → assert) → uncomment the STAGED block at its new position → db:sidecars + db:assert-sidecars → seed → drift check. Staged constraints: Payment.clientIdempotencyKey / OrganizationPayout.idempotencyKey NOT NULL, program_assignment_no_active_overlap, subscription_plan_total_sessions_min / class_plan_total_sessions_min; drop consultant_review_legacy_pair_key; pick one id strategy.

7. Verification gate per PR

prisma validate + generate, cold tsc --noEmit with a 12 GB heap, jest on BOTH __tests__/booking-algorithm and __tests__/payments for every money PR, eslint (warnings block), prettier; chaos 07-real-api-booking + 09-webhook-storm for lock/CAS PRs; a mock-payment dev-server round per PR; no db push against the shared dev DB; CodeRabbit/Gemini triage + SonarCloud PR gate before merge.

[teetangh]

Wave-5 addition (2026-09-02): #1320 — multi-hour consultations are unbookable when availability is stored as adjacent 1-hour rows (prod, verified against the live rows and utils/timeSlotsProcessing.ts:677-684 / checkout.ts:887-951). Ships as PR 3b fix/availability-window-merge, sequenced after PR 3 because both touch checkout.ts. Side finding recorded there: this consultant's rows carry utcOffsetMinutes = -180.