Lazy-init payment SDK clients + #1124 record corrections + Netlify ticket draft

.claude/skills/nextjs-netlify-caching/SKILL.md

@@ -197,6 +197,23 @@ The function runs with `AWS_LAMBDA_FUNCTION_MEMORY_SIZE=1024`, a V8 heap limit o
 
 Do not treat the platform ceiling as a backstop either. Netlify documents 10 s by default and 26 s maximum on paid plans, yet invocations of 26.4 s to 31.9 s were logged on this Pro account. The stall itself is tracked as issue #1124; #1120 is closed by #1123, which established that it is not a database problem.
 
+### The 2026-08-22 memory A/B: assessed and reverted — and what it disproves
+
+The "CPU-starved cold-boot" reading above became testable when Netlify shipped per-function `memory`/`vcpu` config (Credit-based Pro/Ent; `memory` and `vcpu` **scale together**, so setting either tests the same lever). Applied correctly at 2048 MB to the v2 handler and measured under the identical 12-way concurrent unique-key burst protocol:
+
+| Deploy | Handler memory | Result |
+|---|---|---|
+| control (1024 MB) | 1024 | 11/12 slow, TTFB 27.8–31.0 s |
+| treatment | 2048 | 11/12 slow, TTFB 35.9–37.6 s + one platform 500 |
+
+No improvement, possibly worse. Reverted in 08b10ce4. Two conclusions: the CPU-share hypothesis for the stall is **weakened**, not confirmed — doubling per-instance CPU should have shrunk a CPU-bound stall and moved nothing; and any artifact claiming memory "resolved" the stall descends from a misattributed burst that ran during a hyperactive window (three deploys + two agent sessions within nine minutes) where residual warm capacity produced the fast numbers.
+
+**Method traps this cost a day to learn.** Runtime API v2 generates ONE function named `___netlify-server-handler`; overrides targeting v1 names (`___netlify-handler`, `___netlify-odb-handler`) are silently ignored — verify with `netlify api searchSiteFunctions --data '{"site_id":"…"}'` (record field `m`). Netlify deploy IDs do not visibly map to commits: `netlify api listSiteDeploys` → `commit_ref` does, and every cross-deploy claim must use it. Cross-preview A/Bs confound ISR cache freshness with instance-pool age; only same-deploy comparisons count.
+
+**Where this leaves the levers.** Warming workflows (#1148 keep-warm/warm-deploy) protect only the lone-click case — one ping keeps one instance warm and cannot cover bursts. App-side init work is bounded by measurement (solo new instance = full boot + render in ~1.9 s), so shaving SDK init buys fractions of that budget, nothing more. If the tail remains unacceptable after code hygiene, the remaining lever is architectural — always-on compute for SSR — not more warming machinery. A support ticket with the evidence pack lives at `docs/perf/netlify-stall-ticket-draft.md`.
+
+The warm-then-burst close-out (2026-08-23, preview-1148): idle→6-burst stalled 6/6 at 30.6–32.0 s; ~150 s of sustained sequential traffic kept essentially ONE instance warm (every subsequent unique-key RSC fetch served from the ISR/durable cache at ~0.24 s); an immediate 12-burst still stalled 4/12 at 27.8–30.3 s plus a platform 504. Concurrency width alone forces fresh instances into the stall seconds after heavy activity — no ping cadence can prevent it. The same day, a build carrying lazy-initialized Razorpay/Stripe clients (#1221) reproduced the stall at full strength (12/12 slow, 29.5–31.5 s after ≥30 min idle) while its sequential profile was textbook — second independent confirmation, after the CPU-doubling null result, that the stall does not scale with application init work. Do not re-propose bundle-shaving as a stall fix.
+
 ### Fail-open and a cacheable response are safe alone and dangerous together
 
 A fail-open path on an ISR route converts a transient database blip into a cached artefact. `fallbackOnTransientDbError` rethrew during `next build` but degraded at request time, and on `/explore/experts/[consultantId]` that produced HTTP 200 responses carrying the degraded shell at 66 KB against a healthy 104–118 KB. Ten of forty concurrent cold renders came back that way, each with `Cache-Status: "Netlify Durable"; fwd=uri-miss; stored`, and re-fetching them five minutes later returned the same broken page in 0.30–0.64 s with `"Netlify Durable"; hit` and `age: 318–350`. The broken page becomes the *fast* one, which is why nobody notices.

__tests__/payments/confirmation-single-writer.test.ts

@@ -48,6 +48,9 @@ jest.mock("../../lib/payments/core/razorpay", () => ({
   razorpayClient: {
     payments: { fetch: (...a: unknown[]) => paymentsFetch(...a) },
   },
+  getRazorpayClient: () => ({
+    payments: { fetch: (...a: unknown[]) => paymentsFetch(...a) },
+  }),
 }));
 
 const getSession = jest.fn();

__tests__/payments/dispute-earnings-hardening.test.ts

@@ -33,8 +33,12 @@ jest.mock("../../lib/payments/core/razorpay", () => ({
   razorpayClient: {
     payments: { fetch: (...a: unknown[]) => razorpayPaymentsFetch(...a) },
   },
+  // #1221 made utils.ts consume the lazy getter; serve both shapes.
+  getRazorpayClient: () => ({
+    payments: { fetch: (...a: unknown[]) => razorpayPaymentsFetch(...a) },
+  }),
 }));
-jest.mock("../../lib/payments/core/stripe", () => ({ stripeClient: null }));
+jest.mock("../../lib/payments/core/stripe", () => ({ stripeClient: null, getStripeClient: () => null }));
 jest.mock("../../lib/novu", () => ({
   notifyRefundProcessed: jest.fn(),
   notifyDisputeCreated: jest.fn(),

__tests__/payments/dispute-refund-correctness.test.ts

@@ -30,8 +30,9 @@ jest.mock("../../lib/enterprise/system-events", () => ({
 jest.mock("../../lib/payments/core/razorpay", () => ({
   __esModule: true,
   razorpayClient: { payments: { fetch: (...a: unknown[]) => razorpayPaymentsFetch(...a) } },
+  getRazorpayClient: () => ({ payments: { fetch: (...a: unknown[]) => razorpayPaymentsFetch(...a) } }),
 }));
-jest.mock("../../lib/payments/core/stripe", () => ({ stripeClient: null }));
+jest.mock("../../lib/payments/core/stripe", () => ({ stripeClient: null, getStripeClient: () => null }));
 
 // Minimal stubs for the rest of utils.ts's import graph so module load works.
 jest.mock("../../lib/novu", () => ({

__tests__/payments/razorpay-test-key-guard.test.ts

@@ -18,6 +18,9 @@
  *
  * Each test re-requires the modules under jest.resetModules() because both
  * clients initialize at module load / factory-first-call from process.env.
+ * (#1221 made the CORE client construct lazily — the PM-10 guard still fires
+ * at module load as a cheap env check, while SDK construction happens on the
+ * first getRazorpayClient() call. The assertions below follow that split.)
  * Error-code asserts are duck-typed (not instanceof) on purpose: resetModules
  * means the PaymentError class inside the fresh module registry is a
  * different constructor than any top-level import here.
@@ -92,9 +95,7 @@ describe("core checkout/refund client (lib/payments/core/razorpay.ts)", () => {
     process.env.RAZORPAY_KEY_ID = "rzp_test_wrongposture";
     process.env.RAZORPAY_SECRET = "some_secret";
 
-    const thrown = captureThrow(
-      () => requireCoreModule().razorpayClient,
-    );
+    const thrown = captureThrow(() => requireCoreModule());
 
     expect(thrown.message).toMatch(/RAZORPAY_KEY_ID/);
     expect(thrown.message).toMatch(/rzp_test_wrongposture/);
@@ -109,7 +110,7 @@ describe("core checkout/refund client (lib/payments/core/razorpay.ts)", () => {
 
     const mod = requireCoreModule();
 
-    expect(mod.razorpayClient).not.toBeNull();
+    expect(mod.getRazorpayClient()).not.toBeNull();
   });
 
   it("next build phase + prod posture + rzp_test_ key → initializes fine (builds move no money)", () => {
@@ -127,7 +128,7 @@ describe("core checkout/refund client (lib/payments/core/razorpay.ts)", () => {
 
     const mod = requireCoreModule();
 
-    expect(mod.razorpayClient).not.toBeNull();
+    expect(mod.getRazorpayClient()).not.toBeNull();
   });
 });
 

app/api/checkout/verify-signature/route.ts

@@ -33,7 +33,7 @@ import { NextRequest, NextResponse, after } from "next/server";
 import crypto from "crypto";
 import prisma from "@/lib/prisma";
 import { getSession } from "@/lib/auth-server";
-import { razorpayClient } from "@/lib/payments/core/razorpay";
+import { getRazorpayClient } from "@/lib/payments/core/razorpay";
 import { routeCapturedPayment } from "@/app/api/webhooks/razorpay-dispatch";
 import { z } from "zod";
 
@@ -48,6 +48,7 @@ const verifySignatureSchema = z.object({
 
 export async function POST(req: NextRequest) {
   try {
+    const razorpayClient = getRazorpayClient();
     const session = await getSession();
     if (!session?.user) {
       return NextResponse.json({ error: "Unauthorized" }, { status: 401 });

app/api/checkout/verify/route.ts

@@ -2,11 +2,12 @@ import * as Sentry from "@sentry/nextjs";
 import { NextRequest, NextResponse } from "next/server";
 import prisma from "@/lib/prisma";
 import { getSession } from "@/lib/auth-server";
-import { razorpayClient } from "@/lib/payments/core/razorpay";
+import { getRazorpayClient } from "@/lib/payments/core/razorpay";
 import { routeCapturedPayment } from "@/app/api/webhooks/razorpay-dispatch";
 
 export async function GET(req: NextRequest) {
   try {
+    const razorpayClient = getRazorpayClient();
     // Check authentication
     const session = await getSession();
     if (!session?.user) {

app/api/overage/[overageEventId]/order/route.ts

@@ -5,7 +5,7 @@ import { headers } from "next/headers";
 import { NextRequest, NextResponse } from "next/server";
 import {
   createRazorpayOrder,
-  razorpayClient,
+  getRazorpayClient,
 } from "@/lib/payments/core/razorpay";
 import { PaymentStatus } from "@prisma/client";
 import { transitionOverage } from "@/lib/payments/billing/overage-transitions";
@@ -143,6 +143,7 @@ export async function POST(
   // the stored order has already been paid we fall through and mint a new one,
   // because the webhook for the paid order owns that outcome.
   const existingOrderId = event.payment.paymentIntent;
+  const razorpayClient = getRazorpayClient();
   if (razorpayClient && existingOrderId?.startsWith("order_")) {
     try {
       const existingOrder = await razorpayClient.orders.fetch(existingOrderId);

app/api/webhooks/razorpay-dispatch.ts

@@ -32,7 +32,7 @@ import {
   razorpayOrderPaidEventSchema,
   type RazorpayWebhookEnvelope,
 } from "@/schemas/webhooks/razorpay";
-import { razorpayClient } from "@/lib/payments/core/razorpay";
+import { getRazorpayClient } from "@/lib/payments/core/razorpay";
 import { z } from "zod";
 
 // Strict inner-entity schemas used to narrow optional envelope fields at the
@@ -209,6 +209,9 @@ export async function processRazorpayWebhookEvent(
         );
         let paymentIntentId = refundEvent.payment_id;
 
+        // Only the refund family resolves payment_id → order_id via the SDK;
+        // other event branches must not construct a client.
+        const razorpayClient = getRazorpayClient();
         if (razorpayClient) {
           try {
             const rzpPayment = await razorpayClient.payments.fetch(
@@ -259,6 +262,7 @@ export async function processRazorpayWebhookEvent(
         );
         let failedPaymentIntentId = failedRefundEvent.payment_id;
 
+        const razorpayClient = getRazorpayClient();
         if (razorpayClient) {
           try {
             const rzpPayment = await razorpayClient.payments.fetch(

app/api/webhooks/utils.ts

@@ -8,8 +8,8 @@
 } from "@/lib/payments/dispute-status";
 import { Prisma, PaymentGateway } from "@prisma/client";
 import crypto from "crypto";
-import { stripeClient } from "@/lib/payments/core/stripe";
-import { razorpayClient } from "@/lib/payments/core/razorpay";
+import { getStripeClient } from "@/lib/payments/core/stripe";
+import { getRazorpayClient } from "@/lib/payments/core/razorpay";
 import { handlePayoutWebhook } from "@/lib/payments/payouts";
 import {
   notifyRefundProcessed,
@@ -544,6 +544,9 @@
 
   try {
     if (gateway === "stripe") {
+      // Only Stripe verification touches the SDK client; Razorpay verifies
+      // via local HMAC below.
+      const stripeClient = getStripeClient();
       if (!stripeClient) {
         console.error(
           "Stripe client not initialized - cannot verify webhook signature",
@@ -1148,7 +1151,7 @@
 /**
  * Handle dispute created event
  */
 export async function handleDisputeCreated(
   disputeId: string,
   chargeId: string,
   amount: number,
@@ -1159,6 +1162,9 @@
   isChargeRefundable: boolean,
   gateway: "STRIPE" | "RAZORPAY",
 ) {
+  // Only resolve the client the dispute's gateway will use.
+  const stripeClient = gateway === "STRIPE" ? getStripeClient() : null;
+  const razorpayClient = gateway === "RAZORPAY" ? getRazorpayClient() : null;
   // Resolve `chargeId` to OUR paymentIntent BEFORE opening the transaction.
   // This lookup is an external HTTP call to Stripe or Razorpay; leaving it
   // inside the tx held a database transaction open across a network round trip,

docs/perf/netlify-stall-ticket-draft.md

@@ -0,0 +1,55 @@
+# Netlify Pro support ticket — DRAFT (for kaustav to file)
+
+> Subject: Next.js server handler (`___netlify-server-handler`): brand-new instances block their event loop ~24s on first invocation when created concurrently — is this expected scale-out behavior?
+>
+> Site: familiarise.netlify.app (site id `1a1ad7d0-fda0-4efe-9d58-aa0ce0fd6d5c`)
+> Plan: Pro (Credit-based) · Region: `ap-southeast-1` functions · Adapter: `@netlify/plugin-nextjs@5.15.13` (runtime API v2, single consolidated SSR+ISR function) · Next.js 15.5.15, Node 22
+
+## Summary
+
+Since at least July 2026 we have measured a reproducible, bimodal latency pathology on the Next.js server handler. A function instance created **in isolation** boots and serves a real database-backed page end-to-end in **~1.9s**. Instances created **under concurrent load** each stall for roughly **24 seconds with a blocked event loop before executing any application code**, then serve normally. There is nothing between the two modes: across ~90 instrumented cold renders we observed zero samples between ~6s and ~31s.
+
+## Evidence
+
+**1. Bimodality correlates exactly with instance creation count.**
+Four batches against one deploy preview, client-side TTFB via curl, correlated with function logs and an in-app diagnostic route that reports per-instance id + `process.uptime()` + event-loop-lag probe:
+
+| Batch | Concurrency | Instance state | Samples | Result |
+|---|---|---|---|---|
+| A | strictly sequential | new each time | 8 | 1.80–2.72s, no outliers |
+| B | 12 concurrent | ~6 pre-existing | 12 | six at 1.9–4.7s, six at 31.0–33.1s |
+| C | 16 concurrent | ~12 pre-existing | 16 | twelve at 2.6–2.9s, four at 30.8–33.1s |
+| D | 12 concurrent | all warm | 12 | 3.3–5.9s, zero slow |
+
+Slow-count equals newly-created-instance-count in every batch. The diagnostic route confirmed every stalled sample ran on an instance aged <100ms serving invocation #1.
+
+**2. The stall is an event-loop block BEFORE any application work.**
+On stalled first invocations, a diagnostic route that awaits 400ms of idle *before* touching the database reported the idle phase taking **23.9–24.8s**, max loop lag 23.7–24.7s, while instance age was <100ms. The subsequent DB query connected in ~0.9–1.0s. On warm instances the same probe shows 400–453ms / lag 1–70ms. Downstream effects: `pg` connect timers are plain `setTimeout`s, so they fire only after the stall ends (~26s), which initially misdiagnosed this as a database problem.
+
+**3. Memory/CPU scaling does not touch it.**
+We configured the v2 handler correctly by name (`___netlify-server-handler`; verified via `searchSiteFunctions`, field `m`) at **2048 MB** — i.e. doubled vCPU, since your docs state memory and vCPU scale together. Result under the identical 12-way burst protocol:
+
+| Config | Deploy id (ready UTC 2026-08-22) | commit_ref | searchSiteFunctions `m` | Result |
+|---|---|---|---|---|
+| control 1024 MB | `6a894a2398d6…` (07:05) / `6a895a3f4651…` (08:13) | 74f58138 / 08b10ce4 | 1024 | 11/12 slow, TTFB 27.8–31.0s |
+| **treatment 2048 MB** | **`6a8954981e6f…` (07:49)** | **17228d7e** | **2048** | **11/12 slow, TTFB 35.9–37.6s + one platform 500** |
+| post-revert re-run | `6a8974a11d65…` (10:06) | 0646d8f5 | 1024 | 12/12 slow, TTFB 32.6–38.0s |
+
+(An intermediate burst on `6a895c2e7d70…`/58fb03fc at 08:22 came back 16/16 fast — an anomaly attributable to residual warm capacity from three deploys and two concurrent agent sessions within nine minutes, not to the memory setting; recorded for completeness.)
+
+No improvement (possibly worse). We reverted.
+
+**4. Not our bundle's init work.** Sequential brand-new instances complete module loading + init + a full SSR render in <2s total, so first-invocation application work cannot account for 24s; and if the stall were proportional to per-instance init CPU, doubling CPU should have moved it. Confirmed again on 2026-08-23: a build carrying lazy-initialized payment SDK clients (#1221, deploy-preview-1221, commit 353cef1e) still stalled **12/12 at 29.5–31.5s** after ≥30 min idle, while its sequential profile was normal (first-ever request 5.84s settling to ~0.26s warm).
+
+**5. Observability gap:** this function emits no `Init Duration:` log line (only `Duration:`/`Memory Usage:`), so cold starts can't be discriminated from logs; we had to build an in-app instance-age probe. A forum report from May 2025 describes the same absence.
+
+## Questions
+
+1. Is concurrent instance-creation contention (e.g., simultaneous sandbox provisioning, deployment-artifact fetch, or shared-host CPU scheduling during burst scale-out) a known cause of multi-second stalls on runtime-API-v2 handlers? Is there a known incident or fix in flight since mid-2026?
+2. Does Netlify have, or plan, anything equivalent to provisioned concurrency / minimum instances for framework-generated functions like `___netlify-server-handler`? Scheduled keep-warm pings keep at most one instance warm and cannot protect bursts.
+3. Why does the server handler not emit AWS-style `Init Duration` in its logs, and are there plans to expose it? It makes cold-start SLO work impractical.
+4. Any guidance on reducing burst-time instance-creation latency from within the deployment (bundle shape, esbuild vs default bundling, region placement), given memory/vcpu scaling showed no effect?
+
+## Impact
+
+User-visible: landing-page/explore clicks stall 20–30s then render (the "site is down" perception), worst right after deploys and during traffic bursts from a cold pool. We ship ISR-first architecture and deploy-warming workflows, but the tail persists whenever concurrency forces new instances.