A newly created function instance stalls its event loop for ~24s before any application work

[teetangh]

What was measured

A newly created Netlify function instance stalls its event loop for roughly 24 seconds on its first invocation, before any application work. This is the whole of the ~30 s tail on public routes, and it is what made the Prisma connect budget look broken in #1120 — a setTimeout cannot fire on a blocked loop, so pg's 3 s connect timer reported at ~26 s.

Measured on deploy preview 1123 with a temporary diagnostic route that ran 400 ms of pure idle await, touching no database at all, while sampling event-loop lag every 50 ms. Forty invocations across two 20-way concurrent rounds:

instance age	process.uptime()	invocation	400 ms idle phase took	max loop lag	first DB attempt
40 ms	3,363 ms	1	23,905 ms	23,746 ms	ok in 862 ms
39 ms	3,361 ms	1	24,634 ms	24,534 ms	ok in 866 ms
100 ms	3,355 ms	1	24,823 ms	24,655 ms	ok in 1,037 ms

The remaining 37, every one of them on an instance at least 47 s old, completed the same idle phase in 400–453 ms with 1–70 ms of lag. So the stall is bimodal, it is confined to brand-new instances serving invocation 1, and the database is fast on both sides of it.

Why it matters

It is the only remaining cause of 30-second page loads on the public routes, and it scales with instance creation, which means it fires exactly when traffic arrives in a burst. It cannot be mitigated from application code: #1123 established that no timeout value can bound it, and shipped the only two things that help — retrying past it, and serving from the ISR cache so the function is not invoked at all.

What is not yet known

The cause of the stall is not identified. The obvious candidate is cold-instance JavaScript evaluation and GC at Lambda's memory-proportional CPU share, but that is inference. The facts in hand:

• AWS_LAMBDA_FUNCTION_MEMORY_SIZE is 1024, so the function gets roughly 0.58 vCPU.
• V8 reports a heap limit of 1,018 MB, and RSS at rest is 675–795 MB. The instance is running close to its container.
• process.env.NODE_OPTIONS is null inside the function, so netlify.toml's --max-old-space-size=6144 is not leaking into the runtime and letting V8 overrun the container. That hypothesis is ruled out.

Worth trying, in rough order of expected value

1. Raise the function's memory, which raises its CPU with it. Netlify does not expose this on the current plan, so this is a conversation with Netlify rather than a config change, but it is the most direct lever if the stall is CPU-bound.
2. Reduce what the server handler has to evaluate on a cold instance. perf: trim optimizePackageImports to packages Next does not already handle #1109 already trimmed optimizePackageImports; a bundle analysis of the server handler specifically would say whether there is more to take out.
3. Confirm the cause before optimising anything. A CPU profile of a cold invocation would settle it, and the diagnostic-route technique in fix: stop caching degraded renders, and correct the root cause of the 30s tail #1123 is the way to get data out — the function log strips our console.* and does not expose instance identity, so the response body is the only reliable channel.

Follow-up to #1120. Related: #1117 (PG_POOL_MAX=1), #1122 (console.* is stripped in production).

[teetangh]

Is any of this our module-load cost, or is it entirely the platform?

Asked while merging #1123. Answering from the diagnostic data already collected rather than by reasoning about bundle size, and marking clearly what was not measured.

What the data says, and it mostly exonerates our modules

The diagnostic route reported process.uptime() at handler entry alongside the age of its own module, which pins the cold-instance timeline. On all three stalled samples:

	value
process.uptime() at handler entry	3,355 / 3,361 / 3,363 ms
age of our route module at handler entry	100 / 39 / 40 ms
therefore, process start to our module being evaluated	about 3.26–3.32 s
our route module's own evaluation, including constructing the Prisma client	inside that 39–100 ms window
the stall, measured afterwards	23,905 / 24,634 / 24,823 ms

The decisive point is when the stall happens. It happens after our module graph is evaluated, while the handler is doing nothing but await new Promise(r => setTimeout(r, 50)) in a loop. No application code of ours was scheduled during those 24 seconds. So the stall is not our modules being parsed and evaluated, because ours were already parsed and evaluated and sitting idle.

Two caveats that keep this short of proof. Next and the Netlify adapter may evaluate further modules lazily on the first request, which would be framework cost rather than ours but would still be module-load cost. And a garbage collection pause on an instance whose RSS is 745–751 MB against a 1,024 MB container would look identical from the outside.

What our side does measurably cost on a cold instance

Two things, both small relative to 24 seconds, and both real:

• About 3.3 seconds from process start to our first module being evaluated. This is framework bootstrap plus our module graph, and the diagnostic does not separate the two. It is an upper bound on our contribution to that phase, not an attribution.
• Roughly 0.5 to 1 second of Prisma first-query initialisation. The first query on a cold instance took 862–1,037 ms where warm instances took 65–330 ms. lib/prisma.ts constructs PrismaClient and PrismaPg at module scope, so client construction is inside the module-evaluation window above; what is deferred is the engine work on first use.

Not measured

The server handler's module count and bundle size at boot were not measured. serverExternalPackages was not re-examined for this question — the caching skill records it as previously verified clean, which is not the same as verified for boot cost. No CPU profile of a cold invocation was taken.

The experiment that would settle it

Deploy two Route Handlers to one preview and compare them on newly created instances:

1. One that imports nothing from @/lib or @/app — no Prisma, no auth, no Sentry — and runs the same idle-lag probe.
2. One that imports the full application graph the public routes pull in, running the identical probe.

Then read, for each, whether the idle phase stalls and what process.uptime() is when its module is first evaluated.

If the zero-import handler stalls for the same ~24 seconds, our module-load cost is not a contributor and the only levers are Netlify's: function memory, and therefore CPU. If it does not stall, our graph is implicated and reducing what the server handler evaluates at boot becomes worth real effort.

Until that runs, treat "shrink the server bundle to fix the 30 second tail" as unsupported. The evidence in hand points at CPU starvation on a fresh instance, not at how much JavaScript we ship.

Method note for whoever runs it: the function log strips our console.* (compiler.removeConsole) and does not expose instance identity, so the diagnostic has to return its findings in the response body. That is what #1123 did.