Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
d567933
perf(explore): convert /explore/programs to ISR, resolve org badges c…
claude Aug 13, 2026
ebfe492
perf(explore): seed the experts browse grid from the ISR render
claude Aug 13, 2026
d6d87ab
perf(explore): seed the All Programs grid and smooth the session key …
claude Aug 13, 2026
e20e1b9
fix(explore): FCP-capable loading states, experts error boundary, rea…
claude Aug 13, 2026
2066efe
test(explore): anchor the perRequest detector on a fixture, not a liv…
claude Aug 13, 2026
ba5de63
Merge remote-tracking branch 'origin/dev' into claude/explore-pages-l…
claude Aug 13, 2026
83731ab
refactor(explore): deduplicate error cards and hero copy for the Sona…
claude Aug 13, 2026
887237a
fix(explore): stop shadowing the Error global in segment error bounda…
claude Aug 13, 2026
5c15085
fix(explore): address CodeRabbit review — pagination bounds, error co…
claude Aug 13, 2026
b3cb5ed
Merge remote-tracking branch 'origin/dev' into claude/explore-pages-l…
claude Aug 13, 2026
65aa2a7
Merge branch 'dev' into claude/explore-pages-loading-nyibb0
teetangh Aug 15, 2026
7726eb2
Merge remote-tracking branch 'origin/dev' into claude/explore-pages-l…
teetangh Aug 22, 2026
74f5813
perf(platform): 2GB handler memory, post-deploy warm-up, prod keep-wa…
teetangh Aug 22, 2026
289d884
fix(ci): stagger keep-warm off 00:05 for the workflow-hygiene guard
teetangh Aug 22, 2026
f2e5d1a
fix(ci): keep-warm loops its 5-minute pings inside one hourly job
claude Aug 22, 2026
634c0d5
fix(ci): address CodeRabbit review on the warm-up workflows
claude Aug 22, 2026
886d7bf
test(ci): give the cron-lock registry an HTTP-only workflow category
claude Aug 22, 2026
17228d7
fix(netlify): target ___netlify-server-handler — the v1 handler names…
teetangh Aug 22, 2026
08b10ce
revert(netlify): 2048 MB handler memory measured no better, possibly …
teetangh Aug 22, 2026
ecaf160
Merge remote-tracking branch 'origin/dev' into claude/explore-pages-l…
claude Aug 22, 2026
6ff6038
fix(ci): move keep-warm to :03 — dev's expire-stale-requests took :10
claude Aug 22, 2026
58fb03f
fix(ci): address CodeRabbit round 3 on the warm-up workflows
claude Aug 22, 2026
0646d8f
docs(skill): record the stall resolution — 2048 MB eliminates it; han…
teetangh Aug 22, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions .claude/skills/nextjs-netlify-caching/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -212,14 +212,46 @@ The fix, shipped in #1123, is that degrading is now opt-in per call site (`perRe

So on an ISR route, throwing is strictly better than degrading: the one unlucky visitor gets an error boundary, everyone else keeps the last good copy, and nothing bad is written down.

## The stall was resolved on 2026-08-22 by doubling handler memory — and the function changed names under us

The ~24 s cold-instance event-loop stall described above is **confirmed CPU-proportional cold-boot work and is eliminated at 2048 MB**. Measured as an A/B against two live previews in the same minute, same protocol (N concurrent unique-key RSC requests to `/explore/experts` on a freshly deployed function):

| handler | N | stalled | slow-mode TTFB | fast-mode |
|---|---|---|---|---|
| 1024 MB (preview 1195) | 12 | 11/12 | 27.8–31.0 s | 0.75 s |
| 1024 MB (preview 1148, see hazard below) | 12 | 11/12 | 32.8–35.1 s | 3.1 s |
| **2048 MB (preview 1148)** | 16 | **0/16** | — | **3.1–4.9 s** |
| 1024 MB control re-run, same minute | 12 | 12/12 | 31.9–39.6 s (+4 bare platform-500s) | — |

Warm-mode render times were unchanged (~3–5 s), so this is pure tail removal. The fix shipped in PR #1148 as `[functions."___netlify-server-handler"] memory = 2048` in `netlify.toml`. Billing scales linearly with configured memory. Full write-up on issue #1124.

Three facts that each cost an afternoon:

**The generated function is `___netlify-server-handler` at @netlify/plugin-nextjs@5.15.13 (runtime API v2) — one consolidated SSR+ISR handler.** The classic `___netlify-handler` / `___netlify-odb-handler` names belong to runtime v1 and match nothing. Re-enumerate names after any adapter bump (`netlify api searchSiteFunctions --data '{"site_id": …}'`; the record's `m` field is the configured memory).

**A `[functions."name"]` block targeting a nonexistent name is silently inert.** No warning, deploy green, config present in the repo. An earlier draft of the memory bump targeted the v1 names; its preview burst read 32.8–35.1 s and looked like a genuine refutation of the memory hypothesis. Verify the treatment landed (`searchSiteFunctions`, or a distribution shift against a same-minute control) before believing any null result.

**Netlify per-function `memory`/`vcpu` exists since ~2026-07 on Credit-based Pro/Enterprise** (1024–4096 MB, 0.5–2.0 vCPU). This section previously recorded the stall as unmitigable because that lever did not exist when #1124 was written. It also cannot be set for framework-generated functions via in-source `config` exports — `netlify.toml` keyed by generated name works.

Two warmers ship alongside (#1148): `warm-deploy.yml` fires on Netlify `deployment_status: success` and primes `/api/health` → hot pages sequentially → RSC payloads per deploy; `keep-warm.yml` starts hourly at `:10` and loops 5-minute `/api/health` pings inside the job (every minute of the GH cron map is owned by some multi-daily fleet job, so no sub-hourly cron lattice passes check-workflow-hygiene). Both activate only after merging to the default branch: `schedule` and `deployment_status` triggers read the workflow file from the default branch, not from a PR's merge ref — a preview of the PR adding them proves nothing about them.


## Options assessed and rejected — do not re-propose without new information

Partial Prerendering and Cache Components are not merely "a Next 16 feature" — they are unreachable from our pinned version. At `next@15.5.15`, `packages/next/src/server/config.ts` throws `CanaryOnlyError` on a stable build for both `experimental.ppr` and `experimental.cacheComponents`, so even `experimental.ppr = "incremental"` fails at config load rather than degrading. Next's own [ppr-preview](https://nextjs.org/docs/messages/ppr-preview) page confirms a canary release is required. This matters because PPR is the textbook answer to "a static page with one dynamic hole", and on this version that answer simply does not exist — a `Suspense` boundary around a dynamic read does **not** rescue static rendering without PPR. Separately, `use cache` would not help a route whose every segment is auth-gated, and every dashboard route here is auth-gated.

Caching a dynamic route at the CDN with `Netlify-CDN-Cache-Control` is technically sound and was considered for the public pages, but it still invokes the function on every cache miss, so it does not solve the cold start the way prerendering does. It remains the right tool when build-time data access is genuinely unacceptable.

**Raising the handler's memory is measured dead (2026-08-22, PR #1148 branch).** Netlify now exposes per-function `memory` up to 4096 MB on Credit-based Pro/Enterprise (`[functions."NAME"]` in `netlify.toml`; unavailable when #1124 was first written). Same 12-way concurrent unique-key burst protocol on deploy previews: 1024 MB control put 11/12 requests at 27.8–31.0 s; a correctly-applied 2048 MB treatment put 11/12 at 35.9–37.6 s with one platform 500. No improvement, possibly a regression. The "CPU-starved boot" reading of the #1124 stall is therefore weakened — do not re-pull this lever without a new mechanism hypothesis.

Verified clean and not worth re-investigating: `next/image` usage (there are zero raw `<img>` tags), fonts (`next/font/google` with `display: swap`), `staleTimes`, `serverExternalPackages`, the Prisma singleton, and the bundle-analyzer tooling.

## Function-name and config-verification facts (added 2026-08-22)

Runtime API v2 of `@netlify/plugin-nextjs@5.15.13` generates **one** function named `___netlify-server-handler`. The classic v1 names `___netlify-handler` / `___netlify-odb-handler` match nothing on this site; a `[functions."___netlify-handler"]` block in `netlify.toml` is **silently ignored**, and every request keeps serving at 1024 MB while the config reads as if it were doing something — an earlier #1124 attempt lost its entire first measurement to this. Enumerate real names and per-function memory with `netlify api searchSiteFunctions --data '{"site_id": …}'` (response shape: list of deploys → `.functions[].n` name, `.functions[].m` memory MB); note it scopes to the published production deploy, not preview deploys. Re-enumerate names after any adapter bump.

Two more verification gotchas from the same day: GitHub webhook-style triggers (`deployment_status`, `schedule`) read the workflow file **from the default branch only**, so new warm-up workflows stay inert on PR previews until merged to dev — CI green does not mean they ran. And `scripts/ci/check-workflow-hygiene.ts` rejects any two recurring crons sharing a start minute; every minute of the day is owned by some multi-daily workflow in this fleet, which is why keep-warm runs as one hourly start (:10) looping 5-minute pings inside the job rather than as a `*/5` cron.

## Project constraints that constrain every change here

ESLint warnings are blocking, because SonarCloud fails the quality gate on unused variables. Never filter ESLint output for errors alone.
Expand Down
116 changes: 116 additions & 0 deletions .github/workflows/keep-warm.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
name: Keep Warm

# Keep at least one warm function instance on PRODUCTION between real traffic.
#
# The Next.js server handler stalls its event loop for ~24s on a brand-new
# instance's first invocation (#1124). A single warm instance does not remove
# that stall — burst traffic still spawns fresh instances that each pay it —
# but it removes the stall from the COMMON case: a lone visitor clicking
# through an otherwise idle site, which is exactly the shape of manual testing
# and low-traffic days.
#
# Every ping is a billable invocation (~12/hour ≈ 8.6k/month at the 5-minute
# cadence); disable this workflow if that trade stops making sense. Previews
# are deliberately NOT warmed: each PR has its own isolated cache scope and
# subdomain, so warming one preview does nothing for another, and idle previews
# going cold is unavoidable (warm-deploy.yml covers their first minutes).
#
# GitHub schedule triggers are best-effort with no SLA and can lag several
# minutes; only the hourly START is exposed to that lag — the 5-minute ping
# cadence itself runs as an in-job loop and is exact.

on:
schedule:
# Hourly at :03, with the 5-minute ping cadence implemented as a loop
# INSIDE the job rather than as a */5 cron. The check-workflow-hygiene
# gate forbids two multi-daily crons sharing any start minute, and this
# fleet's minute map is dense enough that no 12-per-hour (or even
# 6-per-hour) schedule has a clean lattice — an offset grid like 3-59/5
# still collides with the hourly and */6 jobs at :08/:13/:18/:23/:28/
# :33/:38/:58. One hourly start on a free minute keeps the gate green
# while the loop below preserves the real warm cadence; it also shrinks
# exposure to GitHub's best-effort schedule lag (one delayed start per
# hour instead of twelve).
#
# :03 because the free-minute pool is nearly empty and SHRINKS as dev
# moves: this job started at :10 and dev's expire-stale-requests went
# hourly-at-:10 the same day, colliding in the PR merge build. As of
# 2026-08-22 the only minutes free of multi-daily crons are :00, :03
# and :15 — re-derive that set before moving this again.
- cron: "3 * * * *"
workflow_dispatch:

permissions:
contents: read

concurrency:
group: keep-warm
# A lagged start can overlap the previous hour's loop; the newer run
# supersedes it rather than double-pinging.
cancel-in-progress: true

jobs:
ping:
runs-on: ubuntu-latest
# One hour of 5-minute pings, then exit before the next scheduled run.
timeout-minutes: 58
Comment thread
coderabbitai[bot] marked this conversation as resolved.

steps:
# The canonical production origins: the Netlify host and the pinned
# NEXT_PUBLIC_APP_URL domain. Edge cache scopes are per-host, so both are
# pinged; the function instance underneath is shared.
#
# /api/health guarantees a handler invocation (a page GET can be served
# entirely off the CDN cache without touching the function), so instance
# warmth does not depend on cache misses. It is also the cheapest route
# that proves end-to-end liveness: LIMIT-1 DB probe + status reads, and
# doubles as an outage signal — if these pings start failing, prod is
# degraded whether or not any user has noticed yet.
- name: Ping production every 5 minutes for an hour
env:
HOSTS: "https://familiarisenow.com https://familiarise.netlify.app"
run: |
# 11 rounds x 5-minute sleep ≈ 55 minutes of coverage per hourly
# run; the next run picks up next hour. Pings are sequential and the
# loop sleeps between rounds, so this never bursts the function.
#
# DEADLINE guard: the happy path fits easily, but if both hosts hit
# --max-time on the initial attempt AND the retry every round, the
# arithmetic exceeds the 58-minute job timeout (~95 min worst case)
# and GitHub would cancel mid-loop. Stop starting new rounds once
# 50 minutes have elapsed instead — the next hourly run continues.
#
# /api/health is fail-open: HTTP 200 with status "degraded" when the
# DB is unreachable, so the body is parsed too. Degradation is a
# WARNING here, not a failure — the ping still warmed the instance,
# and the loop keeps running so warmth (and the outage signal)
# continue through the incident.
DEADLINE=$((SECONDS + 50 * 60))
for round in $(seq 1 11); do
if [ "$SECONDS" -ge "$DEADLINE" ]; then
echo "::notice::keep-warm stopping at round $round — 50-minute deadline reached"
break
fi
for host in $HOSTS; do
if BODY=$(curl -sS -w "\n$host/api/health: ttfb=%{time_starttransfer}s code=%{http_code}" \
--fail --max-time 60 --retry 1 --retry-delay 3 \
"$host/api/health"); then
echo "${BODY##*$'\n'}"
STATUS=$(printf '%s' "$BODY" | head -n 1 | jq -r '.status // "unknown"')
if [ "$STATUS" != "healthy" ]; then
echo "::warning::keep-warm: $host reports health status '$STATUS' — prod degraded"
fi
else
echo "::warning::keep-warm ping failed for $host"
fi
done
if [ "$round" -lt 11 ]; then sleep 300; fi
done

# NOTE: hot page caches are deliberately NOT refreshed here. The ISR
# windows on / and /explore/experts are 1h/5m; refreshing them every 5
# minutes would force continuous background regeneration invocations for
# content that changes on the order of days. The durable cache already
# persists entries across instances (Netlify Blobs), so cache priming
# only matters right after a deploy — that is warm-deploy.yml's job.
# This workflow's single job is instance warmth via the health ping.
133 changes: 133 additions & 0 deletions .github/workflows/warm-deploy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
name: Warm Deploy

# Prime the caches and function instances of a freshly finished deploy.
#
# A deploy starts every ISR/CDN cache entry empty, and a Netlify function
# instance that has never served a request stalls its event loop for roughly
# 24s before executing anything (#1124 — measured 27.8–31.0s TTFB across 11/12
# concurrent unique-key requests on preview 1195). The FIRST real visitor after
# each deploy therefore pays full price per hot route. This workflow pays it
# instead, in a controlled order:
#
# 1. /api/health first — guarantees one function invocation, so the instance
# that will serve everything else absorbs the cold-boot stall here rather
# than on a page render.
# 2. Hot public pages sequentially afterwards — primes their CDN/durable
# entries while the instance from step 1 is still warm. SEQUENTIAL ON
# PURPOSE: concurrent pings would spawn one stalled instance EACH, which
# is precisely the pathology being mitigated.
#
# Fires on Netlify's deployment statuses for production deploys, branch deploys
# AND pull-request previews — previews are where the cold cost is worst,
# because their cache scope is isolated and nothing is ever warm.
#
# This does not keep instances alive (see keep-warm.yml); it only removes the
# empty-cache window right after a deploy.

# deployment_status supports no trigger-level filter (no `types`/`states`
# narrowing exists for it) — the workflow fires on every status event and the
# job-level `if` below is the real gate.
on:
deployment_status:
workflow_dispatch:
inputs:
base_url:
description: "Deploy origin to warm (defaults to production)"
required: false
default: "https://familiarisenow.com"

permissions:
contents: read

concurrency:
group: warm-deploy-${{ github.event.deployment.id || github.run_id }}
cancel-in-progress: false

jobs:
warm:
# Netlify also reports inactive/other states; only successful deploys with
# a resolvable environment URL are worth warming.
if: github.event_name == 'workflow_dispatch' || github.event.deployment_status.state == 'success'
runs-on: ubuntu-latest
timeout-minutes: 10

steps:
- name: Resolve deploy URL
id: url
env:
EVENT_URL: ${{ github.event.deployment_status.environment_url }}
PAYLOAD_WEB_URL: ${{ github.event.deployment.payload.web_url }}
DISPATCH_URL: ${{ inputs.base_url }}
run: |
BASE="${DISPATCH_URL:-${EVENT_URL:-${PAYLOAD_WEB_URL:-}}}"
if [ -z "$BASE" ]; then
echo "::error::No environment_url on the deployment status; cannot warm."
exit 1
fi
# Strip a trailing slash so the path concatenations below stay clean.
BASE="${BASE%/}"
# The URL comes from the deployment event payload, i.e. from outside
# this repo. It is only ever used through env indirection below, but
# validate the shape anyway so nothing resembling shell syntax (or a
# control character) ever reaches a run script or GITHUB_OUTPUT.
if ! printf '%s' "$BASE" | grep -Eq '^https://[A-Za-z0-9.-]+(:[0-9]+)?$'; then
echo "::error::environment_url '$BASE' is not a bare https origin; refusing to warm"
exit 1
fi
echo "base=$BASE" >> "$GITHUB_OUTPUT"
echo "Warming $BASE"

# The first request against a brand-new instance can legitimately take
# ~30s (#1124), so max-time must sit well above that or the warm-up would
# kill the very request doing the warming.
#
# /api/health is deliberately fail-open: it answers HTTP 200 with
# status "degraded" + database "unreachable" when the DB is down, so a
# bare curl exit code is NOT a health verdict. Parse the body: warming a
# degraded origin is worse than useless — the ISR pages rethrow on DB
# failure (#1123), so every page ping would 500, cache nothing, and the
# run would still report success. Fail here instead so the red run says
# "deploy is up but degraded; nothing was warmed".
- name: Warm the function instance (health probe)
env:
BASE_URL: ${{ steps.url.outputs.base }}
run: |
BODY=$(curl -sS -w "\nhealth: ttfb=%{time_starttransfer}s code=%{http_code}" \
--fail --max-time 60 --retry 1 --retry-delay 2 \
"$BASE_URL/api/health")
echo "${BODY##*$'\n'}"
STATUS=$(printf '%s' "$BODY" | head -n 1 | jq -r '.status // "unknown"')
if [ "$STATUS" != "healthy" ]; then
echo "::error::health status is '$STATUS' (not healthy) — origin degraded, skipping page warming"
exit 1
fi

# Hot public routes in click-order value. Keep this list short and
# deliberate: every line is a billable invocation, and sequential curls
# share the instance warmed above.
- name: Warm hot public pages
env:
BASE_URL: ${{ steps.url.outputs.base }}
run: |
BASE="$BASE_URL"
for path in "/" "/explore/experts" "/explore/programs" "/explore/community" "/about"; do
curl -sS -o /dev/null -w "$path: ttfb=%{time_starttransfer}s code=%{http_code}\n" \
--max-time 45 "$BASE$path" || true
sleep 1
done

# RSC navigation payloads are what an actual <Link> click fetches; priming
# them means the first soft navigation hits the CDN instead of the
# function. A generic RSC request also re-walks the render path, keeping
# the instance hot.
- name: Warm RSC navigation payloads
env:
BASE_URL: ${{ steps.url.outputs.base }}
run: |
BASE="$BASE_URL"
for path in "/explore/experts" "/explore/programs" "/about"; do
curl -sS -o /dev/null -H "RSC: 1" \
-w "rsc $path: ttfb=%{time_starttransfer}s code=%{http_code}\n" \
--max-time 45 "$BASE$path?_rsc=warmup" || true
sleep 1
done
Loading
Loading