Web Map Client pod crashes (OutOfMemory / OOMKilled) under traffic bursts

[arnoldcastro5000]

TL;DR (the short version)

The production web map client pod runs out of memory and gets killed by Kubernetes (OOMKilled), then restarts. It is not a slow memory leak. It happens when a burst of requests (looks like a web crawler) hits the token detail page (/tokens/<id>). That page is the only page we do not cache, so every single request makes the server do expensive work and talk to our backend API ~5 times. During a burst, hundreds of these pile up at once, the server's memory fills with pending network connections, and the kernel kills the process in about 75 seconds.

This needs both a quick fix (stop the bleeding) and a few follow-up fixes.

---

Background: a few terms you'll see

If you already know these, skip ahead.

• Pod / container – the running copy of our app inside Kubernetes.
• OOMKilled – "Out Of Memory Killed." Each pod has a memory budget (ours is 1792 MiB). If the app uses more than that, the Linux kernel force-kills it. Exit code 137.
• SSR (Server-Side Rendering) – for some pages, our server builds the full HTML on every request instead of in the browser. This is more work per request.
• ISR / caching – Next.js can build a page once and reuse it for a while (e.g. 30 seconds). Most of our pages do this. Cached pages are cheap to serve.
• Event loop – Node.js runs our JavaScript on one single thread. If one task hogs that thread, everything else waits in line, including finishing other requests.
• Heap vs non-heap memory – "heap" is memory for JavaScript objects. "Non-heap" is everything else: open network connections (sockets) and the raw bytes of API responses we're still downloading. Our crash is caused by non-heap memory, which is why we never see a normal "JavaScript out of memory" error.

---

What was observed

From the pod description:

• Image: greenstand/treetracker-web-map-client:2.9.10
• Memory limit: 1792Mi, request: 1300Mi
• Last State: Terminated — Reason: OOMKilled — Exit Code: 137
• Restart Count: 1 — the container ran for ~5 days (Jun 18 → Jun 23) and was killed once. This is not (yet) a crash loop.

From the container logs of the crash (with timestamps):

Note on the log window: kubectl logs --previous only returns the tail of the log that survived rotation, so we only have the final ~2 minutes before the kill (02:10:32 → 02:12:26 UTC, which matches the recorded death time once you convert the describe's EDT timestamps to UTC). This is the end of the story, not the container's whole life.

• 352 requests to /tokens/<id> arrived in ~76 seconds — about 5 per second. This looks like an automated crawler walking through token links.
• 19,000+ log lines in under 2 minutes — the app was drowning in its own logging.
• Many connection errors to the backend API: ECONNRESET (46) and EPIPE (81).
• No "JavaScript heap out of memory" error anywhere. This confirms the memory that overflowed was non-heap (network connections + half-downloaded responses), not JavaScript objects.

From a log of how long backend API calls took:

• 353 of 412 calls took longer than 10 seconds. Some took 70–107 seconds.
• Suspicious clue: eight different API calls (/organizations/179, /194, /7, /9, …) all reported almost exactly 72.5 seconds. Independent slow calls would have varied times. When many different calls all finish at the same instant, it possibly means they were stuck waiting for the single Node.js thread to become free, i.e. the server itself was overloaded, not (only) the backend.

---

Why it happens (the root cause, step by step)

1. A burst of ~5 requests/second hits the token detail page (/tokens/<id>).
2. That page is the only page that is not cached — it uses getServerSideProps, so the server rebuilds it from scratch every single time. (Other pages like trees/wallets/planters are cached and reused.)
3. Each token page render makes ~5 back-to-back calls to the backend API (token → wallet → transactions → tree → planter).
4. On every render we also dump huge objects to the logs (the whole auth config, router, and full error objects). Writing all this is slow and blocks the single Node.js thread.
5. Because the thread is blocked and we set no timeout on our API calls, hundreds of requests and their network connections stay open at the same time, each holding memory for the response it's waiting
   on.
6. All these open connections and buffered responses are non-heap memory. They add up past the 1792 MiB limit.
7. The kernel kills the pod (OOMKilled) and it restarts.

In one sentence: an uncached, expensive page + a request burst + no timeouts + heavy logging = the single server thread gets overwhelmed and memory fills with pending connections until the pod is killed.

Open question: we know a burst killed it at the moment of death. We do not yet know whether memory also crept up slowly over the 5 days (a leak) and left less headroom. To tell the difference we need
the memory-over-time graph (kubectl top history or Prometheus container_memory_working_set_bytes). The fixes below help either way.

---

Impact

• The public map / token / wallet pages go down or become very slow whenever a burst happens.
• The pod gets OOMKilled and restarts, causing a window of downtime. (So far this has happened once; with a single replica there is no backup pod during the restart.)
• Our backend query API also looks slow/overloaded during these events, which may affect other services.

---

How to reproduce (roughly)

1. Send a few requests per second to many different /tokens/<id> URLs at once.
2. Watch memory climb with kubectl top pod -n webmap -l app=treetracker-web-map-client-main --containers.
3. The pod's memory rises toward 1792Mi and it gets OOMKilled within a couple of minutes.

---

Proposed fixes (in priority order)

Highest priority — stop the OOM kills

1. Cache the token detail page like our other pages (switch /tokens/[tokenid] from getServerSideProps to cached/ISR with getStaticProps + revalidate). This means a burst of repeat requests is served from cache instead of re-rendering every time.
   • File: src/pages/tokens/[tokenid].js
2. Limit how many pages we render at once (a concurrency cap), so a burst can't open hundreds of renders + connections simultaneously.
3. Stop dumping large objects to the logs in production (auth config, router, full API error objects). This logging is actively blocking the single thread.
   • Files: src/pages/_app.js, src/models/utils.js, src/models/api.js, and the dynamic page files.
4. Rate-limit / block the crawler at the ingress (it sent 352 requests in ~76s).

Important companion fixes

5. Add a timeout and connection limit to our API calls (e.g. axios timeout, maxContentLength, and a bounded HTTP agent), so slow backend calls fail fast and release memory instead of piling up.
   • File: src/models/utils.js (and the direct axios.get calls in the token page)

Correctness bugs found along the way (smaller, but real)

6. revalidate bug on the token page: the shared error handler returns { revalidate }, which is invalid for getServerSideProps. Every failed API call then throws and renders an extra error page (we saw this 23 times → 82 error renders). Fix the error handling for server-side-rendered pages.
   • Files: src/models/utils.js (the wrapper function), src/pages/tokens/[tokenid].js
7. Crash in the tree page: Cannot read properties of undefined (reading 'id') at src/pages/trees/[treeid].js:468 , happens when the backend returns no tree but we render anyway. Add a guard.

Investigate separately (not the direct cause, but unhealthy)

8. Backend query API is slow. Some calls take 13–107 seconds, e.g. /trees?organization_id=... (107s) and /wallets/.../token-region-count (13.5s). These should be looked at on the backend side.

Things we tried that do NOT fix it

• A previous change (fix: disable ISR memory cache; set prod resource limits) already disabled the in-memory cache and raised limits — the pod still crashes, so that was not the cause.
• Tuning the JavaScript heap size (--max-old-space-size) will not help. The memory that overflows is non-heap (network connections), so changing the heap limit does nothing here.

---

Useful commands for whoever picks this up

NS=webmap
APP=app=treetracker-web-map-client-main

# Memory over time (catch it climbing before the next crash)
watch -n 30 "kubectl top pod -n $NS -l $APP --containers"

# Logs from the container that crashed, with timestamps
kubectl logs -n $NS <pod-name> --previous --timestamps

# See which routes were being hit just before the crash
kubectl logs -n $NS <pod-name> --previous | grep -iE "tokenid:|/tokens|/wallets|/trees" | tail -100

# Confirm the OOM kill
kubectl get events -n $NS --sort-by=.lastTimestamp | grep -iE "OOM|Killed"

Acceptance criteria

• Token detail page is cached (no full re-render on every request).
• A burst of requests no longer drives the pod to OOMKilled.
• API calls have a timeout and bounded connections.
• Production logs no longer dump large objects on every request.
• revalidate and tree-page undefined.id bugs fixed.
• (Follow-up) Backend slow-query investigation tracked in a separate issue.

[arnoldcastro5000]

Issue recurred on June 30, 2026.