[Bug] the container image and Helm chart never set `RUST_MIN_STACK`, so a by-the-book deployment passes every probe and aborts on the first query

[addyCooks]

Summary

RUST_MIN_STACK=33554432 is documented as mandatory on every platform, but neither of the two artifacts operators actually deploy the published container image and the Helm chart ever sets it. A deployment that follows charts/hydradb as written produces nodes that pass their startup, readiness, and liveness probes and then abort on the first query, with the exact symptom the project's own troubleshooting tables describe.

$ grep -rn RUST_MIN_STACK Dockerfile charts/
(no matches)

Why it matters

AGENTS.md documents the requirement without qualification:

| RUST_MIN_STACK=33554432 | every platform: OpenCypher async futures overflow the default stack |

And both troubleshooting tables already name the failure:

README.md Node answers /readyz, then aborts with has overflowed its stack on the first query → RUST_MIN_STACK unset; export 33554432

AGENTS.md Node serves /readyz, then aborts with has overflowed its stack on the first query → RUST_MIN_STACK unset. The node starts fine and dies on first use, so this looks like a query bug and is not.

That last clause is the whole problem: the failure is already known to be misattributed, and the deploy path is the one place where nothing sets the variable.

Current behavior

1. The image has no default. Dockerfile:70-71, runtime-base the stage both graph-node and graph-indexer are built on:

ENV DEBIAN_FRONTEND=noninteractive \
    RUST_LOG=info

RUST_LOG is defaulted here; RUST_MIN_STACK is not. README.md does pass -e RUST_MIN_STACK=33554432 in the docker run example, so a reader who copies that block verbatim is fine but the image carries no safe default for anyone who does not, and nothing in the image fails loudly when it is missing.

2. The Helm chart has no way to set it short of extraEnv. charts/hydradb/templates/configmap.yaml defines roughly fifty environment variables — every storage, cache, indexer, Bolt, TLS, and query-limit tunable, and RUST_LOG and does not define RUST_MIN_STACK:

  GRAPH_GRACEFUL_SHUTDOWN_MS: {{ ... }}
  GRAPH_TRIM_MEMORY_AFTER_HYDRATION: {{ ... }}
  RUST_LOG: {{ .Values.runtime.rustLog | quote }}

Both workloads consume that ConfigMap via envFrom (node-statefulset.yaml:86-88, indexer-deployment.yaml:78-80), so neither inherits it. node.extraEnv and indexer.extraEnv both default to [] (values.yaml:102, values.yaml:158), and charts/hydradb/README.md does not mention the variable at all. Given how exhaustive the ConfigMap is about everything else, this reads as an omission rather than a decision.

Why the probes do not catch it

All three probes hit admin endpoints that execute no query (node-statefulset.yaml:111-129):

• startupProbe → /livez
• readinessProbe → /readyz
• livenessProbe → /livez

So the pod reports Ready, the Service adds it to endpoints, and traffic is routed to it. The stack overflow is an abort, not a recoverable error, so the first real query kills the process. Under sustained traffic the pod restarts, passes its probes again, takes traffic again, and dies again — a CrashLoopBackOff whose visible signal is the query rather than the configuration.

Reproduction

Not executed here this is a static and documentary finding, and the abort itself is the behavior the project already documents. The expected sequence:

1. helm install hydradb charts/hydradb with any otherwise-valid values, leaving node.extraEnv at its default [].
2. Wait for the pod to become Ready. It will.
3. Send any OpenCypher query over Bolt or HTTP.
4. Expect has overflowed its stack and a container restart.

The same applies to docker run ghcr.io/hydra-db/hydradb:latest without -e RUST_MIN_STACK=33554432.

Context

Merged PR #47 ("docs: document running Turbolay locally, add AGENTS.md, fix runtime_smoke.sh stack limit") introduced the variable into AGENTS.md, README.md, and scripts/runtime_smoke.sh in one change. Every developer-facing surface was covered; the Dockerfile and the chart were not touched. The gap looks like it dates from there.

Suggested fix

Two layers, either of which stands alone, both preferably:

Defence in depth default it in the artifacts.

• Dockerfile, runtime-base stage: add RUST_MIN_STACK=33554432 to the existing ENV, so the image is correct under any orchestrator, including plain docker run.
• charts/hydradb/templates/configmap.yaml: add RUST_MIN_STACK: {{ include "hydradb.decimalInteger" .Values.runtime.minStackBytes | quote }} with a values.yaml default of 33554432, matching how every other tunable in that file is handled.

The real fix stop depending on the environment. Set the worker stack size in-process so no deployment can get it wrong. Both binaries currently use #[tokio::main] (src/bin/graph-node.rs:77, src/bin/graph-indexer.rs:762), which offers no way to set it, so this means an explicit builder:

tokio::runtime::Builder::new_multi_thread()
    .enable_all()
    .thread_stack_size(32 * 1024 * 1024)
    .build()?
    .block_on(run())

That makes RUST_MIN_STACK an override rather than a prerequisite, and removes an entire class of "looks like a query bug and is not" reports.

A cheaper interim option, if the in-process change is too invasive for now: have graph-node read RUST_MIN_STACK at startup and refuse to start (or log an error-level warning) when it is unset or below 33554432. Failing at startup is much easier to diagnose than aborting on first use, and it would make the probes meaningful.

Notes

• graph-node is the binary the requirement is documented against; graph-indexer shares runtime-base and the same ConfigMap, so an ENV- or ConfigMap-level fix covers both at no extra cost.
• Worth adding to whatever end-to-end check the project settles on: a test that installs the chart and issues one query would have caught this, whereas nothing that only checks /readyz can.