Measure the heaviest mainnet queries against the new 15s PG_STATEMENT_TIMEOUT (15-20s band now hard-errors)

[SanabriaRusso]

Context

#182 sets PG_STATEMENT_TIMEOUT to a default of 15000 ms. Both known downstream consumers use a 20 s client timeout:

• mina-explorer — src/services/api/http.ts:9, DEFAULT_TIMEOUT_MS = 20_000
• mina-explorer-api — app/config.py:91, upstream_timeout_seconds = 20.0

The 15 s server-side cap therefore sits below the 20 s client-side cap. That ordering is deliberate and correct — the server should give up before the client does, so the failure is a clean cancellation rather than an abandoned query still consuming a connection.

But it creates a band that did not exist before: any query that previously completed in 15–20 s now hard-errors instead of succeeding slowly. Before #182 there was no server-side statement timeout at all, so such a query returned data.

This is not a blocker — #182 was approved on this basis — but it is an unverified assumption, and the only honest way to close it is measurement against real data.

What to measure

The most plausible candidate is mina-explorer's 2000-block analytics query on mainnet. It is the heaviest known consumer query (measured at depth 7, 89 tokens, graphql-armor cost ~161) and it runs against the largest dataset.

Others worth timing while you are set up:

• blocksFullPaginatedBestChain at its maximum limit
• The account-transaction queries at a large block range
• Anything in mina-explorer-api's backfill path that requests a wide range

How to measure

Against a mainnet archive database — devnet and mesa are far too small to be representative:

-- capture the plan and the real execution time
EXPLAIN (ANALYZE, BUFFERS) <the query the resolver emits>;

Or end-to-end through the API, which is more faithful because it includes serialisation:

for i in $(seq 1 20); do
  curl -s -o /dev/null -w '%{time_total}\n' \
    -X POST https://<mainnet-archive-endpoint>/ \
    -H 'content-type: application/json' \
    -d '{"query":"<the 2000-block analytics query>"}'
done | sort -n

Look at p95 and max, not the mean — the tail is what trips a timeout.

Interpreting the result

Observed p99	Action
well under 15 s	No action. Close this issue and record the number in docs/runbook.md's capacity section.
10–15 s	Uncomfortably close. Either raise PG_STATEMENT_TIMEOUT (and correspondingly the consumers' client timeouts, keeping server < client), or optimise the query / add an index. Record the decision.
over 15 s	The query is already broken by this default. Raise PG_STATEMENT_TIMEOUT for that deployment and open a follow-up to optimise, because a >15 s interactive query is a problem independent of the timeout.

PG_STATEMENT_TIMEOUT is env-tunable per deployment, so the mitigation is immediate if the measurement comes back bad — no code change or redeploy of the image required.

Note on how the failure surfaces

A cancelled statement returns SQLSTATE 57014, which yoga masks into a generic error. Confirmed that the resulting message contains none of Cannot query field, Unknown argument, or Unknown type — the three literal strings mina-explorer-api matches at app/upstream/graphql.py:33-42 to classify a permanent schema error.

That matters: a schema-error classification would poison that consumer's capability cache and cause a sticky tier downgrade. Instead a statement timeout classifies as a generic upstream error, which is transient and retried appropriately. So the failure mode is safe — it is only a question of whether it fires at all.

Acceptance criteria

• p95 and p99 of the 2000-block mainnet analytics query recorded
• Confirmed no known consumer query falls in the 15–20 s band, or PG_STATEMENT_TIMEOUT adjusted with the rationale recorded
• The measured figure added to docs/runbook.md's capacity section, so the next person does not have to re-derive it

Related

#182, #197 (runbook capacity section)