Skip to content

P0: Configure Postgres pool limits + statement_timeout (#165) - #182

Open
dkijania wants to merge 5 commits into
mainfrom
feat/pg-pool-timeouts
Open

P0: Configure Postgres pool limits + statement_timeout (#165)#182
dkijania wants to merge 5 commits into
mainfrom
feat/pg-pool-timeouts

Conversation

@dkijania

Copy link
Copy Markdown
Contributor

What & why

Part of the production-readiness epic (#163). Closes #165.

The archive-node Postgres client was created with postgres(connectionString) and no pool sizing or timeouts. Once the API is publicly reachable this is a DoS risk — a single expensive query can hold a connection open indefinitely, exhaust the pool, and cascade into an outage.

Changes

  • New src/db/archive-node-adapter/postgres-options.ts — builds the postgres() options from conservative, env-tunable defaults, isolated so it's unit-testable without a DB.
  • Adapter now calls postgres(connectionString, buildPostgresOptions()).
Env var Default Meaning
PG_MAX_CONNECTIONS 10 Max pooled connections per host
PG_IDLE_TIMEOUT 30 Seconds before an idle connection is closed
PG_CONNECT_TIMEOUT 30 Seconds to wait for a connection before failing
PG_STATEMENT_TIMEOUT 30000 Server-side query cap (ms); longer queries are cancelled by Postgres. 0 disables

Malformed values fall back to defaults rather than throwing, so a stray typo can never silently disable a safety limit (e.g. max → 0). statement_timeout is sent as a startup connection parameter, so it applies to every query on every connection.

Docs (getting-started.md), .env.example.compose, and envionment.d.ts updated; new unit tests cover parsing, fallbacks, clamping, and the options shape.

Testing

  • npm run build — clean
  • npm run test:unit — all pass (6 new assertions in postgres-options.test.ts)
  • npm run lint — clean
  • npx prettier --debug-check . — exit 0

Tests construct their own postgres clients directly, so the new adapter defaults don't affect the integration/live-network suites.

🤖 Generated with Claude Code

@dkijania dkijania added production-readiness Work toward making the API production-ready / publicly available P0 Blocker for public availability labels Jun 28, 2026
@dkijania
dkijania force-pushed the feat/pg-pool-timeouts branch from 58c7d49 to dfcad11 Compare June 28, 2026 09:40
@SanabriaRusso

Copy link
Copy Markdown
Collaborator

Nice — this closes a real P0 cleanly, and isolating the option-building into a pure, unit-tested buildPostgresOptions() is tidy. Sending statement_timeout via porsager's connection startup-parameter object is exactly right — it applies to every query on every pooled connection.

I checked the backwards-compat angle against the mina-explorer client and the defaults here are safe: 30s is generous for its heaviest path (a date-range analytics query that can fetch ~10k blocks), it's env-tunable, PG_STATEMENT_TIMEOUT=0 disables it, and the cancellation surfaces as a normal masked error (PG 57014) that won't collide with the Explorer's Cannot query field fallback trigger. 👍

Two small, non-blocking things:

  1. "per host" wording. PG_MAX_CONNECTIONS maps to porsager's max, which is the total pool size, not per-host — with a multi-host PG_CONN (the documented HA syntax) porsager fails over to the first reachable host rather than fanning out, so it holds a single pool of max connections. Since the README mentions multi-host replicas, "(per host)" in the docs/comment could lead operators to over-size. Suggest just "Max pooled Postgres connections."

  2. Prove acceptance criterion Add Actions resolver support #1 with a tiny integration test. P0: Configure Postgres pool limits + statement_timeout #165 asks that "a query exceeding statement_timeout is terminated and surfaces a clean error" — the new unit tests check the options shape but not the end-to-end cancellation. One assertion against the integration DB would lock it in:

    // client built with PG_STATEMENT_TIMEOUT=200
    await assert.rejects(
      () => sql`SELECT pg_sleep(1)`,
      (err) => err.code === '57014', // canceling statement due to statement timeout
    );

Thanks for tightening this up!

dkijania added a commit that referenced this pull request Jul 17, 2026
Adds the integration coverage #165 actually asks for. The unit tests
assert the options object's shape; these assert that Postgres really
cancels a query past statement_timeout (SQLSTATE 57014), that the pooled
connection stays usable afterwards — a cancelled query must not poison
it, or one slow client would degrade later requests — and that
PG_STATEMENT_TIMEOUT=0 disables the limit as documented.

Also corrects PG_MAX_CONNECTIONS from "per host" to total. porsager's
`max` is the whole pool, and a multi-host PG_CONN fails over rather than
fanning out, so the pool only ever points at one host at a time. Since
the README documents multi-host replicas, "per host" invited operators to
size the pool N times too large.

Addresses review feedback on #182.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@dkijania

Copy link
Copy Markdown
Contributor Author

Thanks @SanabriaRusso — both done in 035c2e8.

"per host" wording. Corrected in the docs table and the code comment; it's the total pool. Your reasoning checks out against the driver: postgres.js scopes hostIndex per Connection (src/connection.js:89), so every pooled connection starts at host[0] and only advances when its own attempt fails — the pool only ever points at one host at a time. Since the README advertises multi-host replicas, "per host" was inviting operators to size the pool N× too large.

(That same mechanism means the README's "the server fans queries across them" claim is wrong too — you flagged the equivalent in #197. Fixed on main's README via #186, and in deploy/README.md via #196.)

Acceptance criterion #1. Added tests/integration/postgres-timeout.test.ts, which asserts the behaviour rather than the options shape:

await assert.rejects(
  () => sql`SELECT pg_sleep(1)`,
  (error) => error.code === '57014',   // canceling statement due to statement timeout
);

The client is built through buildPostgresOptions() exactly as the adapter builds it, so the startup-parameter path is what's under test. Three more alongside it: the pooled connection stays usable after a cancellation (a poisoned connection would let one slow client degrade every later request), a query inside the timeout is unaffected, and PG_STATEMENT_TIMEOUT=0 really does disable rather than cancel immediately. All four pass against a real Postgres.

They're also mutually confirming — the "inside the timeout" and "disabled" cases would still pass if the timeout silently weren't applied, but the 57014 case wouldn't, so the set can't go green vacuously.

@SanabriaRusso

Copy link
Copy Markdown
Collaborator

Verdict: MERGEABLE

Second pass focused on whether the defaults kill real production queries. They don't, and the two maintainer points from the first pass are genuinely addressed in 035c2e8 ("total, not per host" wording in docs/getting-started.md:178 + the PostgresPoolConfig.max docstring; tests/integration/postgres-timeout.test.ts asserts the real 57014).

What I checked

  • The settings are actually applied, not silently ignored. postgres@3.4.3 merges rather than replaces the connection object — node_modules/postgres/src/index.js:482 builds connection: { application_name: 'postgres.js', ...o.connection, ...query }, and src/connection.js:972-981 serialises every entry into the startup packet (k + b.N + v, so a JS number coerces fine). So connection: { statement_timeout } in postgres-options.ts:78-81 is a real GUC on every pooled connection, and application_name is preserved. max / idle_timeout / connect_timeout are top-level in the same file — correct, those are driver options, not server settings. No no-op here.
  • statement_timeout vs the real worst case. 30 000 ms sits above the 20 s client cut on both consumers (mina-explorer src/services/api/http.ts:9 DEFAULT_TIMEOUT_MS = 20_000; mina-explorer-api app/config.py:91 upstream_timeout_seconds = 20.0). So the analytics path (mina-explorer src/services/api/analytics.ts:14, ANALYTICS_BLOCK_LIMIT = 2000) and the 500-page blocks crawl see zero behaviour change — anything the DB kills at 30 s was already a client-side abort at 20 s. That is the safe side of the line to land on; see the nit below for why I'd still tighten it.
  • What the client sees on 57014 — no marker collision. src/server/server.ts:15-27 leaves maskedErrors at its default, and graphql-yoga@4.0.4 cjs/utils/mask-error.js replaces any GraphQLError whose originalError.name !== 'GraphQLError' with the flat string Unexpected error. (cjs/server.js:119). A porsager PostgresError (node_modules/postgres/src/errors.js:1-7) hits exactly that branch, so the body carries errors[0].message === "Unexpected error." — no Cannot query field / Unknown argument / Unknown type / inBestChain substring, so mina-explorer-api app/upstream/graphql.py:32-43 classifies it as a plain UpstreamError, not UpstreamSchemaError. The capability cache is not poisoned and the explorer does not get pinned to MINIMAL. Masking also swallows the one string that would leak topology — Errors.connection() (src/errors.js:18-28) builds write CONNECT_TIMEOUT <host>:<port> — so no connection details reach the wire. Validation errors are untouched (they have no originalError), so the protocolState FULL-tier probe still gets its verbatim Cannot query field.
  • Pool sizing is not a regression. max: 10 is exactly porsager's own default (node_modules/postgres/src/index.js:448), so nothing is being narrowed. connect_timeout: 30 is also the library default; only idle_timeout changes (library default is null = never close). Exhaustion queues rather than errors: connect_timeout is armed only around socket establishment (src/connection.js:76,256-259), and over-max queries land in the queries Queue (src/index.js:55,404) with no deadline — so a burst waits and the 20 s client timeout stays the effective bound. Correct behaviour for a browser-burst + 4 req/s workload.
  • CI is honest. run-tests.sh globs ./build/**/*test.js and skips only live-api / devnet-dump / live-network, so the new integration file runs in Run-Tests, which has Postgres on 5432 (.github/workflows/run-tests.yaml:12-22). The 57014 assertion is the one case in the file that cannot pass if the startup parameter isn't applied, so the set can't go green vacuously — that reasoning in the PR thread holds.
  • .gitignore change is necessary and safe. db/ previously matched src/db/, which would have made the new src/db/archive-node-adapter/postgres-options.ts untracked; /db/ + /data/ are correctly root-anchored and the only such directory in the tree is src/db.
  • No conflict with P1: Add readiness probe distinct from liveness (#169) #187/P1: Graceful shutdown — drain, flush traces, uncaught handlers (#170) #188 as written — see the shutdown note below, which is an input for P1: Graceful shutdown — drain, flush traces, uncaught handlers (#170) #188 rather than a problem here.

Non-blocking nits

  1. PG_STATEMENT_TIMEOUT=0 doesn't do what the docs say (and its test passes vacuously). The startup-packet builder filters falsy values: .filter(([, v]) => v) in node_modules/postgres/src/connection.js:980. With statement_timeout: 0 the parameter is dropped entirely, so the session inherits whatever ALTER ROLE/ALTER DATABASE/postgresql.conf says. On a stock server that is 0 and the doc line is true by accident; on a deployment that sets a role-level statement_timeout, PG_STATEMENT_TIMEOUT=0 silently fails to disable it. tests/integration/postgres-timeout.test.ts would pass either way. One-character-class fix — send it as a string so '0' stays truthy and Postgres gets an explicit session-level statement_timeout=0:

    --- a/src/db/archive-node-adapter/postgres-options.ts
    +++ b/src/db/archive-node-adapter/postgres-options.ts
    @@
         connection: {
    -      // Sent as a startup connection parameter, so it applies to every query.
    -      statement_timeout: config.statementTimeout,
    +      // Sent as a startup connection parameter, so it applies to every query.
    +      // Stringified deliberately: postgres.js drops falsy startup parameters
    +      // (src/connection.js `.filter(([, v]) => v)`), so a numeric 0 would be
    +      // omitted and the session would inherit any role/database-level
    +      // statement_timeout instead of explicitly disabling it.
    +      statement_timeout: String(config.statementTimeout),
         },

    ConnectionParameters is [name: string]: string | number | boolean (node_modules/postgres/types/index.d.ts:343), so this type-checks. Two call-site updates come with it: tests/unit/postgres-options.test.tsassert.deepStrictEqual(options.connection, { statement_timeout: '20000' }), and one extra integration assertion that actually proves the escape hatch:

    test('PG_STATEMENT_TIMEOUT=0 sends an explicit session-level disable', async () => {
      const sql = postgres(connectionString, buildPostgresOptions({ PG_STATEMENT_TIMEOUT: '0' }));
      try {
        const [row] = await sql<{ t: string }[]>`SHOW statement_timeout`;
        assert.strictEqual(row.t, '0');
      } finally {
        await sql.end();
      }
    });

    (SHOW statement_timeout returns '0' for disabled; today, with the numeric 0, that assertion fails against a server whose role default is non-zero.)

  2. Consider PG_STATEMENT_TIMEOUT=15000 as the default rather than 30000. Both consumers abort at 20 s, so no query in the 20-30 s band can ever produce a user-visible success — the only thing the extra 10 s buys is the DB burning a connection on work nobody will read. Worse, mina-explorer-api retries timeouts twice with backoff but never retries GraphQL errors (app/upstream/graphql.py:223-236), so a 25 s query today becomes three 25 s server-side executions of the same scan; at 15 s the first attempt returns a non-retried GraphQL error and the amplification disappears. 15 s also leaves ~5 s of headroom under the client cut for HTTP + JSON serialisation of a 2000-block payload. Purely an improvement over main either way (which has no cap at all), hence non-blocking — but the doc line would then read | PG_STATEMENT_TIMEOUT | 15000 | ... deliberately below the 20s timeout used by known clients so the DB reclaims the connection before the client gives up |.

  3. statement_timeout is per statement, not per request. A blocks query with ENABLE_BLOCK_TRANSACTION_DETAILS=true issues one blocks statement plus three detail statements (src/services/blocks-service/blocks-service.ts:247-251), so worst-case DB time for one GraphQL request is a multiple of the cap. Not a defect in this PR — just worth a sentence in docs/getting-started.md so nobody sizes the value as a request budget.

  4. .env.example.lightnet wasn't updated alongside .env.example.compose. Cosmetic (defaults apply), but the two files otherwise track each other for CORS_ORIGIN / BLOCK_RANGE_SIZE.

  5. Hand-off to P1: Graceful shutdown — drain, flush traces, uncaught handlers (#170) #188: ArchiveNodeAdapter.close()sql.end() with no { timeout } waits for in-flight queries. This PR is what makes that wait bounded (previously unbounded), but the bound is now PG_STATEMENT_TIMEOUT. Whoever lands graceful shutdown should make the drain deadline / terminationGracePeriodSeconds exceed it — 30 s of statement budget under a 30 s grace period means SIGKILL mid-drain.

Automated second-pass review — focus: downstream compatibility with mina-explorer / mina-explorer-api.

SanabriaRusso
SanabriaRusso previously approved these changes Aug 18, 2026

@SanabriaRusso SanabriaRusso left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving on the basis of the second-pass review comment above: no mid-to-high severity security, compatibility, or degradation issue found, and the downstream contract with mina-explorer / mina-explorer-api holds — GraphQL validation error text reaches errors[].message verbatim, the browser SPA's cross-origin access is preserved, and the real consumer query shapes (including the 2000-block analytics query and the 500-row page crawl) still pass.

Two things this approval does not mean:

  • It does not close the non-blocking items in the review comment. Several are worth fixing before or shortly after merge; they are written up there with patches.
  • It does not by itself mean the branch is ready to merge. main requires branches to be up to date, so this needs an update-branch (or a rebase, if the branch is conflicting) first, and a few PRs in this series have cross-PR ordering constraints called out in their review comments.

Automated second-pass review — focus: downstream compatibility with mina-explorer / mina-explorer-api.

@SanabriaRusso SanabriaRusso left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed e986f4d ("fix(db): tighten statement timeout handling") after the earlier approval was dismissed. Both points from the previous review are addressed, and the fix is correct for the right reason. Re-approving.

The falsy-drop fix is genuine — verified against the pinned source

postgres@3.4.3, src/connection.js:972-982, builds the startup packet as:

Object.entries(Object.assign({ user, database, client_encoding: 'UTF8' }, options.connection))
  .filter(([, v]) => v)
  .map(([k, v]) => k + b.N + v)
  .join(b.N)

So .filter(([, v]) => v) is the exact mechanism: numeric 0 is falsy and was silently omitted from the packet, leaving the server default in force rather than explicitly disabling the timeout. String(0)'0' is truthy and survives the filter. And because .map() string-concatenates the value anyway, stringifying is a no-op on the wire for every non-zero value — 15000 and '15000' serialise to identical bytes. No behaviour change except the one intended.

The as unknown as number cast is necessary, and the comment explaining it is accurate

postgres@3.4.3 types/index.d.ts:335 declares statement_timeout: number as an explicit key, which wins over the [name: string]: string | number | boolean index signature two lines below at :342. So a string genuinely cannot be assigned without the cast. This is an upstream typing inaccuracy — the value is serialised as a string regardless — and the four-line comment says so correctly. Fine as written; not worth contorting the code around.

Default 15000 adopted

Documented rationale in docs/getting-started.md:181 is the right one: keeping the server-side cancel below the 20s client timeout means Postgres reclaims the work before the caller abandons it, instead of continuing to burn a connection on a query nobody is waiting for. That matters specifically because mina-explorer-api retries transport timeouts twice but never retries GraphQL errors (app/upstream/graphql.py), so a 30s server-side timeout under a 20s client timeout turned one slow scan into three concurrent ones.

Consistency checked across the whole PR: .env.example.compose, .env.example.lightnet, docs/getting-started.md, and POOL_DEFAULTS all read 15000, with no stale 30000 anywhere in the diff. Sibling docs PRs are unaffected — #197 names PG_STATEMENT_TIMEOUT without quoting a value, and #186/#196 don't reference it.

One residual worth measuring after rollout, not a blocker: 15s now sits below the consumers' 20s client timeout, so any query landing in the 15–20s band flips from slow-but-successful to a hard error. The most plausible candidate is the explorer's 2000-block analytics query on mainnet (ANALYTICS_BLOCK_LIMIT = 2000). The README benchmark (~800 req/s, p99 39ms) says ordinary traffic is nowhere near this, and the value is env-configurable so an operator can raise it — but it's worth timing that one query against mainnet once this is deployed, rather than assuming.

Non-blocking: the new integration test cannot fail

tests/integration/postgres-timeout.test.ts'PG_STATEMENT_TIMEOUT=0 sends an explicit session-level disable' asserts SHOW statement_timeout returns '0'. But a stock Postgres already defaults statement_timeout to 0, so this passes whether or not the parameter was ever transmitted — which is the same vacuity the previous review flagged in the test above it. It would only discriminate against a server with a non-zero default:

ALTER DATABASE archive SET statement_timeout = 5000;

...set before the client connects, then asserting '0' still wins.

Worth noting the regression is nonetheless genuinely covered — by the unit test, not the integration test. tests/unit/postgres-options.test.ts:64 uses assert.deepStrictEqual(options.connection, { statement_timeout: '20000' }), and deepStrictEqual distinguishes '20000' from 20000, so reverting the String(...) fails that assertion. So this is a cleanup suggestion — either make the integration test discriminating or drop it as redundant — not a coverage gap.

Automated re-review of e986f4d — focus: downstream compatibility with mina-explorer / mina-explorer-api.

@SanabriaRusso SanabriaRusso left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correcting my previous review on this PR: I approved on the strength of the production change before checking CI, and Run-Tests — a required check — is red on e986f4d. The failure is caused by the test added in that same commit, so this needs one more push before it can merge. Everything I said about the production fix still stands; only the new test is wrong.

The failure

Run-Tests, run 32191790147:

not ok 5 - PG_STATEMENT_TIMEOUT=0 sends an explicit session-level disable
  location: 'tests/integration/postgres-timeout.test.ts:91:3'
  error: |-
    Expected values to be strictly equal:

    undefined !== '0'

Cause

SHOW statement_timeout returns a single column named statement_timeout. The test reads row.t:

const [row] = await sql<{ t: string }[]>`SHOW statement_timeout`;
assert.strictEqual(row.t, '0');

The <{ t: string }[]> annotation is a compile-time claim that nothing validates at runtime, so row.t is undefined and the assertion fails. SHOW does not accept an AS alias, which is presumably how t was intended to be produced.

Fix

current_setting() is the better instrument here, because unlike SHOW it is an ordinary function call and so does accept an alias:

  test('PG_STATEMENT_TIMEOUT=0 sends an explicit session-level disable', async () => {
    const sql = postgres(
      connectionString,
      buildPostgresOptions({ PG_STATEMENT_TIMEOUT: '0' })
    );
    try {
      const [row] = await sql<{ t: string }[]>`
        SELECT current_setting('statement_timeout') AS t
      `;
      assert.strictEqual(row.t, '0');
    } finally {
      await sql.end();
    }
  });

One more thing, while you are in this file

Once the column name is corrected, this test will pass — but it would also have passed before the String(...) fix, because a stock Postgres already defaults statement_timeout to 0. So it does not actually discriminate the bug it was written to pin: omitting the parameter and explicitly sending 0 produce the same observable result on a default server.

To make it discriminating, give the server a non-zero default first, so that only an explicitly transmitted 0 can win:

  test('PG_STATEMENT_TIMEOUT=0 overrides a non-zero server default', async () => {
    const admin = postgres(connectionString);
    try {
      await admin`ALTER DATABASE ${admin.unsafe(database)} SET statement_timeout = 5000`;
    } finally {
      await admin.end();
    }

    const sql = postgres(
      connectionString,
      buildPostgresOptions({ PG_STATEMENT_TIMEOUT: '0' })
    );
    try {
      const [row] = await sql<{ t: string }[]>`
        SELECT current_setting('statement_timeout') AS t
      `;
      assert.strictEqual(row.t, '0');
    } finally {
      await sql.end();
      const cleanup = postgres(connectionString);
      try {
        await cleanup`ALTER DATABASE ${cleanup.unsafe(database)} RESET statement_timeout`;
      } finally {
        await cleanup.end();
      }
    }
  });

ALTER DATABASE ... SET only affects sessions opened after it runs, which is why the admin connection is closed before the assertion connection is opened.

If that feels heavier than it is worth, the alternative is to delete this integration test entirely — the regression is already genuinely covered at the unit level. tests/unit/postgres-options.test.ts:64 asserts:

assert.deepStrictEqual(options.connection, { statement_timeout: '20000' });

and deepStrictEqual distinguishes '20000' from 20000, so reverting the String(...) fails that assertion with no database required. Either route is fine; the current test is the one option that does not work.

Unchanged from my previous review

The production fix is correct and I verified it against the pinned source — postgres@3.4.3 src/connection.js:979 filters startup parameters with .filter(([, v]) => v), so numeric 0 was dropped and '0' survives, while non-zero values serialise to identical wire bytes. The as unknown as number cast is genuinely required (types/index.d.ts:335 narrows that key to number, overriding the index signature at :342) and the explanatory comment is accurate. The 15000 default and its documented rationale are right, with the residual note that 15s now sits below the consumers' 20s client timeout, so it is worth timing the 2000-block analytics query against mainnet after rollout.

Happy to re-approve as soon as the test is fixed.

Automated re-review of e986f4d — focus: downstream compatibility with mina-explorer / mina-explorer-api.

dkijania added a commit that referenced this pull request Aug 24, 2026
Adds the integration coverage #165 actually asks for. The unit tests
assert the options object's shape; these assert that Postgres really
cancels a query past statement_timeout (SQLSTATE 57014), that the pooled
connection stays usable afterwards — a cancelled query must not poison
it, or one slow client would degrade later requests — and that
PG_STATEMENT_TIMEOUT=0 disables the limit as documented.

Also corrects PG_MAX_CONNECTIONS from "per host" to total. porsager's
`max` is the whole pool, and a multi-host PG_CONN fails over rather than
fanning out, so the pool only ever points at one host at a time. Since
the README documents multi-host replicas, "per host" invited operators to
size the pool N times too large.

Addresses review feedback on #182.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@dkijania
dkijania force-pushed the feat/pg-pool-timeouts branch from e986f4d to eebd4e3 Compare August 24, 2026 17:04
SanabriaRusso
SanabriaRusso previously approved these changes Aug 24, 2026

@SanabriaRusso SanabriaRusso left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Both round-1 findings are fixed, and I verified the second one empirically rather than by inspection.

The SHOW column-name bug is fixed. tests/integration/postgres-timeout.test.ts:136-138 now reads SELECT current_setting('statement_timeout') AS t. Confirmed against a real Postgres 15 that SHOW statement_timeout returns a column named statement_timeout (hence the old undefined !== '0') while current_setting(...) AS t returns { t: ... }. Run-Tests is green and the log shows the subtest actually running, not skipped.

The test is now genuinely discriminating — this was my main concern from round 1, so I reproduced both arms against a throwaway Postgres using the repo's own postgres@3.4.3:

connection.statement_timeout current_setting('statement_timeout') assertion
'0' (current code) "0" passes
0 (the pre-fix regression) "1s" fails

The mechanism is exactly right: setDatabaseStatementTimeout('1s') applies ALTER DATABASE … SET on a separate max:1 client that is await sql.end()-ed in a finally before the assertion connection is created, so the assertion session inherits the non-zero database-level default unless the startup packet explicitly overrides it. Cleanup is safe too — resetDatabaseStatementTimeout() runs in finally, and setupTestDatabase() recreates the DB per run regardless.

Production change re-verified after the rebase. postgres@3.4.3 src/connection.js:980 still builds the startup packet with .filter(([, v]) => v), so numeric 0 is still dropped — the String(...) at src/db/archive-node-adapter/postgres-options.ts:85 is load-bearing, and the as unknown as number cast is still required by the upstream .d.ts. 15000 is consistent across .env.example.compose, .env.example.lightnet, docs/getting-started.md:181, and POOL_DEFAULTS; a repo-wide grep finds no stale 30000.

Good catch on the .gitignore fix, which is load-bearing here rather than incidental: on main, git check-ignore -v src/db/newfile.ts reports .gitignore:27:db/, i.e. every new file under src/db/ — including this PR's own postgres-options.ts — was being silently ignored.

Non-blocking notes:

  • Merge ordering. #134 touches the same .gitignore hunk (it anchors /db/ but leaves data/ unanchored) and #187 touches the same postgres(connectionString) line in archive-node-adapter.ts:44. Whichever lands second needs a trivial conflict resolution; prefer this PR's fully-anchored .gitignore. For the archive-node-adapter.ts collision, resolve to this.client = postgres(connectionString, buildPostgresOptions()); and leave #187's pingClient on its own literal options — passing buildPostgresOptions() to the ping client would give it max: 10 and undo the pool isolation #187 exists to provide.
  • Measure after rollout. 15s now sits below the ~20s client timeout used by both known downstream consumers (mina-explorer src/services/api/http.ts:9 DEFAULT_TIMEOUT_MS = 20_000; mina-explorer-api app/config.py:91 upstream_timeout_seconds = 20.0). Any query in the 15–20s band flips from slow-but-successful to a hard error. The plausible candidate is the explorer's 2000-block mainnet analytics query — worth timing against mainnet rather than assuming. PG_STATEMENT_TIMEOUT is env-tunable if it turns out to be over.
  • A cancelled query surfaces as SQLSTATE 57014 and is masked into a generic error. Confirmed it does not contain Cannot query field, Unknown argument, or Unknown type — the three literal strings mina-explorer-api matches on at app/upstream/graphql.py:33-42 to drive tier fallback and poison its capability cache. So a statement timeout cannot be misread downstream as a permanent schema error. Good.
  • PG_IDLE_TIMEOUT=0 / PG_CONNECT_TIMEOUT=0 are accepted and mean "no timeout" to postgres.js (only max is clamped). Fine as an operator opt-in, just noting it's asymmetric with the max clamp.

dkijania added a commit that referenced this pull request Aug 24, 2026
Adds the integration coverage #165 actually asks for. The unit tests
assert the options object's shape; these assert that Postgres really
cancels a query past statement_timeout (SQLSTATE 57014), that the pooled
connection stays usable afterwards — a cancelled query must not poison
it, or one slow client would degrade later requests — and that
PG_STATEMENT_TIMEOUT=0 disables the limit as documented.

Also corrects PG_MAX_CONNECTIONS from "per host" to total. porsager's
`max` is the whole pool, and a multi-host PG_CONN fails over rather than
fanning out, so the pool only ever points at one host at a time. Since
the README documents multi-host replicas, "per host" invited operators to
size the pool N times too large.

Addresses review feedback on #182.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@dkijania
dkijania force-pushed the feat/pg-pool-timeouts branch from 28d28ba to efae1b6 Compare August 24, 2026 21:56
dkijania added a commit that referenced this pull request Aug 26, 2026
Adds the integration coverage #165 actually asks for. The unit tests
assert the options object's shape; these assert that Postgres really
cancels a query past statement_timeout (SQLSTATE 57014), that the pooled
connection stays usable afterwards — a cancelled query must not poison
it, or one slow client would degrade later requests — and that
PG_STATEMENT_TIMEOUT=0 disables the limit as documented.

Also corrects PG_MAX_CONNECTIONS from "per host" to total. porsager's
`max` is the whole pool, and a multi-host PG_CONN fails over rather than
fanning out, so the pool only ever points at one host at a time. Since
the README documents multi-host replicas, "per host" invited operators to
size the pool N times too large.

Addresses review feedback on #182.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@dkijania
dkijania force-pushed the feat/pg-pool-timeouts branch from efae1b6 to 58da1bb Compare August 26, 2026 15:59
SanabriaRusso
SanabriaRusso previously approved these changes Aug 26, 2026

@SanabriaRusso SanabriaRusso left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving after the rebase (the previous approval was dismissed by the force-push).

Re-verified rather than assumed: the PR content at 58da1bb is byte-identical to the commit I approved (28d28ba) once both are diffed against their respective merge bases. Every +/- line matches. The rebase onto 7361526 was mechanical.

Carrying forward the round-2 verification unchanged:

  • The failing test now reads SELECT current_setting('statement_timeout') AS t, and it genuinely discriminates the bug. ALTER DATABASE ... SET statement_timeout='1s' is applied on a separate max:1 client that is end()-ed before the assertion connection opens, so the assertion session inherits the non-zero DB default. Both arms reproduced against a real Postgres with the repo's own postgres@3.4.3: statement_timeout: '0'"0" (passes); statement_timeout: 0"1s" (fails).
  • Production fix re-confirmed at connection.js:980.filter(([,v]) => v).
  • All checks green on this head.

Residual (measure, not a blocker): 15s sits below both consumers' 20s client timeout, so a query in the 15–20s band flips from slow-but-successful to a hard error. Time the explorer's 2000-block mainnet analytics query post-merge; PG_STATEMENT_TIMEOUT is env-tunable.

Merge-train note: this PR and #134 both fix the unanchored .gitignore:27 db/ entry independently and will conflict on that hunk. Resolve toward the fully anchored /db/ + /data/ form.

dkijania and others added 5 commits August 27, 2026 12:25
The archive-node Postgres client was created with `postgres(connectionString)`
and no pool sizing or timeouts. With the API exposed publicly this is a denial-
of-service risk: one expensive query can hold a connection open indefinitely,
exhausting the pool and cascading into an outage.

Add a small, unit-testable `postgres-options` module that builds the client
options from conservative, env-tunable defaults:

- PG_MAX_CONNECTIONS  (max pooled connections, default 10)
- PG_IDLE_TIMEOUT     (seconds, default 30)
- PG_CONNECT_TIMEOUT  (seconds, default 30)
- PG_STATEMENT_TIMEOUT(ms server-side query cap, default 30000; 0 disables)

Malformed values fall back to defaults rather than throwing, so a stray typo
can never silently disable a safety limit. Docs, env example, and env type
declarations updated; unit tests cover parsing, fallbacks, and the options shape.

Also anchor the `db/` and `data/` .gitignore rules to the repo root (`/db/`,
`/data/`). The unanchored `db/` matched `src/db/` anywhere in the tree, which
silently ignored the new module file.

Closes #165.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QSuak9smCHbp4N17xjjLF6
Adds the integration coverage #165 actually asks for. The unit tests
assert the options object's shape; these assert that Postgres really
cancels a query past statement_timeout (SQLSTATE 57014), that the pooled
connection stays usable afterwards — a cancelled query must not poison
it, or one slow client would degrade later requests — and that
PG_STATEMENT_TIMEOUT=0 disables the limit as documented.

Also corrects PG_MAX_CONNECTIONS from "per host" to total. porsager's
`max` is the whole pool, and a multi-host PG_CONN fails over rather than
fanning out, so the pool only ever points at one host at a time. Since
the README documents multi-host replicas, "per host" invited operators to
size the pool N times too large.

Addresses review feedback on #182.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P0 Blocker for public availability production-readiness Work toward making the API production-ready / publicly available

Projects

None yet

Development

Successfully merging this pull request may close these issues.

P0: Configure Postgres pool limits + statement_timeout

2 participants