Skip to content

fix(timeout): 120s provisioning deadline so slow provisions don't 409#21

Merged
mastermanas805 merged 1 commit into
masterfrom
fix/provisioning-timeout-durability
Jun 10, 2026
Merged

fix(timeout): 120s provisioning deadline so slow provisions don't 409#21
mastermanas805 merged 1 commit into
masterfrom
fix/provisioning-timeout-durability

Conversation

@mastermanas805

Copy link
Copy Markdown
Member

Problem (live-verified)

The SDK's single client-wide ~30 s HTTP timeout was too tight for synchronous provisioning. Under prod hot-pool contention a fresh Postgres provision exceeds 30 s; the client gave up, the server kept working and held a 60 s in-flight idempotency marker, so the caller's retry hit 409 idempotency_key_in_progress. Raising the client-side timeout to ~120 s fixed it in the field.

Fix

  • Provisioning + deploy now default to a 120 s per-request deadline; reads stay at 30 s. The provisioning paths (ProvisionDatabase/Cache/MongoDB/Queue/Storage/Webhook + Deploy) run on a dedicated provisionClient that carries no client-wide Timeout cap; the 120 s budget is enforced per-request via provisionContext. Read calls (list/get/claim/delete/rotate) are unchanged on the existing httpClient.Timeout (30 s), so the guarded read-path timeout tests stay green.
  • WithTimeout still overrides — it now governs both the read and provisioning budgets. A WithHTTPClient whose Timeout is explicitly set governs provisioning too (the SDK won't silently widen a caller's deliberate budget to 120 s).
  • A tighter caller context.Context deadline is always honouredprovisionContext only lengthens an open-ended context, never overrides a shorter caller deadline.
  • The single client-wide cap was the root constraint: http.Client.Timeout is a hard ceiling on every request, so a per-request context deadline alone couldn't extend past 30 s. Splitting into two HTTP clients (sharing the same auth-wrapped transport) is the least-disruptive correct fix and leaves the read-path tests — TestNew_DefaultsAndOptions (Timeout == defaultTimeout) and client_options_test.go's TestWithHTTPClient_PreservesTimeout — untouched.

No 30 s default was intentionally guarded for the provisioning path; the only pinned-30 s tests are read-path and remain valid.

Tests (package instant, run under -race in CI)

  • TestProvisioningTimeoutDefaults — pins read=30 s / provision=120 s split + provisionClient.Timeout==0.
  • TestProvisioningOutlivesReadCap — behavioral proof: a provision succeeds against a server that stalls past the read-path cap, while a read against the same server times out (scaled to ms; same mechanism as 30 s/120 s).
  • TestProvisioningHonorsTighterCallerDeadline, TestWithTimeoutGovernsProvisioning, TestWithHTTPClientTimeoutGovernsProvisioning, TestProvisionContextOpenEndedGetsBudget, TestProvisionContextZeroTimeoutFallsBack, TestProvisionJSONMarshalError.

100% coverage on every added line (provisionContext, provisionJSONWithHeaders, jsonBodyReader, New, WithTimeout all 100%; the only changed line in the shared doWithClient loop body — hc.Do(req) — is covered).

Docs

README Provisioning section gains a "Timeouts: provisioning runs longer than reads" subsection; Client configuration WithTimeout comment updated; CHANGELOG Unreleased/Fixed entry added.

Gate

  • go build ./...go vet ./... ./examples/...
  • go test ./... -race -count=1 ✓ (CI's exact command)
  • golangci-lint run ./...0 issues
  • gofmt (real, $GOROOT/bin/gofmt) clean on all authored/edited files
  • go test -tags integration -run TestProdIntegration ✓ (read-only prod checks, no regression)

🤖 Generated with Claude Code

Provisioning is synchronous server-side: POST /db/new (and the other
/*/new endpoints plus /deploy/new) blocks while the real Postgres /
Redis / Mongo / NATS / bucket / pod is created. Under prod hot-pool
contention a fresh Postgres provision can exceed the old client-wide
30s timeout; when the client gave up, the server kept working and held
a 60s in-flight idempotency marker, so the caller's retry hit
409 idempotency_key_in_progress instead of succeeding.

Give provisioning + deploy a dedicated HTTP client with no client-wide
Timeout cap and a 120s per-request context deadline (provisionContext),
comfortably outliving both the slow provision and the server in-flight
window. Reads (list/get/claim/delete/rotate) stay capped at 30s via the
existing httpClient.Timeout, so the guarded read-path timeout tests are
unchanged. WithTimeout now governs both budgets; a tighter caller
context deadline is always honoured (provisionContext only lengthens an
open-ended context, never overrides a shorter caller deadline).

Tests (package instant, run under -race in CI):
- TestProvisioningTimeoutDefaults: pins read=30s / provision=120s split
  + provisionClient.Timeout==0.
- TestProvisioningOutlivesReadCap: behavioral proof — a provision
  succeeds against a server that stalls past the read-path cap while a
  read against the same server times out (scaled to ms).
- TestProvisioningHonorsTighterCallerDeadline / TestWithTimeoutGoverns
  Provisioning / TestWithHTTPClientTimeoutGovernsProvisioning /
  TestProvisionContext{OpenEndedGetsBudget,ZeroTimeoutFallsBack} /
  TestProvisionJSONMarshalError.

README (Provisioning + Client configuration) and CHANGELOG document the
120s provisioning default and the WithTimeout override.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mastermanas805
mastermanas805 merged commit a71a6e0 into master Jun 10, 2026
11 checks passed
@mastermanas805
mastermanas805 deleted the fix/provisioning-timeout-durability branch June 10, 2026 17:27
mastermanas805 added a commit that referenced this pull request Jun 10, 2026
…ch, presign, pause/resume, wake (#23)

MCP PR #41 shipped 9 operate tools; the SDK covered only 3 of the 11
operate-verb surfaces (RotateCredentials, Capabilities, DeploymentEvents).
This adds the missing 8 methods, each verified against the MCP client and
api openapi.go/handlers for exact endpoint, payload, and envelope:

- SetVaultKey / RotateVaultKey   PUT  /api/v1/vault/:env/:key (+/rotate)
- UpdateDeployEnv                PATCH /deploy/:id/env
- UpdateStackEnv                 PATCH /stacks/:slug/env
- PresignStorage                 POST /storage/:token/presign (broker mode,
                                 token-as-credential, no bearer required)
- PauseResource / ResumeResource POST /api/v1/resources/:id/{pause,resume}
- WakeDeployment                 POST /deploy/:id/wake (surfaces the 501
                                 scale_to_zero_disabled flag gate)

All 8 are quick mutations, so they run on the 30s read-path client (none
are synchronous provisions — the 120s provisioning deadline from #21 stays
scoped to the /*/new + deploy/stack create endpoints). Path segments are
url.PathEscape'd so vault keys with unusual characters round-trip. New
putJSON/patchJSON client helpers carry the same retry + APIError-envelope
semantics as the existing verbs.

Tests: httptest-based per-method suites asserting method, path, body wire
shape, response decode, validation branches, and one APIError branch each
(402 tier gate, 409 not_paused, 410 resource_inactive, 501 flag gate).
100% statement coverage on every new function; package total 98.5%.

README method tables + CHANGELOG updated; vault/env-patch removed from the
"not covered" list (also added the missing ProvisionStorage /
ProvisionWebhook rows).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant