fix(timeout): 120s provisioning deadline so slow provisions don't 409#21
Merged
Merged
Conversation
Provisioning is synchronous server-side: POST /db/new (and the other
/*/new endpoints plus /deploy/new) blocks while the real Postgres /
Redis / Mongo / NATS / bucket / pod is created. Under prod hot-pool
contention a fresh Postgres provision can exceed the old client-wide
30s timeout; when the client gave up, the server kept working and held
a 60s in-flight idempotency marker, so the caller's retry hit
409 idempotency_key_in_progress instead of succeeding.
Give provisioning + deploy a dedicated HTTP client with no client-wide
Timeout cap and a 120s per-request context deadline (provisionContext),
comfortably outliving both the slow provision and the server in-flight
window. Reads (list/get/claim/delete/rotate) stay capped at 30s via the
existing httpClient.Timeout, so the guarded read-path timeout tests are
unchanged. WithTimeout now governs both budgets; a tighter caller
context deadline is always honoured (provisionContext only lengthens an
open-ended context, never overrides a shorter caller deadline).
Tests (package instant, run under -race in CI):
- TestProvisioningTimeoutDefaults: pins read=30s / provision=120s split
+ provisionClient.Timeout==0.
- TestProvisioningOutlivesReadCap: behavioral proof — a provision
succeeds against a server that stalls past the read-path cap while a
read against the same server times out (scaled to ms).
- TestProvisioningHonorsTighterCallerDeadline / TestWithTimeoutGoverns
Provisioning / TestWithHTTPClientTimeoutGovernsProvisioning /
TestProvisionContext{OpenEndedGetsBudget,ZeroTimeoutFallsBack} /
TestProvisionJSONMarshalError.
README (Provisioning + Client configuration) and CHANGELOG document the
120s provisioning default and the WithTimeout override.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mastermanas805
added a commit
that referenced
this pull request
Jun 10, 2026
…ch, presign, pause/resume, wake (#23) MCP PR #41 shipped 9 operate tools; the SDK covered only 3 of the 11 operate-verb surfaces (RotateCredentials, Capabilities, DeploymentEvents). This adds the missing 8 methods, each verified against the MCP client and api openapi.go/handlers for exact endpoint, payload, and envelope: - SetVaultKey / RotateVaultKey PUT /api/v1/vault/:env/:key (+/rotate) - UpdateDeployEnv PATCH /deploy/:id/env - UpdateStackEnv PATCH /stacks/:slug/env - PresignStorage POST /storage/:token/presign (broker mode, token-as-credential, no bearer required) - PauseResource / ResumeResource POST /api/v1/resources/:id/{pause,resume} - WakeDeployment POST /deploy/:id/wake (surfaces the 501 scale_to_zero_disabled flag gate) All 8 are quick mutations, so they run on the 30s read-path client (none are synchronous provisions — the 120s provisioning deadline from #21 stays scoped to the /*/new + deploy/stack create endpoints). Path segments are url.PathEscape'd so vault keys with unusual characters round-trip. New putJSON/patchJSON client helpers carry the same retry + APIError-envelope semantics as the existing verbs. Tests: httptest-based per-method suites asserting method, path, body wire shape, response decode, validation branches, and one APIError branch each (402 tier gate, 409 not_paused, 410 resource_inactive, 501 flag gate). 100% statement coverage on every new function; package total 98.5%. README method tables + CHANGELOG updated; vault/env-patch removed from the "not covered" list (also added the missing ProvisionStorage / ProvisionWebhook rows). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem (live-verified)
The SDK's single client-wide ~30 s HTTP timeout was too tight for synchronous provisioning. Under prod hot-pool contention a fresh Postgres provision exceeds 30 s; the client gave up, the server kept working and held a 60 s in-flight idempotency marker, so the caller's retry hit
409 idempotency_key_in_progress. Raising the client-side timeout to ~120 s fixed it in the field.Fix
ProvisionDatabase/Cache/MongoDB/Queue/Storage/Webhook+Deploy) run on a dedicatedprovisionClientthat carries no client-wideTimeoutcap; the 120 s budget is enforced per-request viaprovisionContext. Read calls (list/get/claim/delete/rotate) are unchanged on the existinghttpClient.Timeout(30 s), so the guarded read-path timeout tests stay green.WithTimeoutstill overrides — it now governs both the read and provisioning budgets. AWithHTTPClientwhoseTimeoutis explicitly set governs provisioning too (the SDK won't silently widen a caller's deliberate budget to 120 s).context.Contextdeadline is always honoured —provisionContextonly lengthens an open-ended context, never overrides a shorter caller deadline.http.Client.Timeoutis a hard ceiling on every request, so a per-request context deadline alone couldn't extend past 30 s. Splitting into two HTTP clients (sharing the same auth-wrapped transport) is the least-disruptive correct fix and leaves the read-path tests —TestNew_DefaultsAndOptions(Timeout == defaultTimeout) andclient_options_test.go'sTestWithHTTPClient_PreservesTimeout— untouched.No 30 s default was intentionally guarded for the provisioning path; the only pinned-30 s tests are read-path and remain valid.
Tests (package
instant, run under-racein CI)TestProvisioningTimeoutDefaults— pins read=30 s / provision=120 s split +provisionClient.Timeout==0.TestProvisioningOutlivesReadCap— behavioral proof: a provision succeeds against a server that stalls past the read-path cap, while a read against the same server times out (scaled to ms; same mechanism as 30 s/120 s).TestProvisioningHonorsTighterCallerDeadline,TestWithTimeoutGovernsProvisioning,TestWithHTTPClientTimeoutGovernsProvisioning,TestProvisionContextOpenEndedGetsBudget,TestProvisionContextZeroTimeoutFallsBack,TestProvisionJSONMarshalError.100% coverage on every added line (
provisionContext,provisionJSONWithHeaders,jsonBodyReader,New,WithTimeoutall 100%; the only changed line in the shareddoWithClientloop body —hc.Do(req)— is covered).Docs
README Provisioning section gains a "Timeouts: provisioning runs longer than reads" subsection; Client configuration
WithTimeoutcomment updated; CHANGELOG Unreleased/Fixed entry added.Gate
go build ./...✓go vet ./... ./examples/...✓go test ./... -race -count=1✓ (CI's exact command)golangci-lint run ./...→ 0 issuesgofmt(real,$GOROOT/bin/gofmt) clean on all authored/edited filesgo test -tags integration -run TestProdIntegration✓ (read-only prod checks, no regression)🤖 Generated with Claude Code