[WIP] Keyless per-pod cloud identity for S3 and GCS

[achudnovskij]

Draft — not ready for review.

Lets a receiver authenticate to object storage with ambient per-pod identity instead of static keys, so each one can run under its own ServiceAccount bound to a scoped cloud role.

• S3 — web-identity (IRSA) via STS AssumeRoleWithWebIdentity, and the container credentials endpoint (EKS Pod Identity / ECS).
• GCS — the GKE/GCE metadata server (Workload Identity), replacing the metadata-server auth not yet supported hard error.

No new dependencies.

Credential chain

src (backup-copy inherit) → static keys → web-identity → container credentials → EC2 IMDS → error if AWS_EC2_METADATA_DISABLED.

This mirrors the AWS SDKs, which is where wal-g gets the same behavior — it writes essentially no credential code of its own (session.NewSessionWithOptions plus the SDK default chain; gcs.NewClient(ctx) for pure ADC on the GCS side). Static keys and GCS SA-JSON still win when present, so existing deployments are unaffected and this can ship independently of the operator work. Incomplete static keys remain a hard error.

Env-var contract

Source	Vars
S3 web-identity	AWS_WEB_IDENTITY_TOKEN_FILE + AWS_ROLE_ARN; optional AWS_ROLE_SESSION_NAME, AWS_ENDPOINT_URL_STS
S3 container creds	AWS_CONTAINER_CREDENTIALS_FULL_URI or ..._RELATIVE_URI; optional AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE / AWS_CONTAINER_AUTHORIZATION_TOKEN
GCS Workload Identity	none — absence of GOOGLE_APPLICATION_CREDENTIALS selects it; GCE_METADATA_HOST overrides the host

On EKS/GKE the platform injects these itself once the pod runs under a bound ServiceAccount.

Notes for the reviewer

• One provider, not two. Both new S3 sources are a single authenticated HTTP call plus the cache/refresh ImdsProvider already had, so they share one AmbientProvider behind an AmbientKind enum — one credentials() body, one client builder, one send/status/body block, and one new CredentialSource arm instead of two. Container creds return the IMDS document shape, so parse_creds is reused; STS returns Query-protocol XML, parsed with s3::first_tag_text (now pub(crate)).
• No GCS auth enum. new() only rewrites host away from STORAGE_HOST for the emulator, so is_emulator() recovers that state and sa: None can mean metadata auth. Both token paths end in the same status check → TokenResp → cache write, so access_token keeps that once and only the request differs.
• No signing changes. x-amz-security-token was already threaded and creds are fetched per request. identity() folds each refreshing source to a constant, so rotation can't spuriously fail server-side-copy eligibility.
• Rotating token files (projected SA JWT, Pod Identity auth token) are re-read on every refresh, never cached. A token file takes precedence over a static token, matching the SDK.
• Non-2xx from STS or the container endpoint maps to StorageError::Http, so a 5xx stays retryable via is_transient.
• GOOGLE_APPLICATION_CREDENTIALS="" now falls back to metadata instead of failing on a read of "".

Breaking change

GcsConfig gains a public metadata_endpoint field, which breaks external struct-literal construction — hence 0.3.0 rather than a patch. It carries GCE_METADATA_HOST and is the seam the metadata tests point at a mock.

Verification

cargo fmt --check, cargo clippy --all-targets --locked -- -D warnings, and cargo test --locked are all clean — 544 tests, 11 of them new (container creds incl. token-file precedence and HTTP errors, web-identity incl. caching, refresh-within-margin and missing-token-file, the STS XML parser, the GCS metadata token, and chain-order/precedence selection in config). The golden SigV4 signature tests are untouched.

No end-to-end cloud validation is possible from a devcontainer, so the mock-HTTP tests are the contract; real-cloud validation happens downstream once a per-tenant ServiceAccount is bound.

🤖 Generated with Claude Code