Add credential auto-refresh support for web_identity/STS chains by HaoXuAI · Pull Request #1051 · duckdb/duckdb-iceberg

HaoXuAI · 2026-06-10T13:50:44Z

Summary

Adds credential auto-refresh support for web_identity and sts credential chains (e.g., IRSA on EKS). Without this, STS credentials expire after ~1 hour and Iceberg queries fail with ExpiredToken.

Problem

The iceberg extension was:

Stripping refresh_info from the catalog secret when building per-table secrets
Snapshotting static credentials for SigV4 catalog API calls without triggering refresh

Even with duckdb-aws#136 (web_identity chain with auto-refresh) and duckdb-httpfs#165 (refresh in more locations), the iceberg extension bypassed the refresh path entirely.

Changes

`sigv4.cpp` / `sigv4.hpp` — Serialized credential refresh before SigV4 signing

Adds MaybeRefreshSecret() method to SIGV4Authorization that:

Checks if enough time has passed since the last refresh (300s interval, configurable via REFRESH_INTERVAL_SECONDS)
Acquires a mutex with try_to_lock — concurrent callers skip instead of blocking
Double-checks timing after acquiring the lock to avoid redundant refreshes
Reconstructs a CreateSecretInput from the secret's refresh_info struct and re-runs the credential chain via SecretManager::CreateSecret to get fresh STS tokens
On failure, skips updating the timestamp so the next request retries

The refresh_info is a STRUCT containing the original named parameters used to create the secret (e.g., chain, region, profile). These are stored by the aws extension when refresh='auto' is set (automatically for web_identity and sts chains).

`iceberg_table_information.cpp` — Propagate refresh_info to per-table secrets

Adds TryRefreshCatalogSecret() helper that mirrors httpfs's refresh logic using public DuckDB APIs (no cross-extension dependency)
Before snapshotting catalog credentials into per-table secrets, triggers a refresh to ensure fresh values
Stops stripping refresh_info so httpfs can auto-refresh per-table secrets during long-running data reads

Dependencies

duckdb-aws#136 (web_identity chain) — merged
duckdb-httpfs#165 (refresh in GLOBs/range reads) — open; our patch handles catalog API calls independently

Testing

Added test_web_identity_credential_refresh.test guarded by mode skip with environment documentation
Tested in production on EKS with IRSA — pods running 10+ hours with zero ExpiredToken errors, credentials refreshing transparently

Tmonster

I would like to see some test for this, even if they are guarded by mode skip + a description of what kind of environment should be set up.

I think there is also a nicer way to handle the Vcpkg ports, I'll come back to this

Tmonster · 2026-06-11T08:13:14Z

+			refresh_input.storage_type = secret_entry->storage_mode;
+			refresh_input.scope = kv_secret_pre.GetScope();
+
+			auto child_count = StructType::GetChildCount(refresh_info.type());


Can you leave a comment as to what children are needed in a secret refresh? I can't think of any

Done — added a comment explaining that refresh_info is a STRUCT containing the original named parameters used to create the secret (e.g., chain, region, profile). These are stored by the aws extension when refresh='auto' is set. Re-passing them to SecretManager::CreateSecret re-runs the credential chain provider, which fetches fresh STS tokens.

Tmonster · 2026-06-11T09:03:48Z

+//! Attempt to refresh the catalog's storage secret if it supports auto-refresh.
+//! Rebuilds the secret via its original provider (e.g., web_identity/sts credential chain)
+//! which fetches fresh STS tokens. No-op if the secret lacks refresh_info.
+static bool TryRefreshStorageSecret(ClientContext &context, const SecretEntry &secret_entry) {


I think this can be renamed to TryRefreshCatalogSecret?

Done — renamed to TryRefreshCatalogSecret.

Tmonster

Thanks! Looks good to me, are we worried at all that this might refresh the secret unnecessarily? I.e if are credentials are valid for an hour, we don't need to refresh every time we read the table within that hour right?

HaoXuAI · 2026-06-16T06:52:08Z

@Tmonster actually the biggest problem for us it got into a race situation when multiple queries landed and all try to refresh the creds. And I'm trying to resolve this issue now

Tmonster

I don't think having one sigv4 refresh for all sigv4 authenticated catalogs is a good idea. Can we make the refresh a per catalog thing?

Also, if you can figure out a way to add tests that would be awesome. I understand it will require some more setup, but if you could include the setup in some script unders scripts, you can guard your test with a mode skip

Tmonster · 2026-06-26T07:59:35Z


+// Static member definitions — shared across all SIGV4Authorization instances
+// AND TryRefreshStorageSecret (via GetRefreshMutex/GetLastRefreshTime accessors)
+std::mutex SIGV4Authorization::refresh_mutex;


Is there a reason we need this shared across all SigV4Authorization instances? If I attach different sigv4 catalogs with different secrets, the refresh logic for both of them should not interfere

Good catch on the wording — the mutex and last_refresh_time are actually instance members of SIGV4Authorization (declared in sigv4.hpp), not static/shared. Each attached catalog gets its own SIGV4Authorization instance, so the refresh logic is already per-catalog and won't interfere between different catalogs with different secrets.

The #include <mutex> at the top of the header is just the standard library include — it doesn't make anything shared.

Tmonster · 2026-06-26T08:01:11Z

+
+	// Fast path: skip if refreshed recently
+	auto now = std::chrono::steady_clock::now();
+	auto elapsed = std::chrono::duration_cast<std::chrono::seconds>(now - last_refresh_time).count();


Seems like we run the if we refreshed recently return check here and also in MaybeRefreshSecret? I think we can gfet away with only doing this once?

You're right — these two serve different purposes but the duplication is confusing:

MaybeRefreshSecret() in sigv4.cpp — called on every catalog API request, time-gated to 300s intervals. This is the hot path.

TryRefreshCatalogSecret() in iceberg_table_information.cpp — called once during GetVendedCredentials() to ensure fresh creds are snapshotted into per-table secrets. This isn't called per-request.

I'll simplify by removing the time-gate from TryRefreshCatalogSecret (it doesn't need one since it's not in the hot path) and just keeping it as a simple "refresh if refresh_info exists" helper. The time-gating only matters in MaybeRefreshSecret where it's called on every request.

- Add per-catalog MaybeRefreshSecret() on SIGV4Authorization with: - Per-instance mutex (catalogs with different secrets don't interfere) - 300s time-gate (avoids refreshing on every request) - try_to_lock (concurrent callers skip instead of blocking) - Call MaybeRefreshSecret before SigV4 signing in CreateAWSInput - Call MaybeRefreshSecret before snapshotting credentials into per-table secrets - Propagate refresh_info to per-table secrets for httpfs auto-refresh

Extends the existing scripts/sigv4_mock_server.py with an STS AssumeRoleWithWebIdentity endpoint that issues rotating credentials (MOCKKEY0, MOCKKEY1, ...) and a /refresh_count endpoint, reusing the upstream mock instead of adding a second one. The test verifies the SigV4 credential refresh is time-gated: it fires on first use (ATTACH) and does not re-fetch on subsequent requests within the 300s window. Guarded by require-env CREDENTIAL_REFRESH_MOCK_AVAILABLE; source scripts/credential_refresh_test_setup.sh to run.

HaoXuAI force-pushed the feature/credential-auto-refresh branch 2 times, most recently from 39b711e to e2d667c Compare June 10, 2026 13:53

Tmonster reviewed Jun 11, 2026

View reviewed changes

HaoXuAI force-pushed the feature/credential-auto-refresh branch 3 times, most recently from 77625e9 to b817338 Compare June 12, 2026 07:44

Tmonster reviewed Jun 15, 2026

View reviewed changes

HaoXuAI force-pushed the feature/credential-auto-refresh branch from b817338 to 89eb657 Compare June 16, 2026 06:48

HaoXuAI force-pushed the feature/credential-auto-refresh branch from 89eb657 to ac82ffd Compare June 18, 2026 23:59

This was referenced Jun 20, 2026

feat: store credential expiration epoch in secret for proactive refresh duckdb/duckdb-aws#157

Merged

fix: unify all CreateSecret paths under single mutex to prevent write-write conflict #1113

Closed

HaoXuAI force-pushed the feature/credential-auto-refresh branch 2 times, most recently from 65b7b7b to b395689 Compare June 25, 2026 20:24

Tmonster reviewed Jun 26, 2026

View reviewed changes

HaoXuAI force-pushed the feature/credential-auto-refresh branch from b395689 to 9e37263 Compare June 26, 2026 22:42

HaoXuAI added 2 commits July 15, 2026 13:40

HaoXuAI force-pushed the feature/credential-auto-refresh branch from 9e37263 to 6926e1a Compare July 15, 2026 20:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add credential auto-refresh support for web_identity/STS chains#1051

Add credential auto-refresh support for web_identity/STS chains#1051
HaoXuAI wants to merge 2 commits into
duckdb:mainfrom
HaoXuAI:feature/credential-auto-refresh

HaoXuAI commented Jun 10, 2026 •

edited

Loading

Uh oh!

Tmonster left a comment

Uh oh!

Tmonster Jun 11, 2026

Uh oh!

HaoXuAI Jun 12, 2026

Uh oh!

Tmonster Jun 11, 2026

Uh oh!

HaoXuAI Jun 12, 2026

Uh oh!

Tmonster left a comment

Uh oh!

HaoXuAI commented Jun 16, 2026

Uh oh!

Tmonster left a comment

Uh oh!

Tmonster Jun 26, 2026

Uh oh!

HaoXuAI Jun 26, 2026

Uh oh!

Tmonster Jun 26, 2026

Uh oh!

HaoXuAI Jun 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

HaoXuAI commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Problem

Changes

sigv4.cpp / sigv4.hpp — Serialized credential refresh before SigV4 signing

iceberg_table_information.cpp — Propagate refresh_info to per-table secrets

Dependencies

Testing

Uh oh!

Tmonster left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tmonster left a comment

Choose a reason for hiding this comment

Uh oh!

HaoXuAI commented Jun 16, 2026

Uh oh!

Tmonster left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

HaoXuAI commented Jun 10, 2026 •

edited

Loading

`sigv4.cpp` / `sigv4.hpp` — Serialized credential refresh before SigV4 signing

`iceberg_table_information.cpp` — Propagate refresh_info to per-table secrets