Skip to content

fix: unify all CreateSecret paths under single mutex to prevent write-write conflict#1113

Closed
HaoXuAI wants to merge 3 commits into
duckdb:v1.5-variegatafrom
HaoXuAI:fix/unified-refresh-mutex
Closed

fix: unify all CreateSecret paths under single mutex to prevent write-write conflict#1113
HaoXuAI wants to merge 3 commits into
duckdb:v1.5-variegatafrom
HaoXuAI:fix/unified-refresh-mutex

Conversation

@HaoXuAI

@HaoXuAI HaoXuAI commented Jun 25, 2026

Copy link
Copy Markdown

Problem

Three code paths call CreateSecret on the same aws_secret concurrently:

  1. CreateAWSInput — inline, no mutex, fired on EVERY S3 request
  2. TryRefreshStorageSecret — no mutex, fired on table access
  3. Multiple SIGV4Authorization instances sharing the same secret name

When two execute simultaneously → write-write conflict on alter with "aws_secret" → secret permanently poisoned → all queries fail until process restart.

Fix

  • Replace inline refresh in CreateAWSInput with MaybeRefreshSecret (static mutex + 300s time gate)
  • TryRefreshStorageSecret shares the same mutex via GetRefreshMutex()/GetLastRefreshTime() accessors
  • Both paths update last_refresh_time on failure (backs off instead of cascading retries)

Reproduction

Any deployment using SIGV4 auth with web_identity credential chain under concurrent queries:

  • ATTACH an Iceberg REST catalog with SIGV4 auth
  • Run 2+ concurrent queries after 300s (credential refresh interval)
  • Result: TransactionContext Error: Catalog write-write conflict on alter with "aws_secret"
  • Secret permanently poisoned — all subsequent queries fail until process restart

Testing

Verified on production (Iceberg Insight Server, 2 replicas, ~10 concurrent queries):

  • Before: 66% HTTP 500 rate under concurrency
  • After: 0% errors over extended testing

HaoXuAI and others added 3 commits June 10, 2026 07:21
- Trigger secret refresh before SigV4 catalog API calls when refresh_info
  is present, ensuring REST catalog requests use non-expired STS tokens
- Propagate refresh_info to per-table secrets so httpfs can auto-refresh
  credentials during long-running queries
- Add TryRefreshStorageSecret() helper that mirrors httpfs refresh logic
  using public DuckDB APIs (no cross-extension dependency)
Builds the patched iceberg.duckdb_extension for linux_amd64 and publishes
to Artifactory (maps-libs-public) for use by iceberg-insight-server.
…-write conflict

Three code paths called CreateSecret on aws_secret concurrently:
1. CreateAWSInput (inline, no mutex, fired on EVERY S3 request)
2. TryRefreshStorageSecret (no mutex)
3. (future) MaybeRefreshSecret

When two hit simultaneously → 'write-write conflict on alter with aws_secret'
→ secret poisoned → all subsequent queries fail until pod restart.

Fix:
- Replace inline refresh in CreateAWSInput with MaybeRefreshSecret (mutex + 300s time gate)
- TryRefreshStorageSecret now uses SIGV4Authorization's shared mutex via accessors
- Both paths back off on failure (update last_refresh_time) to prevent cascading retries

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@HaoXuAI HaoXuAI closed this Jun 25, 2026
@HaoXuAI
HaoXuAI deleted the fix/unified-refresh-mutex branch June 25, 2026 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant