Skip to content

add credential caching#154

Draft
benfleis wants to merge 2 commits into
duckdb:mainfrom
benfleis:cache-credentials
Draft

add credential caching#154
benfleis wants to merge 2 commits into
duckdb:mainfrom
benfleis:cache-credentials

Conversation

@benfleis

@benfleis benfleis commented Mar 3, 2026

Copy link
Copy Markdown
Member

Cache token credentials (by default), so that they endure between queries.
Add CACHE_TOKEN_CREDENTIAL flag (default=true) to SECRET creation, to opt-out in unusual cases, e.g. expecting az account logout to invalidate cache.

Testing so far is [a] confirming that existing behavior works, and [b] manually checking call counts to oauth2. This works via:

AZURE_LOG_LEVEL=verbose <run-test> 2>&1 | tee test.log
grep -c /oauth2 test.log

Specifically:

test original caching
spn_auth.test 3 1
whole suite w/ azure cloud (no slow) 260 26

So it's definitely working within context.

Additionally tested credential invalidations when SECRET replaced, confirmed that the SECRET forces the minting of a new token.

If/when I get around to supporting httplib, I can use http logs to automate these tests

Testing so far is [a] confirming that existing behavior works, and [b]
manually checking call counts to oath2. This works via:

```
AZURE_LOG_LEVEL=verbose <run-test> 2>&1 | tee test.log
grep -c test.log
```

Specifically, spn_auth.test drops from 3 -> 1 oauth2 calls, and across
the entire test suite (non-slow), oauth2 calls drop from 260 -> 26. So
it's definitely working within context.

Additionally tested credential invalidations when SECRET replaced,
confirmed that the SECRET forces the minting of a new token.
@benfleis
benfleis requested a review from samansmink March 3, 2026 11:22

@samansmink samansmink left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @benfleis! thanks for the PR! this looks really good, the results sound really nice. I have added some comments

Comment thread src/azure_storage_account_client.cpp Outdated
auto transport_options = GetTransportOptions(opener, secret);
auto mi_cred = GetManagedIdentityCredential(opener, secret, transport_options);

auto fp = SecretFingerprint(secret);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels like it could be DRY-d up a bit

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was on the edge for me whether to template a function to handle this... felt just below the line. I'll try one and see whether it helps (enough).

static std::string SecretFingerprint(const KeyValueSecret &secret) {
std::string fp;
// NOTE: this key ordering is safe because secret_map is an ordered tree
for (const auto &kv : secret.secret_map) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will use potentially sensitive data (e.g. token) as a fingerprint that is now stored in the clientcontext state. I feel this would make it quite easy to lose track of things and accidentially have this sensitive data be printed to the screen or otherwise accidentally leaked somewhere.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My thinking was less paranoid here, but I can also easily hash it and call it a day.

@@ -388,8 +432,14 @@ static Azure::Storage::Blobs::BlobServiceClient
GetBlobStorageAccountClientFromCredentialChainProvider(optional_ptr<FileOpener> opener, const KeyValueSecret &secret,
const AzureParsedUrl &azure_parsed_url) {
auto transport_options = GetTransportOptions(opener, secret);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the transport options fetch both from a secret and settings. Since you only use the fingerprint based on the secret, this would now result in stale caching potentially

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ay ay ay, good eye! I'll fix this.

- rework caching/invalidation a bit
- add CACHE_TOKEN_CREDENTIAL flag to disable for e.g. `az account logout`
- DRY up cache handling
@benfleis benfleis changed the title add credential caching; invalidate upon secret change add credential caching Mar 4, 2026
@benfleis
benfleis marked this pull request as draft March 10, 2026 11:16
@jbunting

Copy link
Copy Markdown

This would be really useful for us. Can I do anything to help move it along?

@benfleis

Copy link
Copy Markdown
Member Author

Thanks @jbunting , good to know. Nothing atm; it's on my backlog to finish but not forgotten!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants