Skip to content

Expose azure_get_token() scalar to mint AAD bearer tokens#170

Open
djouallah wants to merge 2 commits into
duckdb:v1.5-variegatafrom
djouallah:add-azure-get-token-scalar
Open

Expose azure_get_token() scalar to mint AAD bearer tokens#170
djouallah wants to merge 2 commits into
duckdb:v1.5-variegatafrom
djouallah:add-azure-get-token-scalar

Conversation

@djouallah

@djouallah djouallah commented May 19, 2026

Copy link
Copy Markdown
Contributor

Disclosure: this PR was authored with the assistance of an AI coding agent (Claude Code / Claude Opus 4.7). The code changes were generated by the agent under my direction, reviewed by me, and the commits include a Co-Authored-By: Claude trailer. The motivation, design choices, and security framing below reflect my requirements; the agent did the exploration, drafting, and CI iteration.

Summary

Adds a single scalar function azure_get_token([chain[, scope]]) that mints an Azure AAD bearer token from the same credential-chain providers duckdb-azure already understands (cli, managed_identity, env, default, workload_identity).

Three arities:

SELECT azure_get_token();                                                  -- chain='default', scope='https://storage.azure.com/.default'
SELECT azure_get_token('cli');                                             -- explicit chain
SELECT azure_get_token('cli', 'https://storage.azure.com/.default');       -- explicit scope

Motivation

duckdb/duckdb-iceberg#499 — attaching a OneLake REST iceberg catalog fails because the iceberg extension expects an OAuth2 client-credentials flow, while OneLake wants a plain AAD bearer token (the same one duckdb-azure already mints internally for storage reads).

duckdb/duckdb-iceberg#612 was closed by @Tmonster because it pulled the Azure SDK into duckdb-iceberg. His suggestion was to keep Azure logic in duckdb-azure. This PR follows that direction without modifying duckdb-iceberg at all — duckdb-iceberg's iceberg-typed secret already accepts a static token field, so once duckdb-azure can produce one, ATTACH works untouched:

LOAD azure; LOAD iceberg;

SET VARIABLE onelake_token = (SELECT azure_get_token('cli'));
CREATE OR REPLACE SECRET onelake_cat (TYPE iceberg, TOKEN getvariable('onelake_token'));

ATTACH 'workspace/lakehouse.Lakehouse' AS onelake (
    TYPE ICEBERG,
    ENDPOINT 'https://onelake.table.fabric.microsoft.com/iceberg',
    SECRET onelake_cat
);

Why expose a scalar, not a new secret type

A second iceberg-typed credential_chain secret in duckdb-azure was the obvious alternative but has two downsides:

  1. It cross-registers a secret type that only exists when duckdb-iceberg is loaded — LOAD order matters, and surfaces a confusing "Secret type not found" error otherwise.
  2. It's more code (new CreateSecretFunction, scope handling, redaction wiring) for what's essentially "call GetToken and stash the string".

A plain scalar function decouples duckdb-azure from duckdb-iceberg's secret-type registration entirely. The same primitive is also the natural hook for future auto-refresh.

Security framing — this reduces token exposure compared to current practice

Without this function, the working pattern for OneLake users today is a launcher script that:

  1. Shells out to az account get-access-token --resource https://storage.azure.com/ -o tsv > %TEMP%\az_tok.txt
  2. Reads the file back, embeds the token plaintext in a SET VARIABLE statement inside an init SQL file (often under C:\ProgramData\ so the path doesn't leak the username — which is world-readable)
  3. Launches duckdb -init <init.sql>

That workflow lands the JWT on disk twice (the temp file and the init SQL file), and races a manual DeleteFile between the two. azure_get_token() keeps the token entirely in-memory: no disk write, no init-file plaintext, no temp-file cleanup race. The token surface is narrower, not wider — the function only succeeds when a chain provider is already authenticated (e.g. az login), and anyone with that prerequisite can already mint the same token by hand.

Implementation

  • New function FetchAzureBearerToken(chain, scope) in src/azure_storage_account_client.cpp — directly calls the existing static CreateChainedTokenCredential helper in the same translation unit, so no header churn for the static helper.
  • Three thin scalar callbacks (0/1/2 args) using UnaryExecutor / BinaryExecutor to handle constant/null vectors.
  • One-line addition in src/azure_extension.cpp registers the function set via loader.RegisterFunction.
  • No CMake / vcpkg changes — azure-identity-cpp is already linked.
  • No changes to duckdb-iceberg.

Total: ~60 net lines across 3 files.

Out of scope

  • Auto-refresh. Token validity is ~1h; when it expires the user re-runs SET VARIABLE + CREATE OR REPLACE SECRET. Real auto-refresh would require either duckdb-iceberg to call azure_get_token() on every catalog request / on 401, or a duckdb mechanism to make secret values be live expressions. Both are bigger conversations. This PR is the primitive that makes either approach possible later.
  • Storage-side auth for the actual parquet reads from OneLake — already handled by the existing TYPE azure secret. Users attaching OneLake typically have two secrets: one azure (blob reads) and one iceberg (catalog).

Test plan

  • CI green on format check + Linux amd64/arm64. Mac/Windows-static were green at last check; I'll repost results after the force-push CI run lands.
  • Manual end-to-end against a real Fabric workspace:
    • az login
    • SELECT length(azure_get_token('cli')) returns a JWT
    • ATTACH against OneLake REST catalog succeeds, SHOW TABLES + SELECT ... LIMIT 5 reads rows
  • Negative test: SELECT azure_get_token('cli') without prior az login raises InvalidConfigurationException with the hint message.

Happy to add a sqllogictest verifying registration (SELECT 1 FROM duckdb_functions() WHERE function_name = 'azure_get_token'), mark the function FunctionStability::VOLATILE, and add a README section if you want any of those before merge.

@djouallah
djouallah marked this pull request as ready for review May 19, 2026 01:45
djouallah and others added 2 commits May 19, 2026 11:47
Mints a short-lived Azure AAD bearer token via the existing
CreateChainedTokenCredential helper. Intended use: feed the token
into a duckdb-iceberg `iceberg`-typed secret to attach an OneLake
REST catalog, without coupling either extension's secret types.

Three arities:
  azure_get_token()
  azure_get_token(chain)
  azure_get_token(chain, scope)

Defaults: chain='default', scope='https://storage.azure.com/.default'.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@djouallah
djouallah force-pushed the add-azure-get-token-scalar branch from dcf1154 to 5e7d1c6 Compare May 19, 2026 01:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant