Upgrade duckdb 1.3.0 to 1.4.2 SSO Invalid Configuration Error

[cas--]

I have an SSO AWS account that I have been using locally with duckdb 1.3.0 without issue to access parquet files on private S3 bucket, however when I upgraded to 1.4.2 it fails to find the credentials. I am not sure how to resolve this, I have tried specifying my profile with no luck.

This is fetching parquet with

❯ AWS_PROFILE=prod ~/.duckdb/cli/1.3.0/duckdb
DuckDB v1.3.0 (Ossivalis) 71c5c07cdd
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D CREATE OR REPLACE SECRET secret (
      TYPE s3,
      PROVIDER credential_chain,
      region 'eu-west-1');
100% ▕████████████████████████████████████████████████████████████▏
┌─────────┐
│ Success │
│ boolean │
├─────────┤
│ true    │
└─────────┘
D select timestamp from 's3://xxxx-prod-01/xxxx.parquet' limit 1;
┌─────────────────────┐
│      timestamp      │
│      timestamp      │
├─────────────────────┤
│ 2025-11-11 04:00:00 │
└─────────────────────┘

Testing with 1.4.2 doesn't get past setting up the s3 secret:

AWS_PROFILE=prod ~/.duckdb/cli/1.4.2/duckdb
DuckDB v1.4.2 (Andium) 68d7555f68
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D CREATE OR REPLACE SECRET secret (
      TYPE s3,
      PROVIDER credential_chain,
      region 'eu-west-1');
Invalid Configuration Error:
Secret Validation Failure: during `create` using the following:
Credential Chain: 'config'

Weirdly with duckdb 1.3.1 and 1.3.2 it fails to fetch the parquet which might be clue:

❯ AWS_PROFILE=prod ~/.duckdb/cli/1.3.2/duckdb
DuckDB v1.3.2 (Ossivalis) 0b83e5d2f6
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D CREATE OR REPLACE SECRET secret (
      TYPE s3,
      PROVIDER credential_chain,
      region 'eu-west-1');
100% ▕████████████████████████████████████████████████████████████▏
┌─────────┐
│ Success │
│ boolean │
├─────────┤
│ true    │
└─────────┘
D select timestamp from 's3://xxxx-prod-01/xxxx.parquet' limit 1;
HTTP Error:
HTTP GET error on 'https://xxxx-prod-01.s3.amazonaws.com/xxxx.parquet' (HTTP 403)

Authentication Failure - this is usually caused by invalid or missing credentials.
* No credentials are provided.
* See https://duckdb.org/docs/stable/extensions/httpfs/s3api.html

❯ cat ~/.aws/config
[default]
region=eu-west-3
[profile prod]
sso_session = prod
sso_account_id = 2xxxxxxxxxxx
sso_role_name = prod-readonly
...

[cas--]

From the linked #131 I can confirm the workaround fix on Ubuntu of setting the ca cert path for curl resolves the issue

sudo mkdir -p /etc/pki/tls/certs
sudo ln -s /etc/ssl/certs/ca-certificates.crt /etc/pki/tls/certs/ca-bundle.crt

From duckdb we can need to explicitly provide the chain sso value with profile:

CREATE OR REPLACE SECRET secret (
  TYPE s3,
  PROVIDER credential_chain,
  CHAIN 'sso'
  PROFILE 'dev',
);

Alternatively can export AWS_PROFILE in shell session or prefix command:

AWS_PROFILE=PS-autotest-dev duckdb

CREATE OR REPLACE SECRET secret (
  TYPE s3,
  PROVIDER credential_chain,
  CHAIN 'sso'
);