Skip to content

Endpoint with custom port requires legacy configuration #129

Description

@james-aus

When working with some self-hosted S3 providers (such as Garage), it can be useful to set a custom port.

Using what I think is the recommended way of doing things, I can create the default secret and try to use it like so:

create or replace secret (type s3, provider config, key_id 'abc123', secret 'abc123', region 'garage',
        endpoint 'somehost:3900', url_style 'path', use_ssl 'false';
select count(*) from read_parquet(['s3://bucket/Example/somefile.parquet']);
IO Error:
Could not establish connection error for HTTP HEAD to 'http://somehost:3900/bucket/Example/somefile.parquet'

LINE 1: select count(*) from read_parquet(['s3://bucket/Example/somefile...
                             ^

Digging deeper, I could see no attempt to connect in my logs and I believe that the port number is not being split from the hostname -- but I'm not sure of that because I'm unfamiliar with the AWS SDK and haven't spent a lot of time reading the code.

I did notice however, that running drop secret __default_s3 (or restarting the DuckDB CLI -- just unconfiguring what we've just done) then configuring instead:

set s3_region = 'garage';
set s3_endpoint = 'somehost:3900';
set s3_use_ssl = false;
set s3_url_style = 'path';
set s3_access_key_id = 'abc123';
set s3_secret_access_key = 'abc123';

Allows the previous count(*) command to work correctly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions