Arrow 57→58 bump + DuckLake & Quack remote DuckDB sinks (v0.7.0)

[malon64]

Summary

Add two open remote DuckDB backends to the existing DuckDB accepted-sink, alongside the current Local + MotherDuck targets:

• DuckLake (duckdb/ducklake) — lakehouse: SQL catalog DB (local file / Postgres / MySQL) + Parquet data in object storage. Production-ready (v1.0). Ship stable.
• Quack (duckdb/duckdb-quack) — HTTP client-server RPC exposing full DuckDB over the wire, multi-writer. Beta until DuckDB 2.0 (~fall 2026). Ship experimental, opt-in.

Both are core extensions in DuckDB ≥1.5.3, autoloaded on first ATTACH — one duckdb-crate bump unlocks both.

This is the engine-feature half of the original v0.6.0 plan. The distribution half (lean floe + floe-duckdb companion binary / image / off-PyPI wheel) already shipped in v0.5.1–v0.5.4. This issue tracks the remaining sink work, targeted at a minor release (v0.7.0).

Why this was deferred (the upstream gate)

DuckLake/Quack need DuckDB ≥1.5.3 → duckdb crate =1.10503.1 → DuckDB 1.5.3 / Arrow 58. floe currently pins duckdb = "=1.10500.0" (DuckDB 1.5.0 / Arrow 57) and arrow = "57" on purpose.

The Phase-0 spike (2026-06-05) found iceberg has no published arrow-58 release — latest is 0.9.1 (arrow 57; main is bumped but unreleased). iceberg crosses the arrow type boundary in floe's own crates/floe-core/src/io/write/iceberg/data_files.rs + schema.rs, so dual arrow 57+58 won't compile; and floe-core/floe-cli publish to crates.io with default [delta,iceberg], so the publish verify build would fail. → the arrow bump is hard-blocked upstream until iceberg ships arrow 58 (est. ~July 2026; re-check crates.io/crates/iceberg before starting — as of 2026-06-11 still 0.9.1/arrow57).

Spike-confirmed arrow-58 dep set (ready once iceberg lands): duckdb=1.10503.1, df-interchange 0.3.3 (arrow_58+polars_0_52), deltalake 0.32.3, orc-rust 0.8.0.

---

Phase 0 — Arrow 57→58 / DuckDB ≥1.5.3 compile spike (HARD GATE, redo on a throwaway branch)

The original spike is stale; re-verify against current upstream before any feature code:

1. iceberg arrow-58 release exists (the blocker). If not → still blocked, stop.
2. Confirm duckdb=1.10503.1 still bundles DuckLake + Quack as core extensions; pin exactly (preserve pin discipline) and refresh the comment at crates/floe-core/Cargo.toml:49-51.
3. df-interchange 0.3.3 exposes arrow_58+polars_0_52 (crates/floe-core/Cargo.toml:46).
4. cargo tree -i arrow resolves a single arrow 58 across deltalake/iceberg/polars (no dual 57+58).
5. dataframe_to_record_batch (crates/floe-core/src/io/write/arrow_convert.rs) compiles on arrow 58 (expect no change).
6. Smoke: cargo build -p floe-core --features duckdb + existing duckdb_run:: integration tests green (proves Local + MotherDuck regress-free on DuckDB 1.5.3).

Exit gate: all green → proceed. Red on (1) or (4) → stop and reassess.

Phase 1 — Arrow bump (crates/floe-core/Cargo.toml)

• arrow = "57" → "58" (L42)
• df-interchange features arrow_57→arrow_58 (L46)
• duckdb =1.10500.0 → =1.10503.1 (L51) + comment refresh
• deltalake 0.30.1 → 0.32.3 (L20); iceberg 0.9.x → arrow-58 release
• Flip the two call sites: Interchange::from_arrow_57 → from_arrow_58 at crates/floe-core/src/io/read/orc.rs:93 and crates/floe-core/src/io/unique_seed/mod.rs:88
• ~6 deltalake 0.30→0.32 API fixes (spike-identified): io/write/delta/commit_metrics.rs:85, io/write/strategy/merge/scd1.rs:169, scd2.rs:280, shared.rs:123 (DeltaTable::schema removed) / :319 (Option<NonZero<u64>>) / :327

Phase 2 — DuckLake + Quack sinks (behind the existing duckdb feature)

2a. Target enum + dispatch — crates/floe-core/src/io/write/duckdb/conn.rs

Add to DuckDbTarget (conn.rs:30, keep Local/MotherDuck unchanged):

DuckLake { catalog: String, data_path: String, attach_alias: String,
           secret: Option<DuckLakeSecret>, catalog_token: Option<String>, cache_key: String },
Quack    { endpoint: String, attach_alias: String, token: Option<String>, cache_key: String },

• Extend cache_key() (conn.rs:46) / identity() (never expose secrets) / local_path() (new variants → None).
• resolve_target (conn.rs:86): dispatch by scheme — md:→MotherDuck, ducklake:→DuckLake, quack:→Quack, else updated "unsupported scheme (md:/ducklake:/quack:)" error; no-connection→Local. Add scheme predicates next to is_motherduck_connection (conn.rs:72) and the config-layer copy (validation/manifest run without the duckdb feature).
• Cache keys reuse the motherduck_cache_key token-fingerprint approach so secrets never appear in keys: DuckLake = ducklake:<catalog>#<data_path>#<fp(catalog_token+secret)>; Quack = quack:<endpoint>#<fp(token)>. The OnceLock cache + acquire need no change (key purely on cache_key()).
• open_connection (conn.rs:248): new arms.
  • DuckLake: open_in_memory → if secret, CREATE SECRET … (built in-process, never logged) → ATTACH 'ducklake:<catalog>' AS <alias> (DATA_PATH '<data_path>') → register_arrow_vtab.
  • Quack: open_in_memory → ATTACH '<quack attach string>' AS <alias> (exact syntax + token channel spike-confirmed; never inline a logged token) → register_arrow_vtab.
  • Network-on-first-use caveat: first ATTACH autoinstalls the extension over the network — fine for the shipped duckdb image; breaks air-gapped. Document; offer pre-placed-extension-dir fallback, default to autoinstall.

2b. Catalog-awareness (the one real code change beyond enum/dispatch)

DuckLake/Quack tables live under the ATTACHed alias (alias.schema.table), but today quoted_table/table_exists use 2-part names and information_schema.tables spans all catalogs (crates/floe-core/src/io/write/duckdb.rs). Use explicit 3-part names: add quoted_table_in(catalog, schema, table) and thread Option<&str> catalog alias through apply_write / merge::execute_scd1/execute_scd2 / seed_unique_tracker (~9 call sites). Local/MotherDuck pass None (unchanged). Merge clause SQL unchanged — only the target identifier becomes catalog-qualified.

2c. merge.rs — reused unchanged

execute_scd1/execute_scd2 keep their native MERGE INTO … RETURNING merge_action logic verbatim; only the target-identifier helper gains a catalog arg.

2d. Config / parse / validate

• types.rs DuckDbSinkTargetConfig: add optional data_path, attach_alias (default lake/remote), secret: Option<DuckDbSecretConfig> (type,key_id,secret,region,endpoint,session_token), catalog_token. MotherDuck path unchanged. All secret-bearing fields go through expand_env_token.
• parse.rs parse_sink_duckdb_options: extend validate_known_keys; parse nested secret hash with its own key check; keep blank_to_none.
• validate.rs validate_duckdb_sink (validate.rs:656): generalize the scheme match — md: (unchanged), ducklake: (require data_path; validate secret.type; no sink.accepted.storage binding), quack: (require experimental opt-in else clear error), unknown→updated error. Keep is_duckdb_motherduck (validate.rs:639); add is_duckdb_ducklake/is_duckdb_quack.

2e. Manifest — builder.rs / reconstruct.rs

• redact_duckdb_for_manifest: extend ${ENV}-preserving redaction (reuse is_exact_env_placeholder) to token, catalog_token, every secret.* field; keep non-secret fields (data_path, attach_alias, secret.type/region/endpoint).
• Generalize motherduck_connection + synthetic storage label → remote_duckdb_connection -> (label ∈ {motherduck,ducklake,quack}, connection).
• reconstruct.rs: generalize is_motherduck_placeholder to accept the three labels with matching scheme, round-tripping back to "no filesystem storage".

2f. Quack experimental gating

Runtime opt-in (config flag/env), not a compile feature, so the single duckdb artifact carries it; refuse a quack: sink unless explicitly opted in, with a clear "experimental" message.

Phase 3 — CI / docs / versioning

• ci.yml duckdb job: add a DuckLake local-catalog integration test (catalog = local sqlite/duckdb file in tempdir, DATA_PATH = tempdir) — self-contained once the extension autoinstalls (runners have network; consider caching ~/.duckdb/extensions alongside shared-key: v0-rust-duckdb). Add new test paths to the io_write_duckdb paths-filter and the test invocation. Quack behind a manual env-gate (FLOE_QUACK_TEST_ENDPOINT), skipped by default. Keep --test-threads=1.
• release.yml: the floe-duckdb image + off-PyPI wheel jobs already exist; just bump versions. No new distribution jobs needed.
• Versioning: minor bump to v0.7.0 across floe-core/floe-cli/floe-python Cargo.toml + crates/floe-python/pyproject.toml + pyproject.duckdb.toml (release.yml validate job enforces all manifests == tag).
• Docs: docs/sinks/duckdb.md (DuckLake + Quack config: connection: ducklake:…/quack:…, data_path, secret, ${ENV}, Quack experimental flag, autoinstall network caveat); CHANGELOG.md (DuckLake stable + Quack experimental; Arrow 58 / DuckDB 1.5.3 bump).

---

Acceptance criteria

• Phase-0 spike all green; iceberg arrow-58 confirmed; cargo tree -i arrow = single arrow 58.
• cargo build -p floe-core --features duckdb green; existing duckdb_run:: green (Local + MotherDuck regress-free); cargo clippy -p floe-core --all-targets --features duckdb -- -D warnings clean.
• New DuckLake local-catalog integration test (overwrite/append/SCD1/SCD2 round-trip) green.
• Manifest round-trip test for ducklake:/quack: synthetic storage + secret redaction (literal redacted, ${ENV} preserved).
• Quack manual env-gated test writes to a remote DuckDB and reads back.
• No secret (MotherDuck/Quack/DuckLake token, object-store creds) ever appears in logs, cache keys, or manifests.

Risks

1. iceberg arrow-58 release may still not exist (external, gating) — re-check before starting; blocks the whole release.
2. deltalake 0.30→0.32 API churn — ~6 known fixes; verify against 0.32.3.
3. Catalog-qualification regression — the 2-part-name assumption; mitigated by explicit 3-part names + tests.
4. Secret handling — construct CREATE SECRET/Quack token SQL in-process, never logged; mirror MotherDuck discipline.
5. Autoinstall network dependency for DuckLake/Quack on first ATTACH — fine for the networked image; document air-gapped fallback.
6. Quack is beta until DuckDB 2.0 — shipped experimental/opt-in.