docs: propose native DuckLake layered storage (SQLite-catalog)

[db-tycoon-stephen]

What

Adds docs/proposals/ducklake-layered-storage.md — a storage-engine proposal for giving each warehouse layer its own DuckLake catalog (SQLite metadata + Parquet, S3-able), composed via ATTACH; dbt reads raw read-only and materializes into mart.

Docs-only, one new file.

Why

Came out of a 4-spike investigation (all green; DuckDB 1.5.3, ducklake e6a3bd0a, real Rill 0.86, dbt-duckdb 1.10.1). Headline finding corrects shipped docs (docs/commands/data/analyze.md:68): the read-while-ingest lock that justified the Parquet bridge for Rill is a property of the catalog metadata backend (DuckDB-file locks per-process), not of DuckLake or SQLite. A SQLite-backed catalog supports live Rill reads during ingest — confirmed with the real Rill binary, 0 lock conflicts.

Validated

• Quack and DuckLake do not compose (fronting DuckLake with Quack ruled out).
• DuckDB-file catalog backend locks; SQLite is multi-process safe.
• Real Rill 0.86 reads a ducklake:sqlite: catalog live during ingest, 0 conflicts.
• dbt-duckdb materializes raw-catalog → mart-catalog via profiles.yml attach:.

Where this lands now

Drafted 2026-06-21, when v0.1.10 was still the security pass and DuckLake had no owner in the plan. Both have moved:

• v0.1.10 shipped 2026-07-21.
• The ingestion rewrite's M6 ([Milestone 6] Alternative destinations — MotherDuck, DuckLake, object storage #87) now owns the DuckLake surface — DuckLakeDestination, ObjectStorageDestination, tycoon attach ducklake.

So this is no longer a standalone track competing for a slot. It's the design evidence feeding M6: the corrected lock model, the validated dbt profile shape, and the four spike results are what M6 would otherwise have to rediscover. The "Proposed phasing" section predates the milestone and its sequencing is M6's to set — the doc now says so.

Merging also fixes three dangling references. PTC-41, PTC-42, and PTC-43 all point readers at docs/proposals/ducklake-layered-storage.md for the design and evidence, and that file exists only on this branch.

Freshness re-check (2026-08-02)

Rebased onto v0.1.12, the active release branch (CONTRIBUTING.md: feature and fix PRs target the release branch, not main). This PR previously targeted main; the rebase used --onto so the four PTC-96 commits that are on main but not on v0.1.12 don't ride along. Every code reference in "How it maps to the current code" was verified against both branches — none had rotted:

Reference	Status on main and v0.1.12
ingestion/ducklake_config.py misnamed	Still true — untouched since v0.1.0, still a dlt filesystem/Parquet destination
DatabaseConfig has two fixed fields	Still true — moved (:215 on main, :212 on v0.1.12), so the doc now cites the bare path
observability_dbt.attach_metadata_to_profiles	Still there, unchanged
analyze.md:68 carries the wrong lock rationale	Still uncorrected (#71 open)

mkdocs build --strict and ruff==0.15.20 (the version v0.1.12 pins in its dev group) both pass locally. The new page isn't in mkdocs.yml's not_in_nav list, which is fine — ingestion-architecture.md and proposals/v0.1.10-scope.md aren't either, and unlisted pages are INFO-level, not a strict failure.

Relationship

• Builds on the shipped layer-aware data model (Layer-aware data model: surface sources / staging / intermediate / marts as first-class #30) — logical layers; this adds an optional physical backing. Respects "no layers: block."
• Feeds [Milestone 6] Alternative destinations — MotherDuck, DuckLake, object storage #87 (rewrite M6, the DuckLake destination) and unblocks v0.1.8+: layer-granular backup config (backup.layers replaces skip_raw) #31 (layer-granular backup).
• Tracks: Native DuckLake layered storage (SQLite-catalog): optional physical per-layer catalogs #73 / PTC-41 (this design), Docs: analyze.md:68 mislabels the DuckLake lock as SQLite-backed (it's the DuckDB-file backend) #71 / PTC-42 (analyze.md fix), ducklake_config.py claims DuckLake but only dumps plain Parquet (misnamed) #72 / PTC-43 (ducklake_config.py is misnamed).

Open risk

data_path: s3://… is unproven (all spikes used local paths) — gated as Phase 1, and now a question for M6 rather than for this doc.

Review status

Three rounds from Gemini Code Assist, all five threads resolved (dbt attach field validation, undefined link, READ_ONLY TRUE, raw data_path, missing schema). No further bot review will arrive — Gemini was opted out repo-wide after this PR opened (#173).

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces a proposal for native DuckLake layered storage using a SQLite-catalog to enable a local-first lakehouse on object storage. The review feedback highlights a validation issue with the proposed dbt-duckdb profile configuration, which does not support custom fields like is_ducklake or options out-of-the-box, and identifies a missing markdown reference link for [no-legacy stance].

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[db-tycoon-stephen]

Review triage — commit ae56ca6

Addressed the Gemini Code Assist review:

• ✅ Undefined [no-legacy stance] markdown link → replaced with plain text.
• ↩️ "dbt-duckdb attach rejects is_ducklake/options" → accurate for older dbt-duckdb, but not for 1.10.1 (the version this proposal targets): the Attachment model declares options/is_ducklake as first-class fields, and spike 4 ran a real dbt run with this exact profile (materialized into the mart DuckLake catalog, 3 snapshots). Rather than dismiss it, added an explicit dbt-duckdb ≥ 1.10 version-floor note with the plugin fallback for older versions.

For reviewers: this PR is docs-only (a proposal) and not part of v0.1.10 (the security pass). Tracked as #73 / PTC-41. Open risk noted in the doc: data_path: s3://… is unproven (Phase 1).

[db-tycoon-stephen]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a proposal for a native DuckLake layered storage architecture using a SQLite-backed catalog to enable multi-process, local-first lakehouses. The feedback identifies two key improvements in the proposal's examples: ensuring the SQL ATTACH statement uses READ_ONLY TRUE for better DuckDB version compatibility, and adding the missing data_path option to the raw attachment in the dbt profile configuration.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[db-tycoon-stephen]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds a new proposal document (docs/proposals/ducklake-layered-storage.md) outlining a native DuckLake layered storage architecture using a SQLite-catalog. The feedback points out that the dbt profile YAML example in the proposal is missing the required schema field, which would cause dbt validation to fail, and suggests adding a default schema to make the example valid.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.