diff --git a/CHANGELOG.md b/CHANGELOG.md index 8ec3460..e4a2b2b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,13 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +### Fixed + +- **Docs: the DuckLake read-while-ingest lock was attributed to the wrong backend** ([#71][]). `docs/commands/data/analyze.md` and `docs/recipes/motherduck-cloud-sync.md` both said SQLite-backed DuckLake catalogs hold an exclusive lock that breaks Rill-while-ingesting. The lock belongs to the catalog's metadata database, not to DuckLake or SQLite: a DuckDB-backed catalog (the default) is a DuckDB file and locks per-process, while a SQLite-backed catalog is multi-process safe — Rill 0.86 reads one live during ingest with zero conflicts. Both pages corrected; the v0.1.3 release note that originated the claim carries a dated correction rather than a rewrite. + +[#71]: https://github.com/Database-Tycoon/tycoon-cli/issues/71 + ## [0.1.10] - 2026-07-21 _Headline: the codebase gets dramatically smaller and more focused. The ingestion rewrite's M1 metadata backend lands ([#82][], PRs [#136][]/[#137][]), and the FastAPI server, Dagster orchestration, and Nao/ask AI-agent extras are removed outright ([#139][], PR [#147][]) — the lockfile drops from 174 to 109 packages. Cycle plan in [`docs/proposals/v0.1.10-scope.md`](docs/proposals/v0.1.10-scope.md)._ diff --git a/docs/commands/data/analyze.md b/docs/commands/data/analyze.md index fb092e5..850cf8c 100644 --- a/docs/commands/data/analyze.md +++ b/docs/commands/data/analyze.md @@ -65,7 +65,9 @@ rill/metrics/stg_github__issues_mv.yaml # type: metrics_view, model: stg_githu rill/dashboards/stg_github__issues.yaml # type: explore, metrics_view: stg_github__issues_mv ``` -The Parquet bridge is intentional — see [Rill 0.86 architecture in the v0.1.3 release notes](../../releases/v0.1.3.md) for the full rationale (TL;DR: SQLite-backed DuckLake catalogs hold an exclusive lock that breaks Rill-while-ingesting). +The Parquet bridge is intentional — it needs no DuckLake extension, no catalog attach, and no live warehouse connection, so `--rill` output works anywhere Rill does. See [Rill 0.86 architecture in the v0.1.3 release notes](../../releases/v0.1.3.md) for how it was chosen. + +> **Correction (2026-08-02).** This note previously said the bridge existed because "SQLite-backed DuckLake catalogs hold an exclusive lock that breaks Rill-while-ingesting." That named the wrong backend. The lock belongs to the catalog's **metadata database**, not to DuckLake or to SQLite: a DuckDB-backed catalog (`ducklake:foo.ducklake`, the default) is a DuckDB file, and DuckDB locks a database file to one process. A **SQLite-backed** catalog (`ducklake:sqlite:foo.sqlite`) is multi-process safe — Rill 0.86 was verified reading one live during ingest with zero lock conflicts. The bridge remains a reasonable default for the reasons above, but a DuckLake live connector is not blocked by locking. Evidence and full design in [issue #71](https://github.com/Database-Tycoon/tycoon-cli/issues/71) and [PR #74](https://github.com/Database-Tycoon/tycoon-cli/pull/74). The auto-generated metrics view picks dimensions and measures heuristically: diff --git a/docs/recipes/motherduck-cloud-sync.md b/docs/recipes/motherduck-cloud-sync.md index 3a80998..354b6b6 100644 --- a/docs/recipes/motherduck-cloud-sync.md +++ b/docs/recipes/motherduck-cloud-sync.md @@ -143,7 +143,7 @@ tycoon data sync --from md:other_catalog --to ./snap2.duckdb # one-off sync to - **Pull-only**: there's no `tycoon data sync --reverse` for local → cloud. Deliberate, prevents accidental prod mutation. Use dlt or your existing prod pipeline for that direction. - **Full replace per table**: no incremental sync in v1. Re-sync re-copies every matched table. - **`md:` and local DuckDB only** for sources today. Snowflake / BigQuery / Postgres come later. -- **DuckLake catalogs**: only the `READ_ONLY` ATTACH path is verified. SQLite-backed DuckLake catalogs hold an exclusive lock that conflicts with concurrent Rill — see [Rill 0.86 architecture in the v0.1.3 release notes](../releases/v0.1.3.md). +- **DuckLake catalogs**: only the `READ_ONLY` ATTACH path is verified. Note that a **DuckDB-backed** catalog (`ducklake:foo.ducklake`, the default) is locked to one process at a time, so a concurrent reader such as Rill will conflict; a **SQLite-backed** catalog (`ducklake:sqlite:foo.sqlite`) is multi-process safe and does not. This corrects an earlier note here that attributed the lock to SQLite — see [issue #71](https://github.com/Database-Tycoon/tycoon-cli/issues/71). ## Related diff --git a/docs/releases/v0.1.3.md b/docs/releases/v0.1.3.md index 8702a2a..c294a8c 100644 --- a/docs/releases/v0.1.3.md +++ b/docs/releases/v0.1.3.md @@ -76,6 +76,17 @@ See the detailed "Planned scope" sections below. Quick index: candidate list pending either Rill shared-lock support or a Postgres-catalog story. + > **Correction (2026-08-02), left in place for the record.** + > The deferral above blamed the wrong backend. The v0.1.3 probe + > attached `ducklake:.../catalog.ducklake` — no `sqlite:` prefix, + > so a **DuckDB-backed** catalog, which is a DuckDB file and is + > therefore locked to one process. SQLite was never the cause; + > a `ducklake:sqlite:` catalog is multi-process safe, and Rill + > 0.86 was later verified reading one live during ingest with + > zero lock conflicts. The Parquet bridge is still a sound + > default, but not for the reason given here. See + > [issue #71](https://github.com/Database-Tycoon/tycoon-cli/issues/71). + ### Fixed - **`generate_rill_config` no longer accepts an unused