Skip to content

docs: attribute the DuckLake read-while-ingest lock to the right backend (#71) - #195

Draft
db-tycoon-stephen wants to merge 1 commit into
v0.1.12from
docs/analyze-lock-rationale
Draft

docs: attribute the DuckLake read-while-ingest lock to the right backend (#71)#195
db-tycoon-stephen wants to merge 1 commit into
v0.1.12from
docs/analyze-lock-rationale

Conversation

@db-tycoon-stephen

@db-tycoon-stephen db-tycoon-stephen commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Fixes #71 (PTC-42).

What

docs/commands/data/analyze.md justified the Rill Parquet bridge with:

SQLite-backed DuckLake catalogs hold an exclusive lock that breaks Rill-while-ingesting.

That names the wrong backend. The lock belongs to the catalog's metadata database, not to DuckLake and not to SQLite:

  • DuckDB-backed catalog (ducklake:foo.ducklake, the default) — a .ducklake file is a DuckDB database, and DuckDB locks a database file to one process. Two processes → Conflicting lock.
  • SQLite-backed catalog (ducklake:sqlite:foo.sqlite) — multi-process safe. Rill 0.86 was verified reading one live during ingest with zero lock conflicts.

The v0.1.3 probe that produced the original claim attached ducklake:.../catalog.ducklake — no sqlite: prefix — so it hit the DuckDB-file backend while being labelled SQLite.

The Parquet bridge is still a sound default, just for different reasons: it needs no DuckLake extension, no catalog attach, and no live warehouse connection, so --rill output works anywhere Rill does. The pages now say that instead.

Scope: #71 names one line; there were two live pages

File Change
docs/commands/data/analyze.md Correct rationale + dated correction note
docs/recipes/motherduck-cloud-sync.md Same claim, same fix — not named in #71, found by grepping the assertion rather than the line number
docs/releases/v0.1.3.md The "full rationale" analyze.md links to — fixing only analyze.md would send readers one hop to the same error
CHANGELOG.md New [Unreleased] → Fixed entry

Two judgment calls worth a reviewer's attention

Release notes got an appended correction, not a rewrite. A shipped release note records what was believed at the time; silently editing it erases that the project once held a wrong belief and deferred real work over it. The note is dated and marked "left in place for the record." Same reasoning for the 0.1.3 CHANGELOG entry, which keeps its original wording — [Unreleased] is where the correction belongs.

No link to the DuckLake proposal. The natural destination for the full evidence is docs/proposals/ducklake-layered-storage.md, but that file only exists on #74's branch and mkdocs --strict would fail on the broken link. These pages link to #71 and #74 on GitHub instead. Worth switching to the in-repo path once #74 lands.

Left alone deliberately

docs/proposals/v0.1.6-and-v0.1.7-open-questions.md repeats the belief in four places — including one that has it backwards ("DuckDB-backed catalog, which may avoid the SQLite lock issue"). Archived proposal, not in nav, not user-facing. Happy to add a correction note if reviewers want it.

Verification

  • mkdocs build --strict — clean
  • ruff check / ruff format --check (pinned 0.15.20) — pass, 103 files formatted
  • pytest706 passed, 3 skipped. tests/test_recipe_doctests.py parses docs/recipes/, so the edited recipe is covered.

Note on issue closing

Fixes #71 won't auto-close on merge — GitHub only auto-closes for the default branch, and this targets v0.1.12 per CONTRIBUTING.md. #71 and PTC-42 need closing by hand when this ships, or when the release branch reaches main.

Evidence

Spike investigation of 2026-06-21 (DuckDB 1.5.3, ducklake e6a3bd0a, real Rill 0.86, dbt-duckdb 1.10.1). Full design and results in #74.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

…end (#71)

analyze.md justified the Rill Parquet bridge by claiming SQLite-backed
DuckLake catalogs hold an exclusive lock. Wrong backend. The lock belongs
to the catalog's metadata database: a DuckDB-backed catalog
(`ducklake:foo.ducklake`, the default) is a DuckDB file, and DuckDB locks
a database file to one process. A SQLite-backed catalog
(`ducklake:sqlite:foo.sqlite`) is multi-process safe — Rill 0.86 was
verified reading one live during ingest with zero lock conflicts.

The v0.1.3 probe that produced the original claim attached
`ducklake:.../catalog.ducklake` — no `sqlite:` prefix — so it hit the
DuckDB-file backend while being labelled SQLite.

- analyze.md: replace the rationale with the bridge's actual merits
  (no extension, no attach, no live connection) plus a dated correction.
- motherduck-cloud-sync.md: same claim, same fix. Not named in #71.
- releases/v0.1.3.md: the "full rationale" analyze.md links to. Appended
  a dated correction instead of rewriting a shipped release note.

CHANGELOG entry under a new [Unreleased]. The 0.1.3 CHANGELOG entry keeps
its original wording — released entries are a record, and [Unreleased]
now documents the correction.

Refs #71

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gemini-code-assist

Copy link
Copy Markdown

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant