Refocus README on docs.getdbt.com for setup and configuration#722
Closed
dataders wants to merge 3 commits into
Closed
Refocus README on docs.getdbt.com for setup and configuration#722dataders wants to merge 3 commits into
dataders wants to merge 3 commits into
Conversation
dataders
commented
Apr 21, 2026
Comment on lines
3
to
14
| [DuckDB](http://duckdb.org) is an embedded database, similar to SQLite, but designed for OLAP-style analytics. It is fast and allows you to read and write data stored in CSV, JSON, and Parquet files directly, without requiring you to load them into the database first. | ||
|
|
||
| [dbt](http://getdbt.com) is the best way to manage a collection of data transformations written in SQL or Python for analytics | ||
| and data science. `dbt-duckdb` is the project that ties DuckDB and dbt together, allowing you to create a [Modern Data Stack In | ||
| A Box](https://duckdb.org/2022/10/12/modern-data-stack-in-a-box.html) or a simple and powerful data lakehouse with Python. | ||
| [dbt](http://getdbt.com) is a framework for managing data transformations written in SQL or Python for analytics and data science. `dbt-duckdb` is the adapter that ties DuckDB and dbt together, allowing you to build a [Modern Data Stack In A Box](https://duckdb.org/2022/10/12/modern-data-stack-in-a-box.html) or a local lakehouse with Python. | ||
|
|
||
| ### Installation | ||
|
|
||
| This project is hosted on PyPI, so you should be able to install it and the necessary dependencies via: | ||
|
|
||
| `pip3 install dbt-duckdb` | ||
|
|
||
| The latest supported version targets `dbt-core` versions >= 1.8.x and `duckdb` version 1.1.x, but we work hard to ensure that newer | ||
| versions of DuckDB will continue to work with the adapter as they are released. | ||
| The latest supported version targets `dbt-core` versions >= 1.8.x and DuckDB version 1.1.x, but we work hard to ensure that newer versions of DuckDB continue to work with the adapter as they are released. | ||
|
|
Contributor
Author
There was a problem hiding this comment.
no reason to have touched these sections. editorial overreach!
Comment on lines
+22
to
+24
| - [Materializations](https://docs.getdbt.com/docs/build/materializations) | ||
| - [Incremental models](https://docs.getdbt.com/docs/build/incremental-models) | ||
| - [Python models](https://docs.getdbt.com/docs/build/python-models) |
Contributor
Author
There was a problem hiding this comment.
drop these references
Comment on lines
+19
to
+20
| - [DuckDB setup](https://docs.getdbt.com/docs/local/connect-data-platform/duckdb-setup) | ||
| - [DuckDB configurations](https://docs.getdbt.com/reference/resource-configs/duckdb-configs) |
Contributor
Author
There was a problem hiding this comment.
Suggested change
| - [DuckDB setup](https://docs.getdbt.com/docs/local/connect-data-platform/duckdb-setup) | |
| - [DuckDB configurations](https://docs.getdbt.com/reference/resource-configs/duckdb-configs) | |
| - [connecting to DuckDB with dbt](https://docs.getdbt.com/docs/local/connect-data-platform/duckdb-setup) | |
| - [configuring DuckDB with dbt](https://docs.getdbt.com/reference/resource-configs/duckdb-configs) |
Comment on lines
-32
to
-37
| This will run your dbt-duckdb pipeline against an in-memory DuckDB database that will not be persisted after your run completes. This may | ||
| not seem very useful at first, but it turns out to be a powerful tool for a) testing out data pipelines, either locally or in CI jobs and | ||
| b) running data pipelines that operate purely on external CSV, Parquet, or JSON files. More details on how to work with external data files | ||
| in dbt-duckdb are provided in the docs on [reading and writing external files](#reading-and-writing-external-files). | ||
|
|
||
| To have your dbt pipeline persist relations in a DuckDB file, set the `path` field in your profile to the path |
Contributor
Author
There was a problem hiding this comment.
are we confident that this is represented already on the recently updated docs pages?
| MotherDuck databases generally work the same way as local DuckDB databases from the perspective of dbt, but | ||
| there are a [few differences to be aware of](https://motherduck.com/docs/architecture-and-capabilities#considerations-and-limitations): | ||
| 1. MotherDuck is compatible with client DuckDB versions 0.10.2 and older. | ||
| 1. MotherDuck preloads a set of the most common DuckDB extensions for you, but does not support loading custom extensions or user-defined functions. |
Contributor
Author
There was a problem hiding this comment.
is this mentioned on our docs pages?
Comment on lines
-558
to
-595
| models: | ||
| - name: my_incremental_model | ||
| config: | ||
| materialized: incremental | ||
| incremental_strategy: merge | ||
| unique_key: id # or ['id', 'date'] for composite keys | ||
| ``` | ||
|
|
||
| This generates SQL equivalent to: | ||
|
|
||
| ```sql | ||
| MERGE INTO target AS DBT_INTERNAL_DEST | ||
| USING source AS DBT_INTERNAL_SOURCE | ||
| ON (DBT_INTERNAL_SOURCE.id = DBT_INTERNAL_DEST.id) | ||
| WHEN MATCHED THEN UPDATE BY NAME | ||
| WHEN NOT MATCHED THEN INSERT BY NAME | ||
| ``` | ||
| The `module_paths` profile setting lets you specify a list of filesystem paths containing additional Python modules. These paths are added to the dbt process's `sys.path`, which makes the modules importable within dbt. You can use this to include helper code in your project, such as custom `dbt-duckdb` plugins or shared libraries for Python models. | ||
|
|
||
| **Enhanced Configuration:** | ||
| ### Writing your own plugins | ||
|
|
||
| These options extend the basic merge behavior with additional control over which records get updated or inserted, which columns are affected, and how values are set. | ||
|
|
||
| | Configuration | Type | Default | Description | ||
| | :---: | :---: | :---: | --- | ||
| | `unique_key` | string/list | required | Column(s) used for the MERGE join condition | ||
| | `incremental_predicates` | list | null | Additional SQL conditions to filter the MERGE operation | ||
| | `merge_on_using_columns` | list | null | Columns for USING clause syntax instead of ON for the join condition | ||
| | `merge_update_condition` | string | null | SQL condition to control when matched records are updated | ||
| | `merge_insert_condition` | string | null | SQL condition to control when unmatched records are inserted | ||
| | `merge_update_columns` | list | null | Specific columns to update | ||
| | `merge_exclude_columns` | list | null | Columns to exclude from updates | ||
| | `merge_update_set_expressions` | dict | null | Custom expressions for column updates | ||
| | `merge_returning_columns` | list | null | Columns to return from the MERGE operation | ||
|
|
||
| **Example with Enhanced Options:** | ||
|
|
||
| ```yaml | ||
| models: |
Contributor
Author
There was a problem hiding this comment.
i think this interactive shell stuff stays in the repo
Comment on lines
+91
to
+92
| - Support for Delta and Iceberg external table formats, both as sources and destinations | ||
| - Make dbt incremental models and snapshots work with external materializations |
Contributor
Author
There was a problem hiding this comment.
editorial overreach!
Contributor
Author
|
Addressed review in Changes:
|
Contributor
Author
|
Superseded by #723. Reopened from a fresh branch because GitHub kept showing a stale conflict state on this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This trims the README down so it primarily points readers to the maintained dbt documentation for setup, configuration, and general adapter usage.
It keeps the repository-specific material that still makes sense to live close to the source, especially:
dbt-duckdbpluginsmodule_pathsWhy
The README had grown into a second, partially duplicated copy of the adapter docs. That makes it easy for the README and the dbt docs site to drift.
The dbt docs updates that this PR is designed to point to are being proposed here:
Once those land, the docs site should be the primary place for:
Changes
README.mdwith links to docs.getdbt.compyicebergto Python >= 3.10