New Drizzle Kit config API

[AlexBlokh]

Drizzle ORM is designed to be dialect specific and used with a particular range of database drivers per dialect. While it lets Drizzle move fast and rapidly add support for whatever new dialects/drivers/databases emerged on the market, it makes it harder for newcomers to onboard.

As of now we're going through a major release of Drizzle Kit and adding support for mostly all drivers available, we decided to sit and talk through how we can make it better.

as of now developers need to specify both dialect and driver in the config file. Dialect being either of pg | mysql | sqlite and this list will be joined by mariadb | cockroachdb | mssql | turso(libsql) and on the other hand we have drivers like node-pg | postgresjs | neon-serverless | aws-data-api-pg | xataio and that's only for postgres dialect.
Drizzle config is meant to be typed and driver param presence infers dbCredentials object so that you can enjoy typed params experience. While we already have a variety of different drivers, their naming brings even more confusion and you end up with something like this:

import { Config } from "drizzle-kit";

export default {
  out: "./drizzle",
  dialect: "pg",
  driver: "pg", // or 'node-pg'
} satisfies Config;

in that case driver: "pg" is a node-postgres driver which is named pg on the npm
we also have postgresjs driver which is named postgres on the npm, which will lead to:

export default {
  out: "./drizzle",
  dialect: "pg",
  driver: "postgres", // or 'postgresjs'
} satisfies Config;

and it brings confusion, cause you as a developer install npm i pg but in our config it will be called node-pg which is how the library is called, or driver will be called pg and thus another confusion of why do you have to state dialect: pg and driver: pg. Goes the same with postgresjs driver which is going to be either dialect: pg and driver: postgres or driver: postgresjs.

We're aiming move away from the model of developer passing a driver instance to Drizzle constructor to the knex-like one, to reduce the initial learning curve/complexity for developers to jump into Drizzle:

import { drizzle } from "drizzle-orm"

const db = drizzle("postgres", { 
   ... // connection params
});

Taking in count all the above we've came up with potential solution for the config file to unify driver and dialect fields into one, which will consist of dialect:driver and change pg dialect to postgresql:

import { Config } from "drizzle-kit";

export default {
  out: "./drizzle",
  database: "postgresql:neon",
} satisfies Config;

That would let us keepdbCredentials typed and will end up in a list of database params be

// for postgres databases
// either of
postgresql:node-pg 
postgresql:pg

// either of
postgresql:postgres
postgresql:postgresjs

postgresql:neon
postgresql:aws-data-api
postgresql:xata

// for mysql databases
mysql:mysql2
mysql:planetscale
mysql:tidb

// for sqlite databases
sqlite:turso
sqlite:libsql
sqlite:bun
sqlite:better-sqlite3
sqlite:expo
sqlite:react-native

Big questions:

• this will change the behaviour for people who are not using push, pull and studio commands, since those only require dialect.
• When user provides postgres:neon for example, he might be able to connect to neon database not through @neondatabase/serverless driver, but through pg or postgres drivers. Should we check runtime deps strictly for @neondatabase/serverless and fail if it's not present or also try both pg and postgres

For the first one we might let them just provide database: "dialect" without :driver part?
For the second one we lean heavily towards trying all possible drivers and connect through the first one available as long as all of them compatible

UPD(27 may 2024)

• for better-sqlite and libsql compatibility - we need to automatically prefix files url with file: for libsql if missing and make sure to trim it for better-sqlite
• for neon and vercel serverless drivers we need to check for -pooler in connection strings and automatically create a Pool instead of a Client to not throw errors
• for @neondatabase/serverless use createPool
• @vercel/postgres use createPool (waiting for Vercel to resolve -pooler issue, which prevents pool creation with a non-pooler connection)

[rphlmr]

in that case driver: "pg" is a node-postgres driver which is named pg on the npm

We should keep the name that is in the dependencies because node-postgres is something else in npm.

For the first one we might let them just provide database: "dialect" without :driver part?

So now it will be mandatory?
Is there a reason for that?
What will happen with only database: "dialect"?
Will we have to set dbCredentials too?
If yes, why should we then have to select a driver?

However, it can be hard to realize later that I need to add the driver to unlock specific config for the driver/vendor I use, so, if it is mandatory, better to be explicit.

For the second one we lean heavily towards trying all possible drivers and connect through the first one available as long as all of them compatible

There is a case I know that would instantly fail: PGLite (With dataDir constructor param, the database in the local file system, like SQLite).
It is a 'postgresql' dialect but its configuration will differ from other 'postgresql'.
So it will require its database: "postgresql:pglite" to have a proper configuration typing and its "official" driver to work.

Maintaining a list of cross-dialect compatibilities may be risky and time-consuming?

I would be strict: faster to implement, better configuration (what I set is what I use).
It should not really happen that often, right?

[AlexBlokh]

We went further and potentially managed to find a sweet spot. Our #1 goal is to cover 100% use cases while keeping most commonly used ones the simplest for the developers. pglight, aws-data-api users are edgier and potentially more experienced users as opposed to ones doing their first steps towards databases in nodejs/js world. We decided to not do database param and stick with dialect + optional driver

so lets say for postgres we won't require driver param for either you're using postgres server, postgres in docker container or neon. All you will have to do is specify dialect: postgres and we will connect to database with pg, postgres or @neondatabase/serverless whichever is currently installed in your project and dbCredentials stay fully typed and when you introduce driver: aws-data-api - we automatically infer credentials type for you

[rphlmr]

Looks simpler with this split 👍

[AlexBlokh]

perfect, thank you!

[bacharSalleh]

what if I want to use different driver on localhost. ex: production: neon , dev: pg ?

[adarshmadrecha]

Please add support for bun:sqlite

Currently (v1.3.0 or v1.3.1) gives error when running with bun

bun --bun drizzle-kit push

And running without --bun will run using node js which uses better-sqlite and we get error Bun is not defined.

bun drizzle-kit push

Tip

Hence, can we get a new option in the driver? bun-sqlite?

import { defineConfig } from 'drizzle-kit'
const migrationTableName = 'migrations'

export default defineConfig({
  schema: './src/db/schema/*.ts',
  out: './drizzle',
  dialect: 'sqlite',
  dbCredentials: {
    url: './data.db',
  },
  migrations: {
    table: migrationTableName,
  },
})

[christophkroeppl]

ran into the same issue.
Would be nice if drizzle-kit worked with bun:sqlite.

Bun version 1.3.11

Drizzle deps:
"drizzle-orm": "1.0.0-beta.16-ea816b6",
"drizzle-kit": "1.0.0-beta.16-ea816b6",

$ bun run --bun drizzle-kit push
No config path provided, using default 'drizzle.config.ts'
Reading config file 'MYPATH/drizzle.config.ts'
Using 'better-sqlite3' driver for database querying
 Error  'better-sqlite3' is not yet supported in Bun.
Track the status in https://github.com/oven-sh/bun/issues/4290
In the meantime, you could try bun:sqlite which has a similar API.
error: "drizzle-kit" exited with code 1

$ bun --bun run drizzle-kit push
No config path provided, using default 'drizzle.config.ts'
Reading config file 'MYPATH/drizzle.config.ts'
Using 'better-sqlite3' driver for database querying
 Error  'better-sqlite3' is not yet supported in Bun.
Track the status in https://github.com/oven-sh/bun/issues/4290
In the meantime, you could try bun:sqlite which has a similar API.
error: "drizzle-kit" exited with code 1

$ bunx --bun drizzle-kit push
No config path provided, using default 'drizzle.config.ts'
Reading config file 'MYPATH/drizzle.config.ts'
Using 'better-sqlite3' driver for database querying
 Error  'better-sqlite3' is not yet supported in Bun.
Track the status in https://github.com/oven-sh/bun/issues/4290
In the meantime, you could try bun:sqlite which has a similar API.

It worked once I installed node.