diff --git a/README.md b/README.md index dd6fc865..0178e512 100644 --- a/README.md +++ b/README.md @@ -1,31 +1,46 @@ # pglite-oxide [![CI](https://github.com/f0rr0/pglite-oxide/actions/workflows/ci.yml/badge.svg)](https://github.com/f0rr0/pglite-oxide/actions/workflows/ci.yml) +[![crates.io](https://img.shields.io/crates/v/pglite-oxide.svg)](https://crates.io/crates/pglite-oxide) +[![docs.rs](https://docs.rs/pglite-oxide/badge.svg)](https://docs.rs/pglite-oxide) `pglite-oxide` embeds the [Electric SQL PGlite](https://github.com/electric-sql/pglite) -WASI PostgreSQL runtime in a Rust library. It installs the bundled runtime, starts -Postgres inside Wasmtime, and exposes a small synchronous API for executing SQL -without a separate database server. It can also expose the embedded backend over -a local PostgreSQL socket for Rust clients such as SQLx and `tokio-postgres`. +WASI PostgreSQL runtime in Rust. It gives Rust apps a local Postgres-compatible +database without shipping a native Postgres sidecar. + +Use it when you want: + +- local Postgres semantics in a Rust or Tauri app +- fast Postgres-backed tests without Docker or testcontainers +- a PostgreSQL connection URI for crates such as SQLx or `tokio-postgres` +- a small, embedded database boundary that stays on the Rust side of the app The crate currently targets PostgreSQL 17.x PGlite builds, Rust 1.92+, and Wasmtime 44. -## Quick Start +## Install + +```sh +cargo add pglite-oxide serde_json +``` + +The default `runtime-cache` feature enables Wasmtime's persistent compiled +module cache. Disable default features only if your app cannot write to the +global Wasmtime cache. + +## Direct Embedded API + +Use `Pglite` when your Rust code owns the database calls. ```rust,no_run use pglite_oxide::Pglite; use serde_json::json; -fn main() -> anyhow::Result<()> { - let mut db = Pglite::builder().path("./.pglite").open()?; +fn main() -> Result<(), Box> { + let mut db = Pglite::open("./.pglite")?; db.exec("CREATE TABLE IF NOT EXISTS items(value TEXT)", None)?; - db.query( - "INSERT INTO items(value) VALUES ($1)", - &[json!("alpha")], - None, - )?; + db.query("INSERT INTO items(value) VALUES ($1)", &[json!("alpha")], None)?; let result = db.query("SELECT value FROM items", &[], None)?; println!("{:?}", result.rows); @@ -35,21 +50,27 @@ fn main() -> anyhow::Result<()> { } ``` -Use `Pglite::temporary()?` for an ephemeral database in tests; it clones a -process-local template cluster so repeated tests do not rerun `initdb`. +For tests, use `Pglite::temporary()?`. Temporary databases clone a process-local +template cluster, so repeated tests avoid fresh `initdb` work. -## PostgreSQL Client Compatibility +## PostgreSQL Client URI -Use `PgliteServer` when a library expects a PostgreSQL connection string. The -server owns one embedded backend, so configure downstream pools with a single -connection. +Use `PgliteServer` when an existing library expects a PostgreSQL URL. Configure +client pools with one connection because the embedded runtime owns one backend. + +For SQLx: + +```sh +cargo add sqlx --features postgres,runtime-tokio +cargo add tokio --features macros,rt-multi-thread +``` ```rust,no_run use pglite_oxide::PgliteServer; use sqlx::{Connection, Row}; #[tokio::main] -async fn main() -> anyhow::Result<()> { +async fn main() -> Result<(), Box> { let server = PgliteServer::temporary_tcp()?; let mut conn = sqlx::PgConnection::connect(&server.connection_uri()).await?; @@ -66,169 +87,23 @@ async fn main() -> anyhow::Result<()> { ``` For app persistence, use `PgliteServer::builder().path("./.pglite").start()?`. -For Rust code that does not require a connection URI, prefer the direct -`Pglite` API because it avoids the socket/protocol compatibility layer. -For desktop app shape and state management notes, see -[`docs/TAURI.md`](docs/TAURI.md). - -## Runtime API - -`Pglite` is the main entry point. - -- `Pglite::builder()` configures persistent, app-data, or temporary databases. -- `Pglite::open(path)` opens a persistent database rooted at `path`. -- `Pglite::temporary()` opens a cached ephemeral database for tests. -- `exec(sql, options)` runs simple SQL and returns zero or more result sets. -- `query(sql, params, options)` uses the extended protocol with JSON parameters. -- `describe_query(sql, options)` returns parameter and row metadata. -- `transaction(|tx| ...)` runs `BEGIN`/`COMMIT` with rollback on error. -- `listen`, `unlisten`, and `on_notification` support PostgreSQL notifications. -- `close()` shuts down the embedded backend. -- `PgliteServer` exposes a local PostgreSQL socket for existing client crates. - -Values are passed as `serde_json::Value`. Default parsers and serializers cover -common Postgres types including integers, floats, booleans, JSON/JSONB, bytea, -dates/timestamps, UUIDs, and arrays discovered from `pg_type`. - -## Query Options - -`QueryOptions` controls result parsing and protocol behavior: - -```rust,no_run -use pglite_oxide::{Pglite, QueryOptions, RowMode}; - -fn main() -> anyhow::Result<()> { - let mut db = Pglite::open("./.pglite")?; - - let options = QueryOptions { - row_mode: Some(RowMode::Array), - ..QueryOptions::default() - }; - - let _rows = db.query("SELECT 1, 2", &[], Some(&options))?; - - Ok(()) -} -``` - -For `COPY ... FROM '/dev/blob'`, set `QueryOptions::blob` to the bytes to expose -through the guest `/dev/blob`. For `COPY ... TO '/dev/blob'`, read the returned -`Results::blob`. - -## SQL Templating Helpers - -```rust,no_run -use pglite_oxide::{Pglite, QueryTemplate, format_query, quote_identifier}; -use serde_json::json; - -fn main() -> anyhow::Result<()> { - let mut db = Pglite::open("./.pglite")?; - - let sql = format_query(&mut db, "SELECT $1::int", &[json!(42)])?; - assert_eq!(sql, "SELECT '42'::int"); - - let mut template = QueryTemplate::new(); - template.push_sql("SELECT * FROM "); - template.push_identifier("items"); - template.push_sql(" WHERE value = "); - template.push_param(json!("alpha")); - let built = template.build(); - - assert_eq!(built.query, "SELECT * FROM \"items\" WHERE value = $1"); - assert_eq!(quote_identifier("a\"b"), "\"a\"\"b\""); - - Ok(()) -} -``` - -## Runtime Notes - -The embedded backend uses the same shared-memory CMA protocol as upstream -PGlite. The host preopens: - -- `/tmp` as the runtime root -- `/tmp/pglite/base` as the Postgres data directory -- `/home` for runtime home files -- `/dev` for small device shims such as `urandom` - -The first instance in a process can take a while because Wasmtime compiles the -large PGlite WASM module and the first temporary cluster runs `initdb`. Compiled -modules are cached inside the process so additional `Pglite` instances avoid the -same compile cost. `Pglite::temporary()` also clones a process-local template -cluster, so later temporary databases in the same test process only copy the -prepared filesystem. Use `Pglite::builder().fresh_temporary().open()?` when a -test needs to exercise fresh cluster initialization. - -Opening an existing cluster still invokes PGlite's `initdb` export because the -WASM runtime uses that entry point for in-memory backend setup too. Existing data -is preserved; the full cluster creation work is avoided once `PG_VERSION` exists. - -The default `runtime-cache` feature also enables Wasmtime's persistent compiled -module cache, so later processes can reuse native code for the same PGlite WASM -module. Disable it with `default-features = false` if you need to avoid global -cache writes. - -For fast local test loops in a downstream workspace, add the same profile -override used by this repository. Wasmtime's debug cache otherwise keys entries -by the rebuilt test binary mtime, which defeats reuse after ordinary edits: - -```toml -[profile.dev.package.wasmtime-internal-cache] -debug-assertions = false -``` - -For larger downstream suites, prefer reusing one `Pglite` instance per test when -isolation allows it, and use `fresh_temporary` only for initialization-specific -coverage. - -`PgliteServer` is deliberately blocking and handles one frontend connection at a -time against a single embedded backend. It refuses SSL/GSS negotiation requests -with the standard PostgreSQL `N` response; connection URIs generated by the -crate include `sslmode=disable`. - -```sh -cargo run --bin pglite-proxy -- --root ./.pglite --tcp 127.0.0.1:5432 -psql 'postgresql://postgres@127.0.0.1:5432/template1?sslmode=disable' -``` - -On Unix systems, the default proxy mode is `/tmp/.s.PGSQL.5432`: - -```sh -cargo run --bin pglite-proxy -PGPASSWORD=postgres psql 'postgresql://postgres@/template1?host=/tmp' -``` - -Runtime asset provenance is tracked in `docs/ASSETS.md`. -Release process details are tracked in `docs/RELEASE.md`. - -## Development - -The required local gates are: - -```sh -cargo fmt --all --check -cargo check --all-targets -cargo check --no-default-features --all-targets -cargo clippy --all-targets -- -D warnings -cargo deny check -cargo test --doc -cargo test --test runtime_smoke -- --nocapture -cargo test --test proxy_smoke -- --nocapture -cargo test --test client_compat -- --nocapture -cargo package --allow-dirty -``` -Install the supply-chain gate with `cargo install cargo-deny --locked` if it is -not already available. +## Current Shape -`tests/runtime_smoke.rs` starts the real WASM backend and is intentionally slower -than the protocol unit tests. +`pglite-oxide` is a Wasmtime/WASI embedding of PGlite, not native `libpglite` +bindings. The first process start can spend time compiling the large WASM +module; later starts reuse Wasmtime cache state when the default feature is +enabled. -## Utilities +Prefer the direct `Pglite` API when you do not need a PostgreSQL connection +string. Use `PgliteServer` for compatibility with existing Postgres client +crates. -Two maintenance binaries are included: +## Docs -- `pglite-dump` expands the bundled filesystem manifest/runtime assets. -- `pglite-manifest-sync` syncs `assets/pglite_fs_manifest.json` from the - `pglite.js` bundle published on `electric-sql/pglite-build` `gh-pages`. -- `pglite-proxy` exposes a local PostgreSQL socket backed by the embedded runtime. +- [Usage guide](https://github.com/f0rr0/pglite-oxide/blob/main/docs/USAGE.md) +- [Runtime and performance notes](https://github.com/f0rr0/pglite-oxide/blob/main/docs/RUNTIME.md) +- [Tauri usage](https://github.com/f0rr0/pglite-oxide/blob/main/docs/TAURI.md) +- [Development guide](https://github.com/f0rr0/pglite-oxide/blob/main/docs/DEVELOPMENT.md) +- [Runtime asset provenance](https://github.com/f0rr0/pglite-oxide/blob/main/docs/ASSETS.md) +- [Release process](https://github.com/f0rr0/pglite-oxide/blob/main/docs/RELEASE.md) diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md new file mode 100644 index 00000000..7bf761bf --- /dev/null +++ b/docs/DEVELOPMENT.md @@ -0,0 +1,37 @@ +# Development + +Run the local gates before opening a PR: + +```sh +cargo fmt --all --check +cargo check --all-targets +cargo check --no-default-features --all-targets +cargo clippy --all-targets -- -D warnings +cargo deny check +cargo test --doc +cargo test --test runtime_smoke -- --nocapture +cargo test --test proxy_smoke -- --nocapture +cargo test --test client_compat -- --nocapture +cargo package --allow-dirty +``` + +Install the supply-chain gate when needed: + +```sh +cargo install cargo-deny --locked +``` + +`tests/runtime_smoke.rs` starts the real WASM backend and is intentionally +slower than the protocol unit tests. + +## Maintenance Utilities + +The repository includes maintenance binaries: + +- `pglite-dump` expands the bundled filesystem manifest/runtime assets. +- `pglite-manifest-sync` syncs `assets/pglite_fs_manifest.json` from the + `pglite.js` bundle published on `electric-sql/pglite-build` `gh-pages`. +- `pglite-proxy` exposes a local PostgreSQL socket backed by the embedded + runtime. + +Release process details are tracked in [RELEASE.md](RELEASE.md). diff --git a/docs/RUNTIME.md b/docs/RUNTIME.md new file mode 100644 index 00000000..1efc2e62 --- /dev/null +++ b/docs/RUNTIME.md @@ -0,0 +1,87 @@ +# Runtime and Performance Notes + +`pglite-oxide` runs the upstream PGlite WASI runtime inside Wasmtime. It does +not link a native Postgres library. + +## WASI Layout + +The embedded backend uses the same shared-memory CMA protocol as upstream +PGlite. The host preopens: + +- `/tmp` as the runtime root +- `/tmp/pglite/base` as the Postgres data directory +- `/home` for runtime home files +- `/dev` for small device shims such as `urandom` + +Opening an existing cluster still invokes PGlite's `initdb` export because the +WASM runtime uses that entry point for in-memory backend setup too. Existing data +is preserved; the full cluster creation work is avoided once `PG_VERSION` +exists. + +## Startup Cost + +The first instance in a process can take a while because Wasmtime compiles the +large PGlite WASM module and the first temporary cluster runs `initdb`. + +`pglite-oxide` reduces repeat startup cost in two ways: + +- compiled modules are cached in process, so additional `Pglite` instances avoid + the same compile work +- `Pglite::temporary()` clones a process-local template cluster, so later + temporary databases copy a prepared filesystem instead of rerunning `initdb` + +Use `Pglite::builder().fresh_temporary().open()?` only when a test specifically +needs fresh cluster initialization. + +## Persistent Compile Cache + +The default `runtime-cache` feature enables Wasmtime's persistent compiled module +cache, so later processes can reuse native code for the same PGlite WASM module. + +Disable it if your app cannot write to the global cache location: + +```toml +pglite-oxide = { version = "0.2", default-features = false } +``` + +For fast local test loops in a downstream workspace, add the same profile +override used by this repository. Wasmtime's debug cache otherwise keys entries +by the rebuilt test binary mtime, which defeats reuse after ordinary edits: + +```toml +[profile.dev.package.wasmtime-internal-cache] +debug-assertions = false +``` + +For larger suites, prefer reusing one `Pglite` instance per test when isolation +allows it, and use `fresh_temporary` only for initialization-specific coverage. + +## Socket Server Limits + +`PgliteServer` is deliberately blocking and handles one frontend connection at a +time against a single embedded backend. It refuses SSL/GSS negotiation requests +with the standard PostgreSQL `N` response; connection URIs generated by the +crate include `sslmode=disable`. + +Set SQLx, `tokio-postgres`, Diesel, or framework pools to one connection. + +## Proxy Utility + +Expose a persistent database over TCP: + +```sh +cargo run --bin pglite-proxy -- --root ./.pglite --tcp 127.0.0.1:5432 +psql 'postgresql://postgres@127.0.0.1:5432/template1?sslmode=disable' +``` + +On Unix systems, the default proxy mode is `/tmp/.s.PGSQL.5432`: + +```sh +cargo run --bin pglite-proxy +PGPASSWORD=postgres psql 'postgresql://postgres@/template1?host=/tmp' +``` + +## Runtime Assets + +Runtime asset provenance is tracked in [ASSETS.md](ASSETS.md). The crate bundles +the PGlite runtime files needed to start the embedded backend. diff --git a/docs/USAGE.md b/docs/USAGE.md new file mode 100644 index 00000000..5a869ef3 --- /dev/null +++ b/docs/USAGE.md @@ -0,0 +1,199 @@ +# Usage Guide + +`pglite-oxide` has two public entry points: + +- `Pglite` for direct embedded calls from Rust +- `PgliteServer` for crates that need a PostgreSQL connection URI + +Prefer `Pglite` unless you specifically need a Postgres wire-protocol client. + +## Opening Databases + +Persistent database under an explicit path: + +```rust,no_run +use pglite_oxide::Pglite; + +fn main() -> anyhow::Result<()> { + let mut db = Pglite::open("./.pglite")?; + db.close()?; + Ok(()) +} +``` + +Persistent database under the platform app-data directory: + +```rust,no_run +use pglite_oxide::Pglite; + +fn main() -> anyhow::Result<()> { + let mut db = Pglite::builder() + .app("com", "example", "desktop-app") + .open()?; + db.close()?; + Ok(()) +} +``` + +Temporary database for tests: + +```rust,no_run +use pglite_oxide::Pglite; + +fn main() -> anyhow::Result<()> { + let mut db = Pglite::temporary()?; + db.close()?; + Ok(()) +} +``` + +`Pglite::temporary()` uses the process-local template cluster cache by default. +Use `Pglite::builder().fresh_temporary().open()?` only when a test needs to +exercise fresh cluster initialization. + +## Queries + +`exec` runs SQL without parameters. `query` runs the extended protocol with JSON +parameters. + +```rust,no_run +use pglite_oxide::Pglite; +use serde_json::json; + +fn main() -> anyhow::Result<()> { + let mut db = Pglite::open("./.pglite")?; + + db.exec("CREATE TABLE IF NOT EXISTS items(value TEXT)", None)?; + db.query("INSERT INTO items(value) VALUES ($1)", &[json!("alpha")], None)?; + + let result = db.query("SELECT value FROM items", &[], None)?; + println!("{:?}", result.rows); + + db.close()?; + Ok(()) +} +``` + +Values are passed as `serde_json::Value`. Default parsers and serializers cover +common Postgres types including integers, floats, booleans, JSON/JSONB, bytea, +dates/timestamps, UUIDs, and arrays discovered from `pg_type`. + +## Query Options + +`QueryOptions` controls result parsing and protocol behavior. + +```rust,no_run +use pglite_oxide::{Pglite, QueryOptions, RowMode}; + +fn main() -> anyhow::Result<()> { + let mut db = Pglite::open("./.pglite")?; + let options = QueryOptions { + row_mode: Some(RowMode::Array), + ..QueryOptions::default() + }; + + let rows = db.query("SELECT 1, 2", &[], Some(&options))?; + println!("{:?}", rows.rows); + + db.close()?; + Ok(()) +} +``` + +For `COPY ... FROM '/dev/blob'`, set `QueryOptions::blob` to the bytes exposed +through the guest `/dev/blob`. For `COPY ... TO '/dev/blob'`, read the returned +`Results::blob`. + +## Transactions + +Use `transaction` when several direct calls should commit or roll back together. + +```rust,no_run +use pglite_oxide::Pglite; +use serde_json::json; + +fn main() -> anyhow::Result<()> { + let mut db = Pglite::open("./.pglite")?; + + db.transaction(|tx| { + tx.query("INSERT INTO items(value) VALUES ($1)", &[json!("alpha")], None)?; + tx.query("INSERT INTO items(value) VALUES ($1)", &[json!("beta")], None)?; + Ok(()) + })?; + + db.close()?; + Ok(()) +} +``` + +## SQL Helpers + +`format_query` asks Postgres to quote parameter values. `QueryTemplate` helps +build SQL while keeping identifiers and values separate. + +```rust,no_run +use pglite_oxide::{Pglite, QueryTemplate, format_query, quote_identifier}; +use serde_json::json; + +fn main() -> anyhow::Result<()> { + let mut db = Pglite::open("./.pglite")?; + + let sql = format_query(&mut db, "SELECT $1::int", &[json!(42)])?; + assert_eq!(sql, "SELECT '42'::int"); + + let mut template = QueryTemplate::new(); + template.push_sql("SELECT * FROM "); + template.push_identifier("items"); + template.push_sql(" WHERE value = "); + template.push_param(json!("alpha")); + let built = template.build(); + + assert_eq!(built.query, "SELECT * FROM \"items\" WHERE value = $1"); + assert_eq!(quote_identifier("a\"b"), "\"a\"\"b\""); + + db.close()?; + Ok(()) +} +``` + +## PostgreSQL Clients + +Use `PgliteServer` when another crate expects a PostgreSQL URL. The server owns +one embedded backend, so configure downstream pools with one connection. + +```rust,no_run +use pglite_oxide::PgliteServer; +use sqlx::{Connection, Row}; + +#[tokio::main] +async fn main() -> anyhow::Result<()> { + let server = PgliteServer::temporary_tcp()?; + let mut conn = sqlx::PgConnection::connect(&server.connection_uri()).await?; + + let row = sqlx::query("SELECT $1::int4 + 1 AS answer") + .bind(41_i32) + .fetch_one(&mut conn) + .await?; + assert_eq!(row.try_get::("answer")?, 42); + + conn.close().await?; + server.shutdown()?; + Ok(()) +} +``` + +For app persistence, use: + +```rust,no_run +use pglite_oxide::PgliteServer; + +fn main() -> anyhow::Result<()> { + let server = PgliteServer::builder() + .path("./.pglite") + .start()?; + server.shutdown()?; + Ok(()) +} +``` + +Connection URIs generated by the crate include `sslmode=disable`.