Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ Larva is a deliberate miniaturization of the Delta Lake / Iceberg pattern on top
- The SQL dialect is a closed, documented subset (Design §7). Precise, machine-readable error messages for unsupported SQL are a design feature — agents self-correct from specific errors. Do not quietly accept SQL outside the subset.
- `UPDATE`/`DELETE` without `WHERE` requires explicit `{ allowFullTable: true }`.
- The parser rejects multiple statements per string (injection stacking vector).
- The public API surface (Design §13) must fit on one screen: `larva()`, `db.sql` tagged template (primary API), `db.query(str, params)`, `db.transaction`, `db.asOf`, `db.rollbackTo`, `db.export`, `db.vacuum`, `db.upgrade`. Resist growing it.
- The public API surface (Design §13) must fit on one screen: `larva()`, `db.sql` tagged template (primary API), `db.query(str, params)`, `db.transaction`, `db.asOf`, `db.rollbackTo`, `db.export`, `db.vacuum`, `db.upgrade`, `db.inspect` (read-only layout projection — the one read accessor). Resist growing it.
- On-store format versioning (Design §5): formats 1 (original), 2 (sequences, auto-UUID columns + composite uniques), 3 (ordered commit log), 4 (two-tier writes: intent queues + durable-at-PUT appends). Manifests declare the minimum format their content requires; clients refuse newer formats loudly (`FormatError`). Any future format bump ships with an explicit atomic `upgrade()`; `rollbackTo` preserves the format version.
- Export (SQLite/Postgres/JSON/CSV) is a v1 feature, not an afterthought — the escape hatch is part of the product promise.
- Only Vercel-specific code should be the storage adapter; the `StorageAdapter` contract is exactly four operations: get, put-with-CAS, delete, list-by-prefix.
Expand Down
5 changes: 4 additions & 1 deletion LARVA-DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -294,8 +294,11 @@ await db.rollbackTo(version); // recovery in one line
await db.export({ format: "sqlite" }); // the escape hatch
await db.vacuum(); // reclaim storage outside retention
await db.upgrade(); // one-way flip to format 3 (the commit log)
await db.inspect(version?); // read-only physical layout: chunks + zone maps
```

`inspect()` is the one read-only accessor on the surface, and it earns its place the same way `export()` does: it is a *derived*, format-agnostic projection of the manifest (version, `committedAt`, `formatVersion`, and per-table chunk lists with each chunk's row count and primary-key/partition zone-map min-max), never the raw `Manifest` — so on-store format evolution cannot break its contract. It reads manifest metadata only (strictly lighter than `export`, which also reads every chunk), returns no blob URLs (the private-data invariant holds), and resolves any retained past version exactly as `asOf` does. It exists so a store can be *seen* — it is the primitive under the data viewer (§14) and any store-debugging tool — and it is the deliberate exception to "resist growing the surface": one read-only method, no new write path.

The same surface ships as a shell command — the package's `larva` bin (`npx larva sql | export | upgrade | rollback | vacuum | version`, credentials auto-loaded from `.env.local`). One surface, two doors; the CLI is validated end to end by `scripts/cli-smoke.ts`.

## 14. Roadmap and open questions
Expand All @@ -309,7 +312,7 @@ v1 ships: the storage engine and commit protocol, the SQL subset of Section 7, c

**The SQL dialect roadmap (decided July 2026) — complete.** Four former exclusions, admitted or re-judged against the Section 7 bar. Three shipped in 2.5: additive `ALTER TABLE … ADD COLUMN` (schema-only commit; absent keys read as `NULL` at every read path; additive schema drift auto-migrates at connect — Section 8; renames and drops still wait for a migration design that respects time travel), joins of three or more tables and self-joins (left-deep hash joins in the existing in-memory algebra; distinct alias per occurrence), and uncorrelated subqueries (inner-query-first evaluation rewriting the node into literals, exactly as planned — the result feeds the outer plan and `IN` lists participate in zone-map pruning; correlated subqueries stay out, per Section 7). The fourth, **secondary index blobs**, shipped in 2.6 exactly as sketched — an index is just another immutable blob committed in the same atomic swap, and keying entries by immutable chunk id made it safe without a format bump (Section 8).

Still deferred, unordered: a columnar chunk format; S3/R2/GCS/Azure adapters (the S3 contract harness already exists); a tiny read-only web UI for browsing tables (non-engineers love to *see* their data); an optional Edge Config read-accelerator for flag-shaped tables.
Still deferred, unordered: a columnar chunk format; S3/R2/GCS/Azure adapters (the S3 contract harness already exists); an optional Edge Config read-accelerator for flag-shaped tables. The **read-only web UI** for browsing tables (non-engineers love to *see* their data) is underway in the test-lab app: a `/viewer` route over the demo store that pairs a paged table browser with a version scrubber (time travel via `asOf`) and a chunk/zone-map internals panel — the last of these is why `db.inspect()` (§13) was added, since a store's physical layout lives in the otherwise-private manifest.

**The v2 write path.** The v1 ceiling — every logical commit is one CAS on one global manifest — is a protocol choice, not an object-storage limit, and there is a ladder of known techniques for raising it if multi-process operational workloads (several deployments writing steadily, unique-constraint-bearing inserts, sequence assignment) become a target. In rough order of leverage per cost: **sequence range leasing** (shipped — `t.sequence()`, Section 8); **composite unique constraints** (shipped — `defineSchema` `uniques`, Section 8); **an ordered commit log** (shipped — format 3, Section 6: create-only numbered entries, checkpointed manifest, cheap rebases, explicit `upgrade()`); **lease-elected cross-process group commit** (committed — see above; object-storage CAS doubles as a lease primitive; the leaseholder batches every process's transactions into shared log slots — group commit across processes with no server anyone deploys, safe even under split leadership because the log stays the arbiter; needs the intent queues below as its transport, so the two ship together); **a two-tier write API** (committed — see above; constraint-free appends go to per-writer intent prefixes — zero contention, one-PUT-RTT durability, an LSM memtable made of blobs folded in by a lease-elected compactor; constraint-bearing writes go through the ordered log and wait for a deterministic slot verdict, Calvin-style); and **sharded contention domains** (a two-level manifest so unrelated table groups stop sharing one CAS). Each rung ships independently; the format-versioning guard (Section 5) is what makes adopting any of them a loud, upgradeable break instead of silent corruption. The physics no rung changes: sync-commit latency floors at one PUT round-trip, because durability cannot be faked.

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
[![types](https://img.shields.io/badge/types-included-blue)](packages/larvadb/src/index.ts)
[![license](https://img.shields.io/badge/license-MIT-blue)](LICENSE)

**Current release: 2.6.0.** Real SQL (time series, upserts, JSON, multi-table joins, subqueries), atomic transactions, time travel, and a guaranteed exit path to SQLite *or* Postgres.
**Current release: 2.7.0.** Real SQL (time series, upserts, JSON, multi-table joins, subqueries), atomic transactions, time travel, and a guaranteed exit path to SQLite *or* Postgres.

## Sixty seconds to a database

Expand Down
37 changes: 37 additions & 0 deletions app/api/inspect/route.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
import { NextRequest, NextResponse } from "next/server";
import { SqlError } from "@larva-db/core";
import { demoDb } from "@/app/lib/demo";
import { schemaMeta } from "@/app/lib/viewer";

export const maxDuration = 60;

/**
* Read-only physical layout of the demo store at one version (db.inspect,
* Design §13): per-table chunk lists with zone-map min/max. Powers the
* viewer's internals panel and version scrubber. `?version=N` time-travels;
* `currentVersion` is always returned so the scrubber knows its upper bound,
* and `schema` (every table's columns) so the client can draw the table
* structure before a single row loads.
*/
export async function GET(req: NextRequest) {
const versionParam = req.nextUrl.searchParams.get("version");
try {
const db = await demoDb();
let version: number | undefined;
if (versionParam !== null) {
version = Number(versionParam);
if (!Number.isInteger(version) || version < 0) {
return NextResponse.json({ error: { code: "BAD_VERSION", message: "version must be a non-negative integer" } }, { status: 400 });
}
}
const currentVersion = await db.currentVersion();
const layout = await db.inspect(version);
return NextResponse.json({ ...layout, currentVersion, schema: schemaMeta() });
} catch (err) {
if (err instanceof SqlError) {
const status = err.code === "VERSION_NOT_FOUND" ? 404 : 400;
return NextResponse.json({ error: { code: err.code, message: err.message } }, { status });
}
return NextResponse.json({ error: { code: "INTERNAL", message: err instanceof Error ? err.message : String(err) } }, { status: 500 });
}
}
71 changes: 71 additions & 0 deletions app/api/viewer-rows/route.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
import { NextRequest, NextResponse } from "next/server";
import { SqlError } from "@larva-db/core";
import { demoDb, demoSchema } from "@/app/lib/demo";
import { tableMeta } from "@/app/lib/viewer";

export const maxDuration = 60;
const DEFAULT_PAGE = 50;
const MAX_PAGE = 200;

/**
* A page of rows from one demo table, optionally at a past version (time
* travel via asOf). Read-only. Table and sort column are whitelisted against
* the code-first schema, and page bounds are coerced to integers, so the only
* thing interpolated into SQL are validated identifiers — no user values reach
* the query string (and the parser rejects statement stacking regardless).
* Returns the fetch's pruning stats (chunks scanned vs. total) so the UI can
* show the zone maps doing their job.
*/
export async function GET(req: NextRequest) {
const p = req.nextUrl.searchParams;
const table = p.get("table") ?? "";
const spec = demoSchema[table];
if (!spec) {
return NextResponse.json(
{ error: { code: "UNKNOWN_TABLE", message: `pick a table: ${Object.keys(demoSchema).join(", ")}` } },
{ status: 400 },
);
}
const columns = Object.keys(spec.columns);
const orderByParam = p.get("orderBy");
const orderBy = orderByParam && columns.includes(orderByParam) ? orderByParam : spec.primaryKey;
const dir = p.get("dir") === "desc" ? "DESC" : "ASC";
const limit = Math.min(MAX_PAGE, Math.max(1, Math.trunc(Number(p.get("limit"))) || DEFAULT_PAGE));
const offset = Math.max(0, Math.trunc(Number(p.get("offset"))) || 0);
const versionParam = p.get("version");

try {
const db = await demoDb();
const currentVersion = await db.currentVersion();
const wantsPast = versionParam !== null && Number(versionParam) !== currentVersion;
if (versionParam !== null && !Number.isInteger(Number(versionParam))) {
return NextResponse.json({ error: { code: "BAD_VERSION", message: "version must be an integer" } }, { status: 400 });
}
const version = wantsPast ? Number(versionParam) : currentVersion;
const src = wantsPast ? await db.asOf(version) : db;

const rows = await src.query(`SELECT * FROM ${table} ORDER BY ${orderBy} ${dir} LIMIT ${limit} OFFSET ${offset}`);
const stats = db.lastQueryStats; // capture before the COUNT query overwrites it
const [{ n }] = await src.query<{ n: number }>(`SELECT COUNT(*) AS n FROM ${table}`);

return NextResponse.json({
table,
...tableMeta(table)!,
rows,
total: Number(n),
limit,
offset,
orderBy,
dir: dir.toLowerCase(),
version,
isCurrent: !wantsPast,
stats,
});
} catch (err) {
if (err instanceof SqlError) {
const status = err.code === "VERSION_NOT_FOUND" ? 404 : 400;
return NextResponse.json({ error: { code: err.code, message: err.message } }, { status });
}
return NextResponse.json({ error: { code: "INTERNAL", message: err instanceof Error ? err.message : String(err) } }, { status: 500 });
}
}
28 changes: 28 additions & 0 deletions app/lib/viewer.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
import { demoSchema } from "@/app/lib/demo";

/** The client-facing description of one table's shape — enough to render the
* full table structure (headers, pk/partition badges) before any row loads. */
export interface TableMeta {
columns: string[];
types: Record<string, string>;
primaryKey: string;
partitionColumn: string | null;
}

export function tableMeta(table: string): TableMeta | null {
const spec = demoSchema[table];
if (!spec) return null;
const columns = Object.keys(spec.columns);
return {
columns,
types: Object.fromEntries(columns.map((c) => [c, spec.columns[c].type])),
primaryKey: spec.primaryKey,
partitionColumn: spec.partitionColumn ?? null,
};
}

/** Every table's shape, keyed by name — shipped with the inspect response so the
* viewer can draw the skeleton table immediately. */
export function schemaMeta(): Record<string, TableMeta> {
return Object.fromEntries(Object.keys(demoSchema).map((t) => [t, tableMeta(t)!]));
}
6 changes: 6 additions & 0 deletions app/page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,12 @@ export default function Home() {
<h1 className="grow text-2xl font-semibold tracking-tight">
Larva <span className="text-ink-muted font-normal">/ test lab</span>
</h1>
<Link
href="/viewer"
className="text-ink-secondary hover:text-foreground text-sm underline underline-offset-4"
>
data viewer
</Link>
<Link
href="/docs"
className="text-ink-secondary hover:text-foreground text-sm underline underline-offset-4"
Expand Down
5 changes: 5 additions & 0 deletions app/viewer/layout.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
import { Providers } from "./providers";

export default function ViewerLayout({ children }: { children: React.ReactNode }) {
return <Providers>{children}</Providers>;
}
Loading
Loading