Release 2.4.0 — format 4 tier B: leader batching + verdicts

LARVA-DESIGN.md

@@ -137,7 +137,7 @@ What this buys, measured over the same 40 ms fake-S3 harness as everything else:
 
 Entering format 3 is explicit and atomic: `db.upgrade()` is one final manifest CAS that flips `formatVersion` (older clients then refuse loudly via the Section 5 guard), or `larva({ commitLog: true })` creates new stores in format 3 from birth (retaining v0 as the base-of-time checkpoint). Time travel survives: history checkpoints are sparse (every 8th version) but `manifestAt` reconstructs any retained version by replaying log entries from the nearest base, so `asOf` and `rollbackTo` keep per-version granularity; rollback across the upgrade boundary restores data while preserving the format version. Vacuum treats log entries as history: an entry is dropped only once it is outside the retention window *and* no longer needed to replay from the oldest retained checkpoint.
 
-### Two-tier writes and cross-process group commit (format 4 — tier A shipped; leader batching next)
+### Two-tier writes and cross-process group commit (format 4 — shipped)
 
 Format 3 removed contention *inside* an instance (group commit) and made losing a race cheap (log rebases). The residual ceiling is **cross-instance**: two warm function instances still race for the same log slot, and under sustained multi-instance write load the retry budget is spent on coordination rather than work. Format 4 removes that ceiling with two mechanisms that share one substrate — per-writer intent queues — and one insight about honesty:
 
@@ -153,11 +153,11 @@ Format 3 removed contention *inside* an instance (group commit) and made losing
 
 Entering format 4 requires format 3 first (`upgrade()` raises straight to the top; `larva({ commitLog: true })` births new stores there); it is one-way, explicit, and guarded like every bump — old clients refuse loudly rather than write around the queue or strand it.
 
-**Shipping status.** Tier A is shipped and verified: classification, the one-PUT append path, the overlay (SELECTs scan pending rows alongside chunks, deduped by pk), the ordered-write barrier (an UPDATE/DELETE/transaction folds this instance's pending appends first, so it can never silently miss a row the caller just wrote), the lease-coordinated fold with pk-idempotence plus re-execution when the folded table changed under the plan (the split-lease double-fold guard), and self-healing cleanup (an orphaned intent blob is re-folded harmlessly and deleted by the next cycle). Tier B — ordered intents, leader batching, verdicts in entries — is client-side behavior on this same format and lands as the next increment; until then, ordered commits race slots directly exactly as in format 3.
+**Shipping status.** Both tiers are shipped and verified. Tier A: classification, the one-PUT append path, the overlay (SELECTs scan pending rows alongside chunks, deduped by pk), the ordered-write barrier (an UPDATE/DELETE/transaction folds this instance's pending appends first, so it can never silently miss a row the caller just wrote), the lease-coordinated fold with pk-idempotence plus re-execution when the folded table changed under the plan (the split-lease double-fold guard), and self-healing cleanup (an orphaned intent blob is re-folded harmlessly and deleted by the next cycle). Tier B: single-statement ordered writes escalate to the queue when the contention heuristic trips (two slot losses in one commit, or a rescue after an exhausted retry budget); any waiting writer that finds the lease free elects itself, batches every pending ordered intent into one slot with per-intent verdicts embedded in the entry, and waiting writers learn their fate from the log-tail reads they already do. A verdict on record makes a leftover intent cleanup, never work — the crashed-leader window closes by construction. One measured harness datapoint: 24 concurrent ordered updates from three instances landed as a single log slot. An implementation subtlety worth recording: concurrent request handlers inside one instance share the proto's writer identity, so lease acquisition alone cannot distinguish them — all lease-holding work (folds and leader passes) serializes through one per-instance chain, or N waiters would each 'renew their own lease' and elect N simultaneous leaders (caught by the harness as a 8× over-applied counter).
 
 ### The guarantee, stated honestly
 
-Larva promises: **no silently lost writes, atomic and durable commits, snapshot-isolated reads** — in every format; what changes across formats is the cost of contention, never the guarantee. It does not promise: high write throughput or low write-conflict rates under heavy concurrency. Every commit serializes through one arbitration point — a compare-and-swap on the manifest (formats 1–2) or first-writer-wins on the next log slot (format 3) — so sustained throughput is realistically single-digit commits per second, and heavily concurrent writers will spend time in retry loops (cheaper per retry in format 3, but retries still). This trade was accepted explicitly at design time: for the target workload (small teams, dashboards, agent-built tools) it is invisible; for workloads where it is not invisible, Section 10 tells you to leave, and Section 12 gives you the door. Format 4 raises the envelope: constraint-free appends leave the ordered path entirely (shipped — durable at one PUT), and ordered commits will batch across processes toward ~100–250 transactions per second (leader batching, next increment) — but its floor is stated above too: a constraint-bearing synchronous commit is never faster than one durable PUT round-trip.
+Larva promises: **no silently lost writes, atomic and durable commits, snapshot-isolated reads** — in every format; what changes across formats is the cost of contention, never the guarantee. It does not promise: high write throughput or low write-conflict rates under heavy concurrency. Every commit serializes through one arbitration point — a compare-and-swap on the manifest (formats 1–2) or first-writer-wins on the next log slot (format 3) — so sustained throughput is realistically single-digit commits per second, and heavily concurrent writers will spend time in retry loops (cheaper per retry in format 3, but retries still). This trade was accepted explicitly at design time: for the target workload (small teams, dashboards, agent-built tools) it is invisible; for workloads where it is not invisible, Section 10 tells you to leave, and Section 12 gives you the door. Format 4 raises the envelope: constraint-free appends leave the ordered path entirely (durable at one PUT), and contended ordered commits batch across processes into shared slots (harness-measured: 24 writers' statements in one slot) — but its floor is stated above too: a constraint-bearing synchronous commit is never faster than one durable PUT round-trip.
 
 Cross-statement note: transactions provide atomicity and snapshot reads, and the overlap check at commit time rejects write-write conflicts. This is snapshot isolation, not full serializability — write skew between two transactions reading overlapping data and writing disjoint rows is theoretically possible, exactly as in Postgres's default `READ COMMITTED`/`REPEATABLE READ` modes. This is documented and considered acceptable for v1.
 
@@ -296,7 +296,7 @@ v1 ships: the storage engine and commit protocol, the SQL subset of Section 7, c
 **The committed implementation track (decided July 2026).** The goal it serves: a small app with many concurrent users should not be a graduation reason — only Postgres-shaped needs should be. In order:
 
 1. **Auto-ID columns** — `t.uuid()`: a text column auto-filled with a time-ordered UUID (v7) on `INSERT` when omitted. Contention-free identity — the writer invents the value, so unlike `t.sequence()` there is nothing to coordinate — and time-ordering keeps new rows clustered in chunk zone maps, so primary-key pruning stays effective (a random UUIDv4 scatters). Same agent ergonomics as sequences: omit the column, read it back with `RETURNING`.
-2. **Intent queues + lease-elected cross-process group commit** — the ladder's rungs 4–5, designed and shipping as **format 4** (Section 6, "Two-tier writes"). The queue substrate and tier-A fast appends are shipped (2.3.0); tier-B leader batching — client-side behavior on the same format — is the next increment.
+2. **Intent queues + lease-elected cross-process group commit** — the ladder's rungs 4–5, **shipped in full as format 4** (Section 6, "Two-tier writes"): tier-A fast appends in 2.3.0, tier-B leader batching with verdicts in 2.4.0. The committed track is complete; the SQL dialect roadmap below is what remains.
 
 **The SQL dialect roadmap (decided July 2026).** Four former exclusions, admitted or re-judged against the Section 7 bar, in rough order of leverage: **secondary index blobs** (an index is just another immutable blob committed in the same atomic swap — makes non-key lookups prune); **`ALTER TABLE`, additive first** (`ADD COLUMN` of a nullable column is cheap — absent keys read as `NULL` in existing chunks; renames and drops wait for a migration design that respects time travel); **joins of three or more tables and self-joins** (execute inside the existing in-memory algebra after pruning); and **uncorrelated subqueries** (`WHERE id IN (SELECT …)`, scalar comparisons — planned as inner-query-first evaluation whose result feeds the outer plan; correlated subqueries stay out, per Section 7).
 

README.md

@@ -6,11 +6,11 @@
 
 [![CI](https://github.com/pango07/larva-db/actions/workflows/ci.yml/badge.svg)](https://github.com/pango07/larva-db/actions/workflows/ci.yml)
 [![npm](https://img.shields.io/npm/v/%40larva-db%2Fcore)](https://www.npmjs.com/package/@larva-db/core)
-[![test checks](https://img.shields.io/badge/test_checks-230_passing-brightgreen)](#the-testing-story)
+[![test checks](https://img.shields.io/badge/test_checks-240_passing-brightgreen)](#the-testing-story)
 [![types](https://img.shields.io/badge/types-included-blue)](packages/larvadb/src/index.ts)
 [![license](https://img.shields.io/badge/license-MIT-blue)](LICENSE)
 
-**Current release: 2.3.0.** Real SQL (time series, upserts, JSON), atomic transactions, time travel, and a guaranteed exit path to SQLite *or* Postgres.
+**Current release: 2.4.0.** Real SQL (time series, upserts, JSON), atomic transactions, time travel, and a guaranteed exit path to SQLite *or* Postgres.
 
 ## Sixty seconds to a database
 
@@ -161,7 +161,7 @@ const rows = await db.sql<Customer>`SELECT * FROM customers`;
 
 Format 3 changes how commits land: instead of re-uploading the whole manifest per commit, each commit is a tiny immutable delta in an ordered log, and the manifest becomes a periodic checkpoint. Conflicts get cheap (losing a race costs one small read, not a manifest round-trip), write cost stops scaling with database size, and contention tails shrink — same guarantees, verified by the same stress/property gauntlet. One-way, explicit, and old clients refuse loudly instead of corrupting:
 
-Format 4 adds **fast appends** on top: an INSERT whose outcome is fully client-determined — auto-generated id (`t.uuid()`, `t.sequence()`, or the implicit ULID), no unique constraints — is acknowledged the moment one durable PUT lands in a per-writer queue, then folded into the log in the background. Zero contention with anyone, your own reads see the rows immediately, and ordered writes (UPDATE/DELETE/transactions) fold first so they never miss them. Event logs, activity feeds, and telemetry stop touching the ordered path entirely.
+Format 4 adds **fast appends** on top: an INSERT whose outcome is fully client-determined — auto-generated id (`t.uuid()`, `t.sequence()`, or the implicit ULID), no unique constraints — is acknowledged the moment one durable PUT lands in a per-writer queue, then folded into the log in the background. Zero contention with anyone, your own reads see the rows immediately, and ordered writes (UPDATE/DELETE/transactions) fold first so they never miss them. Event logs, activity feeds, and telemetry stop touching the ordered path entirely. And when many instances hammer the same rows, the contention heuristic stops the retry storms: writers queue their statements and a lease-elected leader lands every waiting writer's work as **one** commit, each statement's result (or precise error) delivered back individually.
 
 ```ts
 await db.upgrade();                      // flip an existing database
@@ -268,7 +268,7 @@ The editable source for these lives at [docs/larva-architecture.excalidraw](docs
 
 ## The testing story
 
-Correctness risk concentrates in the conflict/retry path, so that's where the tests concentrate — **230 checks across seven suites**, all run in CI on every push:
+Correctness risk concentrates in the conflict/retry path, so that's where the tests concentrate — **240 checks across seven suites**, all run in CI on every push:
 
 | Suite | What it proves |
 |---|---|

content/how-it-works/commit-protocol.mdx

@@ -42,6 +42,25 @@ This matters on Vercel specifically: Fluid Compute serves many concurrent reques
   at one storage round-trip, because durability cannot be faked.
 </Callout>
 
+## Format 4: two-tier writes
+
+Format 4 splits writes by one honest question: **is the outcome fully known before the write leaves the process?**
+
+**Tier A — fast appends.** A pure INSERT with an auto-generated id (`t.uuid()`, `t.sequence()`, or the implicit ULID) and no unique constraints cannot fail ordering — nothing about its result depends on other writers. So it's acknowledged the moment **one create-only PUT** lands in this writer's own `queue/` prefix: durable at ack, contention with nobody, `RETURNING` served locally. A lease-elected compactor folds pending intents into ordinary log commits in the background.
+
+- **Read-your-writes**: your instance's un-folded rows are scanned alongside chunk rows — filters, joins, and aggregates see them immediately.
+- **The barrier**: any UPDATE, DELETE, or transaction folds pending appends first, so ordered writes never miss a row you just inserted.
+- **Idempotent folds**: appended rows always carry client-generated ids, so a fold skips anything already present — a folder that crashes mid-cleanup re-folds harmlessly.
+
+**Tier B — contention batching.** Constraint-bearing statements normally race slots exactly as in format 3 (the uncontended fast path is untouched). When the heuristic detects cross-instance contention (repeated slot losses), writers stop racing: each ships its statement as an **ordered intent** and any waiting writer that finds the lease free elects itself leader, plans every pending intent sequentially against a virtual manifest, and lands the whole batch as **one log slot** with per-intent verdicts (result rows or a precise error) embedded in the entry. Waiting writers learn their fate from the log-tail reads they already do.
+
+<Callout type="info">
+  The lease is a **performance mechanism, never a correctness one**: the create-only log slot stays
+  the sole arbiter, so a split lease wastes work but cannot corrupt. A verdict on record makes a
+  leftover intent cleanup, never work — the crashed-leader window closes by construction. Harness
+  datapoint: 24 concurrent updates from three instances landed as a single slot.
+</Callout>
+
 ## Transactions and the consistency model
 
 `db.transaction` runs its callback against one pinned snapshot with read-your-writes; everything lands in one commit. The overlap check rejects write-write conflicts at commit time. This is **snapshot isolation, not serializability** — write skew between two transactions reading overlapping data and writing disjoint rows is theoretically possible, exactly as in Postgres's default mode. Documented, accepted.

content/how-it-works/format-versioning.mdx

@@ -15,7 +15,7 @@ A Larva store is **shared mutable state between independently deployed clients**
 | **1** | the original layout: manifest + chunks + history |
 | **2** | sequence columns (`t.sequence()`), auto-UUID columns (`t.uuid()`), and composite unique constraints — a store only declares 2 when its schema actually uses one, so plain stores stay readable by format-1 clients |
 | **3** | [the ordered commit log](/docs/how-it-works/commit-protocol#format-3-the-ordered-commit-log) — entered only by explicit `db.upgrade()` |
-| **4** | two-tier writes: per-writer intent queues + a compactor lease. Constraint-free INSERTs (auto-generated id, no unique constraints) become **durable at one PUT** and fold into the log in the background — zero contention, read-your-writes preserved. Entered by `db.upgrade()`; `larva({ commitLog: true })` births new stores here |
+| **4** | two-tier writes: per-writer intent queues + a lease. Constraint-free INSERTs (auto-generated id, no unique constraints) become **durable at one PUT** and fold into the log in the background — zero contention, read-your-writes preserved. Under cross-instance contention, ordered writes queue too and a **lease-elected leader batches every waiting writer's statement into one log slot**, verdicts embedded in the entry. Entered by `db.upgrade()`; `larva({ commitLog: true })` births new stores here |
 
 ## The guard
 

content/index.mdx

@@ -30,11 +30,11 @@ That's the whole setup. No migrations to run, no dashboard to visit, no second b
 ## What you get
 
 - **Real SQL** — a deliberately scoped dialect that covers ~90% of what small apps write: joins, `GROUP BY` over expressions, `HAVING`, upserts, JSON functions, date bucketing. Everything outside it is rejected *by name, with an alternative*.
-- **No silently lost writes, ever.** Every commit goes through an atomic protocol; conflicts rebase, re-execute, or fail loudly. Verified by 230 checks including randomized concurrent-writer gauntlets.
+- **No silently lost writes, ever.** Every commit goes through an atomic protocol; conflicts rebase, re-execute, or fail loudly. Verified by 240 checks including randomized concurrent-writer gauntlets.
 - **Atomic transactions** — several statements, one commit, all or nothing.
 - **Time travel** — every commit is a new immutable version; `db.asOf()` reads the past and `db.rollbackTo()` is a one-line undo button.
 - **Auto IDs** — `t.uuid()` fills a time-ordered UUID on insert with nothing to coordinate; `t.sequence()` auto-numbers rows (invoice #42), unique across concurrent writers, without a server.
-- **Fast appends** (format 4) — INSERTs with auto-generated ids and no unique constraints are durable at **one storage round-trip**, contention-free, and fold into the log in the background; your own reads see them immediately.
+- **Fast appends & contention batching** (format 4) — INSERTs with auto-generated ids and no unique constraints are durable at **one storage round-trip**, contention-free; under heavy cross-instance contention, ordered writes batch into shared commits automatically instead of retry-storming.
 - **A guaranteed exit** — `db.export({ format: "postgres" })` produces one pg_dump-shaped file; `psql $DATABASE_URL < export.sql` is the entire migration.
 
 ## Who it's for — and honest limits

content/test-lab.mdx

@@ -50,7 +50,7 @@ bun run dev                       # http://localhost:3000
 vercel deploy --prod --yes        # or ship it
 ```
 
-## The test suites — 230 checks across seven suites
+## The test suites — 240 checks across seven suites
 
 Correctness risk concentrates in the conflict/retry path, so that's where the tests concentrate. All of it runs in CI on every push; merges to `main` publish to npm.