Skip to content

AGENTS.md

Latest commit

 

History

History
333 lines (277 loc) · 17.7 KB

AGENTS.md

File metadata and controls

333 lines (277 loc) · 17.7 KB
Raw

AGENTS.md

Guidance for AI coding agents working in this repository.

Project nature

DoltLite is a fork of SQLite that replaces the B-tree storage engine with a content-addressed prolly tree, giving a SQL database Git-like version control (branch / commit / merge / diff) while staying embeddable as a single library. SQLite's btree.h interface is the architectural seam: the tokenizer, parser, planner, and VDBE remain upstream-derived, while a DoltLite-format primary database replaces SQLite's B-tree, pager, and on-disk format with a prolly-tree engine backed by a single-file, content-addressed chunk store.

The btree.h seam is not a do-not-edit boundary. Changes to upstream-derived code above it are allowed when DoltLite integration or correctness requires them, but they must be narrowly scoped and guarded by DOLTLITE_PROLLY so a DOLTLITE_PROLLY=0 build retains stock SQLite behavior. Prefer solving a storage-engine concern below the seam when the interface permits it.

DoltLite is developed on Git and hosted on GitHub (https://github.com/dolthub/doltlite). Unlike upstream SQLite, it uses pull requests and accepts agentic contributions — opening PRs is the normal workflow (see PR / git workflow below).

Licensing and file headers

DoltLite is licensed Apache-2.0 (LICENSE.md). This differs from upstream SQLite, which is public domain.

  • Upstream SQLite files (src/btree.c, src/vdbe.c, src/where*.c, …) keep their public-domain "blessing" comment. Preserve it unchanged; never add a license header to them.
  • DoltLite source (src/doltlite_*.c, src/prolly_*.c, src/chunk_*.c, all guarded by #ifdef DOLTLITE_PROLLY) carries no per-file license header. Match the surrounding file — do not add one.
  • No issue or PR numbers in code comments; those references belong in commit messages and PR descriptions.

Comments: minimal by default

Do not add explanatory comments. The comment bar in this codebase is very high — aim for effectively none. Only keep a comment that is load-bearing: one that captures a non-obvious invariant, a subtle correctness reason, or a "why it must be this way" that the code cannot express on its own. Everything else — restating what the code does, section banners, narration of obvious steps, TODO chatter — makes the code worse and must be stripped.

Agents over-comment by default; consciously resist it. Write code that reads like the surrounding DoltLite source (which is nearly comment-free), and when in doubt, leave the comment out.

Build

The build uses autosetup from a separate build/ directory.

cd build
../configure
make doltlite                    # CLI shell — the prolly engine (default)
make doltlite-lib                # libdoltlite.a + .so/.dylib + doltlite.h
make DOLTLITE_PROLLY=0 sqlite3   # stock SQLite, for oracle/perf comparison
  • On macOS with Homebrew, link needs LIBRARY_PATH=/opt/homebrew/lib.
  • The test harness runs build/doltlite. Rebuild build/ after any src/ change or you validate a stale engine. A repo-root ./doltlite or ./sqlite3 built over the prolly .o files is DoltLite-in-disguise; to get real stock SQLite for comparison, build with DOLTLITE_PROLLY=0 in a clean directory.
  • make lint runs the layering and raw-file-I/O guards over src/.

Source layout

  • Upstream-derived SQLite core — src/*.c (parser, planner, vdbe.c, where*.c, …), kept close to upstream above btree.h; DoltLite-specific changes there are guarded by DOLTLITE_PROLLY. Master header src/sqliteInt.h.
  • Storage engine — src/prolly_*.c (prolly-tree node, cursor, mutmap, diff, three-way merge, chunker, cache, hashing) and src/chunk_*.c (content-addressed chunk store, WAL, refs, staging, file format).
  • Version-control surfaces — src/doltlite_*.c, roughly one file per feature: doltlite_commit (wire format), doltlite_commit_cmd / doltlite_add / doltlite_reset / doltlite_merge_cmd / doltlite_cherry_pick / doltlite_revert / doltlite_rebase / doltlite_config (SQL commands), doltlite_core (txn seal, catalog flush, create-commit helpers), doltlite_cmd (shared command scaffolding: option errors, peer-branch BUSY, conflict/CV txn-mode outcomes), doltlite_branch (branch/checkout), doltlite_merge (catalog orchestration in doltlite_merge, pass1/pass2 in doltlite_merge_pass1 / doltlite_merge_pass2, rows in doltlite_merge_rows, schema IR in doltlite_merge_schema, plus constraint detectors in doltlite_merge_constraints / _unique / _check / _fk), doltlite_diff / doltlite_diff_stat / doltlite_diff_table, doltlite_log, doltlite_history, doltlite_blame, doltlite_tag, doltlite_conflicts, doltlite_constraint_violations / doltlite_verify_constraints, doltlite_schemas / doltlite_schema_diff, doltlite_patch, doltlite_status, doltlite_workspace, doltlite_ignore, doltlite_docs, doltlite_tests, doltlite_hashof, doltlite_gc, doltlite_remote / doltlite_http_remote / doltlite_remotesrv. Internal header: src/doltlite_internal.h; surfaces are registered in src/doltlite.c.

The version-control surface (vtables + functions)

DoltLite exposes version control as SQLite virtual tables (system tables: dolt_log, dolt_diff, dolt_diff_<table>, dolt_conflicts, dolt_constraint_violations, dolt_status, …) and scalar functions / TVFs (SELECT dolt_commit(...), SELECT dolt_merge(...), SELECT dolt_branch(...)). Dolt exposes the same operations as stored procedures (CALL dolt_commit(...)).

This vtable form, and DoltLite-flavored column names, are intentional — not conformance bugs. When comparing against Dolt, only row-level semantics must match; do not file conformance issues over the vtable/function shape or column naming.

dolt_ignore, dolt_docs, and dolt_tests are lazy user-space system tables: an eponymous writable vtab answers reads while no backing table exists (SELECT never errors on a fresh repo), and the first write statement materializes the real table, which then shadows the module. That is Dolt's lazy-creation UX without the open-time auto-materialization that sank the original dolt_ignore attempt (fcb7720e00). dolt_ignore materializes dolt_ignore(pattern TEXT NOT NULL, ignored TINYINT NOT NULL, PRIMARY KEY(pattern)). dolt_docs materializes dolt_docs(doc_name TEXT NOT NULL, doc_text TEXT NOT NULL, PRIMARY KEY(doc_name)) and serves a default AGENT.md row (DoltLite's own operations guide, not Dolt's embedded text) from the module before materialization; that row is seeded as a stored row when the table materializes. Deliberate divergences from Dolt: the table shows in .tables/.schema once it exists; the default AGENT.md text differs (the docs oracle compares it by name/count, or by full row after a test overwrites it); and because the row is stored rather than synthesized, deleting AGENT.md sticks (Dolt resurrects it on the next read) and it appears in row-level diffs. build.c shape guards keep hand-issued CREATE TABLE statements on the exact schema each module creates.

dolt_tests uses the same lazy user-space system-table pattern: empty reads resolve before a backing table exists, and the first write materializes the six-column table with Dolt's assertion checks. dolt_test_run(...) executes the versioned definitions as read-only single-statement queries and returns the Dolt-compatible test result rows.

Surfaces deliberately not implemented

Absent on purpose, so don't file them as parity gaps or implement them unprompted:

  • dolt_stash — not on the roadmap. Stashing exists because git binds one working tree to a clone, so switching branches forces you to shelve uncommitted work first. DoltLite gives every branch its own working set: dolt_checkout between dirty branches neither refuses nor carries changes over, and connections addressing db/branch hold independent uncommitted state at the same time. The shelve/switch/unshelve dance has nothing to solve here. See Dolt Worktrees.

  • CLI-parity wrappers around plain SQL — dolt_rm, dolt_mv, and anything else whose whole job is to spell a statement SQL already has. These exist in Dolt so its subcommands have a server-side equivalent: dolt_rm's own source calls itself "the stored procedure for the cli command dolt rm", and dolt mv is a CLI command that builds DROP TABLE / rename SQL — dolt_mv is not even a Dolt procedure. DoltLite has no subcommand CLI to be compatible with, so DROP TABLE and ALTER TABLE … RENAME TO are the surface.

  • dolt_statistics — DoltLite already has branch-specific table statistics; they are spelled ANALYZE and sqlite_stat1. Dolt keeps stats in a separate branch-keyed database that is never merged, because it maintains them automatically — continuously regenerated derived data cannot sit in the commit graph. SQLite produces statistics only when the user asks, so sqlite_stat1 is an ordinary versioned table: per-branch for free, and it commits, diffs and merges with everything else. Keeping it accurate is the user's call, same as ANALYZE anywhere else.

  • dolt_query_catalog — DoltHub-specific storage, not engine behavior: it holds saved queries so DoltHub can display them. Nothing local reads it. Worth revisiting if pushing to DoltHub lands, and not before.

Dolt is the reference implementation

Dolt is the authority on what every version-control operation should do. When a behavior is ambiguous, when DoltLite and Dolt disagree, or when implementing a new VC surface, read the Dolt source (Go) to determine the correct semantics — result shape, diff/ merge/conflict rules, dolt_* system-table columns, error conditions — and match it. It's the reference for the oracle suites and for the engine itself. Dolt may be checked out locally as a sibling of this repo; otherwise consult it on GitHub.

Testing

Several independent layers. A change under src/ should run the relevant ones; dolt on PATH is required for the oracle suites.

  • test/run_doltlite_tests.sh — DoltLite-native shell suites (doltlite_*.sh) covering branch/commit/diff/merge/checkout/etc. behavior and parity. Runs build/doltlite. This layer is separate from the regression and C tests and catches branch / default-branch / parity bugs they miss — always run it for version-control changes.
  • test/vc_oracle_*_test.sh — Dolt oracle suites: run identical SQL against DoltLite (SELECT dolt_x) and real Dolt (CALL dolt_x) and diff the normalized output. Invoke as bash test/vc_oracle_X_test.sh build/doltlite dolt. CI auto-runs anything matching vc_oracle_*_test.sh / oracle_*_test.sh, so a new suite with that name needs no wiring. Compare semantics, not vtable shape.
  • Regression buckets — test/regression-buckets/*.txt list the inherited and ported upstream *.test files gated via testfixture. testfixture is built with SQLITE_TEST (the CLI is not), so don't #ifdef SQLITE_TEST- guard a correctness fix, and verify fixes in testfixture, not only the CLI. Every doltlite_*.test suite must appear in a bucket or test/lint_orphaned_suites.sh fails.
  • test/run_c_tests.sh — C unit tests (concurrency, crash recovery, serialize determinism, chunk-store locking, …). Locally it builds the tests it gates and flags binaries older than src/; under CI it runs the build phase's artifacts untouched. Either way not built and stale are counted separately from failed, so a skipped or out-of-date binary can never read as a pass.
  • test/run_sqllogictest.sh — at full pass / 100% parity. test/known_sqllogictest_divergences.txt is empty and must stay empty: a new divergence means fix the engine, never add a known-divergence entry.

Rules that override convenience:

  • Never delete a test or disable a check. If a test or assertion fails, fix the change, not the guardrail.
  • For an observable bug, prove the fix with a fail-before / pass-after test (fails on the unfixed engine, passes with the fix). Latent/defensive fixes may legitimately have no such test.
  • When local validation would take more than ~15 min, push a PR and let CI run the buckets/corpus/suites in parallel; spot-check locally.

PR / git workflow

  • Branch off a freshly pulled master; open the PR against master. Do not stack PRs — a stacked branch leaves commits dangling when its base merges first; rebase onto master instead.
  • Stage by explicit path (git add src/foo.c test/bar.sh). git add -A / git add . sweep in hundreds of untracked build artifacts (*.o, tsrc/*.c, ./doltlite, ./sqlite3, testfixture).
  • Commit or push only when asked. End commit messages and PR bodies with the trailers the harness expects.

Version-control correctness invariants

  • Conflicts are never persisted. A conflicted merge lives only in the transaction that produced it: COMMIT is refused while any conflict remains, an autocommit merge that conflicts is rolled back whole, and the persist path declines to make a conflicted working set durable. Nothing conflicted reaches disk, so no later connection inherits a merge to finish. Dolt allows this behind @@dolt_allow_commit_conflicts, which DoltLite does not implement, so conflict persistence is a deliberate divergence — compare in-transaction behaviour against Dolt, not what survives a commit. Constraint violations are unaffected and still persist.

Subtle rules that are easy to break silently — hold them when touching VC code:

  • Re-confirm under the lock. Multi-step ops (merge / cherry-pick / revert / pull) must re-read HEAD under the lock immediately before advancing a ref, or they clobber a concurrent peer.
  • Scoped ref installs. A pushed ref update must go through the scoped-refs validator (only the declared branch may change); never install a pushed blob wholesale. Clone install is unscoped by design.
  • Canonical catalog. Schema commits adopt the canonical catalog; the live catalog must always equal the persisted one. Constructed catalog arrays (staging / merge / -am) adopt the single master root paired by iTable==1, never by name.
  • Ref resolution for commit / branch / HEAD / WORKING / STAGED goes through doltliteResolveCatalogHashForRef — reuse it, don't re-derive.

Concurrency contract

User-facing concurrency guarantees and the multiproc/multi-connection oracles that pin them live in the README Concurrency section and test/concurrency_contract.tsv. test/concurrency_contract_test.sh fails if a claim loses its evidence needle. When changing graph lock, snapshot pin or write upgrade, locked ref expectation/CAS paths, multiproc harnesses, or conflict durability, update the TSV and keep the multiproc C suites green (multi_process_*, concurrent_* via test/run_c_tests.sh).

The version-control correctness invariants above remain load-bearing for implementors even when a claim is also listed in the contract.

SQLite compatibility contract

The user-facing SQLite compatibility contract lives in the README SQLite Compatibility section; claims and evidence live in test/sqlite_compatibility_contract.tsv. Update the README, contract row, and its evidence together when observable compatibility behavior changes.

Do not infer poor SQLite compatibility from an intentional storage-engine adaptation. DoltLite uses chunks rather than pages, has no SQLite WAL or rollback journal, and cannot use an anonymous rowid as a version-controlled key because identity must be history-independent. Classify inherited-suite exceptions through test/known_testfixture_divergences.txt: engine-gap is a bug to fix, while intentional, unsupported, and harness entries describe known boundaries. SQLLogicTest remains a zero-divergence correctness gate.

Storage format version

Chunk-store version 12 is the beta format freeze. User-facing rules live in the README Storage Format section; claims and evidence live in test/storage_format_contract.tsv, and the golden file lives under test/format-corpus/v12/.

Bumping CHUNK_STORE_VERSION requires:

  1. README Storage Format update
  2. A new test/format-corpus/ entry and MANIFEST
  3. storage_format_contract.tsv evidence update
  4. Explicit open/upgrade policy for version 12 (open-only, migrate, or refuse)

Any incompatible change to a nested write format, including working sets, catalogs, refs, commits, prolly nodes, or key encoding, must bump CHUNK_STORE_VERSION. Do not silently reinterpret another version.

The release workflow also runs test/doltlite_compat_test.sh against prior releases. Run it when changing format recognition, readers, writers, or version handling; version-12 databases already created by Beta users are compatibility fixtures, not disposable test data.

Beta release surfaces

DoltLite is public Beta. Move quickly on engine fixes, but treat documented behavior, version-12 data, the compatibility and concurrency contracts, public C headers and exports, release packages, and binding inputs as shipped surfaces. A change under packaging/, the install/export rules, amalgamation generation, or .github/workflows/release.yml must run the relevant package or artifact smoke test; core engine tests alone do not validate a distributed artifact.

External integration and documentation work is no longer a pre-Beta deprioritized category. Scope validation to the surface being changed, preserve the published contracts, and fix the engine with fail-before/pass-after evidence when behavior is wrong.