feat: add LDBC SNB Interactive short-read benchmark (Lane 1)

.gitignore

@@ -58,7 +58,7 @@ yarn-error.log
 
 # Stryker mutation testing
 .stryker-cache/
-reports/
+packages/typegraph/reports/
 .gstack/
 
 # Local design scratch files

apps/docs/src/content/docs/performance/indexes.md

@@ -76,20 +76,16 @@ export const typegraphTables = createSqliteTables({}, {
 
 **Key options:**
 
-- `fields`: JSON property paths used for filtering/ordering (B-tree expression keys). Optional — an
-  index can instead be keyed entirely by `coveringFields`/`keySystemColumns` — but the declaration
-  must supply at least one of `fields`, `coveringFields`, or `keySystemColumns`.
+- `fields`: JSON property paths used for filtering/ordering (B-tree expression keys). Optional if
+  `coveringFields` or `keySystemColumns` supply the key instead — an index must declare at least
+  one of the three.
 - `coveringFields`: additional properties frequently selected with the same filters. These become
   additional index keys to enable index-only reads when combined with smart select.
-- `keySystemColumns`: system columns (e.g. `"id"`) to include in the index key, alongside
-  `fields`/`coveringFields`. A covering index can only serve a query's join index-only if the
-  index's key matches the join's actual predicate — a query that joins on a system column directly
-  (e.g. a compiled `n.id = e.from_id` for a reverse traversal) needs a matching key column that
-  `fields`/`coveringFields` can't provide, since those only ever accept schema properties. Rejects
-  edge-only system columns on a node index and any column `scope` already prefixes the key with;
-  not supported together with `method: "gin"`/`"trigram"`. Rejects `"id"` combined with `unique:
-  true` — every node's `id` is already unique per row, so a unique index keyed on `id` plus other
-  columns can never enforce a meaningful constraint across those other columns.
+- `keySystemColumns`: system columns (e.g. `"id"`) to include in the key, after the `scope` prefix
+  and before `fields`. See [Keying on system columns](#keying-on-system-columns-keysystemcolumns).
+  Rejects `"id"` combined with `unique: true` — every node's `id` is already unique per row, so a
+  unique index keyed on `id` plus other columns can never enforce a meaningful constraint across
+  those other columns.
 - `unique`: create a unique index.
 - `scope`: prefixes index keys with TypeGraph system columns (default is `"graphAndKind"`).
 - `where`: partial index predicate (portable DSL, compiled per dialect).
@@ -135,6 +131,31 @@ Index `scope` controls which TypeGraph system columns are prefixed ahead of your
 - `"graph"`: prefixes with `graph_id` only (rare; useful for cross-kind queries within a graph).
 - `"none"`: no system prefix (rare; usually only correct for global queries).
 
+### Keying on system columns (`keySystemColumns`)
+
+`fields` and `coveringFields` only ever reference your schema's own properties, and `scope` only
+ever prefixes `graph_id`/`kind` — neither can put a system column like `id` into the index **key**.
+
+Some query shapes need that. A reverse traversal — `.traverse("hasCreator", { direction: "in"
+}).to("Post", "post")` — compiles a join on the target node's own `id` (`n.id = e.from_id`). If you
+also filter, sort, or select a prop on that same node (say `creationDate`), a covering index needs
+`id` in its key to match that join — `graph_id`/`kind` and `creationDate` alone aren't enough,
+because the index can't be chosen for an `id`-equality join it doesn't cover.
+
+```ts
+const postRecent = defineNodeIndex(Post, {
+  keySystemColumns: ["id"],
+  coveringFields: ["creationDate"],
+});
+// -> CREATE INDEX ... ON typegraph_nodes (graph_id, kind, id, (props #>> ARRAY['creationDate']))
+```
+
+**Validation:**
+
+- Node indexes only. Rejects edge-only system columns (`fromKind` / `fromId` / `toKind` / `toId`).
+- Rejects a column already implied by `scope` (e.g. `graph_id` when `scope: "graphAndKind"`).
+- Not supported with `method: "gin" | "trigram"` (same restriction as `coveringFields`).
+
 ## Edge Indexes
 
 `defineEdgeIndex(edgeType, config)` works the same way as node indexes, with one extra option:
@@ -197,6 +218,33 @@ CREATE INDEX idx_person_email_name ON typegraph_nodes
   WHERE deleted_at IS NULL;
 ```
 
+:::caution[PostgreSQL: JSONB expression indexes don't get a true Index Only Scan]
+A covering index like the one above lets PostgreSQL avoid a full **table scan**, but not
+necessarily a **heap fetch** per matching row. PostgreSQL's `Index Only Scan` optimization — skip
+the heap entirely when the index already has everything the query needs — does not extend to
+JSONB extraction expressions (`props #>> ARRAY[...]`), only to real stored columns. Even a
+correctly-shaped `coveringFields` index shows as a plain `Index Scan` in `EXPLAIN`, not
+`Index Only Scan`, and PostgreSQL still visits the heap row for every match.
+
+This matters most for high-fan-in ranked reads — e.g. "each of N friends' most recent post, top 10
+overall" — where a query visits many candidate rows and discards most of them after sorting. If
+`EXPLAIN (ANALYZE, BUFFERS)` shows most of a query's cost coming from a repeated `Index Scan` with
+high `Buffers: shared hit` relative to the rows actually returned, this is likely why — confirmed
+against a real workload at real scale, not a theoretical concern.
+
+TypeGraph doesn't currently offer a built-in way to materialize a hot prop as a real column (this
+is an active area of investigation). Until then, if this is a genuine hot-path bottleneck on
+PostgreSQL, the workaround is maintaining your own
+[stored generated column](https://www.postgresql.org/docs/current/ddl-generated-columns.html)
+alongside `props` with a plain index over it — a real column does get `Index Only Scan`, confirmed
+via `EXPLAIN (ANALYZE, BUFFERS)` showing `Heap Fetches: 0` (run `VACUUM ANALYZE`, not just
+`ANALYZE`, after backfilling — the visibility map needs to be current before PostgreSQL will prove
+it immediately).
+
+We haven't specifically verified whether SQLite's JSON-extraction covering indexes have the same
+limitation for this query shape.
+:::
+
 ## Batched Index Lookup (`bulkFindByIndex`)
 
 `store.nodes.<Kind>.bulkFindByIndex(indexName, items, options?)` takes many in-memory records and

apps/docs/src/content/docs/performance/overview.md

@@ -138,6 +138,19 @@ await store.nodes.Person.bulkInsert(people);
 await store.nodes.Person.bulkUpsertById(itemsWithIds);
 ```
 
+### Batch sizing for large multi-call imports
+
+For a dataset too large for a single `bulkInsert`/`bulkCreate` call (e.g., streaming rows from a
+file in a loop), the *size* of each call matters, not just the total row count. Each call is its
+own transaction, and — per the default
+[`autoRefreshStatistics`](/backend-setup#refreshing-planner-statistics-after-bulk-loads) — can
+trigger a planner-statistics refresh on its own. In a large-scale bulk-load benchmark, batches of
+~2,000 rows per call were consistently ~25-30% slower per row than batches of ~20,000+: fewer,
+larger calls amortize both the per-call transaction commit and the statistics refresh across more
+rows. Prefer batch sizes in the tens of thousands when looping over many calls for a large import,
+and consider `autoRefreshStatistics: false` plus one `store.refreshStatistics()` call after the
+loop if per-call refreshes still dominate.
+
 ### Batch reads
 
 `getByIds()` on node and edge collections uses a single `SELECT ... WHERE id IN (...)` instead of N
@@ -271,8 +284,11 @@ const results = await store
 ```
 
 This optimization pairs well with [covering indexes](/performance/indexes#covering-indexes): if
-your index contains both the filter keys and the selected keys, the database can satisfy the query
-with an index-only scan.
+your index contains both the filter keys and the selected keys, the database can serve the query
+straight from the index instead of scanning the whole table — though on PostgreSQL specifically,
+this stops short of a true `Index Only Scan` for JSONB-extracted fields; see the
+[covering indexes](/performance/indexes#covering-indexes) section for the concrete limitation and
+a workaround.
 
 **When optimization applies:**
 
@@ -490,6 +506,17 @@ Backend-specific overrides:
 
 </details>
 
+### Real-world workload validation
+
+Beyond the synthetic guardrail suite above, TypeGraph is also exercised against the
+[LDBC Social Network Benchmark (SNB) Interactive](https://github.com/ldbc/ldbc_snb_interactive_v1)
+workload — a standard, independently-defined graph benchmark, not a TypeGraph-specific one — at
+SF1 scale (~10k persons, ~1M posts, ~2M comments). This surfaced and fixed two real scaling bugs in
+the library: an unbounded `ANALYZE` cost on bulk SQLite loads, and an N+1 endpoint-existence check
+in batched edge creation. It also directly produced the `keySystemColumns` guidance and the
+PostgreSQL index-only-scan caveat in [Indexes](/performance/indexes#covering-indexes). The
+benchmark source lives in `packages/benchmarks/src/real/` in the repository.
+
 ### Running benchmarks locally
 
 ```bash

packages/benchmarks/.gitignore

@@ -0,0 +1,3 @@
+# Dated real-workload benchmark run bundles (docs/design/benchmark-program-plan.md).
+# The one-line-per-run reports/history.jsonl append log stays in git.
+bench-results/

packages/benchmarks/README.md

@@ -0,0 +1,180 @@
+# TypeGraph benchmarks
+
+Two benchmark programs live in this package:
+
+- **`src/*-bench.ts` + `src/main.ts`** — the synthetic perf-sanity suite
+  (`pnpm bench`, `pnpm perf:check`) that runs in CI as a regression guardrail
+  on a small seeded graph. Keep using this for fast, deterministic
+  before/after comparisons during development.
+- **`src/real/`** — the real-workload benchmark program described below:
+  industry-standard datasets and queries, run against TypeGraph and named
+  competitors, with a fairness harness (row-count + value-level digest
+  parity, noisy-sample detection, a competitor doctor, and a
+  `summary.json` per run). This supersedes `neo4j-compare/` (a 7.2k-node
+  microbenchmark) for anything beyond order-of-magnitude anchors — see
+  that directory's README for what it still remains useful for (a
+  same-machine two-command sanity check with no dataset to fetch).
+
+The full program design lives in `docs/design/benchmark-program-plan.md`
+(gitignored, local to the repo checkout that approved it).
+
+## Real-workload benchmarks (`src/real/`)
+
+### Lane 1 — LDBC SNB Interactive short reads (IS1-IS7)
+
+The headline lane: LDBC SNB Interactive's seven short-read queries (point
+reads, neighbor lists, reply-chain walks), adapted to this schema — the
+closest standard workload to TypeGraph's operational profile. Every
+query's output fields and result ordering match the official LDBC
+reference implementation (verified against
+`ldbc/ldbc_snb_interactive_v1_impls`) — but the schema itself
+deliberately flattens two relationships the official schema models as
+edges into plain properties instead: `Person.cityId` (official IS1
+traverses `IS_LOCATED_IN` to a `City` node; this schema has no `City`
+node at all) and `Forum.moderatorId` (official IS6 traverses
+`HAS_MODERATOR`; see `src/real/schema/snb-graph.ts`'s module doc for why).
+Both are simplifications applied identically across all four engines, not
+a TypeGraph-specific shortcut — call this lane "LDBC SNB-derived," not a
+claim of full official schema conformance.
+
+**Engines** (docs/design/benchmark-program-plan.md's fair pairings):
+
+| Pairing | Product under test | Competitor |
+| --- | --- | --- |
+| Embedded | `typegraph-sqlite` (better-sqlite3, in-process) | `ladybugdb` (`@ladybugdb/core`, Kuzu-family, in-process) |
+| Server | `typegraph-postgres` (node-postgres, imperative docker container) | `neo4j` (pinned image, imperative docker container) |
+
+Every query is implemented through the TypeGraph query builder for the
+TypeGraph engines (never hand-written SQL — see
+`src/real/engines/typegraph-queries.ts`'s module doc) and through
+idiomatic Cypher for Neo4j/LadybugDB. `src/real/schema/snb-graph.ts`
+documents one schema-design note worth knowing before reading the query
+code: `replyOf`'s target is polymorphic (a Comment replies to either a
+root Post or another Comment), which TypeGraph resolves with a
+never-instantiated ontological `Message` supertype
+(`subClassOf(Post, Message)` / `subClassOf(Comment, Message)`) so the
+reply-chain walk can use `includeSubClasses` — Neo4j and LadybugDB don't
+need this, since multi-label nodes / multi-pair relationship tables handle
+polymorphic endpoints natively.
+
+#### Running it
+
+```bash
+# Preflight: which engines are actually runnable on this machine right now
+pnpm --filter @nicia-ai/typegraph-benchmarks bench:snb:doctor
+
+# Tiny committed fixture (31 persons, 40 posts, 105 comments — including a
+# dedicated 25-comment same-creationDate tie cluster, see "Parity gate"
+# below) — fast, always runnable in CI regardless of Docker/optional-package
+# availability
+pnpm --filter @nicia-ai/typegraph-benchmarks bench:snb:smoke
+
+# Same, but exits non-zero on a genuine parity mismatch (row count or
+# value digest) between 2+ engines that ran (the CI-safe form: 0 or 1
+# runnable engines still exits 0)
+pnpm --filter @nicia-ai/typegraph-benchmarks bench:snb:smoke:check
+
+# Adversarial correctness check, independent of cross-engine consensus: one
+# fixture person's IS2 answer is known in advance (see "Parity gate" below),
+# checked against each doctor-runnable engine's actual result directly
+pnpm --filter @nicia-ai/typegraph-benchmarks bench:snb:verify-is2-tie-break
+
+# Real LDBC SF1 (~9.9k persons, ~361k knows (directed), ~1M posts, ~2.05M
+# comments). Takes under an hour on typical hardware — see
+# reports/snb-lane1-results.md for current load times and why they differ
+# this much across engines; that doc is the canonical, current source,
+# not duplicated here since these numbers drift with every run.
+pnpm --filter @nicia-ai/typegraph-benchmarks bench:snb:sf1
+
+# Or run it on a dedicated ephemeral EC2 instance instead of local
+# hardware (docs/ec2-benchmark-runner.md) — useful when local hardware is
+# contended, or for a stable, reproducible hardware profile.
+pnpm --filter @nicia-ai/typegraph-benchmarks bench:snb:sf1:ec2 -- \
+  --region=us-west-2 --subnet-id=<id> --security-group-id=<id> \
+  --iam-instance-profile=<name>
+```
+
+`bench:snb:sf1` looks for an extracted datagen directory at
+`~/.cache/typegraph/fixtures/ldbc-snb/sf1` and prints the exact download
+commands if it's missing:
+
+```bash
+mkdir -p ~/.cache/typegraph/fixtures/ldbc-snb/sf1 && cd ~/.cache/typegraph/fixtures/ldbc-snb/sf1
+curl -L -O https://datasets.ldbcouncil.org/snb-interactive-v1/social_network-sf1-CsvBasic-LongDateFormatter.tar.zst
+zstd -d --stdout social_network-sf1-CsvBasic-LongDateFormatter.tar.zst | tar -xf - --strip-components=1
+```
+
+The archive extracts into its own `social_network-...-LongDateFormatter/`
+subdirectory, not flat — `--strip-components=1` is required, not
+cosmetic; without it, `dynamic/person_0_0.csv` won't be where
+`resolveDatasetRoot` looks for it. Or point at an existing extract with
+`--data-dir <path>`. Real datasets are never committed — only the tiny
+smoke fixture under `fixtures/` is.
+
+Useful flags (all scripts): `--engines=typegraph-sqlite,neo4j` (default:
+all four), `--requests-per-query=N`, `--warmup-requests=N`, `--seed=N`,
+`--output=<dir>` (default `bench-results/current/snb-<profile>`,
+gitignored).
+
+#### What every run writes
+
+- `bench-results/current/snb-<profile>/results.json` — per-query,
+  per-engine latency stats (p50/p95/p99/mean/CV, flagged noisy above 25%
+  CV) plus the parity verdict (row count and value digest).
+- `bench-results/current/snb-<profile>/summary.json` — exact commands,
+  engine versions, dataset parameters, hardware, git commit.
+- `bench-results/current/snb-<profile>/competitor-doctor.json` — which
+  engines were runnable and why not, for the ones that weren't.
+- `reports/history.jsonl` — one append per engine per run, labels
+  `snb:IS1`..`snb:IS7`, alongside the synthetic suite's rows in the same
+  file.
+
+#### Parity gate
+
+Every engine executes the identical seeded request sequence (same sampled
+person/message ids, same order), so per-request results are directly
+comparable index-for-index. A query is "comparable" only when every engine
+that ran agrees on **both** the result-set size (row count) **and** a
+canonical value digest built from the query's actual LDBC-defined output
+fields (message content, names, ids, ordering — not just how many rows
+came back) — see `src/real/harness/parity.ts`'s `evaluateParity()` and
+`src/real/engines/types.ts`'s `canonicalDigest()`. Row-count agreement
+alone previously let a real semantic bug (all four engines measuring the
+wrong IS2 workload) go undetected for as long as it did, which is why the
+gate compares values now, not just counts. `--check` turns a genuine
+mismatch (2+ engines disagreeing, on either signal) into a non-zero exit;
+it never fires when fewer than 2 engines ran, so a no-Docker CI
+environment (only the two embedded engines runnable) still exits 0.
+
+Cross-engine agreement is still only consensus, not proof — this lane has
+shipped bugs (a shared wrong workload, a shared lexicographic-vs-numeric
+tie-break) that every engine reproduced identically, which parity alone
+can never catch. `bench:snb:verify-is2-tie-break` checks against a known
+answer instead: the smoke fixture's dedicated tie-cluster person (25
+same-creationDate comments, `dataset/smoke-fixture-constants.ts`) has an
+IS2 result computable in advance, since an identical creationDate leaves
+ascending message id as the only tie-break — each doctor-runnable engine's
+actual result is checked against that oracle directly, independent of
+what any other engine returned.
+
+The EC2 runner (`bench:snb:sf1:ec2:collect`) additionally requires the
+full four-engine set to have actually produced results — see
+`src/real/ec2/run-sf1-ec2.ts`'s `collect()` doc comment for why a paid,
+multi-hour run can't tolerate the same partial-engine leniency a local/CI
+invocation intentionally allows.
+
+#### Competitor doctor
+
+`src/real/harness/doctor.ts` checks Docker, `pg`/`better-sqlite3`,
+`neo4j-driver`, `@ladybugdb/core`, and image cache status, and records
+every missing piece as an explicit `failed`/`skipped` row — never a silent
+skip. The harness only attempts engines the doctor reports runnable.
+
+## Adding a new lane
+
+Lanes 2-4 (LinkBench request mix, LSQB, adversarial shapes) are planned in
+`docs/design/benchmark-program-plan.md` and land as separate PRs. The
+shared harness in `src/real/harness/` (stats, parity, doctor, summary,
+history, imperative Postgres container launcher) is written to be
+lane-agnostic — a new lane adds its own dataset loader, schema, and engine
+query implementations under `src/real/`, and reuses the harness as-is.