feat: add LDBC SNB Interactive short-read benchmark (Lane 1)

[pdlug]

First lane of the real-workload benchmark program: the LDBC SNB Interactive short-read queries (IS1-IS7), run against four engines under a shared fairness harness, at both SF1 and SF10 scale.

• New packages/benchmarks/src/real/: schema (schema/snb-graph.ts), dataset streaming reader (dataset/ldbc-csv.ts) + a tiny committed smoke fixture, four engine drivers (engines/typegraph-sqlite.ts, typegraph-postgres.ts, neo4j.ts, ladybug.ts), and a lane-agnostic harness (harness/: stats with CV-based noisy-sample flagging, value-level parity gate, competitor doctor, summary.json writer, history.jsonl appender).
• TypeGraph engines drive IS1-IS7 entirely through the query builder — no hand-written SQL (engines/typegraph-queries.ts's module doc explains the .prepare()/.execute() fairness rationale and the Message ontological-supertype workaround for the polymorphic replyOf edge).
• Neo4j and LadybugDB launch imperatively (docker run + tmpfs / in-process embedded) and implement the same 7 queries in native Cypher, using their own native handling of polymorphic node/edge types (multi-label nodes, multi-pair REL TABLEs) instead of TypeGraph's ontology workaround.
• New packages/benchmarks/src/real/ec2/: an EC2 runner (docs/ec2-benchmark-runner.md) so SF1/SF10 can run on a dedicated, ephemeral cloud instance instead of local hardware. SSM Run Command is the default, no-key-pair control channel; generic enough for anyone who clones the repo to point at their own AWS account. An opt-in --ssh-public-key-path diagnostic fallback exists for when SSM itself is what's broken. Hardened over five rounds of review — see below.
• CI-safe: bench:snb:smoke:check runs the tiny fixture end-to-end; a competitor-doctor preflight reports missing Docker/packages as explicit failed rows (never silent), and the run stays green with 0 or 1 runnable engines — it only fails on a genuine parity mismatch (row count or value digest) between 2+ engines that actually ran.
• Incidental fix: the root .gitignore's unscoped reports/ rule (meant for Stryker mutation-testing output) had been silently excluding packages/benchmarks/reports/history.jsonl from every commit since it was introduced — scoped to packages/typegraph/reports/ so both the existing synthetic perf suite's history and this lane's history are actually committed going forward.

One packages/typegraph change: walAutocheckpointPages, a new createLocalSqliteBackend pragma option (see "SF10" below for why). This branch and main independently arrived at the same fix; main's version (merged separately as #251, with an upper-bound validation this branch's original commit didn't have) is now what this branch carries after reconciling with main — see "main reconciliation" below. Everything else is benchmarks-only; three other library fixes were needed along the way but each landed separately in its own PR (#226, #227).

Review history: 5 rounds, all fixed and re-verified with a fresh EC2 run

Five rounds of review found issues serious enough that no query-latency number from before this PR's fixes could be trusted. All findings below are fixed; a fresh SF1 + SF10 EC2 run on the fully-fixed commit (2bc7f74f) has since replaced every invalidated number — see Results below.

Round 1 (6 findings)

1. IS2 implemented the wrong workload, in all three engine drivers. Checked against the official LDBC reference Cypher (ldbc/ldbc_snb_interactive_v1_impls): IS2 is "recent messages of a person" — the given person's own messages, tie-broken messageId ASC. This PR's drivers traversed to that person's friends and measured messages they authored, tie-broken DESC — a materially different, heavier workload. Every engine shared the identical mistake, which is exactly why row-count parity never caught it. Fixed in typegraph-queries.ts, neo4j.ts, and ladybug.ts.
2. materializeIndexes()'s best-effort result was silently discarded in both the SQLite and Postgres drivers. Fixed: assertMessageIndexMaterialized() (schema/snb-graph.ts) checks the result and throws if the index wasn't created or already materialized.
3. collect() (run-sf1-ec2.ts) extracted the benchmark's exit code but never checked it. Fixed: throws (after writing whatever partial artifacts exist) on non-Success SSM status, a missing exit-code marker, or a nonzero exit code.
4. --profile=sf10 always defaulted to c7i.4xlarge, the same 32GB instance type that OOM'd on four separate SF10 attempts. Fixed: profile-aware default (r7i.4xlarge for sf10); --instance-type still overrides explicitly.
5. Neo4j's snb_post_id/snb_comment_id constraints had no current query-time consumer (every by-id match goes through the shared :Message label). Fixed: removed.
6. Release-facing copy didn't match the branch (stale "no typegraph changes," "fastest across the board" overclaim, a nonexistent script reference, a stale "no direct measurement" line). Fixed throughout the results doc, the EC2 guide, and this PR body.

Round 2 (5 more findings — the smoke run in round 1 proved execution and row-count parity, not LDBC semantic parity)

1. IS2 was still incomplete after round 1's fix. The official query also returns message content and root-post-author names; TypeGraph omitted content, Neo4j and LadybugDB omitted both content and author names. IDs were also tie-broken with plain localeCompare ("message:10" sorting before "message:2") despite official ids being numeric. Fixed: content + names added to all three engines' IS2; a shared compareIdsAscending() helper (engines/types.ts) makes every id tie-break numeric-aware.
2. IS3/IS5/IS6/IS7 weren't equivalent across engines either, and row-count-only parity could never have caught any of it: TypeGraph's IS3 had no defined ordering at all; Neo4j's and LadybugDB's IS6 omitted forum id/title; TypeGraph's and LadybugDB's IS7 omitted reply content and author names. Fixed: every query across all three engines now returns exactly the official LDBC output fields, correctly ordered.
3. The parity gate itself only ever compared row counts. Upgraded to a value-level canonical digest per row (SnbQueryResult.digest, canonicalDigest() in engines/types.ts, harness/parity.ts) — engines can no longer agree on count while disagreeing on field values, omitted fields, or order. Verified on the smoke fixture: this immediately caught one real remaining bug (TypeGraph's IS3 digest used the field name personId, Neo4j/LadybugDB used id — same values, incomparable digests) before landing at all 7 queries passing true value-level parity across all four engines.
4. A successful SSM command with no results could still report success (resultsText === "{}" didn't feed the failure predicate). Fixed: !hasParseableResults added to collect()'s failure check.
5. Failed runs could contaminate canonical history — reports/history.jsonl was appended before validating the run succeeded. Fixed: raw history lines are always preserved locally (run-scoped, not shared) for post-mortem; the canonical file is only appended after every success condition passes. A related, smaller finding — four smoke-test history rows this session accidentally generated recorded a git SHA that didn't contain the uncommitted fixes producing them — was caught and reverted, not committed.
6. (P2) Neo4j's fairness label still described the just-removed Post/Comment constraints, and the results doc claimed Neo4j's load time was unaffected by round 1's fixes when constraint creation actually runs inside its timed load(). Both fixed.

Round 3 (2 more P1s — round 2's own IS2 fix wasn't actually correctness-proof — plus 3 P2s)

1. Round 2's IS2_CANDIDATE_LIMIT buffer (a fixed-size native LIMIT re-sorted numerically in JS) wasn't provably correct. Confirmed with a concrete counter-example: a same-creationDate tie cluster larger than the buffer (e.g. 30 messages sharing one timestamp, ids 1-30) causes the native lexicographic LIMIT to exclude genuinely-top-10 candidates (numeric ids 3-9) before the JS resort ever sees them — even a correct numeric resort of a wrong candidate set produces a wrong answer. Fixed by removing the native LIMIT entirely across all three engines: IS2 now fetches all of a person's own messages (unlimited), sorts numerically, and slices to 10 purely in JS — correct regardless of tie-cluster size.
2. collect()'s success predicate still didn't require a parseable summary.json or non-empty history.jsonl lines — only results.json was checked. Fixed: hasParseableSummary/hasHistoryLines added alongside the existing checks; a run is only "successful" when all three reproducibility artifacts are present.
3. (P2) Stale "row-count parity" wording throughout cli.ts, request-plan.ts, snb-short-reads.ts's console output, the README, and the results doc — corrected to describe both row-count and value-digest parity, matching round 2's actual gate upgrade.
4. (P2) README's SF1 load-time numbers were pre-fix stale (75-80 min) — updated to the current numbers (SQLite ~40 min, Postgres ~12 min).
5. (P2) Neo4j's loader was described as "batched UNWIND ... IN TRANSACTIONS" in both the README and results doc when it's actually the offline neo4j-admin database import — corrected in both places. Also softened the lane's "official LDBC" framing: the README now explicitly names the two relationships this schema deliberately flattens into properties (Person.cityId, Forum.moderatorId) instead of modeling as edges, and calls the lane "LDBC SNB-derived" rather than a full schema-conformance claim — the query fields and ordering are still verified against the official reference; the schema-shape simplification is a separate, honestly-disclosed thing.

Round 4 (2 more P1s — round 3's own IS2 fix traded correctness for a new fairness problem)

1. Round 3's fix (removing IS2's native LIMIT entirely) was correctness-proof but changed the measured workload. The official query applies its own ORDER BY messageCreationDate DESC, messageId ASC LIMIT 10 engine-side, before the root-post-author walk. Fetching every message a person ever authored instead — every engine, unconditionally — transferred full message content over the network for engines that pay a real round trip (Postgres, Neo4j), disproportionately penalizing them versus the embedded engines, and diverged from official semantics. The repo's own index documentation notes some IS2 candidate pools run to thousands of rows, so this wasn't a theoretical concern. Root-caused properly instead of patched again: dataset/ldbc-csv.ts now zero-pads every id's numeric portion to a fixed width, so a plain lexicographic ORDER BY id ASC (SQL or Cypher alike) already agrees with numeric order, for any tie-cluster size — this is what actually broke in round 2/3, not the presence of a LIMIT per se. With that fixed, native ORDER BY ... LIMIT 10 is restored everywhere: TypeGraph/LadybugDB fetch each of Post/Comment's own top 10 and merge (any message in the true top 10 of the union must be in its own type's top 10 too, so this is provably equal to the true top 10, for any candidate-pool or tie-cluster size, transferring at most 20 rows regardless of how many messages the person actually authored); Neo4j's unified :Message label needs only one query.
   • Also added: bench:snb:verify-is2-tie-break, an adversarial correctness oracle independent of cross-engine consensus. Every engine agreeing has already gone wrong twice in this lane (the friend-workload bug, the lexicographic-tie-break bug) — consensus alone doesn't prove correctness. The smoke fixture now includes a dedicated person who authors 25 same-creationDate comments; with no other tie-breaking signal, that person's correct IS2 answer (the cluster's 10 smallest message ids) is knowable in advance, and the new script checks each doctor-runnable engine's actual result against that known answer directly. All 4 engines pass.
2. The paid EC2 path could still report success with fewer than the full four-engine set. Doctor-unavailable engines are filtered out of the run rather than recorded as a failure — correct behavior for a local/CI invocation (a no-Docker environment should still exit 0 on the two embedded engines), wrong for a run whose entire point is a complete four-engine comparison: a container failing to start on the instance would silently produce a smaller-than-requested but still "successful" run. Fixed: collect() now parses results.json.engines and requires all four canonical engine names to be present before declaring success, and additionally fetches + preserves competitor-doctor.json locally (success or failure) so an incomplete run is diagnosable without re-connecting to the instance.

Round 5 (1 more P1 — round 4's own new oracle test didn't actually test anything)

1. The tie-break oracle's 25 tie-cluster ids were one contiguous 3-digit range (120..144), so unpadded lexicographic order and numeric order coincide by construction — same-length numeral strings always compare identically both ways. The check would pass whether or not dataset/ldbc-csv.ts's zero-padding fix was actually applied, catching nothing; it was consensus theater dressed up as an oracle. Fixed: split into two blocks of different digit widths (3-digit 120..129, 4-digit 1000..1014) — unpadded lexicographic order now ranks every 4-digit id ahead of every 3-digit one ("1000" < "120" character-by-character), so an unpadded engine returns the wrong answer (1000..1009) instead of the true one (120..129). Verified directly, not just reasoned about: temporarily reverted the padding fix, confirmed all 4 engines fail with exactly that predicted wrong answer, then restored it and confirmed all 4 pass again.

Results

Full writeup: packages/benchmarks/reports/snb-lane1-results.md. Both scales below are fresh runs on commit 2bc7f74f (every fix from all 5 rounds in place): 0 engine failures, 7/7 queries at full value-level digest parity (not row-count alone) at both scales. Single run each — not yet a publishable comparison per the program plan's multi-run bar (tracked in the results doc's Next steps).

SF1 (9,892 persons, 1,003,605 posts, 2,052,169 comments) — 7/7 comparable=yes, 0 failures

Engine	Load time
ladybugdb	41.6 s
neo4j	71.4 s (1.2 min)
typegraph-postgres	669.3 s (11.2 min)
typegraph-sqlite	587.1 s (9.8 min)

SQLite and Neo4j both load dramatically faster than the last SF1 run reported in this PR (SQLite ~4.1x, Neo4j ~4.3x) — not a new optimization, but the SF10 investigation's covering-index-deferral and constraint-removal fixes finally showing up in an SF1 number for the first time (that run predates both). Postgres and LadybugDB, whose load paths weren't touched by either fix, measured 3.5% and 22.7% faster than the previous run, respectively — cross-commit observations that can't establish run-to-run variance or causality on their own. See the results doc for the full query-latency table (IS2 still dominates every engine's latency, as structurally expected — a top-10 selection plus a per-message root-post-author walk).

SF10 (65,645 persons, 7,435,696 posts, 21,865,475 comments) — 7/7 comparable=yes, 0 failures

Getting one clean, fast SF10 run originally took eight EC2 attempts across two root-causing efforts — a memory-exhaustion failure disguised as a networking problem (attempts 1-5), then an EBS gp3 volume's default IOPS ceiling that made a correctly-implemented, locally-validated SQLite checkpoint-tuning fix show zero real-world improvement (attempts 6-8). Full narrative in the results doc. This run is on commit 2bc7f74f (all 5 review rounds); attempt 8 above was commit a58ae38e (load-time fixes only, predating those rounds) — a consistent second load-time data point, not the same commit:

Engine	Load time
ladybugdb	354.0 s (5.9 min, +10.7% vs. attempt 8)
neo4j	404.2 s (6.7 min, +8.8%)
typegraph-postgres	7,127.5 s (1.98 h, +7.4%)
typegraph-sqlite	10,929.4 s (3.04 h, -2.1%, i.e. faster)

The second run differed by -2.1% to +10.7% across engines. These provide a consistent second load-time point, but two cross-commit observations do not establish run-to-run variance or causality — the two runs are on different commits, so a small, real, commit-attributable effect can't be ruled out from two samples alone. With a fresh SF1 run now available under the identical (2bc7f74f) fixes, the results doc also derives a fair SF1-to-SF10 growth-rate comparison for the first time: Postgres grows almost exactly linearly with the ~10.65x comment-count increase; SQLite grows noticeably super-linearly (~18.6x), consistent with the documented wal_autocheckpoint tuning's own caveat that it "holds through roughly 50,000-100,000 pages, then regresses again" — worth a deeper look if pursued further, not investigated in this PR.

[pdlug]

SF10 IS2 latency cliff: root cause found and fixed

The first real SF10 EC2 run (10x SF1 data volume) surfaced a severe regression: IS2 (friends' recent messages) median latency jumped 689x on the SQLite backend (74ms → 51s) and 28x on Postgres (247ms → 7.0s), while Neo4j (7.3x) and LadybugDB (2.6x) scaled near-linearly over the same data growth. Everything else (IS1, IS3-IS7) stayed competitive.

Root cause (independently investigated and cross-checked against this codebase): the covering index this branch already added for the Post/Comment creationDate join (postByCreationDateIndex/commentByCreationDateIndex, keySystemColumns: ["id"]) carried id but not deleted_at/valid_from/valid_to — the three system columns every compiled query's soft-delete/temporal-validity predicate checks. Not truly covering, so every candidate message paid a heap-row fetch. Free at SF1 (table fits in page cache); catastrophic at SF10 (30-50GB table exceeding available cache, so every fetch becomes a random disk read). Measured: 30,897 → 734 page-cache misses per IS2 call (42x) once all three columns are added.

Fixed in two places:

• fix: cover valid_from on default edge indexes; add SQLite cache-size pragmas #245 (core library, off main): the same gap existed in TypeGraph's own default edge traversal indexes (from_idx/to_idx — missing valid_from, used by every edge query in every app, not just this benchmark). Also adds host-aware cacheSizeKib/mmapSizeBytes pragma options to createLocalSqliteBackend (SQLite's own 2MiB default cache being the other half of the problem).
• This branch (commit 92670c5): extends keySystemColumns to all three columns, consolidates the previously-duplicated Post/Comment index declarations (byte-identical, non-partial — pure redundant write cost), sizes SQLite's cache_size host-aware in the engine driver, and runs VACUUM ANALYZE after Postgres bulk load (without it, Postgres can't serve an index-only scan at all regardless of index shape).

Full investigation write-up (ruled-out hypotheses, EXPLAIN QUERY PLAN evidence, the isolated-vs-combined-scale local repros): see the linked artifact in session notes.

Not yet done: a real SF10 re-run to confirm the fix closes the gap at true scale — local repros verified the index-shape change and plan flip, but held off on the ~10-13h EC2 cost pending approval. Tracked as the next step before this branch's results can be trusted for the public comparison draft.

[pdlug]

Synced the PR #245 review fixes onto this branch's copy of the same commit (`a42151c`) — `from_idx`/`to_idx` now also cover the join's target-id column (a second real gap found in review, beyond the `valid_from` fix), plus the `cacheSizeKib` validation and changeset guidance fixes. Full SQLite suite: 4,539 passed, 0 failed.