Revise and improve the benchmarks

Author: habedi

README.md

@@ -69,13 +69,18 @@ That's achieved by:
   never leak between engines, and the peak RSS reported per engine is attributable to that engine alone.
 - **Statistics.** Per query: a cold run (first execution after build) is reported separately; timed rounds
   run with the garbage collector disabled until both a minimum round count and a time budget are met; the
-  report shows median latency with a 95% confidence interval, and the plot carries p25 to p75 whiskers.
+  report shows median latency with a distribution-free 95% confidence interval for the median (the
+  order-statistic method, not a normal approximation on the mean), and the plot carries p25 to p75 whiskers.
 - **Honest comparisons.** Engines are labeled by kind (embedded / in-memory / client-server) and ingestion
   method; load times are never ranked across kinds, the client-server network round-trip caveat is stated
   in every report, and Neo4j's server memory settings are captured from the live server into the results.
+- **Indexing differences.** Index models differ by engine and cannot be fully equalized: IssunDB
+  auto-indexes every scalar property, Neo4j uses a uniqueness index on `id` plus an explicit range index
+  on the filtered column, Ladybug indexes only its primary key, and lance-graph holds no index. The report
+  spells this out so a filtered-query result is read as the engine's indexing model, not raw speed alone.
 - **Determinism.** The dataset is generated from a single seed, byte-for-byte reproducible, with edge rows
   shuffled so no engine gains a locality advantage from sorted insertion order. Hardware (CPU model, cores,
-  and RAM) is recorded in every results file.
+  and RAM) is recorded in every result file.
 - **Scaling.** `make sweep` benchmarks a series of dataset scales and plots median latency vs scale per
   query, so results are never a single-scale snapshot.
 

graphbench/_worker.py

@@ -38,11 +38,32 @@ def _percentile(ordered: list[float], pct: float) -> float:
     return ordered[rank]
 
 
+def _median_ci(ordered: list[float]) -> tuple[float, float]:
+    """Distribution-free 95% CI for the *median* via the binomial (order-statistic)
+    method, normal-approximated.
+
+    The reported point estimate is the median, so its interval should be a median CI,
+    not `1.96*std/sqrt(n)` (a CI for the *mean* that also assumes normality). Latency
+    samples are right-skewed, so this nonparametric interval, read straight off the
+    order statistics, is the honest choice. Returns absolute (lo, hi) bounds.
+    """
+    n = len(ordered)
+    if n < 2:
+        return ordered[0], ordered[0]
+    half = 1.96 * math.sqrt(n)
+    lo_rank = math.floor((n - half) / 2.0)  # 1-indexed lower order statistic
+    hi_rank = math.ceil((n + half) / 2.0) + 1  # 1-indexed upper order statistic
+    lo_idx = min(max(lo_rank - 1, 0), n - 1)
+    hi_idx = min(max(hi_rank - 1, 0), n - 1)
+    return ordered[lo_idx], ordered[hi_idx]
+
+
 def _stats(samples: list[float]) -> dict:
     ordered = sorted(samples)
     n = len(samples)
     mean = statistics.fmean(samples)
     std = statistics.stdev(samples) if n > 1 else 0.0
+    ci_lo, ci_hi = _median_ci(ordered)
     return {
         "rounds": n,
         "min_ms": round(ordered[0], 4),
@@ -52,7 +73,9 @@ def _stats(samples: list[float]) -> dict:
         "p95_ms": round(_percentile(ordered, 95), 4),
         "mean_ms": round(mean, 4),
         "std_ms": round(std, 4),
-        "ci95_ms": round(1.96 * std / math.sqrt(n), 4) if n > 1 else 0.0,
+        # 95% CI for the median (order-statistic method), as absolute bounds.
+        "ci_lo_ms": round(ci_lo, 4),
+        "ci_hi_ms": round(ci_hi, 4),
     }
 
 

graphbench/engines/issundb_engine.py

@@ -1,10 +1,14 @@
 """IssunDB adapter.
 
-IssunDB is an embedded engine with no bulk COPY path, so nodes and edges are inserted
-one row at a time through `add_node`/`add_edge`. IssunDB allocates its own NodeId per
-node, so the build keeps a per-label map from the dataset id to the allocated NodeId
-and resolves edge endpoints through it. The dataset id is stored as a node property so
-queries can return it. The binding returns query results as JSON of the shape
+IssunDB is an embedded engine with a bulk `IMPORT DATABASE` path, so ingestion is one
+import call over a generated Parquet/JSONL bundle rather than per-row inserts. IssunDB
+allocates its own NodeId per node, so the build offsets each label's dataset id into a
+globally unique `_id` (the imported NodeId) while keeping the original `id` as a node
+property, and resolves edge endpoints through the same offsets. The `id` property is
+what the catalog's Cypher filters on; IssunDB auto-indexes every scalar node property,
+so those filters become index/range scans with no explicit index DDL (an explicit
+node `CREATE INDEX` would provision a full-text index instead, which the workload does
+not use). The binding returns query results as JSON of the shape
 `{"columns": [...], "records": [{"values": [...]}]}`.
 """
 
@@ -63,10 +67,12 @@ def build(self, data_dir: Path) -> BuildResult:
             current_offset += (max_id if max_id is not None else 0) + 1
 
         # 3. Process and write node Parquet files with _id column
+        n_node_rows = 0
         for label in self.schema.nodes:
             parquet_path = data_dir / "nodes" / f"{label.name}.parquet"
             dst_parquet = import_dir / f"{label.name}.parquet"
             df = pl.read_parquet(parquet_path)
+            n_node_rows += df.height
             df = df.with_columns(
                 (
                     pl.col(self.schema.id_column).cast(pl.Int64) + offsets[label.name]
@@ -77,10 +83,12 @@ def build(self, data_dir: Path) -> BuildResult:
 
         # 4. Convert and write edge tables with offset endpoints
         edges_start = time.perf_counter()
+        n_edge_rows = 0
         for rel in self.schema.rels:
             parquet_path = data_dir / "edges" / f"{rel.name}.parquet"
             jsonl_path = import_dir / f"{rel.name}.jsonl"
             df = pl.read_parquet(parquet_path)
+            n_edge_rows += df.height
             df = df.with_columns(
                 [
                     (
@@ -118,9 +126,15 @@ def build(self, data_dir: Path) -> BuildResult:
         # Clean up import temp files
         shutil.rmtree(import_dir)
 
-        # Split query time and attribute preparation to edges_seconds and nodes_seconds
-        nodes_seconds = nodes_prep_time + query_time * 0.5
-        edges_seconds = edges_prep_time + query_time * 0.5
+        # `IMPORT DATABASE` is a single bulk call whose internal node and edge phases
+        # are not separately measurable, so its time is apportioned across the two
+        # reported phases by row volume (not an arbitrary 50/50 split). Each phase
+        # also carries its own deterministic preparation cost (the offset rewrite and
+        # the JSONL conversion), which is genuinely per-phase and measured directly.
+        total_rows = n_node_rows + n_edge_rows
+        node_frac = n_node_rows / total_rows if total_rows else 0.5
+        nodes_seconds = nodes_prep_time + query_time * node_frac
+        edges_seconds = edges_prep_time + query_time * (1.0 - node_frac)
 
         return BuildResult(nodes_seconds, edges_seconds)
 

graphbench/engines/neo4j_engine.py

@@ -5,11 +5,14 @@
 environment: NEO4J_URI (default bolt://localhost:7687), NEO4J_USER (default neo4j),
 and NEO4J_PASSWORD (default password).
 
-Ingestion wipes the database, creates a uniqueness constraint per label, then batches
-`UNWIND` + `CREATE` for nodes and edges (the dataset is pre-deduplicated and the
-database starts empty, so the slower MERGE is unnecessary). The dataset id is stored
-as the `id` property on every node, so the catalog's Cypher (which filters on `id`)
-is identical to the other engines and never touches Neo4j's internal element id.
+Ingestion wipes the database, creates a uniqueness constraint per label plus a range
+index on each non-id property the workload filters on, then batches `UNWIND` +
+`CREATE` for nodes and edges (the dataset is pre-deduplicated and the database starts
+empty, so the slower MERGE is unnecessary). The dataset id is stored as the `id`
+property on every node, so the catalog's Cypher (which filters on `id`) is identical
+to the other engines and never touches Neo4j's internal element id. IssunDB
+auto-indexes every scalar property, so indexing the filter columns here keeps the
+filtered-query comparison about execution rather than a missing index.
 
 `server_info` reports the server's memory settings via SHOW SETTINGS so published
 results carry the Neo4j configuration they were measured against.
@@ -37,7 +40,16 @@ def _chunks(rows: list, size: int):
 class Neo4jEngine(Engine):
     name = "neo4j"
     kind = "server"
-    build_method = "batched UNWIND+CREATE over Bolt, uniqueness constraint per label"
+    build_method = (
+        "batched UNWIND+CREATE over Bolt, uniqueness constraint per label, "
+        "range index on filter columns"
+    )
+    # Range indexes on the non-id properties the catalog filters on, so Neo4j seeks
+    # rather than scanning. IssunDB auto-indexes every scalar property, so without
+    # these the filtered queries would measure a missing index, not execution speed.
+    # Person.age (used by age_band_by_country) is the only selective filter at scale;
+    # the name lookups hit tiny fixed-size tables, so they are left unindexed.
+    _FILTER_INDEXES = (("Person", "age"),)
 
     def __init__(self, schema: Schema, workdir: Path):
         super().__init__(schema, workdir)
@@ -67,6 +79,13 @@ def _reset(self) -> None:
                 f"CREATE CONSTRAINT IF NOT EXISTS FOR (n:{label.name}) "
                 f"REQUIRE n.{self.schema.id_column} IS UNIQUE"
             )
+        # Created on the empty database so the index build folds into node load,
+        # the same way the uniqueness constraints (and IssunDB's auto-index) do.
+        for label, prop in self._FILTER_INDEXES:
+            self._session.run(
+                f"CREATE INDEX {label.lower()}_{prop} IF NOT EXISTS "
+                f"FOR (n:{label}) ON (n.{prop})"
+            )
 
     def build(self, data_dir: Path) -> BuildResult:
         self._reset()

graphbench/report.py

@@ -153,9 +153,9 @@ def to_markdown(results: dict, baseline: str | None = None) -> str:
         lines.append(f"| {query} | " + " | ".join(cells) + " |")
     lines.append("")
 
-    # Warm latency: median with 95% CI, speedup vs baseline.
+    # Warm latency: median with a 95% CI for the median, speedup vs baseline.
     lines.append(
-        "## Warm Latency, median ms +/- 95% CI"
+        "## Warm Latency, median ms [95% CI of median]"
         + (f", speedup vs {baseline}" if baseline else "")
     )
     lines.append("")
@@ -176,10 +176,11 @@ def to_markdown(results: dict, baseline: str | None = None) -> str:
                 q = results["engines"][name].get("queries", {}).get(query)
                 cells.append("ERR" if isinstance(q, dict) and "error" in q else "n/a")
                 continue
-            ci = _query_stat(results, name, query, "ci95_ms")
+            ci_lo = _query_stat(results, name, query, "ci_lo_ms")
+            ci_hi = _query_stat(results, name, query, "ci_hi_ms")
             text = f"{val:.2f}"
-            if ci is not None:
-                text += f" +/-{ci:.2f}"
+            if ci_lo is not None and ci_hi is not None:
+                text += f" [{ci_lo:.2f}, {ci_hi:.2f}]"
             if baseline and name != baseline and base_val and val > 0:
                 text += f" ({base_val / val:.1f}x)"
             if min_lat is not None and val == min_lat:
@@ -218,7 +219,23 @@ def to_markdown(results: dict, baseline: str | None = None) -> str:
         "- Parameterized queries rotate their literal values across rounds to "
         "defeat plan/result caches. Whole-graph aggregations (top_followed, "
         "two_hop_paths, ...) have no parameters, so their statement is necessarily "
-        "identical every round."
+        "identical every round; the warmup rounds equalize plan/parse caches across "
+        "engines, and none of the engines benchmarked here enable a result cache by "
+        "default, so the identical statement is recomputed each round rather than "
+        "served from a memoized result. These queries are not parameterized because a "
+        "filter would change which physical operator runs (e.g. a filtered count no "
+        "longer exercises the unfiltered path-count kernel), measuring a different "
+        "query rather than defeating a cache."
+    )
+    lines.append(
+        "- Indexing models differ by engine and are not equalized, because they "
+        "cannot be: IssunDB auto-indexes every scalar node property, so its "
+        "equality and range filters become index/range scans with no user action; "
+        "Neo4j has a uniqueness index on `id` per label plus an explicit range index "
+        "on the filter column (Person.age); Ladybug indexes only its primary key "
+        "(`id`); lance-graph is an in-memory engine with no indexes. Filtered queries "
+        "(point_lookup, age_band_by_country) therefore reflect each engine's indexing "
+        "model, not raw execution speed alone."
     )
     for name in engines:
         server_info = results["engines"][name].get("server_info") or {}