topic 44: e-graphs as a database, and the two testing systems topic 16 only named

.github/workflows/verify.yml

@@ -39,7 +39,7 @@ jobs:
       # takes precedence over .cargo/config.toml's ../.dlp-target. Necessary
       # because actions/cache rejects any path containing ".." — it logs
       # "Invalid pattern" and then silently caches nothing, so every run
-      # recompiles all 45 crates from scratch. The reason the checked-in config
+      # recompiles all 46 crates from scratch. The reason the checked-in config
       # points outside the clone is mdbook, which never runs in this workflow.
       - name: Cache cargo
         uses: actions/cache@v6

CLAUDE.md

@@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 
 ## Project Overview
 
-A self-paced database-internals learning path, rendered as an mdBook (`book.toml`, content in `topics/`, exercises in `capstone/`). `PLAN.md` is the curriculum plan — 44 topics, the source of truth. `PROGRESS.md` tracks status and capstone milestones; `SESSION-LOG.md` is the detailed build log, one entry per topic, newest first. `CONTRIBUTING.md` documents the topic package format and the conventions below.
+A self-paced database-internals learning path, rendered as an mdBook (`book.toml`, content in `topics/`, exercises in `capstone/`). `PLAN.md` is the curriculum plan — 45 topics, the source of truth. `PROGRESS.md` tracks status and capstone milestones; `SESSION-LOG.md` is the detailed build log, one entry per topic, newest first. `CONTRIBUTING.md` documents the topic package format and the conventions below.
 
 ## Working rules
 

CONTRIBUTING.md

@@ -125,7 +125,7 @@ python3 tools/check-reading-depth.py --stats                      # rollout prog
 ```
 
 A second workflow ([verify.yml](.github/workflows/verify.yml)) runs
-`./verify.sh --summary` and a `-D warnings` build of all 45 crates on every push and
+`./verify.sh --summary` and a `-D warnings` build of all 46 crates on every push and
 pull request. It is the gate that keeps the repo's central claim true, so a lane that
 stops running is a red build. Note that `cargo test` is deliberately **not** a gate:
 the stub tests are the specification and are supposed to fail on a fresh clone.

FINDINGS.md

@@ -5,7 +5,8 @@ whole argument for this format over a reading list in one table: a link
 collection cannot be wrong in a way you can detect, and every row below can.
 
 Every figure here comes from a benchmark in this repo, measured on an **Apple M3
-Pro (5P + 6E, 36 GB)** on 2026-07-28. Generators are seeded, so counts,
+Pro (5P + 6E, 36 GB)** on 2026-07-28, except topic 44's row, measured on
+the same machine on 2026-08-26. Generators are seeded, so counts,
 ratios and distributions reproduce exactly; timings will differ on your
 hardware. Run everything with `./verify.sh`, one topic with `./verify.sh 12`, or
 `./verify.sh --list` to see every lane.
@@ -59,6 +60,7 @@ instead.
 | 41 | [On-Chain Analytics](topics/41-onchain-analytics/README.md) | The industry-default haircut rule marks **98% of addresses** tainted from one theft; 658 of them are under 0.1% tainted. An 1816 court case does better. | `./verify.sh 41` |
 | 42 | [Recommendations & Social](topics/42-recommendations-social/README.md) | Recommending bestsellers to everyone gets **34.0% hit-rate@50** with **92.3% overlap** with the global bestseller list. Popularity is not a weak baseline. | `./verify.sh 42` |
 | 43 | [Ops Dependency Graphs](topics/43-ops-dependency-graphs/README.md) | One gray failure: **34 of 55 services alert** and the broken one is not among them — it ranks 35th by failure count, 41st by error rate, at exactly the baseline. | `./verify.sh 43` |
+| 44 | [E-graphs as a Database](topics/44-egraphs-egglog/README.md) | `f(a, g(a))` has **N matches** and costs a backtracking matcher **N²+N+1** units of work; generic join does **5N**. At N=1600: 2,561,601 against 8,000, a measured **21.7×**. Make the pattern linear and generic join is **1.8× slower** — the win is avoided waste, not speed. | `./verify.sh 44` |
 
 ## How to read this table
 

PLAN.md

@@ -42,6 +42,7 @@ flowchart TD
     end
     subgraph CORRECT["Correctness"]
         T16["16 testing"] --> T21["21 formal methods"]
+        T21 --> T44["44 e-graphs as a database"]
     end
     subgraph STREAM["Streaming & temporal"]
         T27["27 incremental views"]
@@ -228,7 +229,7 @@ flowchart TD
 **Why:** The topic that separates hobby DBs from production DBs. Turso and FoundationDB made this their identity.
 
 - **Concepts:** deterministic simulation testing (DST), fault injection, property-based testing (proptest), fuzzing (cargo-fuzz/AFL), metamorphic testing (SQLancer's pivoted queries / TLP), Jepsen & elle (checking linearizability), model checking with TLA+ (taste of), SMT solvers (Z3): proving query rewrites equivalent (Cosette-style), checking optimizer rules and constraint/invariant satisfiability.
-- **Read code:** turso's simulator + DST setup (they blog about it), FoundationDB simulation docs, SQLancer, antithesis blog posts, redis `test/` harness, Z3 (`z3.rs` bindings; skim the tactic/solver architecture — treat Z3 itself as a masterclass codebase: it's a high-performance search engine over logic).
+- **Read code:** turso's simulator + DST setup (they blog about it), FoundationDB simulation docs, SQLancer, `HypothesisWorks/hypothesis` (the `internal/conjecture/` engine — the choice sequence, `sort_key`'s shortlex order, the shrink-pass determinism invariant, the `DataTree`, and swarm testing in `featureflags.py`), `antithesishq/antithesis-sdk-rust` (the open half of Antithesis: `Sometimes` assertions as a coverage property, the linker-assembled assertion catalog, guidance as a fitness signal, and the no-caching contract on `get_random` that betrays a *branching* rather than replaying simulation), redis `test/` harness, Z3 (`z3.rs` bindings; skim the tactic/solver architecture — treat Z3 itself as a masterclass codebase: it's a high-performance search engine over logic).
 - **Papers:** "Testing Database Engines via Pivoted Query Synthesis" (OSDI'20), "Finding Logic Bugs via TLP" (OOPSLA'20), Jepsen analyses (pick redis-raft and a graph DB one), "Z3: An Efficient SMT Solver" (TACAS'08), "Cosette: An Automated Prover for SQL" (CIDR'17).
 - **Build & bench:** add proptest model-checking to the capstone (graph ops vs an in-memory model oracle); build a mini DST harness (simulated clock + fault-injecting IO layer); fuzz your parsers (Cypher + page/SST decoders); use Z3 to verify two of your topic-10 rewrite rules are equivalent (and to find a counterexample when you break one on purpose).
 - **Capstone M16:** openCypher TCK subset runner as the correctness oracle + DST harness + fuzzers (the reference's `fuzz/` and `tck_done.txt` show the bar). Graduation of the correctness spine.
@@ -523,4 +524,16 @@ flowchart TD
 
 ---
 
+## 44. E-graphs as a Database: Relational E-matching & egglog
+
+**Why:** Topic 21 built the e-graph and measured what it repairs; this topic is the sequel, and it belongs in a database course rather than a compilers one, because the fix for the *next* bottleneck came from our literature. E-matching — pattern matching modulo equality — is **60–90% of equality saturation's run time** (POPL'22 §1, citing egg's own measurements), and it is a **conjunctive query**: the e-graph is a set of tables, a pattern is a query, and the *equality constraint* a backtracking matcher checks last is a join key. Measured lane 1 is the whole argument in two columns: on the POPL'22 Figure 2 e-graph (3N e-nodes standing for N²+2N terms) the pattern `f(a, g(a))` has **N matches and costs a backtracking matcher N²+N+1 units of work** — 2,561,601 at N=1600 — while generic join does **5N**, i.e. 8,000, for a measured **21.7×** at that size. And the second table is the honest one: rename the repeated variable and the pattern goes *linear*, every candidate becomes an answer, and generic join is **1.8× slower** — the same result POPL'22 reports in Table 1's `Worst` column (0.76, and 0.03 with index building charged). Then egglog (PLDI'23) takes the last step: stop copying an e-graph into a database whenever you want to match and make the database primary, at which point Datalog's **semi-naive evaluation** applies — measured against a naive re-derivation that finds **20,008 matches and 100,040 probes for 8 new answers**. And SIGMOD'23's Free Join is where the asymptotics get their constants back.
+
+- **Concepts:** the **relational view of an e-graph** (POPL'22 §3.1 — one tuple per e-node, `R_f` of arity k+1, all ids canonical, and the hashcons invariant restated as a **functional dependency** from the children columns to the id column, §4.3); **structural vs equality constraints** and why a top-down walk can only exploit the first (§2.1), with **linear patterns** as the case where there is nothing to win; **unnesting** a pattern into a conjunctive query (Figure 8's `Aux`/`Compile`), which makes **multi-patterns free**; **conjunctive queries, the AGM bound and fractional edge covers** (the triangle at `M^1.5` against a binary plan's `M²`); **generic join** (Algorithm 1) and its two implementation requirements — intersect in `O(min |R_j.x|)` and reach a residual relation in constant time, which is what the trie index buys; the **complexity results** (Theorem 9 worst-case optimality, Theorem 10's `O(√(|Q(I)|·Π|Rᵢ|))`, which predicts *both* of this topic's lanes) and why NP-completeness (data vs query complexity) does not contradict them; **egglog** as Datalog plus two extensions — user-extensible equality and functions with a **`:merge` expression** — where a function is a *map* enforcing a functional dependency, `:merge` is the conflict policy, `min` is the join of a lattice ordered by worseness, and **`:merge = union` is congruence closure**, not implemented but derived (PLDI'23 §3.2–3.4); `:default` as **get-or-make-set**, which is `EGraph::add` arrived at from the other side; the inflationary consequence operator `T_P↑` and the rebuilding operator `R` (§4.2), including the footnote that egglog rules are **not always monotone**; **semi-naive evaluation** as m delta rules per rule (§4.3, Theorem 4.1), the duplicate derivations it necessarily produces, and its production form — a **timestamp column** plus a `GeConst` constraint that a clustered sort order turns into a binary search rather than a filter; the **engine** as an actual database (a sorted-writes table with a sharded hash index and a merge function, dense/sparse row subsets, a planner with **hypertree decomposition** by min-fill variable elimination and Yannakakis-style message variables, then per-bag join planning), **rebuilding compiled to a rule** and planned by the same planner, and a **union-find that unions by min id** rather than by rank because its real cost is the rebuild it triggers; **Free Join** (SIGMOD'23) — the design space parameterised by relations-and-attributes per join step, the **GHT** that is a hash table at two levels and a trie at one-tuple keys, plans as lists of nodes of subatoms with a **cover**, converting an existing binary plan and **factoring** it, **COLT** (a column-oriented *lazy* trie that materialises a level only when probed), vectorized execution, and the measured **2.94×/9.61× geometric means with a 0.85× minimum**.
+- **Read code:** `egraphs-good/egglog` — read it as a database, not as an e-graph library: `core-relations/src/free_join/plan.rs:1-46` (the best short description of a modern join planner in any source file), `core-relations/src/table/mod.rs:1-5` and `:136-152` (the table, and the deliberate ignorance of what a timestamp means), `:445-512` `fast_subset`, `core-relations/src/offsets/mod.rs:333` (`Subset::Dense | Sparse`), `core-relations/src/query.rs:252-256` (semi-naive as a cached plan plus a `GeConst`), `free_join/execute.rs:1418-1560` (the intersect stage, smallest-side-first at `:1464`), `egglog-bridge/src/lib.rs:932-1050` (congruence closure, compiled to rules), `union-find/src/lib.rs` entire (104 lines, and it explains why it declines union-by-rank). Then re-read `~/repos/egg` from topic 21 and note what the two designs share.
+- **Papers:** Zhang, Wang, Willsey & Tatlock, "Relational E-matching" (POPL'22 — Figure 2, §2.1's constraint taxonomy, §3.1–3.2, §3.4's Theorems 9 and 10, Table 1 including its `Worst` column); Zhang, Wang, Flatt, Cao, Zucker, Rosenthal, Tatlock & Willsey, "Better Together: Unifying Datalog and Equality Saturation" (PLDI'23 — §3's language tour, §4.2's two operators and footnote 4, §4.3's semi-naive, §5.3's 3.34×/9.27× attribution, §6's two case studies); Wang, Willsey & Suciu, "Free Join: Unifying Worst-Case Optimal and Traditional Joins" (SIGMOD'23 — Figure 1's design space, §3's GHT and plan language, §4.2's COLT, §5's JOB/LSQB evaluation); revisit "egg: Fast and Extensible Equality Saturation" (POPL'21) from topic 21.
+- **Build & bench:** lane 1 provided — a minimal e-graph (union-find, hashcons, rebuild to fixpoint) whose internals are visible enough to be walked *and* read as tables, egg's `Bind`/`Compare`/`Scan` VM as the backtracking baseline (with the op index, so the comparison is honest), Figure 8's unnesting, trie indexes, a most-constrained-first variable ordering and generic join — measured on both a non-linear and a linear pattern so the negative result is a column rather than a caveat; implement **semi-naive evaluation** (contracts: the delta rules' union, deduplicated, equals `matches(after) − matches(before)` as sets; 8 answers from a 24-tuple delta; probes at least 10× below the naive re-derivation) and **a left-deep binary-join plan for the triangle multi-pattern** (contracts: same substitution set as generic join; the largest materialised intermediate exceeds 4× the output, and grows as `E²/V` while the answer stays at `(E/V)³`), then measure both against the provided columns.
+- **Capstone M44:** the rewrite stage of the planner, priced — replace the capstone's hand-ordered rewrite pass with an e-graph stage whose patterns are matched **relationally** rather than by walking, and report both numbers (plan cost against the hand-ordered pass, match time against a backtracking matcher); timestamp the e-node table and run the saturation loop semi-naively, measuring iterations-to-saturation and total probes against the naive loop on the same rule set; and put one **cyclic** rewrite pattern in the rule set with the binary-join plan measured next to generic join, so the AGM bound is a column rather than a claim.
+
+---
+
 - FPGA / SmartNIC / computational storage offload (beyond GPU)

PROGRESS.md

@@ -4,7 +4,7 @@ Two different things get tracked here, and conflating them is misleading:
 
 - **Package** — does `topics/NN-name/` exist and hold up? That means a study
   guide, four to seven reading guides, `notes.md`, and an experiments crate
-  whose provided lane runs and whose numbers are recorded. All 44 are built;
+  whose provided lane runs and whose numbers are recorded. All 45 are built;
   `./verify.sh` re-derives every one of their measured lanes.
 - **Studied** — have *I* actually worked through the material and the two
   exercise lanes? That is a much smaller number, and it is the honest one.
@@ -60,6 +60,7 @@ is done.
 | 41 | On-Chain & Crypto Analytics (graph use case 4/6) | done | todo |  |
 | 42 | Recommendations & Social Graphs (graph use case 5/6) | done | todo |  |
 | 43 | Network & IT-Ops Dependency Graphs (graph use case 6/6) | done | todo |  |
+| 44 | E-graphs as a Database: Relational E-matching & egglog | done | todo |  |
 
 ## Capstone milestones (falkordb-rs-next-gen from scratch)
 
@@ -109,6 +110,7 @@ is done.
 | M41 provenance & identity (incremental FIFO taint queues in the property layer, maintained union-find cluster index, BlockSci-shaped columnar transaction store for scan queries) | 41 | todo |
 | M42 real-time recommendations (GraphJet-style temporal index segments + doubling edge pools, Pixie random-walk procedure with sub-linear step allocation and early stopping, TAO-shaped association-list API) | 42 | todo |
 | M43 observability path (trace ingest as an incrementally-maintained dependency graph with sketched edge weights, walk + Ferret localization procedures over the CSR, happened-before join operator with Pivot Tracing pushdown) | 43 | todo |
+| M44 relational rewrite stage (e-graph planner pass whose patterns compile to conjunctive queries and run through generic join, a timestamped e-node table driving a semi-naive saturation loop, and one cyclic rewrite pattern measured against a binary-join plan) | 44 | todo |
 
 ## Session log
 

README.md

@@ -2,7 +2,7 @@
 
 **A self-paced curriculum in database internals, where every claim is measured.**
 
-44 topics, from B-trees to GPU query execution to attack graphs. Each one walks you
+45 topics, from B-trees to GPU query execution to attack graphs. Each one walks you
 through the papers and the production code, then hands you a Rust benchmark that
 demonstrates the thing being claimed — so you finish with a number you produced
 yourself, not a fact you read.