bench: compute-bound benchmark harness with hand-written Python baselines

ROADMAP.md

@@ -18,10 +18,10 @@ Keep this a *forward-looking* backlog — do not let it grow back into a changel
   list slicing/splicing, not an append), folds inside in-file `module`s (P8 mangling), and anything the
   occurrence discipline can't prove. Pick one up only when a real hot fold rejects on it. (A
   persistent-map/HAMT `Map` would kill the O(n²) generally but still loses to a bare `dict` on this
-  pattern.) The ceiling framing stands and caps all perf work: Pyfun targets un-JIT'd CPython, so the goal
-  is "as fast as idiomatic hand-written Python," and a genuinely hot inner loop still belongs behind an
-  `extern` — the further lowering tiers (general inlining, fusion, micro-opts) remain **non-goals**
-  (below).
+  pattern.) The ceiling framing stands and caps all *emitted-code* perf work: Pyfun targets un-JIT'd CPython, so
+  the goal is "as fast as idiomatic hand-written Python," and a genuinely hot inner loop still belongs
+  behind an `extern` — the further lowering tiers (general inlining, fusion, micro-opts) remain
+  **non-goals** (below). What runs the output is a separate axis — see **Performance beyond CPython**.
 - **Larger prelude / package manager** — added on demand: prelude functions when a real program misses
   one; the package/façade story (publish typed extern façades once, `import` many) is a whole axis that
   waits for actual users. A future Python-side runtime package could default to `uv`. (Macros are a
@@ -30,6 +30,51 @@ Keep this a *forward-looking* backlog — do not let it grow back into a changel
   measured on a decode-dominated workload; dynamic shapes (`andThen`, decoder-as-value) keep the
   interpreter.)
 
+## Performance beyond CPython (scoped 2026-07-18)
+
+The lowering work above closed the *emitted-code* axis: output within ~1.3× of hand-written Python,
+further tiers measured out (non-goals below). This section is the other axis — changing what runs the
+output. Ordered by effort; each entry carries its own gate. Draft write-up:
+`local/article-draft-how-fast-could-it-get.md`. Measurement infrastructure: `bench/` (added
+2026-07-18) — three compute-bound benchmarks (expr_eval / collatz / map_build), each paired with a
+hand-written Python baseline as the ceiling reference, `bench/run.py` wall-clock runner
+(median-of-N, output-equivalence-checked, `--python` selects the interpreter — the same harness
+measures every option below). CPython 3.14.6 status quo: expr_eval 2.37×, collatz 1.18×,
+map_build 1.64× vs hand-written.
+
+- **Faster host runtimes** (S) — the output is plain, JIT-friendly Python (stable types, no
+  monkey-patching, fixed class shapes), so it runs anywhere Python runs. **GraalPy** advertises a
+  Python 3.12-compliant runtime and should run emitted output unchanged — verify once (incl. PEP 701
+  nested-quote f-strings) and quote a measured number in the docs. **CPython's own JIT** (experimental
+  since 3.13) accrues to every program for free. **PyPy** tops out at Python 3.11 (v7.3.22,
+  2026-04); the *only* 3.12 feature the emitter relies on is PEP 701 (match/case is 3.10), so a
+  `--target 3.11` emission switch that escapes nested quotes in f-strings unlocks PyPy for every
+  program. Deliverables: the switch + a docs note ("compute-bound? run the output on a faster
+  Python"). Worth a real 2–10× on interpreter-bound code for near-zero compiler work.
+- **Typed-emit + mypyc AOT (`--native`)** (M to measure, L to ship; **gated on the measurement**) —
+  the checker knows every binding's inferred type, so the emitter could produce fully annotated
+  Python whose annotations cannot lie, then compile it with mypyc into a C extension — native speed
+  with the interop story intact (the result is still an ordinary extension module). Real blockers
+  make this a feature, not a flag: mypyc does not yet compile `match` statements
+  (python/mypy#12362) and every Pyfun pattern match lowers to one, so native mode needs an alternate
+  `if`/`elif` match lowering; nested closures (partial application), generators (`seq`), and
+  `_pyfun_rt.py` all need a compatibility audit; and mypyc needs a C toolchain on the user's
+  machine, so this is opt-in only — `pip install pyfun` stays toolchain-free. **Gate first:**
+  hand-annotate one compute-bound example's emitted Python, run it through mypyc, measure (an
+  afternoon). Build only if the multiple justifies an L.
+- **Native backend** (not planned — recorded as a design-space note so the property it rests on
+  stays deliberate) — the semantics are AOT-compilable: static HM types (no dynamic dispatch),
+  default immutability (aggressive optimization is sound), tracked effects (pure code may be
+  reordered), exhaustive ADTs (matches become jump tables), units already erase. That is OCaml's
+  profile; nothing in the language *requires* a dynamic runtime, and that stays true by design. The
+  cost center is the boundary: a native Pyfun embeds CPython and every `extern` crosses worlds,
+  where cost = crossing *frequency* × data marshalling, not callee speed (bulk data can share
+  zero-copy via the buffer protocol; chatty per-element crossings are fatal). Pyfun's edge if ever
+  built: externs are typed and effect-tracked, so every crossing is statically known — the compiler
+  could warn on chatty boundaries inside hot loops, or batch them. Two-tier precedent: Codon, Mojo —
+  both multi-year funded-team efforts. Rewriting Python libraries in Pyfun to remove the boundary is
+  rejected outright (the ecosystem is the asset). Reopen only with a funded reason.
+
 ## Verification gaps (things shipped but not exercised on the real surface)
 
 Sweep completed 2026-07-14: Neovim 0.12 (5/5 headless checks: filetype/syntax/LSP attach/hover/

bench/README.md

@@ -0,0 +1,51 @@
+# Benchmarks
+
+Compute-bound benchmarks for measuring Pyfun's emitted code against hand-written
+Python, and for measuring alternative ways of *running* the output (CPython
+versions, GraalPy, PyPy via a future `--target 3.11`, mypyc-compiled — see
+`ROADMAP.md`, "Performance beyond CPython").
+
+Each benchmark exists twice: `<name>.pyfun` (compiled by the runner to
+`out/<name>.py`) and `<name>_baseline.py`, the program a Pythonista would write
+by hand for the same job. The baseline is the **ceiling reference**: emitted
+code at 1.0x has reached "as fast as the Python you would have written".
+Both sides must print byte-identical output — the runner refuses to report
+timings for programs that computed different results.
+
+| benchmark | shape | what it stresses |
+|---|---|---|
+| `expr_eval` | build/simplify/evaluate ADT expression trees | ADT allocation, deep + nested pattern matching |
+| `collatz` | total stopping times, recursive | function-call + integer-op throughput (baseline is iterative, as a Pythonista would write it) |
+| `map_build` | fold a 500k-insert string-keyed Map, then 500k lookups | the fold-pass dict lowering, `Map.tryFind`/`Option.withDefault`, f-string keys |
+
+Deliberately **not** here: I/O-bound workloads. The network-rail example's
+runtime is gzip decompression and long-line scanning, costs every runtime pays
+alike; it measures the shape of that job, not the language (see
+`local/article-draft-leverage-dont-emulate.md`).
+
+## Running
+
+```bash
+python bench/run.py                  # all benchmarks, current CPython
+python bench/run.py --bench collatz  # one benchmark
+python bench/run.py --python graalpy # time both sides on another interpreter
+python bench/run.py --runs 10        # more samples
+python bench/run.py --skip-compile   # reuse out/*.py (e.g. hand-edited for a mypyc experiment)
+```
+
+The runner uses `target/release/pyfun` or `target/debug/pyfun` if built,
+falling back to `cargo run`. Compile time is not measured; only the timed runs
+of the resulting programs are.
+
+## Method
+
+Wall-clock, subprocess-level, median of N runs (default 5) after one warmup,
+with min..max spread reported so drift is visible. No call-graph profilers:
+cProfile charges fixed per-call instrumentation to every function, which makes
+trivial functions called millions of times look hot when they aren't
+(documented in `ROADMAP.md`, "Further lowering tiers"). Interpreter startup
+(~30ms) is included in every measurement on both sides equally; workloads are
+sized so it is noise.
+
+When quoting numbers in docs or articles: pin one machine, note the
+interpreter line the runner prints, and re-run both sides in the same session.

bench/collatz.pyfun

@@ -0,0 +1,12 @@
+# Collatz benchmark: tight recursive arithmetic — function-call and integer-op
+# throughput, nothing else. Sums the total stopping time of every n in
+# [1, 120000). Compute-bound; the printed checksum must match
+# collatz_baseline.py exactly — the runner checks.
+
+let collatz steps n =
+  if n == 1 then steps
+  else if n % 2 == 0 then collatz (steps + 1) (n // 2)
+  else collatz (steps + 1) (3 * n + 1)
+
+let checksum = List.range 1 120000 |> List.fold (fun acc n -> acc + collatz 0 n) 0
+print (f"collatz checksum: {checksum}")

bench/collatz_baseline.py

@@ -0,0 +1,22 @@
+# Hand-written Python baseline for collatz.pyfun: the loop a Pythonista would
+# write (iterative, no recursion). Must print exactly what the compiled Pyfun
+# version prints.
+
+
+def collatz(n):
+    steps = 0
+    while n != 1:
+        if n % 2 == 0:
+            n //= 2
+        else:
+            n = 3 * n + 1
+        steps += 1
+    return steps
+
+
+def main():
+    total = sum(collatz(n) for n in range(1, 120000))
+    print(f"collatz checksum: {total}")
+
+
+main()

bench/expr_eval.pyfun

@@ -0,0 +1,65 @@
+# Expression-evaluator benchmark: ADT allocation + pattern matching, the work
+# CPython's interpreter is slowest at. Builds deterministic expression trees
+# (LCG-seeded, no randomness), evaluates each one raw and after a
+# constant-folding simplify pass, and folds everything into one checksum.
+#
+# Compute-bound on purpose: no I/O beyond the final print. The printed checksum
+# must match expr_eval_baseline.py exactly — the runner checks.
+
+type Expr =
+  | Num int
+  | Add Expr Expr
+  | Mul Expr Expr
+  | Neg Expr
+
+let nextSeed s = (s * 1103515245 + 12345) % 2147483648
+
+let build depth seed =
+  if depth == 0 then Num (seed % 100)
+  else
+    let s1 = nextSeed seed
+    let s2 = nextSeed s1
+    match seed % 4:
+      case 0: Neg (build (depth - 1) s1)
+      case 1: Mul (build (depth - 1) s1) (build (depth - 1) s2)
+      case _: Add (build (depth - 1) s1) (build (depth - 1) s2)
+
+let eval e =
+  match e:
+    case Num n: n
+    case Add a b: (eval a + eval b) % 1000003
+    case Mul a b: (eval a * eval b) % 1000003
+    case Neg a: 0 - eval a
+
+# Constant-folds the easy identities; nested constructor patterns give the
+# match compiler (and later mypyc) something realistic to chew on.
+let simplify e =
+  match e:
+    case Num n: Num n
+    case Neg a:
+      match simplify a:
+        case Num n: Num (0 - n)
+        case sa: Neg sa
+    case Add a b:
+      match (simplify a, simplify b):
+        case (Num 0, sb): sb
+        case (sa, Num 0): sa
+        case (Num x, Num y): Num ((x + y) % 1000003)
+        case (sa, sb): Add sa sb
+    case Mul a b:
+      match (simplify a, simplify b):
+        case (Num 0, _): Num 0
+        case (_, Num 0): Num 0
+        case (Num 1, sb): sb
+        case (sa, Num 1): sa
+        case (Num x, Num y): Num ((x * y) % 1000003)
+        case (sa, sb): Mul sa sb
+
+let iter acc i =
+  let tree = build 12 (nextSeed (i * 7919))
+  let v1 = eval tree
+  let v2 = eval (simplify tree)
+  (acc * 31 + v1 * 17 + v2) % 1000000007
+
+let checksum = List.range 0 400 |> List.fold iter 0
+print (f"expr_eval checksum: {checksum}")

bench/expr_eval_baseline.py

@@ -0,0 +1,83 @@
+# Hand-written Python baseline for expr_eval.pyfun. This is the ceiling
+# reference: the program a Pythonista would write for the same job, using
+# tagged tuples and match statements. Must print exactly what the compiled
+# Pyfun version prints.
+
+M = 1000003
+
+
+def next_seed(s):
+    return (s * 1103515245 + 12345) % 2147483648
+
+
+def build(depth, seed):
+    if depth == 0:
+        return ("num", seed % 100)
+    s1 = next_seed(seed)
+    s2 = next_seed(s1)
+    r = seed % 4
+    if r == 0:
+        return ("neg", build(depth - 1, s1))
+    if r == 1:
+        return ("mul", build(depth - 1, s1), build(depth - 1, s2))
+    return ("add", build(depth - 1, s1), build(depth - 1, s2))
+
+
+def evaluate(e):
+    match e:
+        case ("num", n):
+            return n
+        case ("add", a, b):
+            return (evaluate(a) + evaluate(b)) % M
+        case ("mul", a, b):
+            return (evaluate(a) * evaluate(b)) % M
+        case ("neg", a):
+            return -evaluate(a)
+
+
+def simplify(e):
+    match e:
+        case ("num", _):
+            return e
+        case ("neg", a):
+            sa = simplify(a)
+            if sa[0] == "num":
+                return ("num", -sa[1])
+            return ("neg", sa)
+        case ("add", a, b):
+            sa, sb = simplify(a), simplify(b)
+            match (sa, sb):
+                case (("num", 0), _):
+                    return sb
+                case (_, ("num", 0)):
+                    return sa
+                case (("num", x), ("num", y)):
+                    return ("num", (x + y) % M)
+                case _:
+                    return ("add", sa, sb)
+        case ("mul", a, b):
+            sa, sb = simplify(a), simplify(b)
+            match (sa, sb):
+                case (("num", 0), _) | (_, ("num", 0)):
+                    return ("num", 0)
+                case (("num", 1), _):
+                    return sb
+                case (_, ("num", 1)):
+                    return sa
+                case (("num", x), ("num", y)):
+                    return ("num", (x * y) % M)
+                case _:
+                    return ("mul", sa, sb)
+
+
+def main():
+    acc = 0
+    for i in range(400):
+        tree = build(12, next_seed(i * 7919))
+        v1 = evaluate(tree)
+        v2 = evaluate(simplify(tree))
+        acc = (acc * 31 + v1 * 17 + v2) % 1000000007
+    print(f"expr_eval checksum: {acc}")
+
+
+main()

bench/map_build.pyfun

@@ -0,0 +1,25 @@
+# Map benchmark: builds a string-keyed Map inside a fold — the accumulator
+# shape the fold pass proves, so `Map.add` lowers to an in-place dict insert —
+# then hammers it with lookups. Exercises the fold-pass dict lowering,
+# `Map.tryFind`/`Option.withDefault`, and f-string key construction.
+#
+# The accumulator must stay a bare Map for the pass to fire: threading extra
+# state (e.g. an LCG seed) through a tuple slot is a known rejecting shape
+# (ROADMAP.md, "fold-pass residual shapes") that falls back to copy-per-insert.
+# Keys are therefore hashed from the index directly.
+#
+# Compute-bound; the printed line must match map_build_baseline.py exactly —
+# the runner checks.
+
+let key i = f"key{(i * 1103515245 + 12345) % 2147483648 % 20011}"
+
+let step m i = Map.add (key i) i m
+
+let table n = List.range 0 n |> List.fold step Map.empty
+
+let m = table 500000
+
+let lookupStep acc i = acc + Option.withDefault 0 (Map.tryFind (key (i * 3)) m)
+
+let checksum = List.range 0 500000 |> List.fold lookupStep 0
+print (f"map_build size: {Map.len m} checksum: {checksum}")

bench/map_build_baseline.py

@@ -0,0 +1,22 @@
+# Hand-written Python baseline for map_build.pyfun: a plain dict built in a
+# loop, then the same lookups. Must print exactly what the compiled Pyfun
+# version prints.
+
+
+def key(i):
+    return f"key{(i * 1103515245 + 12345) % 2147483648 % 20011}"
+
+
+def main():
+    d = {}
+    for i in range(500000):
+        d[key(i)] = i
+
+    total = 0
+    for i in range(500000):
+        total += d.get(key(i * 3), 0)
+
+    print(f"map_build size: {len(d)} checksum: {total}")
+
+
+main()