feat(perf): three-mode memory harness + sprint tracker (S1)

TRACKER.md

@@ -0,0 +1,112 @@
+# ExcelBench — Sprint Tracker
+
+> Single source of truth for the **7-Dimension Extension** initiative. Each row tracks one
+> self-contained sprint (one branch, one PR, one row flip). Resume cold by reading this file
+> and the most recent `[*INCOMPLETE*]` marker.
+
+**Last updated**: 2026-04-27 (S1 shipped)
+
+## Status Table
+
+| #  | Dimension                          | Status      | Sprint size | Branch                         | PR  | Acceptance commit range |
+|----|------------------------------------|-------------|-------------|--------------------------------|-----|-------------------------|
+| S1 | Memory honesty + Tracker bootstrap | Shipped     | S (3–5 d)   | `feat/perf-mem-honesty`        | #28 | `50dc104..HEAD@PR#28`   |
+| S2 | Data shape (int/str/date/formula)  | Planned     | M (1 wk)    | `feat/perf-data-shape`         | —   | —                       |
+| S3 | File shape (wide/tall/sparse)      | Planned     | M (1 wk)    | `feat/perf-file-shape`         | —   | —                       |
+| S4 | High-cost operations               | Planned     | M (1 wk)    | `feat/perf-operations`         | —   | —                       |
+| S5 | Workbook complexity perf           | Planned     | M (1 wk)    | `feat/perf-complexity`         | —   | —                       |
+| S6 | Cold-start / warm path             | Planned     | S (3–5 d)   | `feat/perf-cold-start`         | —   | —                       |
+| S7 | Round-trip fidelity (LibreOffice)  | Planned     | L (~2 wk)   | `feat/fidelity-roundtrip`      | —   | —                       |
+
+**Status legend**: `Planned` → `In Progress` → `Shipped` (or `Blocked` with reason).
+
+## How to Flip a Row
+
+When a sprint lands:
+
+1. Update the row's **Status** to `Shipped`.
+2. Fill in the **PR** column (`#NN`).
+3. Fill in **Acceptance commit range** (`abc1234..def5678`).
+4. Bump the **Last updated** line at the top of this file.
+5. Append a sprint acceptance entry (template below) to the **Acceptance Notes** section.
+6. Add the corresponding `DEC-NNN` entry to `decisions.md` if not already done.
+
+If a sprint stalls, switch its status to `Blocked` and add a one-line reason in the row.
+
+## Sprint Acceptance Template
+
+Use this template when appending to **Acceptance Notes** below.
+
+```markdown
+### S<N> — <Dimension> (YYYY-MM-DD)
+
+**Branch**: `feat/...`  ·  **PR**: #NN  ·  **Commit range**: `abc1234..def5678`
+
+**What shipped**:
+- <one-line bullet per major piece>
+
+**Verification**:
+- `uv run pytest tests/` ✓
+- `uv run ruff check src/ tests/` ✓
+- `uv run mypy src/` ✓
+- `excelbench <new-subcommand> ...` ✓ (16 adapters, no crashes)
+- Dashboard regenerated, results.json + history.jsonl appended.
+
+**Decisions**: DEC-NNN logged in `decisions.md`.
+
+**Deferred / out-of-scope**:
+- <items intentionally left for follow-up>
+```
+
+## Acceptance Notes
+
+<!-- Newest first. Append entries here as sprints ship. -->
+
+### S1 — Memory honesty + Tracker bootstrap (2026-04-27)
+
+**Branch**: `feat/perf-mem-honesty`  ·  **PR**: [#28](https://github.com/SynthGL/ExcelBench/pull/28)  ·  **Commit range**: `50dc104..HEAD` (final range fills in on merge)
+
+**What shipped**:
+- `TRACKER.md` (this file) — 7-row sprint table, row-flip protocol, acceptance template.
+- `src/excelbench/perf/memory.py` — three-mode memory harness (`getrusage` / `tracemalloc` /
+  `time` via `/usr/bin/time -l` subprocess + `all` composite). `MemoryProbe` context manager
+  for in-process modes; `parse_time_l_stderr` cross-platform parser (macOS BSD time + GNU
+  time `-l`).
+- `PerfOpResult` extended with `rss_kb_via_time` and `python_heap_peak_kb` fields (existing
+  `rss_peak_mb` preserved — backwards-compatible).
+- `src/excelbench/perf/_iter_subprocess.py` — internal subprocess entrypoint that runs one
+  iteration per invocation; wrapped by parent under `/usr/bin/time -l`.
+- `excelbench perf --memory-mode={getrusage,tracemalloc,time,all}` CLI flag.
+- HTML dashboard renders dual `RSS (MB) — getrusage / time -l` cells with a tooltip
+  explaining divergence whenever any entry has a `time -l` measurement.
+- DEC-018 documents why three modes coexist and what each is honest about.
+
+**Verification** (run on macOS 25.2, Python 3.13):
+- `uv run pytest tests/` ✓ 1140 passed, 32 skipped, 6 xfailed
+- `uv run ruff check src/excelbench/perf/ src/excelbench/cli.py src/excelbench/results/html_dashboard.py` ✓
+- `uv run mypy src/excelbench/perf/` ✓ no issues
+- `excelbench perf --memory-mode=all --feature cell_values --adapter wolfxl --adapter openpyxl --warmup 1 --iters 2`:
+  - All three fields populated as expected.
+  - Python-heap honesty signal landed: openpyxl uses 16× (read) and 227× (write) more
+    Python heap than wolfxl on the same workload, confirming wolfxl pushes allocations into Rust.
+  - `time -l/getrusage` ratio ~0.97x on small fixtures (subprocess startup dominates);
+    expected to diverge meaningfully once Sprint 2 lands ≥1M-cell fixtures.
+
+**Decisions**: DEC-018 logged in `decisions.md`.
+
+**Deferred / out-of-scope**:
+- Tracemalloc reset semantics across nested probes — current code uses `reset_peak()` when a
+  probe re-enters an already-traced context. Should be revisited if any caller starts
+  tracemalloc outside the probe.
+- `time -l` subprocess support on Windows — skipped silently (no `/usr/bin/time`).
+  Sprint 6 (cold-start) will set the precedent for cross-platform subprocess handling.
+- Visualizing the time-l/getrusage divergence as a dedicated chart — single dual-cell
+  with tooltip is sufficient until S2 ships larger fixtures that make the gap visible.
+
+## Reference
+
+- Plan: see the wolfxl session that produced this tracker (multi-sprint roadmap).
+- Architecture: [`architecture.md`](architecture.md)
+- Decisions: [`decisions.md`](decisions.md)
+- Key seams: `src/excelbench/perf/runner.py`, `src/excelbench/harness/adapters/base.py`,
+  `src/excelbench/results/html_dashboard.py`.

decisions.md

@@ -40,6 +40,54 @@ Skip logging for routine bug fixes, refactors, or incremental test additions.
 
 ## Decisions
 
+### DEC-018 — Three coexisting memory-measurement modes (2026-04-27)
+
+**Context**: Until Sprint 1 of the 7-Dimension Extension, the perf runner reported a single
+memory number — peak RSS via `resource.getrusage(RUSAGE_SELF).ru_maxrss`. Two problems with
+that single number: (1) `getrusage` returns the **process-lifetime peak**, so once the first
+heavy iteration has allocated, subsequent iterations report the same sticky max even if they
+allocated less — making per-iteration comparisons misleading; (2) ad-hoc 100k → 1M cell
+benchmarks against openpyxl from the wolfxl 1.0 work showed the `getrusage` peak diverges
+from `/usr/bin/time -l` peaks by 30-300% on large workloads, depending on Rust allocator
+release behavior. There is no single right number — different libraries pay memory cost in
+different places (Python heap vs Rust heap vs OS pages) and the perf dashboard needs to be
+honest about that asymmetry rather than pretending a single column tells the truth.
+
+**Decision**: Coexist three modes via a new `--memory-mode` flag, all populating the existing
+`PerfOpResult` dataclass with separate fields:
+
+- `getrusage` (default, cheap): in-process `RUSAGE_SELF.ru_maxrss`. Preserved as the hot-path
+  default — fast, but documented as lifetime-peak-sticky.
+- `tracemalloc`: in-process `tracemalloc.get_traced_memory()`. Adds the Python heap peak.
+  Misleading for Rust-backed adapters (wolfxl, python-calamine, rust_xlsxwriter) because it
+  cannot see native allocations; honest for pure-Python adapters.
+- `time`: spawn each iteration under `/usr/bin/time -l` and parse peak RSS from stderr.
+  Honest about Rust allocations because the OS reports it. Slow (subprocess startup +
+  adapter import per iteration); quarterly deep-dive only.
+- `all`: composite — every iteration runs in-process (capturing getrusage + tracemalloc) AND
+  in a fresh subprocess (capturing time-l RSS). Used for the memory-deep-dive bench, not CI.
+
+The dashboard renders `RSS (MB) — getrusage / time -l` as a dual cell with a tooltip
+explaining the divergence whenever any entry has the time-l field populated.
+
+**Alternatives considered**: (1) Replace `getrusage` with `psutil.Process().memory_info().rss`
+polling — rejected: still in-process, still subject to allocator-release lag, and adds a
+mandatory third-party dependency to the runner. (2) Drop `getrusage` once `time -l` is
+available — rejected: `time -l` is 50-500x slower per iteration, breaking the CI hot path.
+(3) Single `multi_mode` field instead of three separate fields — rejected: makes the JSON
+schema lossy (can't tell which number came from which mode in a composite run).
+
+**Consequences**: `PerfOpResult` JSON now carries `rss_kb_via_time` and `python_heap_peak_kb`
+in addition to the existing `rss_peak_mb`. Downstream dashboards must accept these as
+optional fields. The `time` and `all` modes spawn one subprocess per iteration via the new
+private `excelbench.perf._iter_subprocess` module; on Windows where `/usr/bin/time` does not
+exist, `rss_kb_via_time` is silently `None`. Comparisons across past results remain valid
+because the existing `rss_peak_mb` field is unchanged.
+
+**Commit(s)**: Sprint 1, branch `feat/perf-mem-honesty`.
+
+---
+
 ### DEC-017 — Do not inject Excel alignment defaults in benchmark comparisons (2026-02-17)
 
 **Context**: Several value-focused adapters return an empty `CellFormat()` for alignment reads/writes.

src/excelbench/cli.py

@@ -260,6 +260,19 @@ def perf(
         "--profile",
         help="Benchmark profile: xlsx (default) or xls.",
     ),
+    memory_mode: str = typer.Option(
+        "getrusage",
+        "--memory-mode",
+        help=(
+            "Memory measurement strategy. 'getrusage' (default) uses the cheap "
+            "in-process RUSAGE_SELF peak. 'tracemalloc' reports getrusage RSS "
+            "plus the Python heap peak (misses Rust allocations). 'time' spawns "
+            "each iteration under /usr/bin/time (-l on macOS, -v on Linux) for "
+            "an OS-honest peak RSS — runs only the subprocess, not double-"
+            "executed in-process. 'all' runs every iteration in all three modes "
+            "— quarterly deep-dive, not the CI hot path."
+        ),
+    ),
 ) -> None:
     """Run performance benchmarks (speed + best-effort memory).
 
@@ -269,15 +282,26 @@ def perf(
 
     from excelbench.harness.adapters import get_all_adapters
     from excelbench.perf import render_perf_results, run_perf
+    from excelbench.perf.memory import VALID_MEMORY_MODES
 
     if not isinstance(iteration_policy, str):
         iteration_policy = "fixed"
+    if not isinstance(memory_mode, str):
+        memory_mode = "getrusage"
 
     profile = profile.strip().lower()
     if profile not in {"xlsx", "xls"}:
         console.print("[red]Error: profile must be one of: xlsx, xls[/red]")
         raise typer.Exit(1)
 
+    memory_mode_normalized = memory_mode.strip().lower()
+    if memory_mode_normalized not in VALID_MEMORY_MODES:
+        console.print(
+            f"[red]Error: --memory-mode must be one of {list(VALID_MEMORY_MODES)}; "
+            f"got {memory_mode!r}[/red]"
+        )
+        raise typer.Exit(1)
+
     if profile == "xls" and test_dir.resolve() == PERF_XLSX_PROFILE_DEFAULT_TEST_DIR.resolve():
         test_dir = XLS_PROFILE_DEFAULT_TEST_DIR
 
@@ -304,6 +328,7 @@ def perf(
     console.print(f"  Iterations: {iters}")
     console.print(f"  Iteration policy: {iteration_policy}")
     console.print(f"  Breakdown: {breakdown}")
+    console.print(f"  Memory mode: {memory_mode_normalized}")
     if adapters:
         console.print(f"  Adapters: {', '.join([a.name for a in selected])}")
     console.print()
@@ -318,6 +343,7 @@ def perf(
             iters=iters,
             iteration_policy=iteration_policy,
             breakdown=breakdown,
+            memory_mode=memory_mode_normalized,  # type: ignore[arg-type]
         )
         render_perf_results(perf_results, output_dir)
 

src/excelbench/perf/_iter_subprocess.py

@@ -0,0 +1,71 @@
+"""Internal subprocess entrypoint for ``time -l`` memory measurement.
+
+Invoked by :func:`excelbench.perf.runner._measure_iteration_under_time_l`. Runs
+exactly one iteration of (adapter, kind, feature) and prints metrics JSON to
+stdout; the parent harness wraps the invocation in ``/usr/bin/time -l`` and
+parses peak RSS from stderr.
+
+Not part of the public CLI surface — only the perf runner calls this.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(prog="excelbench-perf-iter")
+    parser.add_argument("--library", required=True)
+    parser.add_argument("--kind", choices=["read", "write"], required=True)
+    parser.add_argument("--manifest", required=True, type=Path)
+    parser.add_argument("--feature", required=True)
+    parser.add_argument(
+        "--memory-mode",
+        default="getrusage",
+        choices=["getrusage", "tracemalloc", "all"],
+        help="In-process memory mode for the iteration. The 'time' mode is "
+        "implicit — that's why the parent process is calling us under "
+        "/usr/bin/time -l.",
+    )
+    args = parser.parse_args(argv)
+
+    from excelbench.generator.generate import load_manifest
+    from excelbench.harness.adapters import get_all_adapters
+    from excelbench.perf.runner import run_one_iteration
+
+    manifest = load_manifest(args.manifest)
+    matching = [f for f in manifest.files if f.feature == args.feature]
+    if not matching:
+        sys.stderr.write(f"feature {args.feature!r} not in manifest\n")
+        return 2
+    test_file = matching[0]
+
+    adapters_by_name = {a.name: a for a in get_all_adapters()}
+    adapter = adapters_by_name.get(args.library)
+    if adapter is None:
+        sys.stderr.write(f"adapter {args.library!r} not registered\n")
+        return 2
+
+    test_dir = args.manifest.parent
+    metrics = run_one_iteration(
+        adapter=adapter,
+        kind=args.kind,
+        test_file=test_file,
+        test_dir=test_dir,
+        memory_mode=args.memory_mode,
+    )
+    serializable = {
+        k: v
+        for k, v in metrics.items()
+        if k in ("wall_ms", "cpu_ms", "rss_peak_mb", "python_heap_peak_kb")
+        and isinstance(v, int | float)
+    }
+    sys.stdout.write(json.dumps(serializable) + "\n")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())