Skip to content

README.md

Latest commit

 

History

History
300 lines (265 loc) · 16.7 KB

README.md

File metadata and controls

300 lines (265 loc) · 16.7 KB
Raw

Performance benchmarks

This directory holds the performance regression harness that CI runs on every commit and enforces on every release. It exists to keep cronstable fast and small enough for old machines: startup cost, schedule math at 100k-job scale, config parsing, DAG planning, durable-state I/O, memory footprint, the terminal dashboard's per-frame string work, and the web dashboard's render hot paths are all measured, and a release that regresses past a metric's limit does not ship.

The two tools

  • bench.py runs the suite and writes one JSON document. The harness is stdlib-only and benchmarks whatever cronstable the invoking interpreter can import, so the same script can measure an older installed release. A benchmark whose API the measured version lacks is recorded as skipped, never failed. To keep the measurement honest it runs untimed warm-up passes before the timed repeats and (best-effort) pins itself to one CPU and raises its priority; benchmarks split into an in-process tier and a noisier subprocess tier (cold start, import, peak RSS), selectable with --tier.
  • compare.py takes baseline and current JSON files (several rounds per side), merges the rounds, renders a markdown summary and an SVG diverging bar chart of every compared metric, and exits nonzero when a gated metric regressed. A regression gates only when it clears both its declared limit and a couple of its measured noise bands (the per-metric round-to-round scatter), so jitter alone can never fail the gate.

Running locally

python benchmarks/bench.py --quick --json before.json
# ...make your change, then...
python benchmarks/bench.py --quick --json after.json
python benchmarks/compare.py --baseline before.json --current after.json \
    --md diff.md --svg diff.svg

--quick cuts workloads to roughly a tenth for a fast local loop; CI runs the full suite. --only <substring> selects benchmarks by name or group (for example --only cronexpr), --tier inprocess (or subprocess) selects one tier, --warmup N overrides the warm-up passes, --no-stabilize skips the CPU pin, --list prints the inventory, and --smoke is the minimal mode the unit tests use. If cronstable is not installed in the interpreter, the harness falls back to the source tree it lives in and says so on stderr.

Local numbers are only comparable to other runs on the same machine in the same session. The CI comparison is paired for exactly that reason: both versions run interleaved on one runner, in the same weather.

What CI does with this

The perf job in .github/workflows/release.yml runs on every push and PR, in parallel with the build matrix:

  1. installs the current commit into one venv and the latest release tag into another;
  2. runs bench.py against both, interleaved, per tier: five rounds of the in-process tier and two of the subprocess tier (the harness always comes from the current checkout, so both sides run identical measurement code);
  3. runs compare.py over all the result files.

Per metric, rounds merge with the metric's estimator: best-of-rounds for time (the minimum is the least noisy statistic of a fixed workload) and median for memory. A metric fails its gate only when it slows down by more than its declared percentage limit AND by more than its absolute floor AND by more than a couple of its measured noise bands, where the noise band is the two sides' round-to-round scatter combined in quadrature. So microsecond jitter on a sub-millisecond metric can never gate, and neither can a metric's own run-to-run wobble; a change that clears the raw limit but sits inside the noise band is reported (not silently dropped) but does not fail the release. More in-process rounds deepen the best-of-rounds estimate and give the robust scatter statistic enough points to work with; they do not tighten the noise band itself (the band estimates single-round scatter, while the comparison diffs min-of-rounds, so measured bands can even widen as rounds are added).

Three refinements keep the gate both tight and honest:

  • Robust noise band. From three rounds up, the round-to-round scatter is the median absolute deviation (scaled to a standard-deviation equivalent), not a plain standard deviation. One throttled or GC-stalled round can no longer inflate the band and hide a real regression behind it.
  • Interpreter-startup subtraction. The startup.* metrics are dominated by Python's own process spawn and interpreter init, which cronstable cannot regress. Each side's startup.python_baseline is subtracted before the delta is computed, so the gate sees cronstable's OWN contribution and a real couple-of-ms import regression is not diluted below the limit by ~40ms of un-regressable overhead.
  • A tighter default limit. The deterministic in-process compute metrics gate at 15%; the noisier tiers (subprocess process-spawn, real-disk state I/O, peak-RSS, browser render) set a looser limit of their own. The noise band above still protects every one of them from jitter.

On an ordinary commit the comparison prints warnings only. On a release the gate is enforced: the release job requires perf, so a gated regression blocks publishing. The release then appends the comparison to its notes and attaches perf-summary.md (the full table) and perf-results.json (the merged raw numbers). perf-chart.svg (the diff chart) ships in the run's perf-report artifact.

To ship an intentional regression, start a pushed commit's subject with [perf:accept]. The regression is still measured and reported in the release notes, but it does not gate. Only subject lines are scanned, same as the [release] marker. [perf:accept] excuses RELATIVE regressions only: an absolute-budget breach or a gate-integrity failure (below) each has its own ritual, a reviewed edit to the corresponding checked-in file.

Beyond the relative gate

Four properties of the comparator itself, each added after the 2026-07 audits found the relative gate alone could not deliver them:

  • Absolute budgets (budgets.json, compare.py --budgets). The relative gate diffs HEAD against the latest tag only, so a slow drift across many quick releases compounds under a green gate (seven release hops in five days at a legal 15% each is 2.7x). A handful of headline metrics carry absolute ceilings; a breach fails the run even when the relative gate passes. Raising a ceiling is a deliberate edit to budgets.json, made in the same PR as the change that needs it.
  • Gate integrity (expected_gated.txt, compare.py --expected-gated). A metric whose BASELINE side skips is filed as first-release coverage and warned about by nothing, so a gate that died because a private seam drifted was silently ungated forever. The checked-in list names every metric that must actually be compared; a listed metric that was not is an integrity failure. The companion net is tests/test_benchmarks.py's never-skip list, which catches a drifted seam in the ordinary test run.
  • The effective gate column. The percentage and absolute-floor tests are ANDed, so a metric whose value sits near its floor really gates at 100*floor/value, however tight its declared gate_pct reads. The floor is deliberate harness policy (jitter on a tiny metric must never gate); what was missing was anything REPORTING when it binds. The comparison table's "Gate (eff.)" column and a job-log notice now name every floor-bound metric, so an undersized workload is visible and fixable instead of silently ungated.
  • Comparability. The two sides must agree on python version, platform, run mode, and the optional-backend state (orjson, uvloop). A pairing that differs is refused outright (exit 2, never a verdict): a one-sided backend would report a backend swap as a large code regression, or mask a real one. The CI perf job installs orjson into BOTH venvs -- production's default backend in the binaries and Docker images -- and deliberately NOT uvloop (see the waivers below).

bench.py also stamps per-benchmark wall clock (fixtures included) into every result row and prints its ten slowest benchmarks per run, so the CI job's timeout ceiling is triaged from the log rather than by bisecting a timed-out release.

Waivers: what is deliberately not measured

Naming an absence makes it a decision instead of an oversight. A future reader counting metrics against modules should not assume these were missed:

  • uvloop. Every shipped Linux binary and Docker image prefers uvloop; the whole suite runs on stock asyncio. A full uvloop lane would double the perf job and most of any delta would be uvloop's own. Waived; bench.py records a uvloop flag in every result document so the absence stays visible, and comparability (above) refuses a mixed pair.
  • Windows and macOS. The Linux-only pipeline never times the platform-divergent paths. A measured --quick Windows lane was refuted as a dead gate (34 of 38 metrics floor-bound at that scale) and the full-scale shape does not fit any timeout. The one real correctness risk found there -- the halved directory-barrier count -- is a COUNT invariant and is gated by tests/test_perf_invariants.py on every platform instead.
  • The artifact users actually run. Startup/RSS metrics time a CPython venv; every distributed artifact is a PyInstaller onefile binary or a Docker image. A binary startup metric would need the build job's artifact and break the perf job's independence; the SIZE half is not a perf metric at all and belongs as a byte ceiling in the binary job.
  • Tail latency. The suite measures best-of-rounds throughput and peak memory; the user-visible failure for a cron daemon is a scheduler stall, which is a tail. A p99 over 5 rounds is not statistically supportable, so the general case is waived -- and loop.stall_jobs_500 (a MAX scheduling-gap gauge, info for its first release, then armed) covers the one stall class that has actually shipped, five separate times.
  • Leadership backends and operator CLIs (state_admin, jobcli, mcpcli, discovery, tlsutil). Run by hand once, or one-shot startup paths; their pure per-call functions are microsecond-scale. Tests, not metrics.
  • The report/notify delivery pipeline. A reporter metric would be ~95% jinja2/stdlib/socket and would break the no-network rule; the one owned risk (a multi-MB capture rendered into a report body) is a bounds question for a test.
  • resources.py's per-tick process-table walk. It walks the REAL machine's process table, so a timing gate can only measure the runner. The invariant that matters -- one table snapshot per sample batch, however many runs are monitored -- is a count, gated by tests/test_perf_invariants.py.
  • Per-line SSE live-tail fan-out. job.stream_capture_40k deliberately disables the passthrough mirror and excludes the on_line live-tail leg, so per-line x per-client delivery on the shared loop stays untimed (a zero-subscriber LiveLogBuffer variant could close this later).

The rule those waivers keep applying: a COUNT or ORDERING invariant gets a test, never a metric. Five benchmark candidates from the 2026-07 audits (fsync barriers, the once-per-batch process-table walk, 413-before-fetch, the per-run durable-write count, artifact prune residency) are exactly that and live in tests/test_perf_invariants.py, where they gate in both directions, on every platform, in milliseconds.

Terminal and web UI benchmarks

The dashboards have their own hot paths, and both are measured.

The terminal UI (tui.*) is pure Python, so it is benchmarked in process like everything else: the log drawer re-measures, re-cuts and re-inks its whole buffer each frame and the log search re-scans it, so tui.log_restyle_5k and tui.log_search_20k drive text_width / cut_to_width / rewrite_sgr / strip_ansi over a realistic buffer (coloured, plain, wide-glyph and control-character lines). No terminal and no app loop.

The web UI (webui.*) is browser JavaScript, so it is timed inside a headless Chromium via Playwright. The page exposes a window.__perf hook ONLY under the ?perf=1 query string (it is entirely inert otherwise — no global is defined), giving the harness seed helpers and the real render functions; bench.py seeds synthetic jobs / fleet / log data and times renderRows, renderFleet and updateLogCount with the page's own performance.now() (batched, because Chromium clamps that clock to ~100us). The whole webui group skips cleanly when Playwright or its Chromium build is absent, when the page predates the ?perf=1 hook (an older release), and in --smoke (the unit test must not launch a browser). The CI perf job installs Playwright + Chromium into the current-side venv (best-effort) so webui.* runs there; to run them locally:

pip install playwright && playwright install chromium
python benchmarks/bench.py --quick --only webui

Because an older release's page carries no ?perf=1 hook, the webui metrics compare new-against-new (a forward-looking gate and a recorded number), not an old-vs-new delta; the tui.* and backend metrics do diff across releases.

Adding a benchmark

Register a function in bench.py with the @bench(...) decorator:

@bench(
    "group.short_name",       # stable metric id; renaming loses history
    "group",
    detail="one line of what the workload is",
    repeats=(5, 2, 1),        # full / quick / smoke repeats
    gate_pct=25.0,            # regression limit, percent
    gate_floor=0.010,         # and the absolute floor, in the metric's unit
)
def bench_thing():
    ...setup (untimed)...
    t0 = time.perf_counter()
    ...the workload...
    return time.perf_counter() - t0

Ground rules:

  • Time only the workload; do setup outside the timed region, and use fixture(name, builder) for expensive setup shared across repeats.
  • A fixture holding external state that outlives a dropped reference (a subprocess, a session, and above all anything that parks a RUNNING event loop on the harness thread the way Playwright's sync API does) must pass fixture(name, builder, finalizer=...). Fixtures are evicted at group boundaries, and the harness then audits the thread for a leftover running loop: a parked loop hard-fails the run right there, because it would otherwise silently skip every later asyncio.run() benchmark on both sides of the comparison (the 1.2.31 release's six dead gates).
  • Scale the workload with _n(base) so --quick and --smoke stay cheap.
  • Import cronstable inside the function and raise Skip when an API is missing, so the harness still runs against older releases.
  • Keep workloads deterministic: fixed datetimes, fixed inputs, no network.
  • Memory metrics use unit="MB" and compare="median".
  • A benchmark that measures a child process (cold start, import, peak RSS) passes subprocess=True so it lands in the subprocess tier.
  • Size the timed region so it runs long enough (roughly 50ms+) that scheduler and GC jitter are a small fraction; a sub-10ms metric is dominated by noise. Rescaling an existing benchmark is safe for the gate (the comparison re-measures BOTH sides with the current definition, so it never diffs a new workload against a stored old number), but bump the metric id anyway so the name keeps meaning one fixed workload across releases and a release-notes trend is never silently redefined. cronexpr.test_match_200k, schedule.duplicates_20k, dag.plan_claim_10k and schedule.pressure_20k_48h are such rescales: the id suffix carries the new scale, and the old ids drop out.
  • A metric need not be a duration: webapi.jobs_bytes_500 gates the SIZE of a response body (unit="KB"; remember the floor is then in KB), and loop.stall_jobs_500 gates a worst-case scheduling GAP. What matters is that the value is deterministic and the regression it guards moves it.
  • If what you are guarding is a COUNT or an ORDERING (fsyncs per append, calls per batch, a check that must precede a fetch), stop: write a test in tests/test_perf_invariants.py instead. See the waivers above.
  • When a benchmark lands, add it to benchmarks/expected_gated.txt once it is known to compare against the current baseline release. The smoke net's never-skip set is derived from that file, so listing it there is also what makes tests/test_benchmarks.py fail if it ever starts skipping (the guard that matters when it leans on any private seam). A seam-leaning benchmark that cannot enter expected_gated.txt yet (its surface is new this release) has no net until it does; prefer public-surface workloads for anything that must wait.

The suite's own smoke test is tests/test_benchmarks.py; it fails if a headline benchmark starts skipping, so a refactor that breaks a measured API surfaces in the ordinary test run, not at release time.