BenchCore minimizes orchestration overhead, but it does not promise a fixed nanosecond cost across Python versions, platforms, timers, or workloads.
Run:
poetry run python examples/measure_overhead.pyThe script reports its Python and platform identity, then compares:
- repeated direct calls to a no-op Python function; and
- repeated
BenchmarkRunner.run()calls configured for one round and one iteration.
It uses multiple trials, reports the median cost per operation, and subtracts the direct-call median from the BenchCore median. The result estimates complete runner orchestration—two timer reads, result construction, validation, tuple allocation, and statistics—not timer accuracy or the overhead inside a measured region.
The script deliberately has no pass/fail threshold. Scheduler activity, CPU power management, virtualization, instrumentation, and interpreter builds can change results enough to make a universal threshold misleading. Compare commits only on the same controlled machine and report all printed environment data.
BenchCore's deterministic tests use injected timers and call counters to verify calibration, warmup, rounds, iterations, lifecycle boundaries, non-monotonic timer rejection, and bounded execution. Statistical results are checked against Python's standard-library definitions.
Performance changes must not weaken those semantics. A claimed optimization should include the reproduction command, before/after medians from several runs, the tested commit identifiers, and the complete environment.
- BenchCore cannot interrupt a slow callable.
- A timer's resolution and monotonicity constrain useful precision.
- Descriptive statistics summarize observed rounds but do not prove statistical significance.
- Microbenchmarks are sensitive to cache state, background work, thermal state, and input mutation.
- Cross-environment comparisons are rejected by the built-in comparison API.