scripts: Repair and productize matching benchmark#181
Merged
Conversation
The project has a structural matcher benchmark that models several line layouts and reports matcher resource use from a source checkout. The benchmark depends on removed buffer and mapping interfaces, so it cannot run. The project therefore lacks repeatable measurements for a computationally expensive workflow. This commit begins restoring that performance coverage by rebuilding the runner around public line-buffer and matching APIs. It provides seeded quick and full workloads while keeping setup outside the measured phases. Subsequent commits will validate the runner, add ownership attribution coverage, expose the suite in CI and documentation, and provide report comparison.
The structural matching benchmark now runs deterministic quick and full workloads through the public matching interfaces. The project does not yet verify the benchmark schema, fixture shapes, resource boundaries, or adversarial matcher behavior. API drift could therefore make the runner unusable again without a focused failure. This commit addresses that validation gap with smoke coverage for the smallest workload, repeated and reordered inputs, binary exclusion, bounded fixture storage, and separate timing and allocation passes. The next implementation commit will extend the benchmark with ownership attribution measurements.
The benchmark suite measures structural matching across deterministic text layouts and records phase-level time and Python allocation data. Ownership attribution adds Git object resolution, streamed blob loading, shared source mappings, and per-batch claim evaluation to that path. Matching-only results cannot locate regressions in those repository-backed phases. This commit extends the suite with a fixed-size repository fixture whose quick and full modes vary the number of batch claims. Distinct state refs resolve to one source blob so the report exposes deduplicated mapping work. The next commit will validate the attribution workload before CI and documentation expose the complete runner.
The benchmark suite now includes a repository-backed workload that attributes shared source content across many batch claims. The new workload depends on state-ref resolution, source deduplication, unit enumeration, and ownership counts. Without focused assertions, a fixture mistake could produce plausible timings without exercising those paths. This commit addresses that risk by checking the measured attribution phases, the number of resolved expressions, the single blob request, the shared mapping count, and the resulting ownership links. The following CI and documentation commits will make the validated quick and full suites available to contributors.
The matching and ownership attribution benchmarks have focused local tests for their fixtures, schemas, and public API use. The regular CI job exercises the installed project only on Python 3.10 and does not retain benchmark output. Source-checkout drift on the newest supported interpreter can therefore go unnoticed. This commit addresses that gap with a quick-suite smoke job on Python 3.10 and 3.13. Each matrix entry uploads its JSON report without enforcing hardware-dependent timing thresholds. The next commits will explain the benchmark workflow and link that guide from the project website.
The project provides quick and full matching benchmarks, attribution phase measurements, JSON reports, and pull-request smoke coverage. Users cannot discover the available workloads, sampling controls, phase boundaries, or memory interpretation from the command itself. Results are difficult to reproduce without those details. This commit addresses that documentation gap with a benchmark guide covering invocation, deterministic fixture storage, measured phases, report metadata, and baseline refresh expectations. The next commit will expose the guide in website navigation. Later commits will add report comparison and document that workflow.
The documentation site has an advanced-features section for batch operations and storage behavior, while the matching benchmark guide exists outside its navigation. Users browsing the published site cannot reach the benchmark instructions through the documented feature hierarchy. This commit addresses that discoverability gap by listing the matching benchmark guide alongside the other advanced project references. The remaining commits will add machine-readable report comparison, validate it, and finish the guide with comparison policy.
The benchmark suite emits versioned JSON containing phase medians, raw samples, workload hashes, environment details, and source provenance. Users must inspect separate reports by hand and can accidentally compare different workloads or measurement settings. That makes regression decisions difficult to reproduce. This commit adds report comparison with compatibility checks for suite settings and case dimensions. It calculates median time and allocation changes, preserves environment warnings, and can return failure for changes above a chosen threshold. The final two commits will validate the comparison rules and document the baseline workflow, completing the benchmark repair series.
The benchmark command now compares compatible report medians and records regression flags, environment warnings, and unmatched coverage. The comparison path accepts external JSON, so malformed metrics or incompatible fixture dimensions could otherwise produce misleading results instead of a clear rejection. This commit addresses that risk with checks for time and allocation regressions, interpreter warnings, dimension mismatches, and non-finite threshold values. The final commit will document report comparison and conclude the maintained benchmark workflow.
The benchmark guide explains workload selection, phase boundaries, sampling behavior, and report metadata, while the command can now compare saved results. Users cannot discover the compatibility requirements, regression threshold behavior, environment warnings, or baseline refresh policy for that comparison workflow. This commit addresses that gap with a comparison example and guidance for interpreting medians, rejecting mismatched inputs, opting into failure status, and refreshing an accepted baseline. The benchmark workflow now runs against supported public APIs, covers matching and attribution costs, guards API drift in CI, and supports reproducible trend review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The project has a structural matching benchmark intended to report resource use for one of its most computationally expensive workflows.
The benchmark depends on removed buffer and mapping interfaces, so it cannot run from a current source checkout. It also lacks representative attribution coverage, repeatable sampling, compatible-report checks, and CI protection against later API drift.
This pull request addresses those gaps by rebuilding the runner around public matching and attribution APIs. It provides deterministic quick and full workloads, separates timing from Python allocation sampling, records reproducibility metadata, compares compatible JSON reports, and adds focused tests, CI smoke artifacts, and user documentation.
The maintained workflow now covers structural matching, repository-backed ownership attribution, bounded fixture storage, and trend review across revisions. Validation includes 3,190 repository tests, Python 3.10 and 3.13 smoke runs, a 50,000-line workload, a 1,000-batch workload, and a strict documentation build.