usagebench

UsageBench is Bifrost's curated LSP-parity and recurring regression suite for the static-analysis task of discovering usages of source symbols.

The Starlight site under docs/ explains the comparison methodology, current Bifrost-versus-LSP results, and case-level language differences.

Mature language servers provide the baseline and calibration evidence. Bifrost is expected to match them where their behavior agrees with reviewed language semantics, and may preserve justified precision improvements or additional static coverage. The benchmark format remains analyzer-neutral so future competitors can be evaluated against the same source contracts.

LSPs primarily serve developers through interactive editor features. Bifrost serves repository code analysis and navigation for machine consumers, especially coding agents and static-analysis tools. UsageBench measures their overlapping usage and navigation contract: parity preserves the quality developers expect, while reviewed Bifrost-specific edges exercise the analysis substrate agents depend on. It does not compare either product's complete feature surface.

The independently reviewed real-project-v1 evaluation is published as immutable release v0.2.0. It compares Bifrost with gopls, Pyright, and TypeScript language server across 12 source-only sampled repositories and 36 preregistered cases. The broader 24 July 2026 ten-language development run remains available as historical regression evidence and is not pooled with the evaluation result.

The benchmark corpus is authored by source location instead of by an analyzer-specific symbol ID. Each case points at a declaration, expected usage sites, and reverse usage-to-declaration probes using LSP-shaped ranges.

Immutable result pages

Benchmark releases package each raw report, a manifest with its SHA-256 digest, and generated result fragments. Generate (or check) those fragments from an extracted release evidence directory with:

cargo run -- generate-results \
  --manifest evidence/freeze-manifest.json \
  --output-directory generated-results

cargo run -- generate-results \
  --manifest evidence/freeze-manifest.json \
  --output-directory generated-results \
  --check

The generator rejects missing, altered, mismatched, or non-snapshot reports before it writes a page. Evaluation pages additionally verify the freeze ID, bounded claim scope, artifact provenance, denominators, exclusions, and replacements, and cannot mix development documents into an evaluation claim. freeze-manifest writes a sibling *.timings.json even when validation fails; the freeze workflow retains it together with staging, corpus hashing, frozen-input hashing, report validation, and manifest-writing timings. Staged bundles also contain .usagebench-corpus-hashes.json, whose root digest binds the exact input file set and is independently reverified during shard aggregation before its per-file hashes are reused. Location-level precision, recall, and range-quality tables require reports produced by UsageBench 0.2.0 or newer; older reports remain readable for their existing strict and required-destination fields but are never interpreted as having zero location metrics. The current site summary is derived from the hash-bound v0.2.0 release. The hand-authored 24 July summary predates the release evidence pipeline and is retained only as explicitly labelled development history.

Directory Structure

• benchmarks: Authored benchmark case files and corpus documentation.
• fixtures: Small in-repository source corpora used by the baseline cases.
• schema: JSON Schema for benchmark case documents.
• src: Rust validation CLI, schema model, and analyzer runner adapters.
• adapters/lsp: Versioned language-server profiles and reproduction notes.
• containers/reference: Versioned, digest-pinned reference environments.
• scripts: Local image build, offline execution, and report reproduction tools.
• docs: Public Starlight content plus adapter design notes and execution plans.

Validating Benchmark Cases

Benchmark cases use YAML authored around an LSP-shaped location model. Validate them with:

cargo run -- validate benchmarks/cases

CI runs the same Rust test and validation path:

cargo test
cargo run -- validate benchmarks/cases

Git LFS

The pinned real-project evaluation source archives under benchmarks/evaluation/real-project-v1/sources/ are tracked with Git LFS. A checkout must materialize those objects before validating or running the full corpus:

git lfs install
git lfs pull

GitHub workflows that consume the full corpus enable LFS during checkout.

Development Corpus

The current corpus uses small checked-in fixtures for Java, Go, Python, TypeScript, JavaScript, Rust, Scala, C#, PHP, C++, and Ruby. These fixtures are the source of truth for issue #8; the older broad Java/Go/Python generator stack has been removed from the active benchmark path.

Each document is schema v2 and explicitly labeled development, analyzer_informed, and legacy_unattributed. All 158 current cases across 35 documents have completed a first human review, preserved in benchmarks/reviews/2026-07-17-DavidBakerEffendi.md. This remains a regression and diagnosis corpus, not an independently reviewed evaluation partition: promotion still requires a second reviewer, preregistered selection, and a freeze ID.

Evaluation Corpus

benchmarks/cases/evaluation/real-project-v1 is a separate, preregistered, source-locked slice: four repositories and three declarations for each of Go/gopls, Python/Pyright, and TypeScript/TypeScript LS, for 12 repositories and 36 cases. Its source archives, selection, review, adjudication, and source lock are checked in and hash-bound. The historical same-provider review is retained, and all 36 cases now also have blinded OpenAI and Anthropic derivations plus accountable human adjudication under the schema-v2 protocol. Its analyzer evidence is frozen in immutable release v0.2.0. Every published table is evaluation-only and limited to the protocol's descriptive per-profile claim; it is not pooled with development fixtures or presented as language-wide, ecosystem-wide, causal, or performance evidence.

Analyzer Runners

Runner adapters under src/runners translate tool-specific output into one analyzer-neutral report shape. Every report records the requested and resolved tool version plus per-operation capability levels (native, recovered, or unsupported). Reports distinguish exact passes, position-unverified results, hard failures, and runner errors, and separate development from evaluation totals. Print the JSON Schema with:

cargo run -- report-schema

The existing Bifrost command remains stable:

cargo run -- run-bifrost benchmarks/cases \
  --bifrost-repo /path/to/bifrost \
  --bifrost-working-tree

run-bifrost supplies an explicit 300-second wall-clock budget to each scan_usages_by_location call. This batch-oriented default avoids inheriting Bifrost's five-second interactive budget and is recorded in report invocation metadata. Use --scan-usages-max-duration-secs <0-300> to select a smaller per-scan budget for development runs. When --output is set, the runner also atomically updates a sibling *.partial.json checkpoint after each completed benchmark document so interrupted batch runs retain valid evidence. Checkpoints record completed: false plus the full requestedCaseFiles scope and are rejected by snapshot and release tooling as completed evidence.

Source-based runs also retain a Bifrost build cache in --work-dir. A build is reused only when the resolved commit, Cargo and rustc identity, build plan, target/profile/compiler environment, and cached executable metadata still match. The executable SHA-256 is verified on the first use and reused only while its size, modification/change time, device, and inode identity remain unchanged (with the portable metadata subset used off Unix). The same verified provenance cache applies to --bifrost-binary; the required --bifrost-resolved-commit remains report provenance and is not a caller- supplied digest. Reports and CLI output separate checkout/setup, build, provenance hashing, workspace readiness, and analyzer query milliseconds.

The generic LSP adapter starts a versioned stdio language server, opens an isolated fixture workspace, and translates the protocol's native references, definition, and type-definition responses into the same report:

cargo run -- run-lsp benchmarks/cases \
  --profile adapters/lsp/gopls.json \
  --output benchmark-output/gopls-v0.23.0.json

Profiles cover all eleven corpus languages through clangd, Roslyn and csharp-ls, gopls, Eclipse JDT LS, Pyright, Ruby LSP, Metals, Intelephense, rust-analyzer, and typescript-language-server. The executable named by a profile must be installed or available through the profile's package launcher; --server-command can override only the executable while preserving its arguments. See the LSP profile guide for setup and the measured comparison.

The primary public profile registry distinguishes canonical reference runners from native LSP profiles. Bifrost and gopls use the canonical linux/amd64 environment. Each remaining advertised LSP row is frozen from one pinned run whose report records the profile checksum, resolved analyzer identity, executable provenance, environment, and result checksum. Apple clangd 21 and upstream clangd 22 remain separate candidate identities.

UsageBench deliberately stays focused on the LSP-shaped task of finding symbol references and navigating those references back to declarations and types. Broader analysis contracts should use sibling suites—for example, a future callbench for call-graph resolution and taintbench for taint-flow analysis— so each benchmark can model its own ground truth without tool-specific private hooks or weakened semantics.

Releases, Citation, and License

Benchmark corpus releases use immutable SemVer tags such as v0.1.0. The benchmark release version is independent from the Rust package version in Cargo.toml and the benchmark document schemaVersion. See RELEASES.md for the version policy and curated release contents.

Use the root CITATION.cff when citing UsageBench. It describes the latest release and intentionally contains no placeholder DOI. If an archival service assigns a DOI later, the real version-specific identifier will be added to the citation file and that release's notes.

A run report records:

• usagebenchVersion: the Rust CLI and adapter implementation version;
• usagebenchRevision: the exact UsageBench commit, with -dirty when local changes prevent commit-only reproduction;
• usagebenchRelease: the vMAJOR.MINOR.PATCH corpus tag when available; and
• the logical invocation, native or container platform scope, analyzer executable checksum, toolchains, reference-environment definition, and runner's requested and resolved version.

Reference environment version 1 provides the canonical linux/amd64 path for Bifrost and gopls. Save a published container report as report.json, extract the release bundle named by the report, and run one command:

./scripts/reproduce-report.sh report.json reproduced.json

The command restores the checksum-addressed image when its complete identity and immutable registry digest verify, otherwise builds the recorded environment locally, reruns without network access, and compares the reports semantically. Release bundles still contain everything needed for an independent forced rebuild. See ARTIFACT.md for the artifact-review procedure, security boundary, expected build cost, and manual commands. Native runner commands remain available for development but are labeled host-specific and are not the canonical reproducibility claim.

If usagebenchRevision ends in -dirty or usagebenchRelease is absent, the report identifies a development run and is rejected by the canonical reproduction command.

UsageBench is licensed under the permissive MIT License, covering the corpus fixtures, assertions, adapter profiles, and harness code in this repository.

Recurring Bifrost Regression Run

The daily GitHub Actions workflow in .github/workflows/benchmark.yml runs the curated corpus against Bifrost master on ubuntu-latest. This makes every reviewed parity decision, precision edge, and known gap part of a recurring regression signal rather than a one-time comparison.

The workflow:

• validates benchmarks/cases
• checks out BrokkAi/bifrost
• builds usagebench
• runs usagebench run-bifrost benchmarks/cases with the versioned current expected-pass overlay
• uploads the JSON report from benchmark-output
• publishes a GitHub step summary
• optionally posts a payload to Slack

Scheduled runs use Bifrost master. Manual workflow_dispatch runs from the default UsageBench branch can set a specific bifrost_ref and can opt into cases marked unsupported.

Without --bifrost-working-tree, run-bifrost creates an isolated checkout under target/usagebench and checks out --bifrost-commit.

The Bifrost checkout is public and does not use a separate checkout secret. Checkout credentials are removed before repository-controlled commands run.

Slack delivery is best-effort and does not change the benchmark result. To enable it, configure the repository secret SLACK_DAILY_USAGEBENCH_WEBHOOK_URL. The workflow sends a benchmark-specific payload with:

• ok
• error_text
• workflow_run_url
• head_sha_short
• usagebench_revision
• usagebench_release
• bifrost_ref
• bifrost_sha_short
• run_outcome
• cases_count
• passed_count
• improved_count
• total_passed_count
• failed_count
• expected_failures_count
• not_planned_count
• unsupported_count
• skipped_count
• errors_count

Repository-enforced controls and the required administrative settings are documented in GitHub Actions security operations.