[Mark Hildebrand (hildebrandmw)]

diskann-benchmark-runner has supported Regression tests for a while, but they are kind of hard to write. This attempts to make that easier by adding some infrastructure in diskann-benchmark-runner for collecting hierarchical pass/fail results, rolling up the overall success, and displaying results.

The final result is far from perfect, but useable enough that it's not worth investing more time into. Plus, I have a whole host of other things that I need to be working on.

This PR is broken into three commits:

Commit 1

This is the big one, including two new big modules in diskann-benchmark-runner: report/mod.rs and report/tolerance.rs.

tolerances.rs
This introduces fairly robust tools for doing regression checks with tolerance. The checks include:

• MaxRelativeDecrease: Set a bound on how much a value can decrease relative to a baseline using the formula

  new >= old * (1 - max-relative-decrease)
  

  For this, the max relative decrease must be a number in [0, 1] and the values being compared must be positive (no zeros allowed) and finite. All Rust's built-in numeric types up to 64-bits are supported.
  • MaxRelativeIncrease: The flip of the previous one. Here, success is

  new <= old * (1 + max-relative-decrease)
  

  The max relative increase must be non-negative and finite.
• MaxAbsoluteIncrease/MaxAbsoluteDecrease: More straight-forward. An absolute number on how much values can increase or decrease by.

These values serialize by their name, so the tolerance JSON files are a bit more clear. For example, the tolerance file for the newly added regression tests for inmem2 look like the this:

  {
  "input": {
    "type": "",
    "content": {}
  },
  "tolerance": {
    "type": "inmem2-build-and-search-tolerance",
    "content": {
      "build": {
        "build_time": {
          "max-relative-increase": 0.05
        }
      },
      "search": {
        "qps": {
          "max-relative-decrease": 0.05
        },
        "recall": {
          "max-absolute-decrease": 0.01
        }
      }
    }
  }
}

This provides a really nice chain of custody:

• Tolerance JSONs concretely describe what they are checking for.
• Tolerance implementations specify the concrete type of check they are performing. I.E., it's not possible to change "qps" to "max-relative-increase". Deserialization of the JSON will fail.
• Tolerance implementations also don't need to worry too much about actually applying the checks. The (extremely tedious) work has (hopefully) already been done.
• Finally, within the checks themselves, the visually identifiable lower_is_better/higher_is_better help show at the callsite the rough direction that is expected.

report/mod.rs
This adds a Report type that can consist of:

• A single Check.
• A collection of nested Reports with associated keys.
• A Table, which is an alternate presentation of Checks that can make better use of horizontal space.

Checks are leaf values that can display themselves, serialize themselves, and report success or failure. This PR includes several built-in checks:

• Info: Just informative to help when presenting results.
• Error: An error. Unlike Rust's normal error handling, all such errors will be collected and reported. This can help with aggregation across multiple sources instead of just reporting the first one.
• Equal: Assert that two values are equal.
• LowerIsBetter/HigherIsBetter: Checks based on tolerance.rs.

I did my best to make the presentation of the final Report palatable. I'll be the first to say that my skills in text-based presentation are sorely lacking. I'll accept "correct but adequate".

General Flow
Since our benchmark output structs are generally hierarchical, the goal of Report is to make it relatively straight forward to have a unified type that can reflect the structure of the output types, point paths to all failing checks, and present the results in an okay manner.

Commit 2

This replaces the checks in diskann-inmem's integration test with Report. Largely, Report is a souped up version of the Match that was previously used.

Commit 3

Adds regression checks for built time, QPS, and recall for diskann-inmem in diskann-benchmark. This hopefully shows that it is relatively straight-forward to add these checks.

[Copilot]

🟡 Changes recommended

Quantized benchmark registrations bypass regression checks, and saved regression output discards configured tolerance limits, leaving them unavailable when infrastructure errors prevent a result report.

2 open issues

• Preserve tolerance limits in saved regression output · New
• Register quantized benchmarks as regressions · New

What changed in this PR

Adds reusable regression checking and reporting to diskann-benchmark-runner, then adopts it for in-memory integration tests and benchmark comparisons.

Changes:

• Introduces hierarchical and tabular reports with aggregated pass/fail status, serialization, and numeric tolerances.
• Replaces integration tests’ custom Match machinery with shared reports.
• Adds build-time, QPS, and recall regression checks for inmem2 benchmarks.

File	Description
diskann-inmem/​integration/​support/​check.rs	Replaces custom matching infrastructure with shared reports.
diskann-inmem/​integration/​index/​tests.rs	Migrates recall and search-stat checks to reports.
diskann-inmem/​integration/​index/​runner.rs	Uses finished reports for regression results.
diskann-inmem/​integration/​index/​object.rs	Migrates search and counter comparisons to reports.
diskann-benchmark/​src/​index/​inmem2/​quantized/​mod.rs	Adds quantized benchmark regression implementations.
diskann-benchmark/​src/​index/​inmem2/​full.rs	Registers full-precision benchmarks for regression checks.
diskann-benchmark/​src/​index/​inmem2/​benchmark/​mod.rs	Combines build and search tolerances and reports.
diskann-benchmark/​src/​index/​inmem2/​benchmark/​knn.rs	Adds QPS and recall comparisons in tabular reports.
diskann-benchmark/​src/​index/​inmem2/​benchmark/​build.rs	Adds build-time and inserted-vector checks.
diskann-benchmark-runner/​src/​utils/​fmt.rs	Supports forced nesting and formatter size queries.
diskann-benchmark-runner/​src/​report/​tolerance.rs	Implements and tests absolute and relative tolerances.
diskann-benchmark-runner/​src/​report/​mod.rs	Implements and tests reports, checks, tables, and rendering.
diskann-benchmark-runner/​src/​lib.rs	Exposes the report module.
diskann-benchmark-runner/​Cargo.toml	Adds serialization and testing dependencies.
Cargo.lock	Records updated dependency resolution.

_{Review effort: Balanced}

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

❌ Patch coverage is 90.21991% with 169 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.02%. Comparing base (7545fda) to head (db567e4).

Files with missing lines	Patch %	Lines
...iskann-benchmark/src/index/inmem2/benchmark/knn.rs	0.00%	40 Missing ⚠️
diskann-benchmark-runner/src/report/mod.rs	95.40%	32 Missing ⚠️
...iskann-benchmark/src/index/inmem2/benchmark/mod.rs	9.09%	30 Missing ⚠️
...kann-benchmark/src/index/inmem2/benchmark/build.rs	0.00%	21 Missing ⚠️
diskann-benchmark-runner/src/report/tolerance.rs	97.93%	18 Missing ⚠️
...iskann-benchmark/src/index/inmem2/quantized/mod.rs	20.00%	12 Missing ⚠️
diskann-benchmark/src/index/inmem2/full.rs	25.00%	9 Missing ⚠️
diskann-inmem/integration/support/check.rs	60.00%	4 Missing ⚠️
diskann-benchmark-runner/src/utils/fmt.rs	86.36%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1521      +/-   ##
==========================================
- Coverage   91.05%   91.02%   -0.04%     
==========================================
  Files         602      605       +3     
  Lines      124137   125494    +1357     
==========================================
+ Hits       113039   114237    +1198     
- Misses      11098    11257     +159     

Flag	Coverage Δ
miri	91.02% <90.21%> (-0.04%)	⬇️
unittests	90.74% <90.21%> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
diskann-inmem/integration/index/object.rs	98.96% <100.00%> (-0.05%)	⬇️
diskann-inmem/integration/index/runner.rs	91.78% <100.00%> (ø)
diskann-benchmark-runner/src/utils/fmt.rs	98.23% <86.36%> (-0.62%)	⬇️
diskann-inmem/integration/support/check.rs	81.81% <60.00%> (-15.71%)	⬇️
diskann-benchmark/src/index/inmem2/full.rs	56.70% <25.00%> (-3.30%)	⬇️
...iskann-benchmark/src/index/inmem2/quantized/mod.rs	84.41% <20.00%> (-7.14%)	⬇️
diskann-benchmark-runner/src/report/tolerance.rs	97.93% <97.93%> (ø)
...kann-benchmark/src/index/inmem2/benchmark/build.rs	80.40% <0.00%> (-9.49%)	⬇️
...iskann-benchmark/src/index/inmem2/benchmark/mod.rs	9.09% <9.09%> (ø)
diskann-benchmark-runner/src/report/mod.rs	95.40% <95.40%> (ø)
... and 1 more

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

🟡 Changes recommended

Redundant parentheses trigger an unused_parens warning in both floating-point macro expansions, causing CI compilation to fail under -Dwarnings.

1 open issue

• Remove redundant parentheses causing unused_parens build failure · New

2 resolved since last review

• Register quantized benchmarks as regressions
• Preserve tolerance limits in saved regression output

_{Review effort: Balanced}

[Magdalen Dobson Manohar (magdalendobson)]

Looks, good, just caught typos