Repository navigation

Add benchmark pipeline with Rust-native A/B validation - #912

Merged

Yuanyuan Tian (YuanyuanTian-hh) merged 42 commits into

mainfrom

user/tianyuanyuan/benchmark-regression

Apr 17, 2026

Commits on Apr 9, 2026
- Migrate DiskANN benchmark pipeline from ADO to GitHub Actions
  Yuanyuan Tian (from Dev Box)
  committed
  059d857
- Fix openai-100K distance metric and add gitignore patterns
  Yuanyuan Tian (from Dev Box)
  committed
  8e3c7df
- Add push trigger to benchmarks workflow for pre-merge testing
  Yuanyuan Tian (from Dev Box)
  committed
  9101383
- Fix baseline run: use input config from current branch checkout
  Yuanyuan Tian (from Dev Box)
  committed
  5f43fb6
- Fix markdown conversion: replace broken csvtomd with inline Python
  Yuanyuan Tian (from Dev Box)
  committed
  19003b9
- Update benchmark configs: align build/search params to fix low recall
  Yuanyuan Tian (from Dev Box)
  committed
  262bf5d
- Fix recall_at: set to 100 to match groundtruth file K=100
  Yuanyuan Tian (from Dev Box)
  committed
  1699b47
- Remove stale absolute contracts (qps/recall/total_time): calibrated for ADO mimir-enron, not applicable to public datasets on GitHub runners. Threshold calibration tracked in PBI.
  Yuanyuan Tian (from Dev Box)
  committed
  635db8b
- remove comments
  Yuanyuan Tian (from Dev Box)
  committed
  4a0ac3f
- Fix build_ram_limit_gb: reduce 10->4 to fit GitHub runner RAM (7GB limit)
  Yuanyuan Tian (from Dev Box)
  committed
  8f40af3
- Fix wikipedia distance: cosine_normalized->cosine (vectors are not L2-normalized, metric is inner product)
  Yuanyuan Tian (from Dev Box)
  committed
  5d9e0f0
- Fix wikipedia distance: cosine->inner_product (groundtruth uses raw ip, not cosine similarity)
  Yuanyuan Tian (from Dev Box)
  committed
  54ac011
- Align build threads
  Yuanyuan Tian (from Dev Box)
  committed
  8a56a91
- Speed up benchmarks: build threads 1->4, openai pq_chunks 384->192
  Yuanyuan Tian (from Dev Box)
  committed
  0e36b9d
- Temp: disable concurrency cancellation for A/A batch testing
  Yuanyuan Tian (from Dev Box)
  committed
  f7c7f9d
- revert A/A test settings, update OpenAI config to SQ_1_2.0
  Yuanyuan Tian (from Dev Box)
  committed
  18c5faf
- Remove micro-benchmark-iai comments
  Yuanyuan Tian (from Dev Box)
  committed
  5ee4506
- use GitHub Release assets for benchmark datasets
  Yuanyuan Tian (from Dev Box)
  committed
  9ffdefd
- extract csv-to-markdown into reusable script
  Yuanyuan Tian (from Dev Box)
  committed
  18310cb
- calibrate contract thresholds from GitHub runner data
  Yuanyuan Tian (from Dev Box)
  committed
  6c70dfa
- add daily A/A benchmark stability test with failure notification
  Yuanyuan Tian (from Dev Box)
  committed
  046db04
- widen mean_cpus threshold to 15% for shared-runner CPU noise
  Yuanyuan Tian (from Dev Box)
  committed
  128791f
- move benchmark datasets to separate repo (YuanyuanTian-hh/diskann-benchmark-data)
  Yuanyuan Tian (from Dev Box)
  committed
  0c10c8c
- consolidate 3 benchmark scripts into single benchmark_validate.py
  Yuanyuan Tian (from Dev Box)
  committed
  be08e1f
- switch benchmark jobs to self-hosted 1ES runner pool (diskann-github)
  Yuanyuan Tian (from Dev Box)
  committed
  938e0b4
- replace gh CLI with curl for dataset downloads (gh not available on 1ES runners)
  Yuanyuan Tian (from Dev Box)
  committed
  1a510b2
- fix latency_95: read p95_latency instead of p999_latency from benchmark JSON
  Yuanyuan Tian (from Dev Box)
  committed
  3775fa9
- revert to ubuntu-latest runners, switch dataset source to BAB v0.4.0
  Yuanyuan Tian (from Dev Box)
  committed
  128886a
- address PR review: reduce search_list to 200, remove hardcoded Rust version, fix missing-field handling, clean up orphaned thresholds, switch data source to BAB v0.4.0
  Yuanyuan Tian (from Dev Box)
  committed
  59a0343
- widen latency_95 threshold to 15% for shared-runner noise
  Yuanyuan Tian (from Dev Box)
  committed
  92e36f6
- replace push trigger with pull_request trigger targeting main
  Yuanyuan Tian (from Dev Box)
  committed
  2d2a06d
- implement Regression trait for disk-index benchmarks
  Yuanyuan Tian (from Dev Box)
  committed
  6a70f36
- test: set strict tolerances (0.1%) to verify pipeline failure detection
  Yuanyuan Tian (from Dev Box)
  committed
  2b9e57b
- revert tolerances to production values
  Yuanyuan Tian (from Dev Box)
  committed
  e55a8fa
- test for failure
  Yuanyuan Tian (from Dev Box)
  committed
  a1d1e06
- revert test code
  Yuanyuan Tian (from Dev Box)
  committed
  f2c14e8
- address review: refactor workflows to matrix strategy, update build params
  Yuanyuan Tian (from Dev Box)
  committed
  0aefcf7
Commits on Apr 10, 2026
- Address hildebrandmw review: Direction enum, Table display, composite action, workflow renames
  Yuanyuan Tian (from Dev Box)
  committed
  85199c2
- Fix E0521: clone strings for Table::Row::insert ('static bound)
  Yuanyuan Tian (from Dev Box)
  committed
  9d96668
Commits on Apr 14, 2026
- Address review: String metric, remove write perm, rustup show, drop cache
  Yuanyuan Tian (from Dev Box)
  committed
  12d6db8
Commits on Apr 15, 2026
- Change A/A failure notification to @microsoft/diskann-disk-maintainers
  Yuanyuan Tian (from Dev Box)
  committed
  68673a9
Commits on Apr 17, 2026
- Merge branch 'main' into user/tianyuanyuan/benchmark-regression
  Yuanyuan Tian (YuanyuanTian-hh)
  authored
  a553fff

Commits

Commits on Apr 9, 2026

Commits on Apr 10, 2026

Commits on Apr 14, 2026

Commits on Apr 15, 2026

Commits on Apr 17, 2026