Repository navigation

Add DiskANN benchmark pipeline to GitHub Actions - #857

Draft

Yuanyuan Tian (YuanyuanTian-hh) wants to merge 31 commits into

mainfrom

user/tianyuanyuan/add-benchmark-pipeline

Draft

Add DiskANN benchmark pipeline to GitHub Actions#857
Yuanyuan Tian (YuanyuanTian-hh) wants to merge 31 commits into
mainfrom
user/tianyuanyuan/add-benchmark-pipeline

Commits on Mar 19, 2026
- Migrate DiskANN benchmark pipeline from ADO to GitHub Actions
  Yuanyuan Tian (from Dev Box)
  committed
  c6402fb
Commits on Mar 20, 2026
- Fix openai-100K distance metric and add gitignore patterns
  Yuanyuan Tian (from Dev Box)
  committed
  2217353
- Add push trigger to benchmarks workflow for pre-merge testing
  Yuanyuan Tian (from Dev Box)
  committed
  6b7b250
- Fix baseline run: use input config from current branch checkout
  Yuanyuan Tian (from Dev Box)
  committed
  d4a6abd
- Fix markdown conversion: replace broken csvtomd with inline Python
  Yuanyuan Tian (from Dev Box)
  committed
  c6d54a9
- Update benchmark configs: align build/search params to fix low recall
  Yuanyuan Tian (from Dev Box)
  committed
  5775198
Commits on Mar 23, 2026
- Fix recall_at: set to 100 to match groundtruth file K=100
  Yuanyuan Tian (from Dev Box)
  committed
  7ac841e
- Remove stale absolute contracts (qps/recall/total_time): calibrated for ADO mimir-enron, not applicable to public datasets on GitHub runners. Threshold calibration tracked in PBI.
  Yuanyuan Tian (from Dev Box)
  committed
  7d5a72e
- remove comments
  Yuanyuan Tian (from Dev Box)
  committed
  c294c16
- Fix build_ram_limit_gb: reduce 10->4 to fit GitHub runner RAM (7GB limit)
  Yuanyuan Tian (from Dev Box)
  committed
  b8e1d6d
- Fix wikipedia distance: cosine_normalized->cosine (vectors are not L2-normalized, metric is inner product)
  Yuanyuan Tian (from Dev Box)
  committed
  137cae0
- Fix wikipedia distance: cosine->inner_product (groundtruth uses raw ip, not cosine similarity)
  Yuanyuan Tian (from Dev Box)
  committed
  96b63a8
Commits on Mar 24, 2026
- Align build threads
  Yuanyuan Tian (from Dev Box)
  committed
  1b4cfc8
- Speed up benchmarks: build threads 1->4, openai pq_chunks 384->192
  Yuanyuan Tian (from Dev Box)
  committed
  5e6a6e0
- Temp: disable concurrency cancellation for A/A batch testing
  Yuanyuan Tian (from Dev Box)
  committed
  59d25b9
Commits on Mar 26, 2026
- revert A/A test settings, update OpenAI config to SQ_1_2.0
  Yuanyuan Tian (from Dev Box)
  committed
  751e775
Commits on Mar 27, 2026
- Remove micro-benchmark-iai comments
  Yuanyuan Tian (from Dev Box)
  committed
  2c4d235
- use GitHub Release assets for benchmark datasets
  Yuanyuan Tian (from Dev Box)
  committed
  f58e084
- extract csv-to-markdown into reusable script
  Yuanyuan Tian (from Dev Box)
  committed
  0f5c277
- calibrate contract thresholds from GitHub runner data
  Yuanyuan Tian (from Dev Box)
  committed
  b8cddb5
- add daily A/A benchmark stability test with failure notification
  Yuanyuan Tian (from Dev Box)
  committed
  accdf2c
- widen mean_cpus threshold to 15% for shared-runner CPU noise
  Yuanyuan Tian (from Dev Box)
  committed
  5b0e7f7
Commits on Mar 30, 2026
- move benchmark datasets to separate repo (YuanyuanTian-hh/diskann-benchmark-data)
  Yuanyuan Tian (from Dev Box)
  committed
  639d4bb
- consolidate 3 benchmark scripts into single benchmark_validate.py
  Yuanyuan Tian (from Dev Box)
  committed
  1b52de2
Commits on Mar 31, 2026
- switch benchmark jobs to self-hosted 1ES runner pool (diskann-github)
  Yuanyuan Tian (from Dev Box)
  committed
  9779e3c
- replace gh CLI with curl for dataset downloads (gh not available on 1ES runners)
  Yuanyuan Tian (from Dev Box)
  committed
  db7e97a
- fix latency_95: read p95_latency instead of p999_latency from benchmark JSON
  Yuanyuan Tian (from Dev Box)
  committed
  ea91c62
Commits on Apr 1, 2026
- revert to ubuntu-latest runners, switch dataset source to BAB v0.4.0
  Yuanyuan Tian (from Dev Box)
  committed
  360cdc7
Commits on Apr 2, 2026
- address PR review: reduce search_list to 200, remove hardcoded Rust version, fix missing-field handling, clean up orphaned thresholds, switch data source to BAB v0.4.0
  Yuanyuan Tian (from Dev Box)
  committed
  9d635be
- widen latency_95 threshold to 15% for shared-runner noise
  Yuanyuan Tian (from Dev Box)
  committed
  4ca80d1
- replace push trigger with pull_request trigger targeting main
  Yuanyuan Tian (from Dev Box)
  committed
  d168932

Commits

Commits on Mar 19, 2026

Commits on Mar 20, 2026

Commits on Mar 23, 2026

Commits on Mar 24, 2026

Commits on Mar 26, 2026

Commits on Mar 27, 2026

Commits on Mar 30, 2026

Commits on Mar 31, 2026

Commits on Apr 1, 2026

Commits on Apr 2, 2026