[M-Chris]

A change that touches several profiles had two ways to be measured, both wrong for it. One -t run per profile is one approval and one results commit each, posted only after the previous run finishes, because a PR holds one pending run; seven touched profiles is a day of polling the thread. A full --save is one approval but re-measures every profile, and moves the untouched ones by several percent of machine noise (gateway-64 +15%, production-stack -4% in one run), so the real change arrives mixed with drift.

-t now accepts a comma-separated list, one job, one approval, one results commit, and only the named profiles are touched:

/benchmark -f morojs -t gateway-h3,gateway-64,production-stack --save

benchmark.sh resolves the list at argument parsing, in the order given with duplicates dropped, and a name that is not a profile fails there, before any image is built or the host tuned; previously an explicit typo was warned past and the run measured nothing while looking like a run. benchmark-lite.sh does the same against its own table. compare.sh filters on the set.

pr-commands.yml checks every name in the list with the same pattern as before. A list with a name that fails the check is rejected with a comment instead of being reduced to an empty profile: an empty profile means "all tests", and with --save that dispatched a full run nobody asked for, which is exactly how an illustrative example in a PR comment on #1699 queued one. The all-frameworks resolution in both workflows subscribes a framework when it runs any profile in the list; benchmark.sh skips the ones it does not.

README, the PR template and the running-locally docs gain the list form.

Description

---

PR Commands — comment on this PR to trigger (requires collaborator approval):

Command	Description
/benchmark -f <framework>	Run every test the framework subscribes to
/benchmark -f <framework> -t <test>	Run one test only
/benchmark -f <framework> --save	Run and save results (updates the leaderboard on merge)
/benchmark -f <framework> -t <test> --save	Run one test and save results
/benchmark -f <framework> --compare <other>	Measure the deltas against another framework instead of this one
/benchmark-multiple -f <fw1>,<fw2>,...	Benchmark several frameworks in one run — takes -t and --save too; saved results land in a single commit
/benchmark-multiple --save	No -f needed: benchmark and save every framework the PR touches
/benchmark-test -t <test>	Benchmark all enabled frameworks subscribed to <test> and save the results

For /benchmark, always specify -f <framework>; the flags combine in any order. Results come back as a comment with a per-profile table of RPS, p99, CPU and memory — one table per framework on multi runs. A new benchmark comment while a run is in flight queues behind it (one deep) instead of cancelling it. For multi-framework PRs (dependency bumps, same-language refactors) prefer /benchmark-multiple, which runs everything in a single job and commits all saved results together, so no run overwrites another. --compare works on single-framework runs only.

What the deltas are measured against. By default, this framework's own results published on main - answering "did this change help?". When you are tuning a variant or a successor entry, --compare re-bases them on another entry instead:

/benchmark -f genhttp --compare genhttp-kestrel

The reply states which baseline it used, and profiles the other framework does not run show n/a rather than a delta.

---

Run benchmarks locally

You can validate and benchmark your framework locally with the lite script — no CPU pinning, fixed connection counts, all load generators run in Docker.

./scripts/validate.sh <framework>
./scripts/benchmark-lite.sh <framework> baseline
./scripts/benchmark-lite.sh --load-threads 4 <framework>

Requirements: Docker Engine on Linux. Load generators (gcannon, h2load, h2load-h3, wrk) are built as self-contained Docker images on first run.

[M-Chris]

@MDA2AV heres the change we just talked about