Benchmarks in MR/PR: List differences to main branch

[sophie-h]

Maybe this feature exists, and I just can't find it in the docs:

I am running benchmarks on bencher.dev's bare-metal runners for my main branch. For merge requests, I have a manual job that can be started to check the performance changes the merge request brings. However

• The output does not give me a comparison of the performance changes towards the main branch. --start-point sounds like the thing to use ("Historical Metrics and optionally Thresholds are copied over from the Start Point."). But I still don't get a comparison in the output of bencher run
• I also don't want to store the benchmarks for MRs permanently, so I'm not sure what to supply as a --branch because no matter what I supply, it will be stored under that branch permanently.
• The bencher run output links me to the isolated results of the run. But I would like to have an overview of all results, in comparison to the main branch. Maybe as graphs with the new merge request data point marked?

The goal is to post these results as comment in GitLab, which is trivial to add as a curl command if I have the output from bencher. The GitLab metrics report would probably solve all my issues, but GNOME runs the free version of GitLab, which doesn't have the feature.

[epompeii]

Hey @sophie-h thank you for using Bencher, and I'm sorry things aren't clear.

I would recommend checking out how to use Bencher in GitLab CI/CD, since you're using GitLab. That should get you started.

There's also a CI provider agnostic guide for how to track benchmarks in CI with Bencher.

I am running benchmarks on bencher.dev's bare-metal runners for my main branch.

The only difference for Bencher Bare Metal is that you'll need to include the --image option. Otherwise, bencher run should work just the same as a local run.

The goal is to post these results as comment in GitLab, which is trivial to add as a curl command if I have the output from bencher.

Adding built-in GitLab MR support is something I've had on my todo for a while. PRs very welcome!

In the meantime, you can just use the --format html option and the --quiet flag in order to get an HTML comment to add to the MR.

If you're looking to accept MRs from public forks, you will probably need to do a something similar to handling pull request from forks on GitHub.

The GitLab metrics report would probably solve all my issues

My understanding is that GitLab still doesn't support graphing metrics over time nor does it have configurable thresholds and alerts. Those are two of the main use cases for most Bencher users.

Does any of that help?

[sophie-h]

I have all the material that you linked, and there is no new information for me.

1. The output still doesn't show me a comparison to the main branch (like +15%)
2. There is still no link to a plot that includes the main branch and my branch data points
3. I still don't know what to supply as a branch to not keep data of merge requests around forever

Here is an example output

---

Bencher Report

Branch	wip/sophie-h/bencher-output
Testbed	Intel v1

⚠️ WARNING: No Threshold found!

Without a Threshold, no Alerts will ever be generated.

• Duration (nanoseconds (ns))

Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the --ci-only-thresholds flag.

Click to view all benchmark results

Benchmark	Duration	microseconds (µs)
tiny.png/GdkPixbuf	📈 view plot ⚠️ NO THRESHOLD	727.88 µs
tiny.png/glycin	📈 view plot ⚠️ NO THRESHOLD	606.08 µs

🐰 View full continuous benchmarking report in Bencher

---

Bencher Report

Branch	wip/sophie-h/bencher-output
Testbed	Intel v1

⚠️ WARNING: No Threshold found!

Without a Threshold, no Alerts will ever be generated.

• file_size (Measure (units))

Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the --ci-only-thresholds flag.

Click to view all benchmark results

Benchmark	file_size	Measure (units) x 1e3
glycin-heif	📈 view plot ⚠️ NO THRESHOLD	2,623.65 units x 1e3
glycin-image-rs	📈 view plot ⚠️ NO THRESHOLD	6,146.34 units x 1e3
glycin-jxl	📈 view plot ⚠️ NO THRESHOLD	2,619.55 units x 1e3
glycin-svg	📈 view plot ⚠️ NO THRESHOLD	2,164.88 units x 1e3
glycin-thumbnailer	📈 view plot ⚠️ NO THRESHOLD	471.82 units x 1e3

🐰 View full continuous benchmarking report in Bencher

[epompeii]

Thank you for the example output! That's super helpful.

1. The output still doesn't show me a comparison to the main branch (like +15%)

You need to create a Threshold. In order to show you something like +15%, Bencher needs to know what the baseline ought to be, and Thresholds are the way to define that baseline. That's what all of the ⚠️ WARNING: No Threshold found!s are about.

2. There is still no link to a plot that includes the main branch and my branch data points

I'm looking at the commit here: GNOME/glycin@98aebb4#diff-037ea159eb0a7cb0ac23b851e66bee30fb838ee8d0d99fa331a1ba65283d37f7R69-R70

--start-point "$CI_MERGE_REQUEST_TARGET_BRANCH_NAME"

As long as you have data on the target branch, ie main and you use the --start-point option (like you are) then the plot links like the ones you provided include historical data from the target branch: 📈 view plot

If you click on the first Metric in the plot you can see that it originates from the main branch:

You probably want to also add the --start-point-reset flag. This basically resets your MR branch onto the latest target branch's results each run. This is useful to prevent drift in Statistical Continuous Benchmarking data.

3. I still don't know what to supply as a branch to not keep data of merge requests around forever

Once you're done with a branch you can Archive it. With the CLI this is the bencher archive command. There is an example in the GitLab CI/CD Merge Request docs.

      if [ "$MR_STATE" = "closed" ]; then
        bencher archive \
        --project project-abc4567-wxyz123456789 \
        --token "$BENCHER_API_TOKEN" \
        --branch "$CI_COMMIT_REF_NAME"
      else
        echo 'Merge request is not `closed`. Skipping archival.'
      fi

I just merged in a PR to allow the project keys to be able to archive branches (#845), so you will be able to use --key bencher_run_abc123WXYZ here after the next release.

[sophie-h]

It doesn't seem to work as described.

As soon as I set --branch to something different than main, the threshold is no longer recognized (WARNING: No Threshold found!). However, for this to make sense, it should use the threshold for the branch from --start-point t, right?

If I set --branch main, I have a threshold and relative data output. But of course, this then gets recorded into the main stats.

The --start-point-reset flag is for when multiple benchmarks are run in the same merge request, to not compare the data to the previous run in the MR but to main?

[epompeii]

As soon as I set --branch to something different than main, the threshold is no longer recognized

Ah! In order to clone the Thresholds from your Start Point (ie main) onto your MR Branch, you need to use the --start-point-clone-thresholds flag.

This is what we're doing in the GitLab CI/CD Merge Request example.

You can also construct Thresholds for the MR Branch explicitly if you want them to be different from the Start Point Thresholds using the --threhsold-* options.