Can we make the bot positive?

[davidlange6]

The k4beanch-bot messages are really useful..

However, they suggest that observed performance improvements are regressions (eg, key4hep/k4geo#578 (comment)).

Could it look at the sign of the performance change and make an appropriate message? (which could be performance improvement and/or physics bug introduced when workflows get significantly faster..)

[jbeirer]

Hi @davidlange6,

The wording here was meant neutrally: k4Bench flags any confirmed step beyond the baseline in either direction, without judging whether it's good or bad. I was using "regression" in the regression-testing sense, i.e. a deviation from the reference, not as a claim that things got worse.

The reason I kept it neutral is that k4Bench can't tell the two cases apart. A big drop in runtime or memory can be a real optimization, like in k4geo#578, or it can mean part of the workload isn't being exercised any more. They look the same in the measurements, so calling a speed-up an improvement would say more than the benchmarks actually show.

I did think about letting the AI reviewer decide, since it reads the diff. On #578 it already got it right and pointed at the phi-symmetry change with the shared logical volumes as the reason the memory went away. But that's in the comment already, in the assessment section, where it's clearly a model estimate rather than a measurement. Putting it in the heading wouldn't add anything, it would just state it more strongly, and I'd rather keep the heading to what was measured.

However, I did change the wording of the heading, which now says "Possible performance change traced to this pull request", and the arrows in the table are monochrome instead of both red, so they only show the sign of the step. The signed percentage was already there. Existing comments will pick this up on the next nightly.

Best,
Joshua