Feature Request (CLI) - Show Failing Scorers

[nicoalbanese]

Would love to be able to see which scorers have failed from the CLI. Particularly for use in CI.

[mattpocock]

@nicoalbanese Is this when you're running with a threshold? Otherwise what would we consider a 'failure'?

[nicoalbanese]

@nicoalbanese Is this when you're running with a threshold? Otherwise what would we consider a 'failure'?

Yes, exactly!

[mattpocock]

I think this isn't the job of the CLI to provide - instead, diving through the UI via evalite export is the preferred mode.

[nicoalbanese]

But what about if this is being run in CI in an action?

[mattpocock]

Mmm, I guess both is probably the answer.