Make AbstentionBench pip-installable

[celiawaggoner]

As mentioned in a previous issue, @jeqcho is porting AbstentionBench to inspect evals in this PR. He's currently copying some files over from this source code since it isn't structured as a package. I'm wondering if you'd be open to updating AbstentionBench to be a pip-installable Python package.

[PolinaKirichenko]

Hi! We are very excited you are porting AbstentionBench into inspect evals!
We would be open to convert it into a pip package in the longer run, but due to other priorities we might not be able to do it very soon. If this option works on your end, maybe you could go ahead with copying the source code files for now?

Separately, we also have AbstentionBench data on huggingface if that would be of any help (https://github.com/facebookresearch/AbstentionBench?tab=readme-ov-file#abstentionbench-data-on-huggingface, https://huggingface.co/datasets/facebook/AbstentionBench).

Please let me know though if you think it would be critical to convert AbstentionBench into a pip package for your integration!

[celiawaggoner]

Thank you for the response @PolinaKirichenko! I'll let the PR author know about the dataset on HuggingFace.

And we can definitely go ahead with copying the source code files for now, glad to hear you're open to converting to a pip package though!

[eleweek]

@PolinaKirichenko as another option, would you accept a PR from the Inspect Evals team to make it pip-installable? Or would reviewing one also not be a priority (completely understandable if that's the case)?

I am not saying that we want to submit a PR, but it'd be good to be aware if we have such an option.

[PolinaKirichenko]

Thank you so much! I will take a look at the PR within the next ~2 days.