In evaluation/benchmarks.py:322-330:
class BoolQ(HFMCQBenchmark):
def __init__(self, split: str = "validation", num_shots: int = 5):
super().__init__(
path="google/boolq", split=split, shot_split="train", num_shots=num_shots,
row_to_doc_fn=lambda r: MCQDoc(
f"{r['passage']}\nQuestion: {r['question'].capitalize()}?",
["Yes", "No"], 1 - int(bool(r['answer']))
)
)
The google/boolq dataset has answer: bool where True means the correct answer to the question is "yes". With choices ["Yes", "No"], the gold index for True should be 0, not 1 - 1 = 0… wait, 1 - int(True) == 0, so True → 0 → "Yes". That's correct.
But for False: 1 - int(False) == 1 → "No". Also correct.
OK — false alarm on the label inversion, sorry. However, there's a real bug nearby: r['question'].capitalize() will lowercase every character after the first one, so a question containing acronyms like "Was DNA discovered..." becomes "Was dna discovered...". Capitalize is destructive here; a simple r['question'][0].upper() + r['question'][1:] would preserve case correctly.
Severity: Low (changes the prompt distribution, may slightly hurt benchmark scores).
In
evaluation/benchmarks.py:322-330:The
google/boolqdataset hasanswer: boolwhereTruemeans the correct answer to the question is "yes". With choices["Yes", "No"], the gold index forTrueshould be 0, not1 - 1 = 0… wait,1 - int(True) == 0, soTrue → 0 → "Yes". That's correct.But for
False:1 - int(False) == 1 → "No". Also correct.OK — false alarm on the label inversion, sorry. However, there's a real bug nearby:
r['question'].capitalize()will lowercase every character after the first one, so a question containing acronyms like "Was DNA discovered..." becomes "Was dna discovered...". Capitalize is destructive here; a simpler['question'][0].upper() + r['question'][1:]would preserve case correctly.Severity: Low (changes the prompt distribution, may slightly hurt benchmark scores).