Skip to content

[bug] Choice letter: [B], ***B***, and 答案:B never extract (MMLU/GPQA FN) #491

Description

@marktech0813

Problem

Three common final-answer spellings for multiple-choice benchmarks extract as None:

extract_choice_letter("[B]")       # None
extract_choice_letter("***B***")   # None  (**B** works; triple-star bold+italic does not)
extract_choice_letter("答案:B")    # None  (Chinese "answer:" cue)
score_text("mmlu", "[B]", "B")     # 0.0
score_text("mmlu", "***B***", "B") # 0.0
score_text("mmlu", "答案:B", "B")  # 0.0

**B** already works via _strip_choice_md_emphasis; the MD regex only allows */**, not ***. Bracketed letters and non-English answer cues are missing from committed/fallback patterns.

Expected

All three extract B and score 1.0 against gold B.

Environment

  • origin/main

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:adaptersBenchmark adapters and loadersbugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions