A public prototype for testing how AI models play Twenty Questions.
Results · Method · Data · Discussions
Deep20Bench measures how many questions an AI model needs to identify a hidden subject: a person, a character, a kind of thing, or a specific thing. Each question builds on the earlier answers. Fewer questions give a better (lower) score.
The current edition, 1.1, allows 40 questions. Questions are answered with Yes, Rather yes, Rather no, No, or Unknown; guesses get Yes, No, or Unknown. Edition 1.0 used only Yes, No, and Unknown with a 50-question limit, and its published results keep their original scores.
Within an edition, every model plays the same subjects with the same question limit and scoring rules. Edition 1.1 accepts a few reviewed versions of the answer-checking prompts and subject descriptions; each run records which one it used. The subject set is small, so the results support only limited conclusions. All results, transcripts, and scoring data are public.
Python 3.14.6 and uv are required.
uv syncPlace an OpenRouter key in the ignored file private/openrouter.yml:
api:
api_key: your-openrouter-keyRun one experimental round. This makes paid OpenRouter calls:
uv run deep20 benchmark run B-0001 \
--model M-0001 \
--benchmark-mode experimental \
--targets T-0001 \
--iterations 1 \
--run-id BX-local-exampleYou can also set the key in OPENROUTER_API_KEY instead of using the file.
- Source layout
- Architecture
- Running benchmarks
- Game engine
- Oracle, Reviewer, and Judge
- Publication package
- Documentation index
Deep20Bench was created by Patrick Heusser and Markus Tuor. See CITATION.cff for citation details.
The software is source-available under a dual-license model: PolyForm Noncommercial 1.0.0 for noncommercial use, with separate commercial licenses available. Project-authored documentation and result data use Creative Commons Attribution 4.0. See LICENSE.md for the exact scope and third-party exclusions.