GAMUT🌈: Two-Level Meta-Rubrics for Evaluating Open-Ended Generation

📄 Paper: https://arxiv.org/abs/2607.19322

🤗 Dataset: https://huggingface.co/datasets/facebook/GAMUT

GAMUT (Grounded Assessment of Multimodal Factuality) is a multimodal everyday deep-research benchmark and a two-level meta-rubric framework for evaluating open-ended, long-form generation.

Two-level meta-rubric framework — resolving a tension in rubric-based evaluation.

• Judging open-ended generation requires structure, because a complete answer often does not decompose into independent, individually required facts:
  • answers may draw from open-ended pools of valid alternatives where no single item is strictly required (the supporting ingredients in the figure);
  • or describe ordered processes where the sequence itself matters (the preparation steps).
• But LLM judges are far more reliable and lower-variance scoring flat, binary checks — the very structure needed to describe a complete answer is what makes it hard to grade one consistently.
• GAMUT resolves this with a two-level rubric: a structured meta-rubric (left) for authoring the grading criteria — what a complete answer requires — which is then mechanically compiled into flat binary checks (right) that a judge scores one at a time.

Everyday deep-research benchmark.

• 1,813 image-grounded questions across 10 diverse domains, grounded in real wearable imagery.
• Complex, real-world questions that require multi-step information gathering and multi-paragraph answer synthesis.
• Questions and rubrics are built through a multi-round human + LLM annotation process, each rubric backed by cited web evidence and verified by expert annotators.

Challenging and discriminative.

• Evaluated 14 proprietary and open-weight models; the best (Gemini 3.1 Pro) reaches only 58.7%, so the benchmark is far from saturated.
• Scores spread widely with large gaps between models, and the ranking is robust to the choice of judge (Gemini and Claude judges agree on an identical ranking).

This repository provides the tooling to work with GAMUT:

• download_images.py — the GAMUT dataset ships text only; this self-contained script reconstructs each example's image from CRAG-MM.
• evaluation/ — the rubric-graded evaluation harness (LLM-as-judge over an OpenAI-compatible API), which scores model responses against the compiled binary checks and reports the final metric.

Dataset

The GAMUT dataset is hosted on the HuggingFace Hub: facebook/GAMUT.

Each example has three fields:

field	type	description
session_id	string	Unique id; also the join key to CRAG-MM for the image.
question	string	The question about the image.
rubrics	struct	The two-level meta-rubric (Answer Critical / Valuable / Context, plus source snippets).

Installation

All tooling shares one set of dependencies, declared at the repo root:

pip install -r requirements.txt

Images

Images are not redistributed here. Reconstruct them from CRAG-MM:

python download_images.py --output-dir ./gamut_data \
    --contact "https://your-project.example"   # or an email

About --contact: the URL-only images are hosted on Wikimedia Commons, whose User-Agent policy requires requests to identify the client and include a contact (a URL or email). Pass your own contact via --contact; the script adds it to the HTTP User-Agent. Without it, Wikimedia may throttle (HTTP 429) or block (403) the downloads. (You can alternatively override the full string with --user-agent.)

This writes gamut_data/images/<session_id>.png and a gamut_data/gamut_with_images.jsonl that adds an image_path field to every example. Some CRAG-MM images are fetched from URLs that may become invalid over time; the script is non-fatal on such misses and prints an explicit summary (and a missing_images.txt) so any gaps are unmistakable.

Resuming: the script resumes by default — re-running it skips any example whose images/<session_id>.png already exists, so an interrupted run (or a retry after some URLs failed) only fetches what's still missing. Downloads are written to a .part file and renamed into place only once complete, so an existing final .png is guaranteed intact (no re-verification needed); any leftover .part files from an interrupted run are cleaned up automatically at startup. Pass --overwrite to re-download everything from scratch.

Evaluation

See evaluation/README.md for the full harness. In short (dependencies already covered by the root requirements.txt above):

python evaluation/evaluate.py \
    --dataset facebook/GAMUT \
    --responses your_model_responses.jsonl \
    --judge-model gpt-4o \
    --output-dir results/your_model

The judge speaks the OpenAI Chat Completions API, so it works unchanged against OpenAI and any OpenAI-compatible server (vLLM, SGLang, TGI, Together, etc.) via --base-url.

License

The GAMUT Dataset is licensed CC-by-NC and is intended for benchmarking purposes only. Third party content pulled from other locations are subject to its own licenses and you may have other legal obligations or restrictions that govern your use of that content.

See the LICENSE file for the full license text.

Citation

@misc{chen2026gamut,
      title={Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: {GAMUT}, a Benchmark for Factual Completeness}, 
      author={Xilun Chen and Zhaleh Feizollahi and Ross Goodwin and Seungwhan Moon and Scott Yih and Pinar Donmez and Babak Damavandi and Luna Dong},
      year={2026},
      eprint={2607.19322},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2607.19322}, 
}