Clarification on Leaderboard Submission

[Gucci233]

Hi, thanks for releasing MBench and maintaining the leaderboard.

We would like to add our model results to the MBench leaderboard. I have finished running the evaluation, and each metric directory contains units.jsonl, items.jsonl, and summary.json. I would like to confirm which level of results should be included in the leaderboard submission JSON.

Should the submission only contain the aggregated scores from each metric's summary.json plus total_m_score, or should it also include item-level results from items.jsonl / units.jsonl for verification?

Could you also provide an example leaderboard submission JSON, including the format of total_m_score?

Thanks!

[study-overflow]

Hi, thanks for your interest in submitting results to the MBench leaderboard!

For leaderboard submission, please include a top-level leaderboard_submission.json at the root of your ZIP file. This file contains the aggregate scores used for leaderboard display and ranking.

Required fields

{
  "model_name": "ExampleWorldModel",
  "model_link": "https://example.com/model",
  "model_type": "text-conditioned",
  "total_m_score": 52.46
}

• model_type should be either text-conditioned or action-conditioned.
• total_m_score should be a numeric value on the 0–100 scale, e.g. 52.46.

Optional aggregate scores

You may also include optional aggregate scores corresponding to the leaderboard columns, such as:

{
  "entity_score": 66.58,
  "environment_score": 68.17,
  "causal_score": 25.38,
  "object_geometry": 0.0,
  "object_texture": 0.0,
  "human_identity": 0.0,
  "human_appearance": 0.0,
  "epipolar_geometry": 0.0,
  "reprojection_consistency": 0.0,
  "lighting_consistency": 0.0,
  "style_consistency": 0.0,
  "state_progress": 0.0,
  "physical_plausibility": 0.0,
  "text_interaction": 0.0,
  "action_interaction": 0.0,
  "trigger_coverage": 0.0,
  "memory_reliability": 0.0
}

Verification files

The detailed evaluation outputs such as summary.json, items.jsonl, and units.jsonl are not directly used for public leaderboard ranking.

If your evaluation pipeline generated them, please include them in the ZIP under a verification/ directory for official verification and reproducibility.

Recommended ZIP structure

submission.zip
├── leaderboard_submission.json
└── verification/
    ├── object_geometry/
    │   ├── summary.json
    │   ├── items.jsonl
    │   └── units.jsonl
    ├── object_texture/
    │   ├── summary.json
    │   ├── items.jsonl
    │   └── units.jsonl
    ├── ...
    └── action_interaction/
        ├── summary.json
        ├── items.jsonl
        └── units.jsonl

If your evaluator organizes outputs differently, that is fine as long as leaderboard_submission.json is present at the ZIP root.

The additional files are used only as supporting materials for review.

Review process

Submitted results are saved as pending submissions and are not automatically added to the public leaderboard.

Official results will be added after review by the MBench team.