Skip to content

DATA.md

Latest commit

 

History

History
86 lines (61 loc) · 2.69 KB

DATA.md

File metadata and controls

86 lines (61 loc) · 2.69 KB
Raw

Data schemas

The Agent Forest task loaders read each dataset family in a slightly different format. We perturb only the question text; ground-truth labels are left untouched so clean / perturbed accuracy is directly comparable.

GSM8K — data/perturbed/gsm/gsm_dataset_<dtype>/test.jsonl

JSON Lines, one record per line:

{"question": "Natalia sold clips ...", "answer": "She sold ... #### 72"}

The answer field uses the upstream #### <numeric> convention. The AgentForest GSM8K task extracts the trailing number for grading.

MATH — data/perturbed/math/math_dataset_<dtype>/math_subset_20.json

Single JSON file, nested by difficulty level → subject → list of records:

{
  "0": {
    "algebra": [{"problem": "Expand ...", "level": "Level 1", "type": "Algebra", "solution": "..."}],
    "geometry": [...],
    ...
  },
  "1": {...},
  ...
}

The math_subset_20.json file ships exactly 20 problems per (level, subject) pair, mirroring the subset used by the original Agent Forest paper.

MMLU-Math — data/perturbed/mmlu/mmlu_dataset_<dtype>/<subject>_test.csv

Multiple CSV files, one per math subject, no header. Columns:

question, choice_A, choice_B, choice_C, choice_D, answer

We include the five MMLU math splits used in the paper: abstract_algebra, college_mathematics, elementary_mathematics, high_school_mathematics, high_school_statistics.

MultiArith — data/perturbed/multiarith/multiarith_dataset_<dtype>/test.json

Single JSON file, list of records:

[
  {"index": 0, "question": "Cody had 45 dollars ...", "final_ans": 11.0},
  ...
]

Dataset variants

Subset Examples
GSM8K (test.jsonl) 1 319
MATH (math_subset_20.json) 7 × 5 × 20 = 100
MMLU-Math (5 CSVs) ~1 100
MultiArith (test.json) ~180

Six perturbation variants per subset (clean, punctuation_10/30/50, wikitypo, r2ata).

Zenodo archive

The perturbed datasets and WikiTypo dictionaries used in the paper are published as a single tarball on Zenodo:

Fetch with the helper script (the record id is wired in as the default):

bash scripts/download_data.sh
# or, to pin a specific version:
ZENODO_RECORD=19851341 bash scripts/download_data.sh

If you'd rather skip the network roundtrip, regenerate with bash scripts/prepare_data.sh — the AEDA punctuation and WikiTypo perturbations are deterministic at seed=0 and produce the same files.