Merlin-Arthur pipeline (persistent) - #508

Draft

AuguB wants to merge 87 commits into

mainfrom

trace/ma-pipeline

Commits on Jun 24, 2026

Commits on Jun 25, 2026

fix: implement post_process_generated_completion and _get_ground_truth for squad2_ma
AuguB
committed
e903f57

Commits on Jun 29, 2026

feat: add correct sysprompt to SQuAD2_MA and add no-sysprompt version of SQuAD2_MA
AuguB
committed
99ac523
chore: correct benchmark name and add to task_names
AuguB
committed
544bd5a

Commits on Jul 3, 2026

fix: correct SQUAD system prompt
AuguB
committed
baa7f08

Commits on Jul 6, 2026

Improve squad system prompt again
Stijn de Boer
committed
3b815e9

Commits on Jul 8, 2026

Add instruction to use 'final answer' marker.
Stijn de Boer
committed
fc09e31

Commits on Jul 10, 2026

Smaller token limit for squad
AuguB
committed
dcedad5

Commits on Jul 20, 2026

fix: I am stupid
AuguB
committed
24b44dc
chore(deps): update dependency pytest to >=9.1.1,<10 (#409)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
e3aac16
chore: refresh third-party license reports (#411)

authored and
AuguB
committed
6316325
ci: adding claude code as a reviewer (#400)

authored and
AuguB
committed
c9a58fb
refactor: Introduce EvalFactory.display_name (#413)

pacman82
authored and
AuguB
committed
b1c2fa3
chore(main): release 0.5.2 (#407)

github-actions[bot]
authored and
AuguB
committed
0f13e86
ci: drive release-please with public version-bump bot (#382)

authored and
AuguB
committed
4765982
chore(deps): update pre-commit hook astral-sh/uv-pre-commit to v0.11.24 (#415)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
99bcebf
chore(deps): update ruff (#416)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
f93ddbf
chore(deps): update actions/cache action to v6 (#417)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
4fba895
chore(deps): update dependency uv_build to >=0.11.24,<0.11.25 (#414)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
fb07d66
refactor: removing ruler accuracy metrics (#412)

prabhuteja12
authored and
AuguB
committed
de16e9d
chore: bump up cuda version to match pytorch defaults (#418)

prabhuteja12
authored and
AuguB
committed
69aa6d3
fix: corrected path to dataset_revisions (#419)

prabhuteja12
authored and
AuguB
committed
d0e3424
chore: pin dataset revisions (#420)

authored and
AuguB
committed
8c167c6
chore(main): release 0.5.3 (#421)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
71c1449
refactor: extract dataset_path and display_name (#422)

pacman82
authored and
AuguB
committed
c69ef75
chore: remove infinitebench tasks (#423)

prabhuteja12
authored and
AuguB
committed
d751258
refactor: migrate collect_data_set_revisions to take eval_factory (#424)

martinreinhardt01
authored and
AuguB
committed
10c3185
ci: auto-merge release-please PRs (#425)

martinreinhardt01
authored and
AuguB
committed
97bdf12
chore(deps): lock file maintenance (#342)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
2ae3a68
refactor!: remove dead code from unused benchmarks (#430)

martinreinhardt01
authored and
AuguB
committed
877b6f6
chore(deps): update anthropics/claude-code-action digest to 37b464c (#427)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
013f7d9
fix(deps): update dependency openai to >=1.109.1,<3 (#397)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
e223df8
chore(deps): update pre-commit hook astral-sh/uv-pre-commit to v0.11.27 (#434)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
0cf1c5f
chore(deps): update dependency uv_build to >=0.11.27,<0.11.28 (#433)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
9eeb279
chore: refresh third-party license reports (#428)

authored and
AuguB
committed
c611261
chore(main): release 0.6.0 (#432)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
bdaad17
chore(deps): update renovatebot/github-action action to v46.1.18 (#435)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
24fdfbd
fix(deps): update dependency mistral-common to >=1.11.5,<2 (#436)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
581fe07
chore(deps): update dependency uv_build to >=0.11.28,<0.11.29 (#438)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
c4acf40
fix(deps): update dependency wandb to >=0.28.0,<1 (#439)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
df625bb
chore(deps): update dependency mypy to >=2.2.0,<3 (#441)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
f3db6e4
chore(deps): update anthropics/claude-code-action digest to e90deca (#440)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
0ade403
chore: refresh third-party license reports (#442)

authored and
AuguB
committed
52b3809
chore(main): release 0.6.1 (#437)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
c9344f5
chore(deps): update pre-commit hook astral-sh/uv-pre-commit to v0.11.28 (#443)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
c1374bc
fix(deps): update dependency nltk to >=3.10.0,<4 (#444)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
4e5793c
chore(deps): update astral-sh/setup-uv action to v8.3.2 (#446)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
28cf9f0
fix(deps): update dependency tensorboard to v2.21.0 (#447)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
17e975a
chore: refresh third-party license reports (#450)

authored and
AuguB
committed
b0ca51e
chore(main): release 0.6.2 (#445)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
c2dc858
refactor: use dataset_revision_collection and extend registry methods (#429)

fsschneider
authored and
AuguB
committed
66de43b
refactor: start switching dataset version lock file from task class name to dataset name (#431)

fsschneider
authored and
AuguB
committed
e9c5d26
chore: remove dead code, registering extra dependencies (#426)

authored and
AuguB
committed
17a5274
refactor: each leaf task has an associated revision lockfile (#451)

pacman82
authored and
AuguB
committed
ff2b81b
refactor: pinning job now updates hf-dataset-revisions.json (#453)

pacman82
authored and
AuguB
committed
4debbd4
fix: space prefixing mathminerva bpb tasks (#454)

prabhuteja12
authored and
AuguB
committed
3af21d6
fix: limit memory used by code exec sandbox (#457)

prabhuteja12
authored and
AuguB
committed
02daf54
chore(main): release 0.6.3 (#455)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
0c06ad4
feat: allow user prompt suffix (#452)

NickyHavoc
authored and
AuguB
committed
40734e5
feat: register FullTextMMLU task for use in data ablations (#459)

authored and
AuguB
committed
40c87e1
chore(main): release 0.6.4 (#458)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
87a8173
refactor: use hf-dataset-revision.json's pinning (#461)

pacman82
authored and
AuguB
committed
87147d5
chore: update generated task docs (#460)

authored and
AuguB
committed
d5ead33
chore: remnants of legacy pinning mechanism removed (#462)

prabhuteja12
authored and
AuguB
committed
9b45453
ci: failing docker build removed (#463)

prabhuteja12
authored and
AuguB
committed
8f5a007
ci: cache invalidation of HF datasets is tied to lockfile (#464)

pacman82
authored and
AuguB
committed
1877927
fix: missing revision lockfile declared for mbpp (#465)

pacman82
authored and
AuguB
committed
fd5bb6f
test: add tests to mbpp (#467)

prabhuteja12
authored and
AuguB
committed
b9fd720
refactor: pinning of hf datasets lives in update_hf_datasets.py (#468)

pacman82
authored and
AuguB
committed
0d1c717
feat!: sacrebleu completly removed (#469)

pacman82
authored and
AuguB
committed
f8b1f6c
chore(deps): update dependency torch to v2.13.0 [security] (#471)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
073013a
chore(deps): update anthropics/claude-code-action digest to 3553f84 (#473)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
bd8374b
chore: refresh third-party license reports (#474)

authored and
AuguB
committed
8fe235c
chore(main): release 0.7.0 (#466)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
605f74a
chore(deps): update renovatebot/github-action action to v46.1.19 (#475)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
ac0ce79
chore(deps): update dependency uv_build to >=0.11.29,<0.11.30 (#476)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
76d1324
chore(deps): update anthropics/claude-code-action digest to af0559e (#477)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
90e7506
chore(deps): update pre-commit hook astral-sh/uv-pre-commit to v0.11.29 (#478)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
bdbbf09
chore(deps): update ruff (#479)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
a351fa9
fix(deps): update dependency wandb to >=0.28.1,<1 (#480)

aar-public-version-bump-bot[bot]
authored and
AuguB
committed
77ee166
ci: hf datasets are cached lazily per job. Global warmup job is gone (#482)

pacman82
authored and
AuguB
committed
93ea63f

Commits on Jul 21, 2026

fix: fix squad benchmark
AuguB
committed
63ee09a

Commits on Jul 23, 2026

Merge remote-tracking branch 'origin/main' into trace/ma-pipeline
AuguB
committed
a12cf00

Commits on Aug 24, 2026

Merge remote-tracking branch 'origin/main' into trace/ma-pipeline
Johann Higl
committed
9051210