Merlin-Arthur pipeline (persistent) - #508

Draft

AuguB wants to merge 87 commits into

mainfrom

trace/ma-pipeline

AuguB commented Jul 22, 2026 •

edited

Loading

Contributor

Persistent branch baked into the shiraz image for the Merlin-Arthur pipeline. Based on merlin_arthur_reasoning_fix.

Made with Cursor

Sibling PRs: merlin-arthur Aleph-Alpha-Research/merlin-arthur#393, eval-framework #508, shiraz Aleph-Alpha-Research/shiraz#1016, prey Aleph-Alpha-Research/prey#705, metal-tiger Aleph-Alpha-Research/metal-tiger#971, savanna Aleph-Alpha-Research/savanna#3164

AuguB and others added 30 commits

June 24, 2026 12:14


          fix: add reasoning fixes to Squad_ma and TriviaQA_ma tasks

b2d9703


          fix: add wide max_tokens to squad and triviaQA.

0079ec7


          fix: implement post_process_generated_completion for squad


          Merge branch 'main' into merlin_arthur_reasoning_fix

20c43c7


          fix: implement post_process_generated_completion and _get_ground_trut…

e903f57

…h for squad2_ma


          feat: add correct sysprompt to SQuAD2_MA and add no-sysprompt version…

99ac523

… of SQuAD2_MA


          chore: correct benchmark name and add to task_names

544bd5a


          fix: correct SQUAD system prompt

baa7f08


          Improve squad system prompt again

3b815e9


          Add instruction to use 'final answer' marker.

fc09e31


          Smaller token limit for squad

dcedad5


          fix: I am stupid

24b44dc


          chore(deps): update dependency pytest to >=9.1.1,<10 (#409)

e3aac16

Co-authored-by: aar-public-version-bump-bot[bot] <286693160+aar-public-version-bump-bot[bot]@users.noreply.github.com>


          chore: refresh third-party license reports (#411)

Co-authored-by: prabhuteja12 <11191577+prabhuteja12@users.noreply.github.com>


          ci: adding claude code as a reviewer (#400)

c9a58fb

Co-authored-by: Martin Reinhardt <martin.reinhardt@aleph-alpha.com>


          refactor: Introduce EvalFactory.display_name (#413)

b1c2fa3


          chore(main): release 0.5.2 (#407)

0f13e86

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>


          ci: drive release-please with public version-bump bot (#382)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>


          chore(deps): update pre-commit hook astral-sh/uv-pre-commit to v0.11.…

99bcebf

…24 (#415)

Co-authored-by: aar-public-version-bump-bot[bot] <286693160+aar-public-version-bump-bot[bot]@users.noreply.github.com>


          chore(deps): update ruff (#416)

f93ddbf

Co-authored-by: aar-public-version-bump-bot[bot] <286693160+aar-public-version-bump-bot[bot]@users.noreply.github.com>


          chore(deps): update actions/cache action to v6 (#417)

4fba895

Co-authored-by: aar-public-version-bump-bot[bot] <286693160+aar-public-version-bump-bot[bot]@users.noreply.github.com>


          chore(deps): update dependency uv_build to >=0.11.24,<0.11.25 (#414)

fb07d66

Co-authored-by: aar-public-version-bump-bot[bot] <286693160+aar-public-version-bump-bot[bot]@users.noreply.github.com>


          refactor: removing ruler accuracy metrics (#412)

de16e9d


          chore: bump up cuda version to match pytorch defaults (#418)

69aa6d3


          fix: corrected path to dataset_revisions (#419)

d0e3424


          chore: pin dataset revisions (#420)

8c167c6

Co-authored-by: prabhuteja12 <11191577+prabhuteja12@users.noreply.github.com>


          chore(main): release 0.5.3 (#421)

71c1449

Co-authored-by: aar-public-version-bump-bot[bot] <286693160+aar-public-version-bump-bot[bot]@users.noreply.github.com>


          refactor: extract dataset_path and display_name (#422)

c69ef75


          chore: remove infinitebench tasks (#423)

d751258


          refactor: migrate collect_data_set_revisions to take eval_factory (#424)

10c3185

pacman82 and others added 30 commits

July 20, 2026 14:14


          refactor: pinning job now updates hf-dataset-revisions.json (#453)

4debbd4


          fix: space prefixing mathminerva bpb tasks (#454)

3af21d6


          fix: limit memory used by code exec sandbox (#457)

02daf54


          chore(main): release 0.6.3 (#455)

0c06ad4

Co-authored-by: aar-public-version-bump-bot[bot] <286693160+aar-public-version-bump-bot[bot]@users.noreply.github.com>


          feat: allow user prompt suffix (#452)

40734e5


          feat: register FullTextMMLU task for use in data ablations (#459)

40c87e1

Co-authored-by: Cursor <cursoragent@cursor.com>


          chore(main): release 0.6.4 (#458)

87a8173

Co-authored-by: aar-public-version-bump-bot[bot] <286693160+aar-public-version-bump-bot[bot]@users.noreply.github.com>


          refactor: use hf-dataset-revision.json's pinning (#461)

87147d5


          chore: update generated task docs (#460)

d5ead33

Co-authored-by: fsschneider <12153723+fsschneider@users.noreply.github.com>


          chore: remnants of legacy pinning mechanism removed (#462)

9b45453


          ci: failing docker build removed (#463)

8f5a007


          ci: cache invalidation of HF datasets is tied to lockfile (#464)


          fix: missing revision lockfile declared for mbpp (#465)

fd5bb6f


          test: add tests to mbpp (#467)

b9fd720


          refactor: pinning of hf datasets lives in update_hf_datasets.py (#468)

0d1c717


          feat!: sacrebleu completly removed (#469)

f8b1f6c


          chore(deps): update dependency torch to v2.13.0 [security] (#471)

073013a

Co-authored-by: aar-public-version-bump-bot[bot] <286693160+aar-public-version-bump-bot[bot]@users.noreply.github.com>


          chore(deps): update anthropics/claude-code-action digest to 3553f84 (#…

bd8374b

…473)

Co-authored-by: aar-public-version-bump-bot[bot] <286693160+aar-public-version-bump-bot[bot]@users.noreply.github.com>


          chore: refresh third-party license reports (#474)

8fe235c

Co-authored-by: prabhuteja12 <11191577+prabhuteja12@users.noreply.github.com>


          chore(main): release 0.7.0 (#466)

605f74a

Co-authored-by: aar-public-version-bump-bot[bot] <286693160+aar-public-version-bump-bot[bot]@users.noreply.github.com>


          chore(deps): update renovatebot/github-action action to v46.1.19 (#475)

ac0ce79

Co-authored-by: aar-public-version-bump-bot[bot] <286693160+aar-public-version-bump-bot[bot]@users.noreply.github.com>


          chore(deps): update dependency uv_build to >=0.11.29,<0.11.30 (#476)

76d1324

Co-authored-by: aar-public-version-bump-bot[bot] <286693160+aar-public-version-bump-bot[bot]@users.noreply.github.com>


          chore(deps): update anthropics/claude-code-action digest to af0559e (#…

90e7506

…477)

Co-authored-by: aar-public-version-bump-bot[bot] <286693160+aar-public-version-bump-bot[bot]@users.noreply.github.com>


          chore(deps): update pre-commit hook astral-sh/uv-pre-commit to v0.11.…

bdbbf09

…29 (#478)

Co-authored-by: aar-public-version-bump-bot[bot] <286693160+aar-public-version-bump-bot[bot]@users.noreply.github.com>


          chore(deps): update ruff (#479)

a351fa9

Co-authored-by: aar-public-version-bump-bot[bot] <286693160+aar-public-version-bump-bot[bot]@users.noreply.github.com>


          fix(deps): update dependency wandb to >=0.28.1,<1 (#480)

77ee166

Co-authored-by: aar-public-version-bump-bot[bot] <286693160+aar-public-version-bump-bot[bot]@users.noreply.github.com>


          ci: hf datasets are cached lazily per job. Global warmup job is gone (#…

93ea63f

…482)


          fix: fix squad benchmark

63ee09a


          Merge remote-tracking branch 'origin/main' into trace/ma-pipeline

a12cf00

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	.github/workflows/claude-code-review.yaml
#	.github/workflows/generate_task_docs.yml
#	.github/workflows/renovate.yaml
#	.github/workflows/tests.yml
#	.release-please-manifest.json
#	CHANGELOG.md
#	THIRD_PARTY.md
#	THIRD_PARTY_LICENSES.md
#	docs/tasks/README.md
#	pyproject.toml
#	src/eval_framework/tasks/base.py
#	src/eval_framework/tasks/benchmarks/squad.py
#	src/eval_framework/tasks/registry.py
#	src/eval_framework/tasks/task_names.py
#	uv.lock


          Merge remote-tracking branch 'origin/main' into trace/ma-pipeline

Main removed the TriviaQA task (9b3b969); this branch keeps TRIVIAQA and
TriviaQA_MA for the MA eval family: the task and its test are restored,
registration moves into main's per-module structure as
register_triviaqa_tasks, the trivia_qa revision pin returns to
hf-dataset-revisions.json, and the TriviaQA_MA prompt hashes are
regenerated against main's fixed MA prompt. The branch's SQuAD2_MA tasks
graduated to main and are dropped here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet