Skip to content

feat: experiment overlays (-x flag) and benchmark correctness fixes - #42

Merged
jordanauge merged 1 commit into
mainfrom
feat/experiment-overlays
Aug 27, 2026
Merged

feat: experiment overlays (-x flag) and benchmark correctness fixes#42
jordanauge merged 1 commit into
mainfrom
feat/experiment-overlays

Conversation

@jordanauge

Copy link
Copy Markdown
Collaborator

Adds -x / --experiment-overlay to benchmark run; fixes 3 benchmark bugs
found during development.

New ExperimentOverlay manifest kind: a YAML file that deep-merges into
experiment.yaml before validation, keeping operational concerns out of
the experiment spec.

mas-lab benchmark run experiment.yaml
-x overlays/local-dev.yaml \ # cache + n_runs override
-x pipelines/standard-eval.yaml # add reusable eval pipeline

Semantics:

  • Precedence: experiment.yaml < overlay(-x) < --infra < --set
  • RFC 7386 JSON Merge Patch: dicts merge recursively, everything else
    (scalars, lists) replaces. To extend a pipeline list, provide the
    full desired list in the overlay.
  • Paths resolved relative to experiment.yaml directory

Files: execution/experiment_overlay.py, engine.py, run_batch/{init,
load}.py, worker.py, cli/commands/benchmark/run.py

  • cache: runs with status=error in result.json are no longer treated as
    cache hits; absent or unreadable result.json is also a miss (a run
    that crashed before writing its result must not be reused)
  • plotnine: force matplotlib.use('Agg') before import so the step works
    from benchmark worker threads, not only from the main thread
  • bootstrap.py: remove stale import of mas.runtime.boundary.context.skills
    deleted in cb8709d; the dead import crashed every agent run

Signed-off-by: Jordan Augé augjorda@cisco.com

Adds -x / --experiment-overlay to benchmark run; fixes 3 benchmark bugs
found during development.

New ExperimentOverlay manifest kind: a YAML file that deep-merges into
experiment.yaml before validation, keeping operational concerns out of
the experiment spec.

  mas-lab benchmark run experiment.yaml \
      -x overlays/local-dev.yaml \        # cache + n_runs override
      -x pipelines/standard-eval.yaml      # add reusable eval pipeline

Semantics:
- Precedence: experiment.yaml < overlay(-x) < --infra < --set
- RFC 7386 JSON Merge Patch: dicts merge recursively, everything else
  (scalars, lists) replaces. To extend a pipeline list, provide the
  full desired list in the overlay.
- Paths resolved relative to experiment.yaml directory

Files: execution/experiment_overlay.py, engine.py, run_batch/{__init__,
load}.py, worker.py, cli/commands/benchmark/run.py

- cache: runs with status=error in result.json are no longer treated as
  cache hits; absent or unreadable result.json is also a miss (a run
  that crashed before writing its result must not be reused)
- plotnine: force matplotlib.use('Agg') before import so the step works
  from benchmark worker threads, not only from the main thread
- bootstrap.py: remove stale import of mas.runtime.boundary.context.skills
  deleted in cb8709d; the dead import crashed every agent run

Signed-off-by: Jordan Augé <augjorda@cisco.com>
@jordanauge
jordanauge merged commit b59ab1e into main Aug 27, 2026
3 checks passed
@jordanauge
jordanauge deleted the feat/experiment-overlays branch August 27, 2026 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant