Skip to content

fix(benchmark): allow baseline scenarios with no overlays - #44

Draft
Tehsmash wants to merge 1 commit into
outshift-open:mainfrom
Tehsmash:fix/benchmark-baseline-no-overlays
Draft

fix(benchmark): allow baseline scenarios with no overlays#44
Tehsmash wants to merge 1 commit into
outshift-open:mainfrom
Tehsmash:fix/benchmark-baseline-no-overlays

Conversation

@Tehsmash

Copy link
Copy Markdown

Summary

Fixes a bug where benchmark scenarios declared with empty overlays (overlays: {}) were silently skipped, making it impossible to run an app "baseline" scenario with no configuration changes.

Root causes:

  1. OverlayStack.from_dict injected logic=[scenario_id] whenever all overlay lists were empty, defeating any explicit attempt to declare a no-overlay scenario. The default should only fire when the overlays key is absent entirely — which still happens via MASScenarioSpec.from_dict.

  2. The base-manifest loading path in preload_scenario_configs was gated on configs_dir is None, so it was unreachable in experiments mixing overlay and no-overlay scenarios.

Fix:

  • Remove the fallback injection in OverlayStack.from_dict
  • Extend the base-manifest path to also trigger when the scenario's flattened overlay list is empty

A baseline scenario can now be declared as:

scenarios:
  - id: baseline
    overlays: {}

OverlayStack.from_dict silently injected logic=[scenario_id] when the
user explicitly declared empty overlays (overlays: {}), making it
impossible to express a no-overlay baseline. The fallback in
preload_scenario_configs that loads the base manifest directly was also
gated on configs_dir being None, so it was unreachable in experiments
that mix overlay and no-overlay scenarios.

Remove the fallback injection in OverlayStack.from_dict — the default
still fires in MASScenarioSpec.from_dict when the overlays key is
absent entirely. Extend the base-manifest path in
preload_scenario_configs to also trigger when the scenario's flattened
overlay list is empty, regardless of whether configs_dir is set.

Signed-off-by: Sam Betts <1769706+Tehsmash@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant