Prototype config-driven large ensemble support

[Copilot]

Summary

Objectives:

• Allow one cfg to drive multiple ensemble-member runs
• Reuse the existing single-run/task pipeline instead of adding per-task ensemble logic
• Keep per-member case, input, output, and provenance isolated
• Surface the queue-volume risk called out in issue discussion

Select one: This pull request is...

• a bug fix: increment the patch version

• a small improvement: increment the minor version

• a new feature: increment the minor version

• an incompatible (non-backwards compatible) API change: increment the major version

• Config expansion

  • Adds [default] ensemble_members
  • Expands one cfg into one resolved run per member
  • Replaces {member} across string cfg values before entering the existing launch flow

• Isolation / safety

  • Requires {member} in [default] case, input, and output
  • Prevents member collisions in script output and case-scoped web/provenance paths
  • Rejects duplicate member IDs
  • Warns when the prototype may enqueue a large number of jobs

• Execution model

  • Keeps task generation, templates, dependencies, and bundles unchanged
  • Runs the current top-level pipeline once per resolved member config

• Provenance / coverage

  • Writes member-resolved provenance cfgs
  • Adds focused unit coverage for ensemble expansion
  • Updates config-shape assertions for the new default key

Example cfg shape:

[default]
ensemble_members = "0101", "0102"
case = v3.LR.historical_{member}
input = /archive/v3.LR.historical_{member}
output = /post/v3.LR.historical_{member}
www = /www/diagnostics

[global_time_series]
active = True
experiment_name = v3 ensemble {member}

Small Change

• To merge, I will use "Squash and merge". That is, this change should be a single commit.
• Logic: I have visually inspected the entire pull request myself.
• Pre-commit checks: All the pre-commits checks have passed.

Big Change

• To merge, I will use "Create a merge commit". That is, this change is large enough to require multiple units of work (i.e., it should be multiple commits).

1. Does this do what we want it to do?

Required:

• Product Management: I have confirmed with the stakeholders that the objectives above are correct and complete.
• Testing: I have added or modified at least one "min-case" configuration file to test this change. Every objective above is represented in at least one cfg.
• Testing: I have considered likely and/or severe edge cases and have included them in testing.

If applicable:

• Testing: this pull request introduces an important feature or bug fix that we must test often. I have updated the weekly-test configuration files, not just a "min-case" one.
• Testing: this pull request adds at least one new possible parameter to the cfg. I have tested using this parameter with and without any other parameter that may interact with it.

2. Are the implementation details accurate & efficient?

Required:

• Logic: I have visually inspected the entire pull request myself.
• Logic: I have left GitHub comments highlighting important pieces of code logic. I have had these code blocks reviewed by at least one other team member.

If applicable:

• Dependencies: This pull request introduces a new dependency. I have discussed this requirement with at least one other team member. The dependency is noted in zppy/conda, not just an import statement.

3. Is this well documented?

Required:

• Documentation: by looking at the docs, a new user could easily understand the functionality introduced by this pull request.

4. Is this code clean?

Required:

• Readability: The code is as simple as possible and well-commented, such that a new team member could understand what's happening.
• Pre-commit checks: All the pre-commits checks have passed.

If applicable:

• Software architecture: I have discussed relevant trade-offs in design decisions with at least one other team member. It is unlikely that this pull request will increase tech debt.

• Fixes Add support for large ensembles #127