Skip to content

Experiment resume can skip changed configurations due to run_id collisions #19

Description

@rschumann

Description

ExperimentRunner.run_single() uses run_id to decide whether a run has
already completed. The current ID contains the model alias, regime, condition,
and seed, but not the requested case count or other result-affecting
configuration.

For example, these two configurations currently produce the same ID:

from mech_gov.experiment.runner import RunConfig, _make_run_id

for n in (10, 1000):
    config = RunConfig(
        model_name="mock",
        regime_name="R2",
        condition="S0",
        seed=42,
        cases_per_condition=n,
    )
    print(
        n,
        _make_run_id(
            config.model_name,
            config.regime_name,
            config.condition,
            config.seed,
        ),
    )

Output:

10 mock-R2-S0-seed42
1000 mock-R2-S0-seed42

Because the completion check happens before execution, skip_completed=True
can treat the changed experiment as already completed.

Relevant code:

  • _make_run_id() in src/mech_gov/experiment/runner.py
  • the completion check near the start of ExperimentRunner.run_single()

Proposed direction

Would you be open to separating the readable run label from a deterministic,
versioned identity derived from allowlisted result-affecting inputs?

A possible identity could include the case count, resolved model ID, condition,
regime, seed, semantic distribution fingerprint, and any explicitly supported
result-affecting parameters. Existing JSONL records would remain untouched.

Before submitting a patch, I would like to confirm your preferred compatibility
behavior for legacy records:

  1. Warn and execute when only a legacy label matches; or
  2. Preserve legacy label-based resume behind an explicit compatibility option.

I can adapt the patch to the agreed approach and include offline tests for
collision prevention, determinism, legacy records, and secret exclusion.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions