Description
ExperimentRunner.run_single() uses run_id to decide whether a run has
already completed. The current ID contains the model alias, regime, condition,
and seed, but not the requested case count or other result-affecting
configuration.
For example, these two configurations currently produce the same ID:
from mech_gov.experiment.runner import RunConfig, _make_run_id
for n in (10, 1000):
config = RunConfig(
model_name="mock",
regime_name="R2",
condition="S0",
seed=42,
cases_per_condition=n,
)
print(
n,
_make_run_id(
config.model_name,
config.regime_name,
config.condition,
config.seed,
),
)
Output:
10 mock-R2-S0-seed42
1000 mock-R2-S0-seed42
Because the completion check happens before execution, skip_completed=True
can treat the changed experiment as already completed.
Relevant code:
_make_run_id() in src/mech_gov/experiment/runner.py
- the completion check near the start of
ExperimentRunner.run_single()
Proposed direction
Would you be open to separating the readable run label from a deterministic,
versioned identity derived from allowlisted result-affecting inputs?
A possible identity could include the case count, resolved model ID, condition,
regime, seed, semantic distribution fingerprint, and any explicitly supported
result-affecting parameters. Existing JSONL records would remain untouched.
Before submitting a patch, I would like to confirm your preferred compatibility
behavior for legacy records:
- Warn and execute when only a legacy label matches; or
- Preserve legacy label-based resume behind an explicit compatibility option.
I can adapt the patch to the agreed approach and include offline tests for
collision prevention, determinism, legacy records, and secret exclusion.
Description
ExperimentRunner.run_single()usesrun_idto decide whether a run hasalready completed. The current ID contains the model alias, regime, condition,
and seed, but not the requested case count or other result-affecting
configuration.
For example, these two configurations currently produce the same ID:
Output:
Because the completion check happens before execution,
skip_completed=Truecan treat the changed experiment as already completed.
Relevant code:
_make_run_id()insrc/mech_gov/experiment/runner.pyExperimentRunner.run_single()Proposed direction
Would you be open to separating the readable run label from a deterministic,
versioned identity derived from allowlisted result-affecting inputs?
A possible identity could include the case count, resolved model ID, condition,
regime, seed, semantic distribution fingerprint, and any explicitly supported
result-affecting parameters. Existing JSONL records would remain untouched.
Before submitting a patch, I would like to confirm your preferred compatibility
behavior for legacy records:
I can adapt the patch to the agreed approach and include offline tests for
collision prevention, determinism, legacy records, and secret exclusion.