fix(run): make run_id an experiment fingerprint, excluding output sinks - #31
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
_run_idhashed the entire config, including report output paths (out,trace_db,live). So two runs of the same experiment that only differ in where their artifacts are written got different run ids — even though they produce byte-identical findings. In practice this is a foot-gun: writing the same run toreport.htmlvs/tmp/r.htmlyields different ids, which reads as "different runs" when nothing about the experiment changed.Fix
The run id should identify the experiment — target, population, goals, concurrency, chaos, seed — not the paths its report happens to land in. Exclude the output-sink fields from the fingerprint:
budget_usdstays in — it can change outcomes via the hard-stop, so it's genuinely part of the experiment.Why it's safe
run_idis passed to the tracer and report for labeling only; it never feeds the simulation RNG (that's seeded separately fromconfig.seed). Verified empirically: findings are byte-identical with and without this change — only the id string differs. Determinism (NFR-REPRO-01) is unaffected.Test
test_run_id_is_an_experiment_fingerprint_not_an_output_pathpins both halves of the invariant: output sinks must not change the id; the experiment config (seed, population, budget) must. Full suite + ruff + mypy green.