Toy dataset generation for pseudo-experiments - #130
Draft
rbarrue wants to merge 9 commits into
Draft
Conversation
The PDFDerivativeProvider/EFTDerivativeProvider dispatch-on-job-block pattern was duplicated at four call sites (pdf/eft calibration and modification-plot entry points). Collapse to one factory function and switch all four over.
…servationArrays _compute_T_chunk (reading the cache) and setObservation's inline observation-mode T loop (reading by_class) were the same computation written twice. Extract _compute_T_from_columns(rid, columns_by_class, ...), which both now delegate to; it takes either an in-memory cache dict or an observation/toy by_class block, since both support the same start:stop slicing. Keeps _compute_T_chunk's tolerance of a (N,0) BIT column (rate_shift-only / POI-less classes), which the old inline observation copy did not have. Move _eval_region_surrogates from N2LLExtensions up into the base N2LL class (setObservation needs it too, not just evaluate_ratio), and switch it to the permissive missing-BIT-predictor branch setObservation already used instead of N2LLExtensions' hard raise -- documented in its docstring as a real weakening for any other caller of evaluate_ratio. Add setObservationArrays(unbinned_blocks, binned_counts) to register an observed/toy dataset from already-evaluated arrays instead of loaders; setObservation now ends by calling it, so one place owns the observed/Asimov mode switch. This is what will let toy generation feed a fit without a second copy of the observation wiring. Verified with a synthetic in-memory region (BIT + PNN + lnN + rate_shift, no trained surrogates are available in this environment) that _compute_T_chunk reproduces the pre-refactor reference bit-for-bit, and that the deterministic-injection identity (observed w == cached w0 implies the observation branch equals the Asimov branch) holds to float roundoff.
Toy generation throws global observables nu_obs ~ Normal(nu_true, 1) per penalized nuisance, so the fit's constraint term needs to be centred on the thrown value rather than always on zero. Add ModelParameter.constraint_center (default 0.0), change Hypothesis.penalty() to sum((val - center)**2, ...), and add Hypothesis.set_constraint_centers(mapping) to set it by name (raises on an unknown or non-penalized target). Rotated mirrors constraint_center on its passthrough nuisance copies for correct printing; Rotated.penalty() already forwards to the base so needs no change. At the default center of 0 this is bit-identical to before -- reconfirmed with the same synthetic Likelihood check used for the T refactor.
Implements the generator described in user/ricardo/claude/plans/toy-generation-for-pseudo-experiments.md: cache-mode toys (n_i ~ Poisson(w0*(1+T)) on the trained surrogates, "is the model calibrated against itself") and truth-mode toys (exact PDF/EFT reweighting, a per-process scale factor, or an alternate sample, while the fit still uses the trained surrogates -- "does the surrogate mismodel the truth"), for both unbinned and binned regions, with UID-split-based holdout and weight-fraction renormalization for truth mode. - fit/ToyGenerator.py (new): ProcessSource, throw_constraint_centers, generate_unbinned_toy_from_cache/from_truth, generate_binned_toy, generate_toy, save_toy/load_toy (plain-dict toy, no Toy class), and a __main__ that reads a standalone toy spec YAML + --toyPoint/--seeds and writes one HDF5 file per seed. Exact PDF/EFT weight reconstruction reuses common/derivative_providers.build_derivative_provider and expand_pois_linear_quadratic (fit/Likelihood.py) rather than re-deriving the Taylor contraction. RNG is a stable hash of (seed, region_id[, class_id]), not sequential spawning, so adding a region/class doesn't perturb siblings. - fit/Likelihood.py: add N2LL.setToy(toy, hypothesis) (setObservationArrays + applying the toy's thrown constraint centres) and a --toyFile flag, inserted before the --data/--asimov branches since nominal Asimov is the fall-through default. Generation only ever happens in ToyGenerator.py; the fit entry point loads a toy and never generates one. - ML/Calibration/calibration_runner.py: generalize the hardcoded 'c2st_train'+'c2st_val' UID interval helper to _uid_split_interval(split_cfg, *split_names), so toy generation's 'final_eval' split reuses the same bucket-interval arithmetic instead of duplicating it; _uid_c2st_intervals is now a one-line wrapper and its call site is unchanged. - configs/unbinned_v6/toys_example.yaml, configs/Eta_unbinned/toys_example.yaml: example specs (single- and multi-process) matching the plan's spec format. No trained surrogates or MC caches are available in this environment (0/64 ready artifacts even on the unmodified branch), so this is verified with a synthetic in-memory N2LL fixture rather than a real config: cache-mode yield closure over 300 toys, throw_constraint_centers mean/width, binned cache-mode closure, generate_toy orchestration + seed-determinism, and a full save_toy/load_toy round-trip through setToy giving a bit-identical n2ll(hyp). The exact PDF/EFT weight reconstruction formula is separately checked against an independent finite-difference derivative. Truth-mode's ROOT-streaming path (UID masking, per-source diagnostics, multi-process concatenation) is implemented per the plan but not exercised end-to-end here, for lack of sample data in this environment -- flagged for a first real run against configs/unbinned_v6/unbinned_2018.yaml and configs/Eta_unbinned/Eta_unbinned_2016.yaml.
Two bugs found running cache-mode generation for real against configs/unbinned_v7/unbinned_2018.yaml: 1. generate_unbinned_toy_from_cache crashed unconditionally on any negative Poisson intensity, but NLO-generated samples routinely carry negative-weight MC events (confirmed: 0.33% of events in every SR_2018 class have negative w0), which trips this even at the nominal hypothesis where T is identically 0. Extend the allow_negative_weights escape hatch (so far only wired for truth mode) to generate_unbinned_toy_from_cache and generate_binned_toy's cache branch, and thread it through generate_toy's cache-mode calls -- same crash-by-default/clip-when-allowed semantics, now covering both the negative-MC-weight cause and the T < -1 cause. 2. Importing common.derivative_providers.build_derivative_provider at module level pulled in data/samples_eft.py (which eagerly constructs RDataLoaders for every declared EFT sample at import time), making cache-mode generation depend on EFT sample data reachability just to import the module. Moved the import inline into _materialize_truth_weights, its only (truth-mode-only) call site. Added configs/unbinned_v7/toys_cache_example.yaml, the spec used to validate this against real trained surrogates: 5 cache-mode toys generated, yield closure within 1.14 sigma of sum(clip(w0,0,None)), and the full save_toy/load_toy/setToy/--toyFile chain confirmed against a real fit setup (finite -2lnL, correct response to hypothesis changes, bit-identical repeat evaluation). See user/ricardo/claude/plans/toy-generation-status.md for the full record.
f"{self.val:.6e}" raises TypeError when self.val is an autograd ArrayBox
(during gradient tracing) rather than a plain float -- ArrayBox doesn't
support the format-spec protocol. Unwrap with getval() (already imported,
identity when autograd isn't available) before formatting.
Pre-existing bug, unrelated to the toy generation work; surfaced because
run_autograd_fit's --verbosity 2 calls Hypothesis.print() -> ModelParameter
.__repr__() from inside grad(fcn), a code path apparently never exercised at
the default --verbosity 1. Found running a --verbosity 2 fit against a real
generated toy (configs/unbinned_v7/unbinned_2016.yaml).
Overlay each toy seed's weighted distribution of the config's default_features per unbinned region, binned via data/plot_options. Cache-mode toys store only surrogate columns + row indices (not raw kinematics), so plotting them re-streams the region's samples once at the union of requested seeds' indices to recover X -- costly by design, opt-in only. Truth-mode toys already carry X directly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Draw the model's expected w0*(1+T) at the toys' generation hypothesis, summed over the WHOLE cached MC (not just the sampled toy events), as a grey reference band behind each toy seed's overlay -- what the toys should fluctuate around, useful e.g. to check the negative-weight clipping isn't biasing the shape. Always available regardless of toy source (cache/truth), since the surrogate cache is always built. Refactors materialize_cache_region_features into materialize_cache_region_diagnostics, which accumulates the Asimov histogram in the same re-streaming pass that recovers cache-mode toys' raw kinematics -- no extra full pass needed. Cache intensity computation factored out of generate_unbinned_toy_from_cache into _compute_cache_intensity, shared with the new diagnostics path. Real-run verified against unbinned_v7/unbinned_2016.yaml (SR_2016): Asimov band tracks the toy seeds' mean, with visibly larger Poisson scatter in the low-statistics tail, as expected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
rbarrue
marked this pull request as ready for review
July 29, 2026 13:50
rbarrue
marked this pull request as draft
July 29, 2026 14:45
… alternative samples
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements toy/pseudo-experiment generation per
user/ricardo/claude/plans/toy-generation-for-pseudo-experiments.md:common/derivative_providers.build_derivative_provider(job)factory, replacing four duplicated call sites._compute_T_chunk(cache) andsetObservation's inline observation-modeTloop into one_compute_T_from_columns; moves_eval_region_surrogatesto baseN2LL; addssetObservationArrays.ModelParameter.constraint_center/Hypothesis.set_constraint_centers(...)for centring the nuisance penalty term on a toy's thrown global observables.fit/ToyGenerator.py: cache-mode (model-is-truth) and truth-mode (exact reweighting / per-process scale factor / alternate sample) toy generation, unbinned + binned,save_toy/load_toy, a standalone spec-file__main__, andN2LL.setToy+--toyFileinfit/Likelihood.py.allow_negative_weightsextended to cache mode (NLO negative-weight MC events trip the negative-intensity guard even at the nominal hypothesis), and a lazy import so cache-mode generation no longer depends on EFT sample data being reachable.ModelParameter.__repr__crashed on autogradArrayBoxvalues (surfaced by--verbosity 2, never exercised before).Test plan
Real-config validation against
configs/unbinned_v7/unbinned_2018.yaml/unbinned_2016.yaml(trained surrogates, real cache):fval,edm,niter, all 42 parameter values/errors, and the full covariance + correlation matrices bit-for-bit identical between pre- and post-refactor code.--toyFile/load_toy/setToywiring: finite-2lnL, correct response to hypothesis changes, bit-identical repeat evaluation.run_autograd_fithitsNaN likelihoodon its first optimizer step for a real thrown toy (pre-existing fitter fragility:T(x=0)=0identically for any dataset, so an unscaled first L-BFGS-B step from the origin can pushT < -1; not toy-generation-specific). Logged intoy-generation-status.mdwith repro steps, left unfixed for now.See
user/ricardo/claude/plans/toy-generation-status.md(gitignored, local) for the full validation log.Still WIP — opened as draft, more validation to follow.