feat: add return_variance parameter to run_simulation_pure and run_scenarios - #6
Merged
Conversation
…enarios Adds a new `return_variance = FALSE` parameter (fully backward compatible) that, when TRUE, attaches 10 variance-of-the-mean columns to the output: - Var_Exp_N, Var_Exp_Events: computed from raw per-replicate storage, var(raw) / n_rep - Var_Exp_Time: trial-level duration variance, same method - Var_Type_I_Error_or_Power, Var_PET_Efficacy, Var_PET_Futility, Var_Pr_Reach_Max_N, Var_Pr_Final_Efficacy, Var_Pr_Final_Futility, Var_Pr_Final_Inconclusive: analytical Bernoulli, p*(1-p)/n_rep Motivation: BATON's hetGP surrogate requires per-evaluation variance estimates. Previously, default_variance_estimator returned NA for continuous metrics (EN, ET), forcing homoskedastic DiceKriging for those objectives. With return_variance = TRUE, simulators can pass known variances to mleHetGP for more accurate GP fitting. Implementation reuses the existing collect_raw_n / all_final_n_raw infrastructure (shared with return_percentiles). Two new raw vectors (all_final_events_raw, all_final_time_raw) are allocated only when return_variance = TRUE. Also adds 47 new tests covering backward compatibility, column presence, analytical correctness, monotonicity, and run_scenarios pass-through. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Collaborator
Author
|
The below checks failed for a pre-existing test unrelated to the changes made for this pull request. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
return_variance = FALSEparameter torun_simulation_pure()andrun_scenarios()— fully backward compatible, no behaviour change when omittedTRUE, attaches 10 variance-of-the-mean columns to the output data frame: 3 raw-based continuous metrics (Var_Exp_N,Var_Exp_Events,Var_Exp_Time) and 7 analytical proportion variances (Var_Type_I_Error_or_Power,Var_PET_Efficacy,Var_PET_Futility,Var_Pr_Reach_Max_N,Var_Pr_Final_Efficacy,Var_Pr_Final_Futility,Var_Pr_Final_Inconclusive)run_scenariospass-throughMotivation
BATON's hetGP surrogate (
mleHetGP) requires per-evaluation variance estimates asknown = list(Delta = ...). Previously,default_variance_estimatorreturnedNAfor continuous metrics (EN, ET) because no variance was available fromrun_scenarios, forcing BATON to fall back to homoskedastic DiceKriging for those objectives. This is suboptimal becauseVar(N)is heteroskedastic across the design space.With
return_variance = TRUE, simulator wrappers can extract and attach variance attributes so BATON uses hetGP for all objectives.Implementation Details
Continuous metrics (
Var_Exp_N,Var_Exp_Events,Var_Exp_Time): raw per-replicate values are stored during the simulation loop andvar(raw) / n_repis computed after aggregation. The existingcollect_raw_n/all_final_n_rawinfrastructure (shared withreturn_percentiles) is reused for N; two new vectors (all_final_events_raw,all_final_time_raw) are added for events and time.Proportion metrics: analytical Bernoulli variance
p * (1 - p) / n_rep— no additional storage required.Memory: at most
n_rep × n_arms × 3doubles of additional allocation (~240 KB for 5000 reps, 2 arms), only whenreturn_variance = TRUE.Test Plan
return_variance = FALSE(default): noVar_*columns, identical structure to pre-change outputreturn_variance = TRUE: exactly 10Var_*columns present and non-negativereturn_variance = TRUE(same seed)p*(1-p)/nanalytically (tolerance 1e-10)Var_Exp_Ndecreases asnum_simulationsincreasesreturn_percentiles = TRUEsimultaneouslyrun_scenarios()backward compat and forward pass withreturn_variance = TRUEtest-rcpp-equivalence.R)🤖 Generated with Claude Code