Skip to content

feat: add return_variance parameter to run_simulation_pure and run_scenarios - #6

Merged
naimurashid merged 1 commit into
mainfrom
feat/return-variance
Apr 15, 2026
Merged

feat: add return_variance parameter to run_simulation_pure and run_scenarios#6
naimurashid merged 1 commit into
mainfrom
feat/return-variance

Conversation

@ayoung31

@ayoung31 ayoung31 commented Apr 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Adds return_variance = FALSE parameter to run_simulation_pure() and run_scenarios() — fully backward compatible, no behaviour change when omitted
  • When TRUE, attaches 10 variance-of-the-mean columns to the output data frame: 3 raw-based continuous metrics (Var_Exp_N, Var_Exp_Events, Var_Exp_Time) and 7 analytical proportion variances (Var_Type_I_Error_or_Power, Var_PET_Efficacy, Var_PET_Futility, Var_Pr_Reach_Max_N, Var_Pr_Final_Efficacy, Var_Pr_Final_Futility, Var_Pr_Final_Inconclusive)
  • Adds 47 new tests covering backward compatibility, column correctness, analytical accuracy, and run_scenarios pass-through

Motivation

BATON's hetGP surrogate (mleHetGP) requires per-evaluation variance estimates as known = list(Delta = ...). Previously, default_variance_estimator returned NA for continuous metrics (EN, ET) because no variance was available from run_scenarios, forcing BATON to fall back to homoskedastic DiceKriging for those objectives. This is suboptimal because Var(N) is heteroskedastic across the design space.

With return_variance = TRUE, simulator wrappers can extract and attach variance attributes so BATON uses hetGP for all objectives.

Implementation Details

Continuous metrics (Var_Exp_N, Var_Exp_Events, Var_Exp_Time): raw per-replicate values are stored during the simulation loop and var(raw) / n_rep is computed after aggregation. The existing collect_raw_n / all_final_n_raw infrastructure (shared with return_percentiles) is reused for N; two new vectors (all_final_events_raw, all_final_time_raw) are added for events and time.

Proportion metrics: analytical Bernoulli variance p * (1 - p) / n_rep — no additional storage required.

Memory: at most n_rep × n_arms × 3 doubles of additional allocation (~240 KB for 5000 reps, 2 arms), only when return_variance = TRUE.

Test Plan

  • return_variance = FALSE (default): no Var_* columns, identical structure to pre-change output
  • return_variance = TRUE: exactly 10 Var_* columns present and non-negative
  • Core OC columns unchanged when return_variance = TRUE (same seed)
  • Proportion variances equal p*(1-p)/n analytically (tolerance 1e-10)
  • Var_Exp_N decreases as num_simulations increases
  • Compatible with return_percentiles = TRUE simultaneously
  • run_scenarios() backward compat and forward pass with return_variance = TRUE
  • Full existing test suite: 462 passed, 0 regressions (1 pre-existing timing flake in test-rcpp-equivalence.R)

🤖 Generated with Claude Code

…enarios

Adds a new `return_variance = FALSE` parameter (fully backward compatible)
that, when TRUE, attaches 10 variance-of-the-mean columns to the output:

- Var_Exp_N, Var_Exp_Events: computed from raw per-replicate storage,
  var(raw) / n_rep
- Var_Exp_Time: trial-level duration variance, same method
- Var_Type_I_Error_or_Power, Var_PET_Efficacy, Var_PET_Futility,
  Var_Pr_Reach_Max_N, Var_Pr_Final_Efficacy, Var_Pr_Final_Futility,
  Var_Pr_Final_Inconclusive: analytical Bernoulli, p*(1-p)/n_rep

Motivation: BATON's hetGP surrogate requires per-evaluation variance
estimates. Previously, default_variance_estimator returned NA for
continuous metrics (EN, ET), forcing homoskedastic DiceKriging for
those objectives. With return_variance = TRUE, simulators can pass
known variances to mleHetGP for more accurate GP fitting.

Implementation reuses the existing collect_raw_n / all_final_n_raw
infrastructure (shared with return_percentiles). Two new raw vectors
(all_final_events_raw, all_final_time_raw) are allocated only when
return_variance = TRUE.

Also adds 47 new tests covering backward compatibility, column
presence, analytical correctness, monotonicity, and run_scenarios
pass-through.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@ayoung31

ayoung31 commented Apr 2, 2026

Copy link
Copy Markdown
Collaborator Author

The below checks failed for a pre-existing test unrelated to the changes made for this pull request.

@naimurashid
naimurashid merged commit 77c7f83 into main Apr 15, 2026
1 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants