monte_carlo_layer_means default seed makes a small-n draw an exact prefix of a large-n draw, and the compare_against_monte_carlo MSE column is easy to misread

[joshmeee]

DRAFT — starter-kit feedback for AICrowd (NOT POSTED; needs Josh's approval)

Where this would go: GitHub issue on AIcrowd/whest-starterkit (the Phase-1 update post says
"Reproducible bugs: open an issue on flopscope, whestbench, or the starter kit"). Community
contribution prizes are USD 500-5,000.

Honest framing note: item 1 is a proven aliasing bug with a modest measured effect (~1.5x).
Item 2 is a documentation/UX issue, but it is the one that actually cost us weeks of a wrong
conclusion. Do not oversell either.

---

Title

monte_carlo_layer_means default seed makes a small-n draw an exact prefix of a large-n draw, and the compare_against_monte_carlo MSE column is easy to misread

1. Prefix aliasing from the shared default seed

local_engine.monte_carlo_layer_means(mlp, n_samples, seed=0) defaults to seed=0, and
rng.standard_normal((n_samples, width)) fills row-major. So two calls with the same seed and
different n_samples are not independent draws: the smaller one is an exact prefix of the
larger one.

import numpy as np, flopscope.numpy as fnp
a = fnp.random.default_rng(0).standard_normal((5, 8), dtype=fnp.float32)
b = fnp.random.default_rng(0).standard_normal((200, 8), dtype=fnp.float32)
assert np.array_equal(np.asarray(a), np.asarray(b)[:5])   # passes: max abs diff 0.0

The natural way to self-evaluate a sampling estimator is to build a high-n reference and compare a
cheap draw against it. With the default seed that measures a subsample against its own superset,
so the apparent error is systematically too small.

Measured on build_mlp(width=64, depth=8, seed=0), final-layer MSE against an 80,000-sample
reference, shared default seed versus independent seeds (single draws, so these are noisy):

n_cheap	shared seed	independent seeds	understated by
2,000	4.81e-05	7.07e-05	1.47x
20,000	4.14e-06	6.57e-06	1.59x
40,000	2.38e-06	3.61e-06	1.52x

Suggested fix: derive the default seed from n_samples (or require an explicit seed), and note
in the docstring that two calls intended as independent must use different seeds.

2. The compare_against_monte_carlo MSE column is easy to read backwards

The table prints one row per n_samples with columns sampling_flops, estimator_flops, MSE:

 n_samples | sampling_flops | estimator_flops |        MSE
     1,000 |  4,231,456,768 |   3,263,522,174 |   0.000237
    10,000 | 42,314,272,768 |   3,263,522,174 |   0.000098
   100,000 | 423,142,432,768 |   3,263,522,174 |   0.000070

estimator_flops is constant down the column while MSE falls with n_samples, which reads as if
MSE describes the sampler. It does not — it is the estimator's error measured against an
increasingly accurate MC reference, converging to the estimator's true MSE.

We misread this and concluded in our own notes that our analytic estimator "beats Monte-Carlo
sampling per-FLOP; sampling needs ~10k samples to match." Running an actual MC estimator through
whest run on the public split showed the opposite by a wide margin: plain MC at n=5,000 scored
final_layer_mse 1.17e-05 against our analytic estimator's 8.37e-05. The wrong conclusion sat in
our notes for weeks and steered the whole approach.

Suggested fix: rename the column to something like estimator_mse_vs_mc_reference, and add a
line to the printed header stating that it is the estimator's error against the reference and that
it converges to the estimator's own MSE as n_samples grows.

Why these two are worth one report

Both push a participant toward the same wrong belief: that analytic estimation is beating sampling
when it is not. Item 1 flatters a sampling baseline's measured error; item 2 mislabels an analytic
estimator's error as the sampler's. The published example benchmarks are safe because they come from
whest run against baked ground truth, but local self-evaluation via these two helpers is not.

Environment: flopscope 0.10.0, whestbench 0.14.0, starter kit at bb90562, Python 3.10.20,
NumPy 2.2.6, Windows.