Non-stationarity breaks permutation surrogates in multi-agent RL

Analysis code and generated data for the transfer-entropy experiments reported in Non-Stationarity Breaks Permutation Surrogates in Multi-Agent Reinforcement Learning: Diagnosis and Remedies.

Archived release: 10.5281/zenodo.22658530 · Preprint: arXiv:2604.23716

What the experiment shows

Transfer entropy is routinely validated against permutation surrogates. That test assumes the series are stationary, and agents that anneal exploration are not: each one's action distribution drifts on a schedule, independently of every other.

Against a null control where the true answer is zero by construction, pairs of agents drawn from runs that never met, omitting one preprocessing step drives the false-positive rate to 100.00% and 99.95% in two games. Every null pair in every seed is flagged as showing directed influence.

The mechanism is not about agents or learning. Synthetic Bernoulli sequences with a shared trend and no game reproduce it exactly, and holding the marginal entropy fixed while removing only the trend returns the rate to nominal.

The remedy that works changes the null model rather than the data. Permuting the source within blocks of training time breaks the alignment at a resolution where the drift is flat, reaches the size of the test in both games, and leaves the series, the statistic and the estimand alone. It is also the most sensitive of the three remedies at matched size.

Layout

experiments/
  run_*.py             one experiment each; each writes results/<name>_summary.json
  guard.py             56 regression checks over the manuscript and these results
  verify_*.py          independent reimplementations and cross-checks
  check_*.py           consistency checks over the manuscript's numbers
  provenance.py        stamps every result with the script, its hash and library versions
  results/             every stored result, plus per-pair records as .jsonl
figures/               the figures as published

Running

The experiments import GameBrains for the game engine and agents; nothing in that repository is modified. Place the two side by side:

parent/
  gamebrains/
  <this repository>/

Then, from experiments/:

py -3 run_blockwise_surrogate.py     # the four-way decomposition, 100 seeds
py -3 guard.py                       # verify every reported number against the stored results

Requires Python 3.11+, numpy, scipy, statsmodels. pyinform is optional and enables a third-party cross-check of the transfer-entropy values.

Full seed sweeps take hours. Each script prints per-seed progress and can be re-run; results are written atomically at the end.

Verification

Every quantitative claim in the manuscript is checked mechanically rather than by reading:

• verify_paper_numbers.py traces each figure to a stored result, and fails on any it cannot trace that is not listed as a known non-result.
• check_pvalues.py checks all 53 quoted p-values against the stored ones, accepting a value only within half a unit of the last significant figure the manuscript commits to.
• verify_conditional_te.py and independent_verification.py check the estimators against independently written implementations and against pyinform.
• guard.py runs all of the above plus its own assertions, in about 11 seconds.

Licence

GPL-3.0-or-later, matching GameBrains, which these scripts import.