Frontier AI models typically struggle with conditional probability and regime-switching systems, often resorting to naive Monte Carlo simulations or falling into algebraic traps.
This repository demonstrates a classic failure mode in LLMs when calculating dispersion in mixed-state models (such as MS-HMMs), specifically regarding the Law of Total Variance.
Consider a system with two hidden states:
-
State A:
$N(\mu=0, \sigma^2=1)$ occurring$90%$ of the time. -
State B:
$N(\mu=5, \sigma^2=4)$ occurring$10%$ of the time.
When asked to calculate the total variance of this system, LLMs frequently compute only the expected value of the conditional variances (1.3. They completely hallucinate or ignore the variance of the expected values ($Var(E[X|Y])$), which accounts for the anomalous dispersion between the two states.
The red dashed line shows the narrow distribution the LLM assumes by ignoring the variance decomposition. The solid curve represents the true probability space.
To grade the LLM's output automatically, this script does not rely on approximations. It uses a purely theoretical derivation via SymPy, utilizing exact rational fractions to eliminate floating-point inaccuracies.
By applying the full variance decomposition theorem, the script establishes the absolute mathematical truth (3.55), serving as a rigid evaluation benchmark for frontier AI models.
Tech Stack: Python, SymPy (Symbolic Mathematics), NumPy, Seaborn (Data Visualization).