Skip to content

Add VaR model validation: rolling forecasts and backtests - #4

Merged
ult666666 merged 1 commit into
mainfrom
feat/var-backtesting
Nov 2, 2026
Merged

ult666666 merged 1 commit into
mainfrom
feat/var-backtesting

Conversation

@ult666666

Copy link
Copy Markdown
Owner

What changes

quantkit computed VaR and CVaR but could not tell whether a VaR model is any good. This PR adds quantkit.var, the validation loop risk teams and bank supervisors use:

  • Rolling forecasts (rolling_var_es): one-day VaR and expected shortfall (ES) by historical simulation, a Gaussian model, EWMA (RiskMetrics, decay 0.94) and filtered historical simulation (FHS), with a configurable window and confidence level. A forecast for day t uses only returns t - h .. t - 1. Every method is one location-scale model, r[t] = mu[t] + sigma[t] e[t], which defines VaR, ES and RiskForecast.simulate.
  • Exception backtests: Kupiec's proportion-of-failures test (LR_uc), Christoffersen's independence (LR_ind) and conditional coverage (LR_cc) tests, with p-values. Statistics stay finite with no exceptions, only exceptions or no consecutive pairs (0 log 0 = 0). null_rejection_rates computes each test's exact size with a dynamic program over the exception sequence.
  • Basel traffic light (basel_traffic_light): zones and plus factors from Table 2 of the Basel Committee's 1996 backtesting framework, for 250 days of 99% VaR.
  • ES backtest (es_backtest): Acerbi and Székely's (2014) Z2, with a Monte Carlo p-value from a null simulated from the forecasts themselves.
  • CLI and examples: quantkit var-backtest prices.csv --method ewma --level 0.99, examples/var_backtest.py (figure on synthetic GARCH data) and examples/var_size_power.py (size and power study). CI runs both scripts and the new command.
  • README, DESIGN (new section 6), CHANGELOG, limits and roadmap updated. The historical-VaR tail count moved to a shared helper (no result changes).

Verification

  • uv run pytest --cov=quantkit: 434 passed (345 before), coverage 99.62% (99.50% before), every new module at 100%. ruff check, ruff format --check, mypy --strict clean. README doctests include a new VaR example.
  • Independent checks: each rolling window equals historical_var / parametric_var on that window; EWMA equals the RiskMetrics recursion; FHS equals a Hull-White rescaling loop; Kupiec equals SciPy binomial log-likelihoods (Hypothesis); p-values equal scipy.stats.chi2.sf; LR_cc = LR_uc(days 2..T) + LR_ind; exact sizes equal brute-force enumeration of all 2^T sequences; Basel cumulative probabilities equal binom.cdf(k, 250, 0.01) to the published two decimals.
  • No look-ahead: changing the scored return and all later ones leaves every forecast through that day bit-identical for all four methods; a peeking forecaster fails the same check. Returns older than the history do not matter either.
  • Closed forms on i.i.d. data: historical simulation's exception probability is m / (window + 1) (3/251 at 99%, 250 days); the Gaussian method's is a Student-t prediction probability (1.479% for a 30-day window).
  • Size (returns drawn from the EWMA model, so forecasts are exact): the forecasts recover the model volatility to 1e-12 and exceptions equal z < -2.326 exactly; rejection rates match the exact sizes, 9.5% (LR_uc), 1.4% (LR_ind), 1.1% (LR_cc) at 250 days, and 5% for Z2.
  • Power (uv run python examples/var_size_power.py, 1,000 paths): Gaussian VaR on Student-t(4) returns over 1,000 days is rejected by LR_uc in 46.9%, LR_cc in 31.3% and Z2 in 87.3% of paths, against 2.2%, 0.9% and 3.9% on normal returns. At 97.5% the exception tests reject 3.4% while Z2 rejects 43.3%.
  • Mutation check: 15 injected bugs (look-ahead window, reversed EWMA weights, ddof=0, FHS standardized by the next day's volatility, ES formula, missing factor 2, wrong pooling in LR_ind, chi2(1) for LR_cc, <= exceptions, DP pair counting, two Basel table errors, Z2 without +1, wrong p-value tail, null not simulated) each fail at least one test.

Limits

  • One-day horizon and a single return series in return units: no multi-day scaling, position-level P&L or option revaluation.
  • EWMA decay is fixed, not estimated; FHS filters with EWMA, not a fitted GARCH.
  • LR p-values are asymptotic. null_rejection_rates reports the true size (9.5% for nominal 5% Kupiec at 250 days, 99%), but exact or Monte Carlo p-values are on the roadmap, as are duration-based tests.
  • The traffic light covers the 1996 table (250 days, 99%), not the FRTB 97.5% zones or P&L attribution.
  • Z2's null is simulated from each method's own forecast distribution, which is discrete for historical simulation and FHS.

Generated by Claude Code

quantkit computed historical and Gaussian VaR and CVaR, but nothing tested
whether a VaR model is any good. Risk teams validate VaR models with exception
backtests, and supervisors map exception counts to the Basel traffic-light
zones.

What changed:
- quantkit.var.rolling_var_es: rolling one-day VaR and expected shortfall by
  historical simulation, a Gaussian model, EWMA (RiskMetrics, decay 0.94) and
  filtered historical simulation. Each forecast uses only the window before the
  day it forecasts. Every method is one location-scale model, which also
  defines RiskForecast.simulate.
- Kupiec, Christoffersen independence and conditional coverage tests with
  p-values, finite with no exceptions, only exceptions or no consecutive pairs.
  null_rejection_rates gives each test's exact size by dynamic programming.
- basel_traffic_light: the zones and plus factors of the Basel Committee's 1996
  backtesting framework, Table 2.
- es_backtest: Acerbi and Szekely's Z2 with a null simulated from the forecasts.
- quantkit var-backtest command, two example scripts (figure and size/power
  study), README, DESIGN and CHANGELOG updates; CI runs the new command and
  examples. The historical-VaR tail count moved to a shared helper.

Verification:
- uv run pytest --cov=quantkit: 434 passed (345 before), coverage 99.62%
  (99.50% before); ruff, ruff format and mypy --strict clean.
- Rolling forecasts equal historical_var/parametric_var on every window, the
  RiskMetrics recursion, and a Hull-White loop for FHS. A no-look-ahead test
  changes the scored return and later ones; a peeking forecaster fails it.
- Statistics match hand calculations and SciPy (binom, chi2); exact sizes match
  brute-force enumeration of all 2^T sequences.
- Size: on returns drawn from the EWMA model, simulated rejection rates match
  the exact sizes (9.5%, 1.4%, 1.1% at 250 days, 99%) and Z2's 5%.
- Power (examples/var_size_power.py, 1,000 paths): Gaussian VaR on Student-t(4)
  returns over 1,000 days is rejected by LR_uc in 46.9% and Z2 in 87.3% of
  paths, against 2.2% and 3.9% on normal returns.
- 15 injected bugs (look-ahead window, reversed EWMA weights, ddof, missing
  factor 2, wrong chi2 dof, <= exceptions, Basel table, Z2 tail, and others)
  each fail at least one test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016h2RvhZFj77hPxFn8TQUqq
@ult666666
ult666666 merged commit 7efb979 into main Nov 2, 2026
6 checks passed
ult666666 added a commit that referenced this pull request Nov 2, 2026
Review of the VaR model validation release (#4) found three problems.

The README quick start told users to pip install the v0.1.0 release
wheel. That release predates quantkit.var. Followed as written, the
third README doctest block fails with ModuleNotFoundError: No module
named 'quantkit.var', and `quantkit var-backtest` is an invalid choice.
Cutting a release is out of scope, so the quick start now installs from
GitHub and says what the v0.1.0 wheel lacks. CI gains a step that builds
the wheel, installs it in a clean environment and runs the README
doctests against it.

`quantkit var-backtest` printed its table, then exited with status 1 when
the sample was too long for null_rejection_rates to enumerate. One case
is 20 years of daily returns at 95%: "enumerating 4789 days with up to
369 exceptions is too slow". The command now only leaves out the
exact-size note. `quantkit --help` now mentions VaR model validation.

Verification:
- New test_var_backtest_handles_samples_too_long_for_the_exact_size
  fails on main (main() returns 1) and passes with the fix.
- The v0.1.0 wheel, installed in a fresh venv: the README doctests fail
  in the VaR block and var-backtest is rejected. The wheel built from
  this branch, and pip install of git+https://github.com/ult666666/
  quantkit.git, pass the README doctests (about 6 s for the CI step).
- uv run pytest --doctest-glob=README.md --cov=quantkit: 435 passed,
  99.62% coverage. ruff check, ruff format --check and mypy pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016h2RvhZFj77hPxFn8TQUqq
@ult666666
ult666666 deleted the feat/var-backtesting branch November 4, 2026 23:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants