Repository navigation
Add VaR model validation: rolling forecasts and backtests - #4
Merged
Merged
Conversation
quantkit computed historical and Gaussian VaR and CVaR, but nothing tested whether a VaR model is any good. Risk teams validate VaR models with exception backtests, and supervisors map exception counts to the Basel traffic-light zones. What changed: - quantkit.var.rolling_var_es: rolling one-day VaR and expected shortfall by historical simulation, a Gaussian model, EWMA (RiskMetrics, decay 0.94) and filtered historical simulation. Each forecast uses only the window before the day it forecasts. Every method is one location-scale model, which also defines RiskForecast.simulate. - Kupiec, Christoffersen independence and conditional coverage tests with p-values, finite with no exceptions, only exceptions or no consecutive pairs. null_rejection_rates gives each test's exact size by dynamic programming. - basel_traffic_light: the zones and plus factors of the Basel Committee's 1996 backtesting framework, Table 2. - es_backtest: Acerbi and Szekely's Z2 with a null simulated from the forecasts. - quantkit var-backtest command, two example scripts (figure and size/power study), README, DESIGN and CHANGELOG updates; CI runs the new command and examples. The historical-VaR tail count moved to a shared helper. Verification: - uv run pytest --cov=quantkit: 434 passed (345 before), coverage 99.62% (99.50% before); ruff, ruff format and mypy --strict clean. - Rolling forecasts equal historical_var/parametric_var on every window, the RiskMetrics recursion, and a Hull-White loop for FHS. A no-look-ahead test changes the scored return and later ones; a peeking forecaster fails it. - Statistics match hand calculations and SciPy (binom, chi2); exact sizes match brute-force enumeration of all 2^T sequences. - Size: on returns drawn from the EWMA model, simulated rejection rates match the exact sizes (9.5%, 1.4%, 1.1% at 250 days, 99%) and Z2's 5%. - Power (examples/var_size_power.py, 1,000 paths): Gaussian VaR on Student-t(4) returns over 1,000 days is rejected by LR_uc in 46.9% and Z2 in 87.3% of paths, against 2.2% and 3.9% on normal returns. - 15 injected bugs (look-ahead window, reversed EWMA weights, ddof, missing factor 2, wrong chi2 dof, <= exceptions, Basel table, Z2 tail, and others) each fail at least one test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016h2RvhZFj77hPxFn8TQUqq
ult666666
added a commit
that referenced
this pull request
Nov 2, 2026
Review of the VaR model validation release (#4) found three problems. The README quick start told users to pip install the v0.1.0 release wheel. That release predates quantkit.var. Followed as written, the third README doctest block fails with ModuleNotFoundError: No module named 'quantkit.var', and `quantkit var-backtest` is an invalid choice. Cutting a release is out of scope, so the quick start now installs from GitHub and says what the v0.1.0 wheel lacks. CI gains a step that builds the wheel, installs it in a clean environment and runs the README doctests against it. `quantkit var-backtest` printed its table, then exited with status 1 when the sample was too long for null_rejection_rates to enumerate. One case is 20 years of daily returns at 95%: "enumerating 4789 days with up to 369 exceptions is too slow". The command now only leaves out the exact-size note. `quantkit --help` now mentions VaR model validation. Verification: - New test_var_backtest_handles_samples_too_long_for_the_exact_size fails on main (main() returns 1) and passes with the fix. - The v0.1.0 wheel, installed in a fresh venv: the README doctests fail in the VaR block and var-backtest is rejected. The wheel built from this branch, and pip install of git+https://github.com/ult666666/ quantkit.git, pass the README doctests (about 6 s for the CI step). - uv run pytest --doctest-glob=README.md --cov=quantkit: 435 passed, 99.62% coverage. ruff check, ruff format --check and mypy pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016h2RvhZFj77hPxFn8TQUqq
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
quantkit computed VaR and CVaR but could not tell whether a VaR model is any good. This PR adds
quantkit.var, the validation loop risk teams and bank supervisors use:rolling_var_es): one-day VaR and expected shortfall (ES) by historical simulation, a Gaussian model, EWMA (RiskMetrics, decay 0.94) and filtered historical simulation (FHS), with a configurable window and confidence level. A forecast for daytuses only returnst - h .. t - 1. Every method is one location-scale model,r[t] = mu[t] + sigma[t] e[t], which defines VaR, ES andRiskForecast.simulate.0 log 0 = 0).null_rejection_ratescomputes each test's exact size with a dynamic program over the exception sequence.basel_traffic_light): zones and plus factors from Table 2 of the Basel Committee's 1996 backtesting framework, for 250 days of 99% VaR.es_backtest): Acerbi and Székely's (2014) Z2, with a Monte Carlo p-value from a null simulated from the forecasts themselves.quantkit var-backtest prices.csv --method ewma --level 0.99,examples/var_backtest.py(figure on synthetic GARCH data) andexamples/var_size_power.py(size and power study). CI runs both scripts and the new command.Verification
uv run pytest --cov=quantkit: 434 passed (345 before), coverage 99.62% (99.50% before), every new module at 100%.ruff check,ruff format --check,mypy --strictclean. README doctests include a new VaR example.historical_var/parametric_varon that window; EWMA equals the RiskMetrics recursion; FHS equals a Hull-White rescaling loop; Kupiec equals SciPy binomial log-likelihoods (Hypothesis); p-values equalscipy.stats.chi2.sf;LR_cc = LR_uc(days 2..T) + LR_ind; exact sizes equal brute-force enumeration of all 2^T sequences; Basel cumulative probabilities equalbinom.cdf(k, 250, 0.01)to the published two decimals.m / (window + 1)(3/251 at 99%, 250 days); the Gaussian method's is a Student-t prediction probability (1.479% for a 30-day window).z < -2.326exactly; rejection rates match the exact sizes, 9.5% (LR_uc), 1.4% (LR_ind), 1.1% (LR_cc) at 250 days, and 5% for Z2.uv run python examples/var_size_power.py, 1,000 paths): Gaussian VaR on Student-t(4) returns over 1,000 days is rejected by LR_uc in 46.9%, LR_cc in 31.3% and Z2 in 87.3% of paths, against 2.2%, 0.9% and 3.9% on normal returns. At 97.5% the exception tests reject 3.4% while Z2 rejects 43.3%.ddof=0, FHS standardized by the next day's volatility, ES formula, missing factor 2, wrong pooling in LR_ind, chi2(1) for LR_cc,<=exceptions, DP pair counting, two Basel table errors, Z2 without+1, wrong p-value tail, null not simulated) each fail at least one test.Limits
null_rejection_ratesreports the true size (9.5% for nominal 5% Kupiec at 250 days, 99%), but exact or Monte Carlo p-values are on the roadmap, as are duration-based tests.Generated by Claude Code