Claims Reserving Benchmark

When should actuaries trust machine learning for claims reserving?

This project compares classical actuarial reserving methods with distributional GLMs, tabular machine learning, a PyTorch neural network, and a constrained Chain Ladder residual hybrid. Every model is evaluated under the same rolling-valuation protocol; the project measures where flexibility helps instead of assuming that AI wins.

Artifacts: working paper · recruiter guide · research protocol · reproducibility contract

The manuscript is an independent working paper and has not been peer reviewed.

Author and scope: Amine Manai develops and maintains this benchmark as an independent research project. It uses public actuarial datasets and does not contain insurer or client data.

Current research snapshot: v0.3.0

• deterministic company-level panel from all 6 CAS CLRD lines, in paid and incurred views
• 13 model implementations across classical, statistical, ML, neural, and hybrid families
• company-LOB clustered inference with bootstrap confidence intervals, paired Wilcoxon tests, Friedman ranks, and Holm family-wise error correction
• 7 controlled scenarios x 30 independent seeds x 3 valuation dates
• one-year emergence diagnostics and separate probabilistic calibration analysis
• hybrid shrinkage ablation and canonical RAA external reference validation
• 49 tests, 95%+ coverage, a strict 85% coverage gate, and clean Ruff checks
• source-pinned, SHA-256 verified data plus configuration and run fingerprints
• Streamlit dashboard, static diagnostics, generated report, LaTeX table, and social summary

Repository map

config/                       Effective experiment choices
data/raw/                     Verified public downloads (ignored by Git)
docs/                         Methodology, data card, and recruiter guide
paper/                        Research manuscript source
publication/                  Generated report, LaTeX table, and communication draft
results/                      Detailed/summary CSVs, metadata, and diagnostics
scripts/                      Direct workflow entry points
src/claims_reserving/
  defaults/                   Wheel-safe default YAML configuration
  evaluation/                 Backtests, metrics, and tail analysis
  models/                     Classical, GLM, ML, neural, and hybrid models
  visualization/              Static report diagnostics
  benchmark.py                Configuration-driven orchestration
  research.py                 Company panel, scenarios, ablations, and inference
  research_reporting.py       Research tables, figures, and manuscript macros
  dashboard.py                Interactive Streamlit application
  metadata.py                 Hardware, package, Git, data, and config provenance
  paths.py                    Portable runtime directory resolution
tests/                        Numerical, integration, CLI, plotting, and packaging tests

For a short technical tour, read docs/RECRUITER_GUIDE.md. For actuarial details, read docs/METHODOLOGY.md.

Installation

Python 3.11-3.13 is tested in CI.

git clone https://github.com/aminemanai2003/claims-reserving-benchmark.git
cd claims-reserving-benchmark
python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements.lock
python -m pip install -e . --no-deps

For development without the lock:

python -m pip install -e ".[all]"

Reproduce the benchmark

claims-benchmark download all --data-dir data/raw
claims-benchmark benchmark --data-dir data/raw --output-dir results --config-dir config
claims-benchmark scenarios --output-dir results --config-dir config
claims-benchmark report --results-dir results --publication-dir publication

Run the complete research protocol and build its publication assets:

claims-benchmark research --data-dir data/raw --output-dir results --config-dir config
claims-benchmark research-report --results-dir results --publication-dir paper/generated --figures-dir paper/figures

The full protocol uses 48 company-LOB clusters, paid and incurred views, 30 simulation replications per scenario, and neural models. Expect a materially longer runtime than the portfolio smoke benchmark. Use claims-benchmark research --quick for CI-style validation.

The quick smoke run is:

claims-benchmark benchmark --quick --without-neural --output-dir results/quick

Runtime directories never depend on site-packages. They can also be set through:

• CLAIMS_RESERVING_DATA_DIR
• CLAIMS_RESERVING_OUTPUT_DIR
• CLAIMS_RESERVING_CONFIG_DIR

Configuration

The four files under config/ control datasets, model hyperparameters, scenario seeds, valuation splits, metrics, and tail assumptions. benchmark.py consumes those values directly. A wheel contains identical defaults under src/claims_reserving/defaults/, so the CLI remains usable outside a source checkout.

Every run writes:

• effective_config.yaml
• dataset SHA-256 values
• configuration SHA-256
• Git commit and dirty-state flag
• Python and package versions
• CPU, RAM, CUDA availability, and selected compute device
• successful, numerical-failure, and response-domain non-applicability counts

Models

Family	Models
Classical	Chain Ladder, Mack, Bornhuetter-Ferguson, Cape Cod, residual-bootstrap Chain Ladder
Statistical	ODP Poisson GLM, Gamma GLM, Tweedie GLM
Machine learning	Ridge, Random Forest, histogram gradient boosting
Neural	PyTorch feed-forward network with early stopping
Hybrid	ChainLadder-ResNet with horizon-dependent shrinkage

BF and Cape Cod require genuine premium exposure. They are deliberately excluded from RAA rather than substituting first-development claims as fake exposure. BF uses either an explicit expected loss ratio or a leakage-safe prior estimated from the most mature origin periods; Cape Cod uses all observable exposure-weighted periods.

Full research results

The primary score is mean absolute symmetric reserve error (MASRE), a bounded measure that remains defined when incurred triangles show zero or negative future emergence. Run the full protocol above to generate the versioned company-panel, scenario, ablation, calibration, and paired-inference tables under results/; the manuscript consumes those fresh artifacts. Results are conditional benchmark findings, not universal rankings. The detailed audit trail distinguishes successful fits, numerical failures, and valid model/data non-applicability.

Scenario evidence

The full scenario study uses 210 independently seeded simulated triangles. The leading model can change by regime, supporting model selection by reserving conditions rather than branding. Log-link GLMs are only evaluated when their observed incremental claims meet the model's response domain; otherwise the attempt is retained as not_applicable, never silently clipped or transformed.

Tail and uncertainty diagnostics

tail_sensitivity.csv compares selected tail factors with a log-linear extrapolation of positive excess link ratios. Mack and bootstrap models report raw origin-level coverage, company-cluster bootstrap calibration intervals, and width. Generated plots include:

• model comparison;
• reserve-error curves through valuation dates;
• tail-factor sensitivity;
• triangle heatmaps;
• development factors;
• actual-versus-projected triangles;
• reserve intervals by origin.

Dashboard and publication

claims-dashboard

The dashboard selects RAA, CAS, or simulated triangles; valuation date; claim basis; line of business; and model. It displays completed triangles, reserves, diagnostics, and saved benchmark tables.

claims-benchmark report generates:

• publication/benchmark_report.md
• publication/results_table.tex
• publication/linkedin_post.md

paper/main.tex consumes the generated LaTeX tables and figures.

The research edition additionally generates cluster-level statistical summaries, paired comparisons, scenario and ablation tables, vector figures, manuscript macros, the final PDF under output/pdf/, and a self-contained arXiv source archive under output/arxiv/. See docs/RESEARCH_PROTOCOL.md.

Quality checks

python -m ruff check .
python -m pytest
python -m build

GitHub Actions runs these checks on Python 3.11 and 3.13. Tests include canonical RAA Chain Ladder factors and reserve, checksum failures, CAS aggregation, BF exposure controls, all models, rolling leakage controls, metadata, CLI workflows, all plots, report generation, neural determinism, and package-safe paths.

Hardware

Published runs record CPU, memory, package versions, CUDA availability, and selected compute device in their metadata. The hash-locked environment uses the official CPU-only PyTorch wheels for portable CI and local reproduction. An explicitly installed compatible CUDA build can still be selected with device="auto".

Limitations

This is research software, not an actuarial opinion. Schedule P data is historical and aggregated. Mack intervals are approximate, residual bootstrap assumptions require validation, and tail conclusions depend on the chosen extrapolation. Deep sequence models remain intentionally excluded until pooled data justifies their parameter count. The accompanying manuscript is a non-peer-reviewed working paper.

Sources

• Casualty Actuarial Society research resources
• Chainladder-python sample datasets
• CAS-maintained chainladder-python repository

Code is MIT licensed. Public datasets retain their original source terms.