[P2][VALIDATION] Create model cards, dataset cards, validation protocols, backtesting, and uncertainty gates

[andreazedda]

Problem

The repository contains mechanistic ODE/PKPD simulation, Twin calibration, toxicity signals, counterfactual runs, uncertainty/sensitivity work and an experimental therapy-design/Pareto module. These components do not yet share a formal validation framework.

A numerical output can currently exist without a canonical statement of:

• intended use;
• biological assumptions;
• units;
• parameter sources;
• calibration domain;
• validation dataset;
• uncertainty coverage;
• failure modes;
• prohibited interpretation.

Objective

Adopt a lifecycle for scientific models in which evidence and validation are first-class versioned artifacts.

Required model card

For every executable scientific model record:

model_id and version
owner
intended use and non-use
state variables and equations
input/output schemas
units and dimensional checks
parameter definitions, defaults and bounds
source/evidence references
calibration method
validation datasets
performance metrics
uncertainty method
known biases and failure modes
compatibility/invalidation rules
change history

Required dataset card

For every synthetic, private or external dataset record:

dataset identity/version/hash
provenance and license/access
population/domain
inclusion/exclusion
variables/units/missingness
quality/validation status
known biases
allowed uses
split strategy
privacy/storage/retention

Private patient data remains outside Git; Git stores schemas/cards and non-identifying metadata only.

Validation protocol

Create a versioned protocol covering:

Structural verification

• equation/code correspondence;
• unit and dimensional consistency;
• parameter bounds;
• invariants such as non-negative populations;
• solver convergence and step sensitivity;
• deterministic seed behavior.

Calibration

• pre/post residual metrics;
• holdout or time-respecting evaluation;
• identifiability diagnostics;
• parameter-bound saturation;
• sensitivity to weights/initialization;
• calibration rejection rules.

Prediction/backtesting

• rolling-origin evaluation;
• naive and simpler-model baselines;
• leakage prevention;
• subgroup/domain analysis;
• error distributions, not only averages;
• failure-case reporting.

Uncertainty

• parameter versus observation versus structural uncertainty;
• interval coverage;
• calibration of probabilities where applicable;
• robustness of scenario ranking;
• sensitivity and ablation analysis.

Counterfactuals

Explicitly distinguish:

mechanistic scenario simulation
association
causal hypothesis
identified causal effect

No causal-effect claim without a documented identification argument.

Therapy-design toy model

The existing bulk/reservoir/subclone, ADC/CAR-T and Pareto logic must remain labelled as an implemented mechanistic toy model until coefficients are estimated/validated. Hand-authored relapse and toxicity weights require sensitivity analysis and evidence linkage.

Acceptance criteria

• Every model used in a report has a versioned model card.
• Every dataset used has a dataset card and legal/access status.
• Validation protocol is versioned and linked in run manifests.
• Calibration has explicit acceptance/rejection gates.
• Temporal backtests prevent leakage and include baselines.
• Uncertainty intervals have coverage diagnostics.
• Ranking robustness is reported under parameter/model changes.
• Model changes require scientific regression/output-diff reports.
• Failures are reported as failures, not hidden by a successful process exit.
• External benchmark claims are made only after reproducible external evaluation.

Milestone suggestion

M3 — Scientific Validation Layer.

Dependencies

Depends on run manifests/model registry and the closed Research Loop v0.1. Supports Research v1 release.