08.evaluation.md

Evaluation of Context-Adaptive Models

The preceding sections described how context-adaptive models are built, explicitly through a structured $f(c)$ and implicitly through emergent computation in large models. Both paradigms raise the same three practical questions once a model is deployed: how cheaply it adapts, how stably it routes a context to the right behavior, and how well that behavior holds up when the context shifts. This section collects the design principles and proposes formal evaluation metrics that answer these questions.

Measuring Context-Conditional Performance

We base every metric below on a single straightforward quantity: performance conditioned on context. Let $\mathcal{C}$ denote the context space and $\mathcal{D}_{\mathrm{test}}$ a test distribution over $(x, y, c)$. For a predictor $\hat{f}$, define the context-conditional risk as

$$ \mathcal{R}(\hat{f}\mid c) = \mathbb{E}!\left[, \ell\big(\hat{f}(x,c), y\big) ,\middle|, c ,\right], \qquad \mathcal{R}(\hat{f}) = \mathbb{E}_{c\sim \mathcal{D}_{\mathrm{test}}}!\left[, \mathcal{R}(\hat{f}\mid c) ,\right]. $$

A single aggregate risk hides exactly the heterogeneity these models exist to capture, so a context-stratified evaluation reports $\mathcal{R}(\hat{f}\mid c)$ across predefined bins or via a smoothed estimate $\int \mathcal{R}(\hat{f}\mid c),\mathrm{d}\Pi(c)$ for a reference measure $\Pi$ that weights regions of the context space.

Adaptation Efficiency

The efficiency of a context-adaptive method hinges on design choices that trade computational tractability against statistical accuracy, and the same choices govern whether a method scales to large data while remaining interpretable.

One central principle is the use of sparsity assumptions to limit the number of context-dependent parameters. This can be achieved through group sparsity, which encourages entire groups of parameters to be zero simultaneously [@doi:10.1111/j.1467-9868.2005.00532.x], hierarchical regularization that applies different strengths of shrinkage to varying levels of context specificity [@doi:10.1017/CBO9780511790942], and adaptive thresholding that adjusts sparsity levels in accordance with context complexity.

Efficiency can also be enhanced through computational strategies that allocate resources adaptively. Early stopping terminates optimization for contexts where convergence occurs rapidly [@doi:10.48550/arXiv.1606.04838], while context-dependent sampling employs different sampling schemes across contexts [@doi:10.48550/arXiv.1809.09582]. Caching and warm-starting further accelerate optimization by reusing solutions from similar contexts, which is particularly effective when contexts vary smoothly [@doi:10.1561/2200000016]. Related context-dependent computation appears in adaptive batching, per-context learning rates, and multi-fidelity pipelines that allocate compute and precision by context complexity.

To make these gains measurable, let $S_k(c)={(x_j, y_j, c)}_{j=1}^k$ denote $k$ examples available within context $c$, and define the adaptation efficiency as

$$ \mathrm{AE}_k(c) = \mathcal{R}(\hat{f}_0 \mid c)

• \mathcal{R}(\hat{f}_{S_k} \mid c), \qquad \mathrm{AE}k = \mathbb{E}{c}!\left[, \mathrm{AE}_k(c) ,\right], $$

where $\hat{f}0$ is the non-adapted baseline and $\hat{f}{S_k}$ the adapted predictor; the curve $k \mapsto \mathrm{AE}k$ summarizes few-shot adaptation gains across context sizes. A closely related quantity measures how much of this efficiency carries between contexts. For a shared representation $\phi$ transferred from source to target contexts $\mathcal{C}{\mathrm{src}} \to \mathcal{C}_{\mathrm{tgt}}$, the transfer gain is

$$ \mathrm{TG}(\phi) = \mathcal{R}{\mathcal{C}{\mathrm{tgt}}}!\big(\hat{f}_{\mathrm{scratch}}\big)

• \mathcal{R}{\mathcal{C}{\mathrm{tgt}}}!\big(\hat{f}_{\phi}\big), $$

the risk reduction from transferring $\phi$ rather than training from scratch, so that positive values indicate useful transfer, matching the sign convention of $\mathrm{AE}_k$. This is the same efficiency argument that motivates the amortized context encoders of the previous sections.

Formalization: data limits on adaptation efficiency

These gains are ultimately bounded by how much data each context supplies. Let contexts lie in a metric space $(\mathcal{C}, d)$, let the per-context parameter be $\theta(c)$, and for an observation $(x,y,c)$ let $p_\theta(y\mid x,c)$ be a conditional model with loss $\ell(\theta;x,y,c)$. For a neighborhood $\mathcal{N}_\delta(c)={c':d(c,c')\le\delta}$, the effective sample size available to estimate $\theta(c)$ is

$$ N_{\text{eff}}(c,\delta)=\sum_{i=1}^n w_\delta(c_i,c), \qquad w_\delta(c_i,c)\propto K!\left(\frac{d(c_i,c)}{\delta}\right), \qquad \sum_i w_\delta(c_i,c)=1, $$

for a kernel $K$. A kernel-regularized estimator with smoothness penalty

$$ \mathcal{R}(\theta)=\int|\nabla_c\theta(c)|^2,\mathrm{d}c $$

solves

$$ \widehat{\theta}=\arg\min_{\theta\in\Theta};\frac{1}{n}\sum_{i=1}^n \ell(\theta;x_i,y_i,c_i)+\lambda,\mathcal{R}(\theta). $$

Under local Lipschitzness in $c$ and an $L$-smooth, $\mu$-strongly convex risk in $\theta$, a standard bias–variance decomposition yields for each component $j$

$$ \mathbb{E}!\left[|\widehat{\theta}_j(c)-\theta_j(c)|^2\right] \lesssim \underbrace{\frac{\sigma^2}{N_{\text{eff}}(c,\delta)}}_{\text{variance}} +\underbrace{\delta^{2\alpha}}_{\text{approx. bias}} +\underbrace{\lambda^2}_{\text{reg. bias}}, \qquad \alpha>0. $$

where $\sigma^2$ is the noise variance, $L$ and $\mu$ the smoothness and strong-convexity constants of the risk, and $\alpha>0$ the context-smoothness exponent governing the approximation bias. Finer locality (small $\delta$) sharpens resolution but shrinks $N_{\text{eff}}$ and inflates variance; this is the quantitative form of the adaptation–data tradeoff, and amortized encoders raise $N_{\text{eff}}$ by sharing $\theta(c)=f_\phi(c)$ across contexts. Compute enters through the same neighborhood: an early-stopped first-order method with step size $\eta$ and $T(c)$ context-dependent iterations satisfies

$$ \mathcal{L}!\big(\theta^{(T(c))}\big)-\mathcal{L}!\big(\theta^{\star}\big) \le (1-\eta\mu)^{T(c)}!\left(\mathcal{L}!\big(\theta^{(0)}\big)-\mathcal{L}!\big(\theta^{\star}\big)\right) +\frac{\eta L\sigma^2}{2\mu,N_{\text{eff}}(c,\delta)}, $$

where $\mathcal{L}$ is the population objective, $\theta^{(0)}$ the initialization, $\theta^{\star}$ its minimizer, and $\eta$ the step size. This links the compute budget $T(c)$ and data availability $N_{\text{eff}}(c,\delta)$ to the attainable excess risk at context $c$.

Routing Stability

Both explicit and implicit models adapt by choosing which parameters or which experts govern the prediction. Explicit models make this decision overtly, through the partitions or context encoders that map context to model parameters. Implicit models make the same decision internally, through the gating of a mixture of experts or the attention weights of a transformer. In either case the decision is a function of context, and the stability of this function plays a key role in generalization: small perturbations of the context should not cause erratic changes in which behavior is selected, or predictions become unpredictable near routing boundaries.

Instability is easiest to see at the boundaries. A partition model can flip a sample between adjacent leaves under a negligible change in a splitting variable, and an in-context learner can change its answer under a reordering or reformatting of the same prompt examples, a sensitivity documented for in-context learning where the distribution and arrangement of examples can matter more than their content [@doi:10.48550/arXiv.2202.12837]. In sparse expert models routing instability can cause experts to collapse or remain underused, undermining reliability [@doi:10.48550/arXiv.2406.18219]. Reporting routing stability therefore means probing the model with perturbations that should be behavior-preserving, such as reordering exchangeable examples, jittering a continuous context near a learned split, or relabeling an equivalent measurement policy, and measuring how much the selected behavior and the resulting prediction move. Smoothly routed models degrade gracefully across these perturbations while sharply partitioned or brittle ones do not. The difference is a design choice as much as a diagnostic.

This admits a single measure across both discrete and continuous perturbations. Let $\rho(c)$ denote the routing decision the model exposes, whether gate weights, a leaf assignment, or the adapted parameters $\theta(c)$, and let $\Pi_c$ be a family of behavior-preserving perturbations of context $c$. The routing stability at $c$ is the expected movement of that decision under perturbation,

$$ \mathrm{RSt}(c) = \mathbb{E}_{\pi \sim \Pi_c}, d\big(\rho(c),, \rho(\pi(c))\big), $$

with a supremum over $\Pi_c$ giving the worst-case variant. Discrete invariances such as reordering exchangeable examples enter through $\Pi_c$ directly; for continuous perturbations, normalizing by the context displacement $D(c, \pi(c))$ recovers a local Lipschitz constant of the routing map, which is the smoothness reading of stability for continuous contexts.

Context-Specific Robustness

The final major evaluation type asks whether adaptation still helps when the context distribution at test time differs from training. Context shift takes many forms, including new subpopulations, drifting covariates, and unseen measurement policies. For explicit models it is easiest to make concrete when context is the pattern of observed measurements, since the deployment environment routinely presents measurement policies that were rare or absent during training; we use missingness-as-context as the running example (i.e. input dropout). Evaluation of missingness-as-context models should report mask-stratified metrics, including worst-group performance, following group-robust evaluation practice [@doi:10.48550/arXiv.1911.08731; @doi:10.48550/arXiv.2012.07421]. Robustness should be probed with mask-shift stress tests, training under one measurement policy and testing under another, to quantify degradation and the benefit of contextualization, as formalized in the Domain Adaptation under Missingness Shift (DAMS) setting [@doi:10.48550/arXiv.2211.02093; @doi:10.48550/arXiv.2012.07421]. When imputation is used, authors should assess imputation realism by holding out observed entries under realistic mask distributions and reporting MAE/RMSE and calibration for $p(x_{\text{missing}}\mid x_{\text{observed}})$ [@doi:10.48550/arXiv.1806.02382; @doi:10.48550/arXiv.1806.02920]. For causal or estimation applications, conduct ignorability sensitivity analyses, contrasting MAR-based results with pattern-mixture or selection-model analyses under plausible MNAR mechanisms [@doi:10.2307/2337120; @doi:10.48550/arXiv.2301.05043]. Finally, include ablations that remove mask or indicator inputs, and, for trees, disable default-direction routing, to confirm that gains derive from modeling the mask signal rather than from artifacts [@doi:10.48550/arXiv.1603.02754; @doi:10.48550/arXiv.2211.09259]. Practical implementations of these ideas are widely available: GRU-D [@doi:10.48550/arXiv.1606.01865] and BRITS [@doi:10.48550/arXiv.1805.10572] provide mask- and time-aware sequence models, while GAIN [@doi:10.48550/arXiv.1806.02920] and VAEAC [@doi:10.48550/arXiv.1806.02382] offer open-source code for imputation under arbitrary masks. For tree ensembles, XGBoost supports sparsity-aware default-direction splits, making it straightforward to treat missing values as context without preprocessing [@doi:10.1145/2939672.2939785].

Implicit models face the same issue in a different form. In-context learning adapts within a single forward pass, but that adaptation is opaque and can be brittle: performance is sensitive to small changes in the prompt, and reliability under distribution shift is hard to guarantee or to audit. Recent surveys frame the open questions as developing a more complete theoretical account of when in-context learning fails, improving its reliability, and establishing methods for controlling its behavior in high-stakes applications [@doi:10.48550/arXiv.2301.00234]. The evaluation logic carries over from the explicit case: stratify performance by the kind of context, stress-test under context shift, and ablate the adaptive signal to confirm that the model is using context as intended rather than exploiting an artifact. From this perspective, the mask-shift stress test for an explicit model and the prompt-perturbation test for an in-context learner are the same experiment applied to two realizations of $f(c)$.

The degradation these tests reveal can be summarized in a single score. Let $Q$ denote a family of admissible context shifts, for example $f$-divergence or Wasserstein balls over context marginals; the robustness score

$$ \mathrm{RS}(\hat{f}; Q) = \sup_{\widetilde{\mathcal{D}}\in Q} \left[ \mathcal{R}_{\widetilde{\mathcal{D}}}(\hat{f})

• \mathcal{R}{\mathcal{D}{\mathrm{test}}}(\hat{f}) \right] $$

reports the worst-case excess risk over that family, with higher values indicating greater sensitivity to contextual change.