Skip to content

02_evals.md

Latest commit

 

History

History
117 lines (63 loc) · 7.18 KB

02_evals.md

File metadata and controls

117 lines (63 loc) · 7.18 KB
Raw

Evaluation Suite

How Scoring Works

For a set of model completions $\mathcal{C} = {c_1, \dots, c_n}$ and a set of reference answers $\mathcal{D} = {d_1, \dots, d_n}$ drawn from the same underlying question corpus, VOICE computes an alignment score in $[0, 1]$ measuring how stylometrically similar the completions are to the reference author. Both $\mathcal{C}$ and $\mathcal{D}$ are responses to the same set of prompts, so any observed distributional difference reflects style rather than content.

Step 1: Wasserstein Distance

For each metric $f$, apply it to every text to get empirical distributions:

$$X = {f(c_i)}_{i=1}^n \quad \text{and} \quad Y = {f(d_i)}_{i=1}^n$$

and compute their Wasserstein-1 distance $W(X, Y)$. A smaller distance means the two distributions are closer in shape.

Step 2: Calibration

To interpret $W(X, Y)$ we need to know what distance is expected even between two samples of the same size from the same author. VOICE estimates this from the training split of $\mathcal{D}$ by bootstrap resampling: repeatedly drawing two non-overlapping subsets of size $n$ and computing their mutual Wasserstein distance. This yields a reference distribution $\mathcal{W}_0$ of self-distances that characterises natural within-author variation under the same sample size as the completions being evaluated.

Step 3: Tail Value

The observed distance is located within $\mathcal{W}_0$ via its empirical CDF:

$$p = \hat{F}_{\mathcal{W}_0}(W(X, Y)), \qquad t = 1 - p$$

$t$ is the tail value: the fraction of self-distances that exceed the observed distance. A high $t$ means the model's completions are no further from the reference than typical same-author samples, i.e. they are stylistically consistent with the reference author.

Step 4: Group and Overall Score

Metrics are organised into groups (see Stylometric Metrics). Each group captures a distinct linguistic object (word length, vocabulary richness, etc.) and the metrics within it represent different ways of characterising that same object. Because they study the same underlying property, metrics in the same group are highly correlated; averaging within groups before aggregating across them prevents any single linguistic dimension from dominating the score simply by having more metrics defined for it.

Within each group the tail values are averaged:

$$\bar{t}_g = \frac{1}{|g|} \sum_{f \in g} t_f$$

The overall alignment score $\mathcal{S}$ is then the geometric mean over groups, weighting each group equally. Group tail values are floored at $\varepsilon = 10^{-3}$ before the product is taken, so that a model which genuinely scores zero on one dimension receives a heavy penalty without collapsing scores for all other groups:

$$\mathcal{S} = \left(\prod_{g=1}^{G} \max(\bar{t}_g,, \varepsilon)\right)^{1/G} \in (0, 1]$$

This rewards consistent performance across all groups; a high score on one dimension cannot compensate for near-zero performance on another. The harmonic mean would impose an even harsher penalty on low groups, but it collapses discrimination when any group score is exactly zero, which occurs frequently in practice, making it unsuitable here.

A score of 1 would mean the completions are stylistically indistinguishable from the reference even relative to same-author variation; in practice this is unrealistic but the directionality is still informative.

This Is Not a Hypothesis Test

The tail value $t$ resembles a p-value but is not one and should not be interpreted as such.

A classical hypothesis test asks whether there is sufficient evidence to reject a null hypothesis. That framing entails a binary decision and is sensitive to sample size: with enough data, even negligible stylistic differences will cross any fixed significance threshold.

VOICE is not concerned with whether two distributions are statistically indistinguishable. It asks how similar they are, on a continuous scale, relative to the natural within-author variation present in the training corpus. The calibration distribution $\mathcal{W}_0$ serves as a normalisation baseline, not a null to be rejected. There is no significance threshold and no p-value, only a score.

Uncertainty Quantification

The uncertainty in $\mathcal{S}$ arises from the finite number of completions $n$. It is estimated using jackknife resampling over the completion set, operated directly at the level of tail values so that no analytical propagation through the calibration CDF is required. Operating at the tail value level naturally produces asymmetric confidence intervals that respect the $[0, 1]$ boundary of the score.

Jackknife Replicates

For each $i = 1, \dots, n$, remove $c_i$ and its paired reference $d_i$ and run the full pipeline (recomputing the Wasserstein distance and locating it within $\mathcal{W}0$) for every metric $f$. This yields a matrix of leave-one-out tail values ${t{f,-i}}$, or equivalently $n$ replicate vectors $\mathbf{t}_{-i} \in \mathbb{R}^M$ where $M$ is the total number of metrics across all groups.

Confidence Intervals

Per-metric, group, and overall confidence intervals are all read directly from the empirical distribution of the corresponding jackknife replicates.

For an individual metric $f$, the replicates ${t_{f,-i}}$ give a distribution of leave-one-out tail values from which percentile intervals are taken directly.

For a group $g$, the replicate group scores are computed as:

$$\bar{t}_{g,-i} = \frac{1}{|g|} \sum_{f \in g} t_{f,-i}$$

and percentile intervals are read from ${\bar{t}_{g,-i}}$.

For the overall score, the replicate scores are the geometric mean of the per-group replicate averages:

$$\mathcal{S}_{-i} = \left(\prod_{g=1}^{G} \max(\bar{t}_{g,-i},, \varepsilon)\right)^{1/G}$$

and percentile intervals are read from ${\mathcal{S}_{-i}}$.

Example

Suppose a model achieves an alignment score of 0.2. A natural reading is:

On average, across stylometric groups, the model's completions sit at the 80th percentile of same-author self-distances, meaning the completions are stylistically closer to the author reference than 20% of within-author sample pairs drawn from the training corpus.

Diagnostics

The API exposes intermediate results for deeper inspection:

results = make_comparison(completions, true_ds)  # uncertainty=True by default

# Per-metric tail values
results.metric_tails       # dict[str, float]

# Per-group average tail values
results.group_tails        # dict[MetricGroup, float]

# Overall score
results.score              # float

# Jackknife percentile confidence intervals (default 90%)
results.metric_tail_cis()   # dict[str, tuple[float, float]] | None
results.group_tail_cis()    # dict[MetricGroup, tuple[float, float]] | None
results.score_ci()          # tuple[float, float] | None

Per-metric tails are useful for identifying which stylometric dimensions are misaligned; per-group tails aggregate correlated metrics and correspond directly to the terms in the geometric mean.

The confidence interval methods accept a confidence keyword (default 0.90, see UNCERTAINTY_DEFAULTS). Passing uncertainty=False to make_comparison skips the jackknife pass, in which case the interval methods return None.