Add uncertainty quantification to evaluation suite

.pre-commit-config.yaml

@@ -74,16 +74,3 @@ repos:
       - id: pydoclint
         args: ["--config=pyproject.toml"]
         exclude: ^tests/
-
-  # Cognitive complexity limit via flake8-cognitive-complexity (keep for src; exclude tests)
-  - repo: https://github.com/pycqa/flake8
-    rev: 7.3.0
-    hooks:
-      - id: flake8
-        additional_dependencies:
-          - flake8-cognitive-complexity
-        args:
-          - --max-line-length=79
-          - --extend-ignore=E203
-          - --max-cognitive-complexity=10
-        exclude: ^tests/

README.md

@@ -1,3 +1,14 @@
-# VOICE - Fine-tuning for Stylistic Fidelity
+# 🗣️ VOICE
+
+...
 
 ---
+
+## Documentation
+
+```
+docs/
+├── 00_data.md         — Included datasets
+├── 01_evals.md        — Evaluation suite: scoring and interpretation
+└── 02_stylometry.md   — Stylometric metrics: definition and catalogue
+```

cspell/library-words.txt

@@ -9,6 +9,9 @@ ylabel
 frameon
 allclose
 linalg
+mathcal
+mathbb
+mathbf
 dotenv
 wandb
 jinja

cspell/project-words.txt

@@ -17,3 +17,6 @@ resumability
 resumable
 hyperparams
 redef
+evals
+submatrix
+vocab

docs/00_data.md

@@ -0,0 +1,22 @@
+# Datasets
+
+---
+
+Two example datasets are included, each containing press conference Q&A transcripts split into train, validation and test sets:
+
+| President | HuggingFace |
+|---|---|
+| Barack Obama | [`AccelerateScience/bo-press-conference-qa`](https://huggingface.co/datasets/AccelerateScience/bo-press-conference-qa) |
+| George W. Bush | [`AccelerateScience/gwb-press-conference-qa`](https://huggingface.co/datasets/AccelerateScience/gwb-press-conference-qa) |
+
+Each example is a single JSONL record in chat format:
+
+```json
+{
+  "messages": [
+    {"role": "system",    "content": "You are president Barack Obama ..."},
+    {"role": "user",      "content": "<press conference question>"},
+    {"role": "assistant", "content": "<president's answer>"}
+  ]
+}
+```

docs/01_stylometry.md

@@ -0,0 +1,52 @@
+# Stylometric Metrics
+
+---
+
+A **stylometric metric** is any function
+
+$$f : \mathcal{T} \rightarrow \mathbb{R}$$
+
+that maps a text string to a real scalar capturing some surface property of writing style. VOICE treats each metric as a distribution over a corpus: given a set of texts, $f$ is applied to each one to produce a sample from the author's stylometric distribution for that feature.
+
+## Implemented Metrics
+
+### Word Length Distribution
+Moments of the per-word character length distribution.
+
+| Metric | Description |
+|---|---|
+| `avg_word_length` | Mean word length |
+| `std_word_length` | Standard deviation of word length |
+| `skew_word_length` | Skewness of word length |
+| `kurtosis_word_length` | Kurtosis of word length |
+
+### Vocabulary Richness
+Type–token and word statistics measuring lexical diversity.
+
+| Metric | Description |
+|---|---|
+| `type_token_ratio` | Unique tokens / total tokens |
+| `moving_avg_type_token_ratio` | TTR averaged over a sliding window (MATTR) |
+| `hapax_legomena_ratio` | Fraction of words appearing exactly once |
+| `dis_legomena_ratio` | Fraction of words appearing exactly twice |
+| `tri_legomena_ratio` | Fraction of words appearing exactly three times |
+
+### Function Words
+| Metric | Description |
+|---|---|
+| `function_word_ratio` | Proportion of tokens drawn from a closed function-word list |
+
+### Character N-gram Diversity
+Type–token ratio and MATTR computed over character $n$-grams for $n \in \{3, 4, 5\}$.
+
+| Metric | Description |
+|---|---|
+| `char_{n}gram_type_token_ratio` | Character $n$-gram TTR |
+| `char_{n}gram_moving_avg_type_token_ratio` | Character $n$-gram MATTR |
+
+### Text Length
+| Metric | Description |
+|---|---|
+| `num_words` | Total word count |
+
+> **Note:** `num_words` may be deprecated in a future release. It does not tend to be used as a signature for authorship attribution in the broader stylometry literature.

docs/02_evals.md

@@ -0,0 +1,115 @@
+# Evaluation Suite
+
+---
+
+## How Scoring Works
+
+For a set of model completions $\mathcal{C} = \{c_1, \dots, c_n\}$ and a set of reference answers $\mathcal{D} = \{d_1, \dots, d_n\}$ drawn from the same underlying question corpus, VOICE computes an **alignment score** in $[0, 1]$ measuring how stylometrically similar the completions are to the reference author. Both $\mathcal{C}$ and $\mathcal{D}$ are responses to the same set of prompts, so any observed distributional difference reflects style rather than content.
+
+### Step 1: Wasserstein Distance
+
+For each metric $f$, apply it to every text to get empirical distributions:
+
+$$X = \{f(c_i)\}_{i=1}^n \quad \text{and} \quad Y = \{f(d_i)\}_{i=1}^n$$
+
+and compute their Wasserstein-1 distance $W(X, Y)$. A smaller distance means the two distributions are closer in shape.
+
+### Step 2: Calibration
+
+To interpret $W(X, Y)$ we need to know what distance is *expected even between two samples of the same size from the same author*. VOICE estimates this from the **training split** of $\mathcal{D}$ by bootstrap resampling: repeatedly drawing two non-overlapping subsets of size $n$ and computing their mutual Wasserstein distance. This yields a reference distribution $\mathcal{W}_0$ of self-distances that characterises natural within-author variation under the same sample size as the completions being evaluated.
+
+### Step 3: Tail Value
+
+The observed distance is located within $\mathcal{W}_0$ via its empirical CDF:
+
+$$p = \hat{F}_{\mathcal{W}_0}(W(X, Y)), \qquad t = 1 - p$$
+
+$t$ is the **tail value**: the fraction of self-distances that *exceed* the observed distance. A high $t$ means the model's completions are no further from the reference than typical same-author samples, i.e. they are stylistically consistent with the reference author.
+
+### Step 4: Group and Overall Score
+
+Metrics are organised into groups (see [Stylometric Metrics](01_stylometry.md)). Each group captures a distinct linguistic object (word length, vocabulary richness, .etc) and the metrics within it represent different ways of characterising that same object. Because they study the same underlying property, within-group metrics are highly correlated; averaging within groups before aggregating across them prevents any single linguistic dimension from dominating the score simply by having more metrics defined for it.
+
+Within each group the tail values are averaged:
+
+$$\bar{t}_g = \frac{1}{|g|} \sum_{f \in g} t_f$$
+
+The overall **alignment score** $\mathcal{S}$ is then the mean over groups, weighting each group equally:
+
+$$\mathcal{S} = \frac{1}{G} \sum_{g=1}^{G} \bar{t}_g \in [0, 1]$$
+
+A score of **1** would mean the completions are stylistically indistinguishable from the reference even relative to same-author variation; in practice this is unrealistic but the directionality is still informative.
+
+---
+
+## This Is Not a Hypothesis Test
+
+The tail value $t$ resembles a p-value but is not one and should not be interpreted as such.
+
+A classical hypothesis test asks whether there is sufficient evidence to reject a null hypothesis. That framing entails a binary decision and is sensitive to sample size: with enough data, even negligible stylistic differences will cross any fixed significance threshold.
+
+VOICE is not concerned with whether two distributions are statistically indistinguishable. It asks how similar they are, on a continuous scale, relative to the natural within-author variation present in the training corpus. The calibration distribution $\mathcal{W}_0$ serves as a normalisation baseline, not a null to be rejected. There is no significance threshold and no p-value, only a score.
+
+---
+
+## Uncertainty Quantification
+
+The uncertainty in $\mathcal{S}$ arises from the finite number of completions $n$. It is estimated using jackknife resampling over the completion set, operated directly at the level of tail values so that no analytical propagation through the calibration CDF is required. Operating at the tail value level naturally produces asymmetric confidence intervals that respect the $[0, 1]$ boundary of the score.
+
+### Jackknife Replicates
+
+For each $i = 1, \dots, n$, remove $c_i$ and its paired reference $d_i$ and run the full pipeline (recomputing the Wasserstein distance and locating it within $\mathcal{W}_0$) for every metric $f$. This yields a matrix of leave-one-out tail values $\{t_{f,-i}\}$, or equivalently $n$ replicate vectors $\mathbf{t}_{-i} \in \mathbb{R}^M$ where $M$ is the total number of metrics across all groups.
+
+### Confidence Intervals
+
+Per-metric, group, and overall confidence intervals are all read directly from the empirical distribution of the corresponding jackknife replicates.
+
+For an individual metric $f$, the replicates $\{t_{f,-i}\}$ give a distribution of leave-one-out tail values from which percentile intervals are taken directly.
+
+For a group $g$, the replicate group scores are computed as:
+
+$$\bar{t}_{g,-i} = \frac{1}{|g|} \sum_{f \in g} t_{f,-i}$$
+
+and percentile intervals are read from $\{\bar{t}_{g,-i}\}$.
+
+For the overall score, the replicate scores are:
+
+$$\mathcal{S}_{-i} = \frac{1}{G} \sum_{g=1}^G \bar{t}_{g,-i}$$
+
+and percentile intervals are read from $\{\mathcal{S}_{-i}\}$.
+
+---
+
+## Example
+
+Suppose a model achieves an alignment score of **0.2**. A natural reading is:
+
+> On average across stylometric groups, the model's completions sit at the 80th percentile of same-author self-distances, meaning the completions are stylistically closer to the reference than 20% of within-author sample pairs drawn from the training corpus.
+
+---
+
+## Diagnostics
+
+The API exposes intermediate results for deeper inspection:
+
+```python
+results = make_comparison(completions, true_ds)  # uncertainty=True by default
+
+# Per-metric tail values
+results.metric_tails       # dict[str, float]
+
+# Per-group average tail values
+results.group_tails        # dict[MetricGroup, float]
+
+# Overall score
+results.score              # float
+
+# Jackknife percentile confidence intervals (default 90%)
+results.metric_tail_cis()   # dict[str, tuple[float, float]] | None
+results.group_tail_cis()    # dict[MetricGroup, tuple[float, float]] | None
+results.score_ci()          # tuple[float, float] | None
+```
+
+Per-metric tails are useful for identifying which stylometric dimensions are misaligned; per-group tails aggregate correlated metrics and correspond directly to the terms summed in the overall score.
+
+The confidence interval methods accept a `confidence` keyword (default 0.90, see `UNCERTAINTY_DEFAULTS`). Passing `uncertainty=False` to `make_comparison` skips the jackknife pass, in which case the interval methods return `None`.

pyproject.toml

@@ -42,8 +42,6 @@ where = ["src"]
 [dependency-groups]
 dev = [
     "detect-secrets>=1.5.0",
-    "flake8>=7.3.0",
-    "flake8-cognitive-complexity>=0.1.0",
     "isort>=7.0.0",
     "mypy>=1.19.1",
     "notebook>=7.5.3",

src/voice/_defaults.py

@@ -31,17 +31,12 @@ class MetricGroup(str, Enum):
     .. attribute :: CHAR_NGRAM_DIVERSITY
 
         Character n-gram TTR and MATTR across n = 3, 4, 5.
-
-    .. attribute :: TEXT_LENGTH
-
-        Raw token count.
     """
 
     WORD_LENGTH_DISTRIBUTION = "word_length_distribution"
     VOCABULARY_RICHNESS = "vocabulary_richness"
     FUNCTION_WORDS = "function_words"
     CHAR_NGRAM_DIVERSITY = "char_ngram_diversity"
-    TEXT_LENGTH = "text_length"
 
 
 @dataclass(frozen=True)
@@ -111,6 +106,26 @@ class CalibrationDefaults:
 CALIBRATION_DEFAULTS: CalibrationDefaults = CalibrationDefaults()
 
 
+@dataclass(frozen=True)
+class UncertaintyDefaults:
+    """
+    Default parameters for uncertainty quantification.
+
+    These defaults control the confidence intervals reported alongside
+    comparison results.
+
+    .. attribute :: confidence
+
+        Confidence level for percentile intervals read from the
+        jackknife replicate distributions.
+    """
+
+    confidence: float = 0.90  # Safe default for validation/test size = 150
+
+
+UNCERTAINTY_DEFAULTS: UncertaintyDefaults = UncertaintyDefaults()
+
+
 @dataclass(frozen=True)
 class PlottingDefaults:
     """

src/voice/comparison/_utils.py

@@ -35,6 +35,38 @@ def calibrated_percentile(value: float, dist: np.ndarray) -> float:
     return float(np.mean(dist <= value))
 
 
+# -----------------------------------------------------------------------------
+# Percentile interval
+# -----------------------------------------------------------------------------
+
+
+def percentile_interval(
+    values: np.ndarray, confidence: float
+) -> tuple[float, float]:
+    """
+    Compute a central percentile interval from an empirical distribution.
+
+    For a confidence level c, this returns the (alpha/2, 1 - alpha/2)
+    empirical percentiles of `values` with alpha = 1 - c. The interval is
+    asymmetric in general and bounded by the range of `values`.
+
+    :param values: Array of samples from an empirical distribution
+    :param confidence: Confidence level, strictly between 0 and 1
+    :return: (lower, upper) percentile interval
+    :raises ValueError: If `values` is empty or `confidence` is not in [0, 1]
+    """
+    values = np.asarray(values, dtype=float).ravel()
+    if values.size == 0:
+        raise ValueError("values must be non-empty.")
+    if not 0.0 < confidence < 1.0:
+        raise ValueError("confidence must be strictly between 0 and 1.")
+
+    alpha = 1.0 - confidence
+    lower = float(np.percentile(values, 100.0 * alpha / 2.0))
+    upper = float(np.percentile(values, 100.0 * (1.0 - alpha / 2.0)))
+    return lower, upper
+
+
 # -----------------------------------------------------------------------------
 # Bootstrap null distribution
 # -----------------------------------------------------------------------------