English · Italiano

Geometry Sets the Budget

A clean-residual scale for additive interventions in language models

Key result. How far can you push a language model's residual stream before its continuations become improbable under the model itself? That boundary varies by ~4600× across models — yet it is predicted, with no strength sweep and no training, by a cheap robust statistic: s = σ_med·√d. A random-direction control shows the relation is geometric; a shuffled control isolates a smaller, direction-specific effect.

---

What this is

The measurement harness and all raw data behind two findings about activation steering (adding a unit-norm vector to an LLM's residual stream, h ← h + α·v̂):

1 — A geometric tolerance relation. The self-NLL boundary α_NLL — the largest strength at which a steered continuation's mean NLL under the unsteered model stays within ln τ of baseline — is predicted a-priori by the clean scale s = σ_med·√d: the median per-dimension standard deviation of residual activations at the injection layer (robust, by construction, to the few massive outlier dimensions), times the square root of the residual width. Across 11 open checkpoints spanning six architecture types (including a mixture-of-experts and a reasoning-distilled model), raw boundaries span ~4600× while κ = α_NLL / s stays within [0.2, 1.2]. The relation holds for random unit vectors too — it captures perturbation tolerance, not steering per se.

2 — The Semantic Tolerance Ratio. Semantic directions reach the boundary at or before the random median in 10 of 11 checkpoints (ρ_sem ∈ [0.6, 1.0]; the eleventh sits at 1.25, one grid step the other way). Shuffled controls — same norm and marginal statistics, alignment destroyed — match the random level on two models and move toward it on a third. An exploratory, protocol-dependent diagnostic of direction-specific sensitivity.

Everything ran on consumer Apple hardware (M4 Pro 24 GB, M2 Max 32 GB) in MLX. The instrumented forward passes are written from scratch and validated against reference implementations with the included oracle test (code/test_families.py, threshold ≥ 0.999; measured runs gave 0.9997–0.99999), identical argmax, including the MoE router path.

Repository layout

Path	Contents
data/	83 raw JSON sweep curves. Per-α: {alpha, nll, tone, sample}; top-level: tag, model_type, d, layer, sigma_med, scale_clean, full_norm, kappa_tau2, kappa_tau3. Tag suffixes: canonical (tone, EN); _alt prompt-set retest; _formal / _polite / _negtone trait axes; _ita Italian probes; _rnd<seed> random controls; _shuf<seed> shuffled controls; _Lf<frac> injection-depth sweeps; _n32 32-prompt replications.
code/	Instrumented MLX forward passes (scope_auto.py for qwen2/llama/mistral; scope_families.py adds OLMo-2, Gemma-3, and the OLMoE mixture-of-experts), the sweep harness (kappa_sweep.py), contrastive corpora (steer_gate*.py), the figure (fig_kappa.py), the oracle-validation test (test_families.py), and two early single-model scopes kept for provenance.
MANIFEST.json	The 11 checkpoints with exact HuggingFace IDs (main branch, downloaded 17 Jul 2026; commit SHAs not pinned), injection layers, and per-model statistics.
refs.bib	Bibliography; every 2026 arXiv entry verified against its primary source.
docs/	The scaling figure shown above.
paper/	The preprint PDF (English and an Italian reading translation).

Reproduce

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Regenerate the figure from the released data (no model download needed):
KAPPA_DATA=./data KAPPA_OUT=./docs python code/fig_kappa.py

# Re-measure a κ on a local model (download it via the IDs in MANIFEST.json):
KAPPA_DATA=./data python code/kappa_sweep.py --fine <local_model_dir>

Controls and variants

python code/kappa_sweep.py --fine --random 0   <dir>   # random-direction control
python code/kappa_sweep.py --fine --shuffle 0  <dir>   # shuffled-direction control
python code/kappa_sweep.py --fine --trait polite <dir> # another trait axis
python code/kappa_sweep.py --fine --layerfrac 0.25 <dir>  # injection depth
python code/kappa_sweep.py --fine --ita        <dir>   # Italian probes (Minerva / Mistral)

Requirements: Apple Silicon (MLX), Python ≥ 3.11, models in HuggingFace safetensors format (bf16 — the method deliberately avoids quantized weights). Every κ in the write-up is recomputable from data/. One incidental social-media-style handle in a model-generated sample was redacted; no other edit was made to the raw curves.

Note. Code comments and some runtime log lines are in Italian (the lab's working language); the documentation is in English.

Citation

Archived on Zenodo with a DOI — please cite:

Cappello, D. (2026). Geometry Sets the Budget: A Clean-Residual Scale for Additive Interventions in Language Models (v0.3) [Software]. Zenodo. https://doi.org/10.5281/zenodo.21442383

The concept DOI 10.5281/zenodo.21442383 always resolves to the latest version. Machine-readable metadata: CITATION.cff. The preprint PDF is in paper/.

@software{cappello2026geometry,
  author    = {Cappello, Daniele},
  title     = {Geometry Sets the Budget: A Clean-Residual Scale for Additive Interventions in Language Models},
  year      = {2026},
  publisher = {Zenodo},
  version   = {v0.3},
  doi       = {10.5281/zenodo.21442383},
  url       = {https://doi.org/10.5281/zenodo.21442383}
}

License

Apache-2.0. The measured models remain under their own licenses; see MANIFEST.json for their HuggingFace identifiers.

AI-assistance disclosure

This work was designed, directed, and supervised by the author, who takes full responsibility for all of its content. The instrumented forward passes and analysis code, the execution of all experiments, and the drafting of the manuscript were carried out with Claude Code (running Claude Opus 4.8) under the author's direction and review; OpenAI Codex (GPT-5.6) was used for independent adversarial review of the claims. No collaboration with, or endorsement by, the tools' vendors is implied. Every factual assertion and reference was verified by the author against its primary source.