Code + data behind the AIES 2026 paper "Auditing Alignment Controllability in LLMs via Political Axes."
Just want to run it? Jump to the reproduction recipe. Just want the finding? Read the next section.
Ask the usual question, "is this AI left-wing or right-wing?", and you get a single dot on a political compass. Every model gets its dot. Case closed.
Except that dot barely means anything.
A model's resting position only tells you where it sits when nobody is pushing. But nobody deploys a model in a vacuum. The moment it gets a system prompt (a persona, a role, an instruction) it moves. So the honest question isn't where does it stand, it's:
How far, and in which directions, can you walk it?
Picture the dot as a dog sitting in a field. Where it happens to sit tells you little. What matters is the leash: how far it roams, which directions it goes happily, and where it digs in its heels. That reachable area, not the resting spot, is what decides whether a model can actually voice your views or only ever drifts back toward its own.
Here is exactly that, measured. Each star is a model's resting spot (no steering). Each colored loop is the area we could walk that same model into by changing only the prompt. The grey ✕ are the extreme corners we asked for but no model reached: the built-in limit on what it will say.
Where each model sits vs. how far you can walk it. Same
model, prompt-only steering. No loop fills the square. (Compound-quadrant
pilot, n=5 per corner; regenerate with
analysis/scripts/viz_compound_pilot_scty.py. Full study below.)
What that picture says, in numbers:
- The leash is long. The persona you set explains about 90% of where a model lands; the brand you chose explains under 3%. Steering, not training, dominates what you see.
- Not every dog walks the same. Some roam far, some barely leave the spot, some hit an invisible fence. Where a model won't go is itself a bias, a limit on which views it will voice for you.
- Two fair ways to measure the walk can disagree about which direction a model favors. So "this model leans X" is a shakier claim than it sounds.
- Models almost never refuse (about 1 in 100), and none can be walked into every corner of the map.
Why this matters, and to whom. If you audit AI, govern it, or care about political pluralism, the reachable area is the real thing to scrutinize: it shows whose views a model can and can't be made to express, a fairness question a single score can't touch.
If you ship AI, this is a test you may be skipping. The bias that bites you isn't where a model sits out of the box. It's where a few words (a system prompt, or just a user's phrasing) can push it. The dimension worth measuring is your model's reachable area on the axes that matter for your product:
- An AI tutor should not swing on a value-laden topic because a student rephrases the question, or quietly slant a lesson under a mild role prompt.
- A customer-facing assistant should not be steerable into off-brand, partisan, or non-compliant answers by an adversarial prompt.
The procedure here (define an axis, score responses, measure how far the model moves) is exactly that test. It tells you what your deployment can be pushed to say before someone else finds out for you.
7 leading models × 13 persona prompts × 70 Political Compass items × 10 repeats = 63,700 answers (plus saturation, ablation, and compound-quadrant lanes). Models: GPT-5, Claude Sonnet 4.5, Grok-4.3, Gemini 2.5 Flash Lite, DeepSeek-Chat v3.1, Kimi K2, Qwen3.6 Max Preview.
We used the Political Compass because it is simple and widely understood, but it is only the instrument. The transferable part is the procedure: define a value-sensitive axis, build an instrument that scores responses along it, then measure how far a model can be steered across it (its reachable area, not its resting point). Any domain where you can name an axis and score answers along it fits the same template, and the same measurement supports two uses: auditing how much a model can be aligned to a target, and tuning toward one.
- Reproduce every number. A short recipe (below) regenerates the processed
data and re-checks 51/51 paper claims against a live recomputation
(
paper_claim_audit.py). - Reuse the data for new questions.
raw/is the unmodified model responses: every persona × item × repeat, with failures kept honest (never silently counted as "neutral"). It's a ready-made substrate for questions we didn't ask: other axis definitions, per-item patterns, cross-model geometry, refusal behavior, better metrics. - Reuse the harness.
ingestion/is a small, standard-library pipeline you can point at new models or new prompt families.
If you build on this, a citation is appreciated. See Citation.
.
├── requirements.txt Python dependencies
├── CITATION.cff How to cite (repo artifact + paper)
├── LICENSE / LICENSE-data.txt MIT (code) / CC-BY-4.0 (data)
│
├── raw/ Unmodified collection artifacts
│ ├── core/ 910 core replicates (single bundle)
│ ├── sat_topup/ Saturation supplement (4 named × 20 runs)
│ ├── r3_ablations/ Length / temperature / refusal pilots
│ ├── compound_pilot/ Compound pilot, econ × government plane
│ └── compound_pilot_scty/ Compound pilot, econ × society plane
│
├── processed/ Analysis-ready, NULL-aware cell scores
│ (also regenerable from raw/, see below)
│
├── ingestion/ Collection + processing pipeline
│ ├── 01_collect/ OpenRouter collectors (re-collect raw)
│ ├── 02_process/ Raw → processed transformers (stdlib only)
│ └── tests/ pytest contracts
│
└── analysis/ Paper numerics + figures
├── scripts/ Analysis scripts (one per Results section)
└── tests/ Smoke tests
processed/ is committed for convenience and is byte-for-byte regenerable
from raw/ (the test suite checks this). Generated, not shipped:
analysis/figures/: paper/appendix PNGs, rendered by theviz_*scripts_output/:run_all.pylogs + figure copies
The manuscript and appendix (.tex / PDF) are released separately via the
Zenodo DOI flow and are not part of this repository.
All commands assume your cwd is the repository root.
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txtPython ≥ 3.10. The 02_process/ scripts only use the stdlib; the analysis
scripts need numpy / pandas / scipy / matplotlib / statsmodels.
processed/ is already committed, so you can skip straight to step 3. To
regenerate it from raw/ and confirm reproducibility:
./ingestion/02_process/process_all.shRe-derives all 10 processed JSON files deterministically (sorted keys, no
timestamps, byte-identical for byte-identical input). The pytest suite
(pytest ingestion/tests analysis/tests) asserts the regenerated files match
the committed ones. See ingestion/README.md for the per-lane breakdown.
# All paper-claim values vs. live recomputation:
python analysis/scripts/paper_claim_audit.py
# Main-paper figures (batch_steerability.png, batch_compass_2d.png):
python analysis/scripts/viz_paper_figures.py
# Appendix figures (main_study_compass_grid.png, compound_pilot_compass_grid.png):
python analysis/scripts/viz_appendix_grids.py
# Pilot B figure (compound_pilot_scty_compass.png):
python analysis/scripts/viz_compound_pilot_scty.pyPer-section numeric scripts (01_variance_decomposition.py, 02_dispersion_tiers.py,
…) are documented in analysis/README.md.
Costs USD and depends on third-party model availability. Skip unless you want a fresh end-to-end run.
export OPENROUTER_API_KEY=sk-or-...
./ingestion/01_collect/collect_core.sh --execute # ~$25–35, ~2 h
./ingestion/01_collect/collect_sat_topup.sh --execute # ~$5, ~30 min
./ingestion/01_collect/collect_r3_ablations.sh --execute # ~$2, ~10 min
./ingestion/01_collect/collect_compound_pilot.sh --execute
./ingestion/01_collect/collect_compound_pilot_scty.sh --execute- No silent imputation. Unparseable / refusal / empty / api-error
responses are stored as
numeric_score = nullwith explicitresponse_status. - One identical-prompt retry at collection time
(
repair_invalid_once = True,repair_prompt_suffix = ""). A second failure yieldspersistent_<x>withnumeric_score = null. - NULL excluded from both numerator and axis denominator in cell-score recomputation. Missing data never pulls a cell toward Neutral.
- R3 e3 refusal pilot is exempt from retry. Refusals there are the experimental signal, not a transient failure.
The precise figures behind the plain-words claims above:
- Design: 7 models × 13 contexts, every core cell n = 10; 910 core replicates + 160 saturation supplement + 16 R3 ablation cells.
- Variance decomposition: context (persona) η² ≈ 0.88–0.93; model η² < 0.03; context × model interaction η² ≈ 0.05–0.08. (This is "the prompt matters far more than the brand.")
- Cross-model convergence: under authoritarian framing, per-question shifts agree across the seven endpoints at mean pairwise Spearman r̄ = 0.79 (permutation p < 0.001).
- Metric non-equivalence: displacement and proximity can rank steering directions oppositely under non-centered baselines, so directional "bias" claims are measurement-dependent.
- Refusals: ≈ 1.19% overall; refusals cluster rather than spread.
- Data honesty: 192 NULL responses (0.22% of 87,360 question rows), 0 imputed; repair-retry recovery 159 / 192 ≈ 82.8% (same-prompt, one attempt). In the core lane specifically: 64 rows repair-folded, 54 recovered, 10 still NULL (6 refusal, 4 unparseable).
- Missing-data policy cannot carry a finding:
13_null_sensitivity.pyrecomputes every context × axis aggregate with NULL excluded and again with NULL scored as Neutral. Max movement is 0.034 pp on the released data and 0.113 pp with the repair rolled back — both inside the 0.124 pp bound the paper quotes from the V3.6 pre-repair audit.
paper_claim_audit.py confirms 51 / 51 claims HOLD against the current data.
| Question | Start here |
|---|---|
| End-to-end reproduction recipe | this file |
| Per-lane raw → processed contracts | ingestion/README.md |
| Script ↔ Results-section map | analysis/README.md |
| Raw dataset cards (per lane) | raw/<lane>/README.md |
- Code (
analysis/,ingestion/, scripts): MIT. SeeLICENSE. - Data (
raw/and any derivedprocessed/outputs): CC-BY-4.0. SeeLICENSE-data.txt.
Machine-readable metadata is in CITATION.cff; GitHub's
"Cite this repository" button reads it. It points to the AIES 2026 paper as
the preferred citation. Please cite the paper if you use the data or code.
To cite this dataset/code archive directly, use the version-stable Zenodo concept DOI: 10.5281/zenodo.21489805.