Is your LLM judge measuring taste, or is it measuring length, formatting, and familiarity?
Misthos answers that with an experiment instead of a vibe. It generates minimal counterfactual pairs (two responses that make identical claims but differ on exactly one surface property), shows them to your judge in both orders, and reports how much each property moved it, with confidence intervals, measured against a control.
Then it does the other half: it turns a small number of an expert's pairwise judgments into an explicit, auditable rubric, and runs that rubric through the same battery. A rubric is not trustworthy because a human wrote it down. It is trustworthy because it survives the same falsification.
misthos (μισθός): Greek for wage, reward, what is owed.
pip install misthos
misthos demo # runs the whole loop offline, no API keyReinforcement learning from human feedback is trained on preference data, and preference data is contaminated. Annotators systematically favour text that feels familiar (typicality bias) and this is now understood as a data-level driver of mode collapse, not an artifact of the optimiser (Zhang et al., 2025).
The practical consequence has its own well-worn anecdote: a team scales up preference collection, agreement with the collected feedback goes up, and the outputs get blander: better on the metric, worse to the people who actually know the domain. The optimiser is working. The target is wrong.
That is a measurable claim, and measuring it is the whole job here. Everyone has the anecdote. Almost nobody has the instrument.
There is no shortage of adjacent work: rubric-based reward modelling (RaR, OpenRubrics, Auto-Rubric), static LLM-judge bias benchmarks, reward-hacking benchmarks. All of it is either a paper artifact evaluated on a fixed public dataset, or a benchmark with a frozen set of items. None of it tells you what your judge does on your prompts.
That is the gap Misthos fills.
| Axis | Question |
|---|---|
length |
Does it reward more words for the same content? |
scaffolding |
Does it reward markdown structure over plain prose? |
typicality |
Does it reward familiar, conventional phrasing? (the anti-slop probe) |
hedging |
Does it reward or punish epistemic hedging? |
confidence |
Does it reward assertive framing? |
sycophancy |
Does it reward flattery aimed at the reader? |
identity |
Does it reward a response for who is said to have written it? |
paraphrase |
Control. How much does it move when nothing meaningful changes? |
Position bias is measured across all axes at once, because it belongs to the judge rather than to any one perturbation.
1. Every pair passes a content-preservation gate. A rewrite model told to
"make this longer" will happily also sharpen the argument. Keep that pair and
your length number is quietly a quality number. So each pair faces a
deterministic axis check (did the rewrite actually do the thing?) and a
claim-level equivalence check (did it smuggle in content?). Failures are
dropped and counted; the card reports 153 of 160 generated pairs survived, because "we measured 153 pairs" and "we generated 160 and kept 153"
are different claims and only one is honest.
2. Nothing is scored against zero. Judges are noisy; a paraphrase that
changes nothing still moves one a little. So every axis is scored against the
paraphrase control with a two-sample bootstrap, and a confound is only
reported when that difference interval excludes zero. Moving a judge is easy.
Moving it more than a meaning-preserving rewrite does is the finding.
3. Every pair is judged in both orders. Order is itself a confound. Average over it, or it leaks into every other axis' number instead of being reported as its own.
A Taste Card, a shareable page describing what a judge responds to:
| Axis | Sensitivity | 95% CI | vs control | 95% CI | Reading |
| ------------- | ----------- | ---------------- | ---------- | ---------------- | ------------------------- |
| `hedging` | -0.900 | [-1.000, -0.750] | -1.067 | [-1.517, -0.583] | confound detected |
| `length` | +1.000 | [+1.000, +1.000] | +0.833 | [+0.400, +1.300] | confound detected |
| `scaffolding` | +1.000 | [+1.000, +1.000] | +0.833 | [+0.400, +1.300] | confound detected |
| `typicality` | +0.789 | [+0.526, +1.000] | +0.623 | [+0.107, +1.114] | confound detected |
| `identity` | +0.375 | [+0.100, +0.650] | +0.208 | [-0.308, +0.733] | not distinguishable |
| `paraphrase` | +0.167 | [-0.300, +0.600] | n/a | n/a | control |
Sensitivity is signed, in [-1, +1]: 0 is the target for every axis. Note
identity (a placebo that changes only the claimed author) correctly comes
back as not distinguishable from control. An instrument that fires on
everything is not an instrument.
The headline number is the confound index: root-mean-square sensitivity across the non-control axes. It counts surface sensitivity in either direction, so a judge deliberately built to punish clichés scores high on it. That is a design choice showing up as a measurement, not automatically a defect; read the signs before concluding.
# 1. Audit a judge you already have
misthos audit --corpus responses.json --provider anthropic:claude-opus-5 \
--out card.json --markdown card.md
# 2. Elicit an expert's taste (active selection picks which pairs to ask about)
misthos elicit --corpus responses.json --out session.json -n 40
misthos compile --session session.json --out rubric.json
misthos curve --session session.json # active vs random sample efficiency
# 3. Audit the rubric with the same battery
misthos audit --corpus responses.json --rubric rubric.json --out card-rubric.json
# 4. Watch for reward hacking as you optimise against it
misthos guard --before card-week1.json --after card-week8.json \
--score-before 0.61 --score-after 0.83 --fail-on-hackingguard is the piece that matters once a judge becomes a reward signal. The
interesting failure is not that scores stop rising; it is that they keep rising
for the wrong reason. If the optimised score went up and confound sensitivity
went up with it, some of that win is in the confounds; --fail-on-hacking
exits non-zero so CI can say so.
Expert attention is the scarce input in this entire pipeline. Misthos fits a
Bradley-Terry model over the judgments collected so far and asks next about the
pair it is least able to predict, weighted toward items it has seen least.
misthos curve reports what that buys against a random baseline on your own
data: if active selection does not win there, the curve says so.
Comparisons are only ever proposed within a prompt: preferring one answer over another is meaningful, preferring an answer about Rome over an answer about recursion is not.
Provider-agnostic from the first commit. One method, three backends:
misthos audit --provider anthropic:claude-opus-5 ...
misthos audit --provider openai:gpt-4.1 ...
misthos audit --provider mock ... # deterministic, offline, no key
export MISTHOS_PROVIDER=anthropic:claude-opus-5 # or set it onceYou can also split them: perturb with one model, audit a judge running on
another (--judge-provider). Implementing a backend is one method:
class MyProvider:
name = "mine"
def complete(self, prompt, *, system=None, max_tokens=4096, schema=None, task=None) -> str:
...The core package has zero dependencies; vendor SDKs are optional extras
(pip install 'misthos[anthropic]').
It is not a stub, and it is the reason the test suite means anything. It plants specific biases; it likes length, scaffolding and familiar phrasing, dislikes hedging, and has a mild position bias, and the tests assert that the battery recovers them with the right sign and ordering, and does not fire on the placebo axis. If a refactor of the statistics quietly stops detecting a planted confound, CI goes red.
misthos demo runs the entire loop against it with no API key. Everything it
produces is a demonstration of the instrument, not a measurement of any real
judge, and the generated cards say so in their own notes. In particular the
mock's rubric judge is a simulation of partial instruction-following: do not
read the generic-vs-rubric comparison as evidence that rubrics reduce confounds.
Measuring that is what a real provider is for.
Alpha, and honest about what it does not yet do:
- Prose only. The axes are prose-specific. Code and UI need their own:
axes are data (
misthos/audit/axes.py), so adding a domain is additive. - Power. With ~20 base responses per axis, only effects above roughly 0.3 clear the control reliably. The intervals show this; do not read a non-significant axis as "no bias", read it as "not resolved at this n".
- The gate is a model call. A perturbation model that consistently changes content in a way the checker consistently misses would bias results. The gate pass rate is on the card so you can see when it is doing suspiciously little.
- Single judge per run. Ensembles and reward-model APIs are wrapper work, not new machinery.
New axes, new domains, and adversarial attacks on the gate are the most useful contributions. See CONTRIBUTING.md.
MIT.