Decide whether an LLM judge can be trusted to grade without a human watching.
No runtime dependencies. Python 3.14+.
You have 10,000 model outputs and you cannot read them. You want a language model to grade them for you. Before believing a single number it produces, you check it against a sample you graded yourself.
The obvious check — how often did we agree? — is misleading, and the more imbalanced your data, the more misleading it gets.
Suppose 95 of 100 transcripts contain no special-category personal data. A judge that answers "no" every single time is right 95% of the time and has detected nothing. Accuracy rewarded it for the base rate.
The same trap catches agreement between two raters. Two people who each answer "no" 95% of the time will match roughly 90% of the time by luck alone:
both say no = 0.95 × 0.95 = 0.9025
both say yes = 0.05 × 0.05 = 0.0025
------
chance agreement 0.905
So "we agreed 95% of the time" means you got 0.045 above a floor of 0.905. Not impressive; barely above two people guessing.
Kappa subtracts the free agreement and reports the share of what was left that you actually closed.
observed − chance
kappa = -----------------
1 − chance
- observed — how often the two raters actually matched. Counted.
- chance — how often they would have matched had each answered at their own usual rate without looking at the item. Calculated from their totals alone.
- 1 − chance — the room that was available above luck.
A multiple-choice exam with four options each. Guessing already scores 25%. So a score of 50% is not "half right" — it closed 25 points of the 75 that were actually yours to win.
(50 − 25) / (100 − 25) = 0.33
Kappa is that exam score, with guessing replaced by what two raters would match by luck.
The formula is min–max normalisation, (x − min) / (max − min), with
min = chance and max = 1. The same shape appears in R² in regression (floor:
predicting the mean), normalised gain in education research, and efficiency in
physics. Faced with a new metric, the question to ask is: what is its floor,
and what is its ceiling?
| kappa | Meaning |
|---|---|
| 1.0 | Perfect agreement |
| 0.8 – 1.0 | Strong |
| 0.6 – 0.8 | Good |
| 0.4 – 0.6 | Moderate |
| below 0.4 | Poor |
| 0.0 | No better than luck |
| negative | The raters contradict each other more often than luck would |
Totals do not give you observed agreement. Knowing one rater said yes 40 times and the other said yes 80 times does not say which items each chose. The overlap could be anything. Observed agreement must be counted item by item; chance agreement is the opposite, calculated from the totals alone precisely because it pretends nobody looked.
Kappa is symmetric — neither rater is the reference. Swap the two raters and the number is identical. So the choice of metric is a claim about the question:
| Question | Does a right answer exist? | Metric |
|---|---|---|
| Does this text contain a national insurance number? | Yes | Precision / recall / F1 |
| Is this rationale biased? | No, two careful people would differ | Kappa |
Which means kappa can never tell you a judge is correct, only that it is consistent with a human. If both are wrong in the same way, kappa is 1.0 and you have learned nothing.
A kappa of 0.457 from 40 items and from 4,000 items are not the same claim:
items kappa 95% interval width
40 0.457 [0.158, 0.730] 0.572 ← could be poor, could be good
80 0.457 [0.249, 0.647] 0.398
200 0.457 [0.334, 0.581] 0.247
800 0.457 [0.393, 0.518] 0.126
4000 0.457 [0.429, 0.484] 0.055 ← reliably mediocre
The point estimate never told you how much to believe it. The width does.
Note the two different reasons to distrust a judge, which are easy to conflate: at 40 items you cannot tell; at 4,000 you can tell, and the answer is no.
The interval is obtained by bootstrapping — drawing new samples of the same size from your labelled pairs, with replacement, scoring each, and reporting the middle 95%. It assumes nothing about the shape of the underlying distribution.
Two of the three ways this can go both look like "a low kappa", and only one of them is fixed by labelling more items. The lower bound decides whether a claim can be made; the upper bound decides whether more data could still rescue the judge.
Against a threshold of 0.6:
| Interval | Verdict | What is wrong |
|---|---|---|
| [0.65, 0.79] | TRUST |
nothing — the worst case already clears the bar |
| [0.41, 0.88] | LABEL_MORE |
too few labelled items to tell either way |
| [0.43, 0.48] | REJECT |
the judge — narrowing the range will not lift it |
The same judge, at three sample sizes, shows why the distinction is not cosmetic:
40 items kappa=0.457 [0.16, 0.73] -> LABEL_MORE
800 items kappa=0.457 [0.39, 0.52] -> REJECT
8000 items kappa=0.457 [0.44, 0.48] -> REJECT
Identical point estimate throughout. At 40 items you cannot tell; at 800 you can, and the answer is no.
A LABEL_MORE verdict says the estimate is too wide to act on without saying
how much more work would settle it. Larger samples are simulated from the labels
already in hand, and the smallest size whose verdict stops straddling the
threshold is reported — along with what that verdict would be.
good judge, undersampled
now: 32 items, kappa 0.75 [0.50, 0.94] -> LABEL_MORE
estimate: label 128 total -> would read TRUST
poor judge, undersampled
now: 40 items, kappa 0.46 [0.16, 0.73] -> LABEL_MORE
estimate: label 160 total -> would read REJECT
The second row is the one that saves work. Both say "label more" today, but the simulation gives away the ending: 120 further transcripts would buy a rejection. Replace the judge instead.
The assumption, which is the whole limitation: simulated items inherit the agreement pattern of the sample in hand, so the figure is only as sound as that sample is representative. A judge that behaves differently on cases not yet labelled will not obey it.
A kappa says how much two raters differ and never what about. The items behind the number are the only part of this tool that improves a judge rather than grading it, so they are kept and split by direction — what the judge flags that a human would not, and what it misses. The two usually have different causes: consistent over-flagging is often a threshold, consistent misses on one kind of case are often an ambiguous rubric.
kappa 0.00 | agreed 3/6
MISSED (human flagged, judge did not):
- I take Fridays off for prayer
- my wife and I moved here in 2019
OVER-FLAGGED (judge flagged, human did not):
- I use Python daily
The score alone says only "no better than luck". The report says the judge misses disclosures made obliquely, with no keyword present — which is a finding you can act on.
from judgecal.agreement import kappa
from judgecal.disagreement import disagreements
from judgecal.interval import kappa_interval
from judgecal.sample_size import items_needed
from judgecal.verdict import assess
human = [True] * 20 + [False] * 480
judge = [True] * 5 + [False] * 15 + [False] * 480
result = kappa(human, judge)
result.observed # 0.97 — looks excellent
result.kappa # 0.39 — is not
result.table # the four cells, so the number can be audited
interval = kappa_interval(human, judge, seed=0)
interval.low, interval.high
interval.width # how much the estimate is allowed to move
decision = assess(interval, threshold=0.6)
decision.verdict # Verdict.TRUST | LABEL_MORE | REJECT
decision.reason # the bound and threshold that produced it
report = disagreements(human, judge, transcripts)
report.only_first # the judge missed these
report.only_second # the judge flagged these, you did not
report.agreed # counted, not listed
estimate = items_needed(human, judge, threshold=0.6, seed=0)
estimate.needed # total items at which the verdict resolves
estimate.projected # what it would resolve tokappa raises rather than returning a number when both raters used a single
label throughout: chance agreement is then total, there is no room above it, and
a silent 0.0 would be a lie.
| Path | Contents |
|---|---|
judgecal/agreement.py |
Cross-tabulation and kappa |
judgecal/interval.py |
Bootstrap confidence interval |
judgecal/verdict.py |
Trust, label more, or reject |
judgecal/disagreement.py |
The items the raters split on |
judgecal/sample_size.py |
How much more labelling a decision needs |
tests/ |
Hand-computed examples as specification |
uv sync
uv run pytest- Weighted kappa, for ordered categories such as a 1–5 score.
- Fleiss' kappa, for more than two raters.