judgekeeper ^_™

Check your LLM-as-a-judge in 5 steps.
Find. Mark. Measure. Fix. Re-check.

Website · Guide · Reference · Changelog

An LLM-as-a-judge is an AI model that grades another model's answers. judgekeeper shows how often your LLM-as-a-judge agrees with you, and where it goes wrong.

Install

In your project folder, in its virtual environment:

pip install judgekeeper
judgekeeper

Or, with no virtual environment: uvx judgekeeper. Python 3.11 to 3.14.

Reads promptfoo, DeepEval, Inspect AI, MLflow, or your own CSV, TSV or JSONL file with input, output and verdict columns. A verdict can be pass/fail, true/false, yes/no, correct/incorrect, right/wrong or 1/0, in any case; --label-map reads other words. Inspect AI .eval logs and MLflow need an extra: pip install "judgekeeper[inspect]" or pip install "judgekeeper[mlflow]" (the uv forms are in the reference).

Using a coding agent? Give it this page and tell it: "Follow this to check my LLM-as-a-judge." It runs the steps and gives you the marking page link; you mark, then the coding agent explains the result.

The 5 steps

1. Find: judgekeeper finds your eval results.
2. Mark: a page opens in your browser. Mark answers Pass or Fail; the LLM-as-a-judge's grade stays hidden. "See the numbers so far" appears with your first numbers, and "See your result" once there is a result.
   • No browser here, or a coding agent runs it? judgekeeper --no-browser prints the link to open instead.
   • Already have Pass/Fail labels? Run judgekeeper --labels your-labels.csv, or judgekeeper --labels if they are in your results. Then mark more if needed.
3. Measure: run judgekeeper again to see how often it agrees with you.
4. Fix: judgekeeper fix helps you write a better rule and tests it on marks kept aside. It makes no AI call: you grade the marked answers again with your own eval tool, which may use your API key.
5. Re-check: after you change the model or the rule, grade your marked answers again (judgekeeper fix writes them to .judgekeeper/fix/marked-answers.csv). judgekeeper check <new results file> shows if it still agrees with you.

In your project, judgekeeper reads your results files and writes only in .judgekeeper/, plus judgekeeper.toml after you say yes. It makes no AI call and needs no API key. See SECURITY.md.

The numbers

• TPR: of the answers you passed, how many it passed too. The result page calls it "When you say Pass, your LLM-as-a-judge also says Pass".
• TNR: of the answers you failed, how many it failed too. The result page calls it "When you say Fail, your LLM-as-a-judge also says Fail".
• Kappa: how much you agree beyond what matching at random would give.

TPR and TNR each come with a range, such as "about 90% (between 85% and 93%)" (when you mark every answer, the numbers have no range). The result says "good enough to use" only when a wider range (99%) starts at 80% or more for both. That way you can look after every mark. The numbers are about the answers in your results, worked out from the ones you marked. They are not a promise about new answers.

How the numbers are worked out: reference.

Checked on public data

These runs use public datasets that people already labelled. One adds made-up empty answers that should fail, and one is a simulation. Where the true number is known, they test whether judgekeeper's ranges hold it. They also show what judgekeeper says about real LLM-as-a-judge setups. Each line links its report.

• JailbreakBench: the ranges held the true values in 96 of 100 random tests with 100 marks each (TPR, TNR and kappa at once; another 100 tests give a slightly different count).
• RAGTruth: news summaries. Three LLM-as-a-judge setups, 3 runs each, and 3 more runs at a lower effort setting: needs work in all 12. Changing the model did not close the gap.
• RAGTruth question answers and data-to-text: "needs work" in all 4 runs.
• JailbreakBench graded by two other setups: "needs work" for both; each passed more than half of the expert-failed answers it graded (setup B could not grade 61 of 300).
• Empty answers: one model failed 0 of 20 empty summaries that should fail.
• Simulation: the ranges held the true value 96.7 times in 100 on average.

More

A guide on the website walks through the 5 steps. judgekeeper --help lists every command, and the reference explains each one. Made by one person; issues welcome, pull requests are not taken.

License

judgekeeper and the judgekeeper logo are trademarks of Sathvik Thota. The Apache-2.0 license covers the code, not the name or logo.