An LLM-as-a-judge is an AI model that grades another model's answers. judgekeeper shows how often your LLM-as-a-judge agrees with you, and where it goes wrong. It is open source (Apache-2.0) and runs on your own computer.
- Find. Find your eval results and the LLM-as-a-judge that graded them.
- Mark. Mark your answers Pass or Fail in your browser.
- Measure. See how often your LLM-as-a-judge agrees with you.
- Fix. Write a better rule and test it on marks kept aside.
- Re-check. Change your LLM-as-a-judge and see if it still agrees with you.
In your project folder, in its virtual environment:
pip install judgekeeper
judgekeeper
Or, with no virtual environment: uvx judgekeeper.
Works with promptfoo, DeepEval, Inspect AI, MLflow or your own CSV, TSV or JSONL file.