Docket — LLM document classification

A small, runnable reference implementation of a classifier. classify banking/investment documents into a typed taxonomy with an LLM, route low-confidence cases to a human, and measure every prompt change against a labelled set so you never silently break a category.

It runs offline out of the box (no API key) using a mock model, so you can read real output immediately, then swap in OpenAI or Anthropic with one line.

Run it

cd docclass
python run_demo.py            # end-to-end demo, offline
python tests/test_classifier.py   # logic tests, offline

To use a real model, set OPENAI_API_KEY (or ANTHROPIC_API_KEY), pip install openai (or anthropic), and swap the backend in run_demo.py:

from docclass.backends import OpenAIBackend
clf = Classifier(backend=OpenAIBackend(), prompt_version="v2")

How it fits together

document text
   │
   ├─ taxonomy.py     defines the classes + which ones are confusable
   ├─ prompts/        versioned prompt templates (classify_v1, v2, ...)
   ├─ backends.py     swappable LLM (Mock | OpenAI | Anthropic)
   ├─ classifier.py   build prompt → call model → parse JSON → confidence route
   └─ evaluation.py   accuracy, per-class P/R/F1, confusion matrix, regression gate

The key separation: adding a document type or editing a prompt is a data change. No pipeline code changes. That is what makes scaling from 15 to 33 categories safe.

Design decisions

Hallucinated labels are always rejected. classifier._parse rejects any label not in the taxonomy and downgrades it to UNKNOWN. Unparseable output is handled the same way. The model can never auto-route a junk label.

Confusable classes are modelled explicitly. taxonomy.py records confused_with for each class, and the prompt injects "Do not confuse with..." guidance. The confusion matrix in evaluation.py then tells you whether errors are the expected hard pairs (fine) or something new (a real bug). In the demo, the errors are exactly CONTRACT_NOTE↔TRADE_CONFIRMATION and KYC_ADDRESS↔BANK_STATEMENT — the predicted hard cases.

Confidence threshold as the human-in-the-loop control. review_threshold controls the AUTO/REVIEW split. The metric that matters is auto_error_rate: of the documents auto-processed, how many were wrong. Trade auto_rate (coverage) against auto_error_rate (safety) by moving the threshold.

Regression gate on every prompt change. evaluation.regressions() compares per-class F1 between two runs and surfaces any class that got worse. It runs in CI because the mock backend needs no network.

Provider-agnostic by design. The backend abstraction means switching providers (OpenAI → Anthropic → Azure OpenAI → self-hosted) touches one class, not the pipeline — important in regulated environments where data-processing agreements may constrain which provider you can use.

What to build next

• Expand taxonomy.py from 15 toward the full 33 types. Each addition should come with 2–3 labelled examples in the test set.
• Write classify_v3.txt, run the demo against a real model, and use the regression gate to prove it beats v2 (or doesn't).
• Add confidence calibration: bin predictions by stated confidence and check whether "0.8" really means 80% correct. Adjust the threshold from data.
• Add a Langfuse hook in backends.py to trace latency and token cost per call.
• Handle OCR noise: add a few garbled documents and see what breaks.

Note on data

Every document in data/labeled_test_set.jsonl is synthetic and invented for this project. No real customer data is used — which is also the right instinct for a regulated environment.