RouteIQ

A cost-aware decision layer for routing text requests. RouteIQ classifies incoming requests (emails, support tickets, purchase requests, form submissions), decides with a cheap and fast model first, escalates to a stronger LLM only when confidence is low, falls back to a human when even that is unsure, and writes the outcome to a target system over REST. Every step is measured for accuracy, cost and latency.

At a glance (measured on 300 held-out CLINC150 examples, see Results):

• The cascade matches the accuracy of an LLM-only setup (98.3%) at about 54% of its cost ($0.0064 against $0.0118 per 1,000 requests).
• Every request the baseline accepted was right (139 of 139), and 2% of requests went to a person.
• Python, FastAPI, SQLite and scikit-learn behind a React and TypeScript dashboard, packaged with Docker, with more than 1,100 automated tests.

The numbers in this README come from real runs, and tests check that they can be reproduced. The Roadmap lists what is not done.

---

Why this exists

Many business processes start with someone reading a message and deciding where it goes: which team, which category, which priority. Sending every one of those decisions to a large LLM works, but it is slow and expensive, and most requests are easy.

RouteIQ explores a simple idea: use the cheapest model that is confident enough, and spend LLM budget only where it changes the outcome. It also makes that trade-off measurable instead of guessed.

How it works

                 incoming request
                        |
                        v
             +---------------------+
             |  Tier 1: fast model |   sklearn baseline / small decision model
             |  label + confidence |
             +----------+----------+
                        |
          confidence >= t1 ?
             yes /            \ no
                /              \
               v                v
        +-----------+   +---------------------+
        |   accept  |   |  Tier 2: LLM        |
        +-----------+   |  via OpenRouter     |
               |        +----------+----------+
               |                   |
               |        confidence >= t2 ?
               |           yes /        \ no
               |              /          \
               |             v            v
               |        +---------+  +--------------+
               |        | accept  |  | human review |
               |        +---------+  |    queue     |
               |             |       +------+-------+
               +------+------+--------------+
                      v
          +------------------------+
          |  Integration adapter   |   webhook / mock ERP (OData-style REST) / JSON
          +------------------------+
                      |
                      v
          metrics store (SQLite): label, confidence, tier, cost, latency

The dashboard in pictures

The screenshots show the CLINC150 test set replayed through the cascade (scripts/replay_demo.py, see Demo data): the decisions are the ones the real models made, while the times and the reviewers' decisions are simulated. The Try it screenshot is a live answer of the trained baseline model.

Overview. Totals for a time window: requests, cost, p95 latency, how requests ended and the delivery status. Reviewer agreement is shown with a note that it is not accuracy.

Review queue. The requests the cascade was unsure about, oldest first, with one button per label. The model's suggestion is marked, but a person picks the label.

History. Every request, with filters, search and paging. A row opens to show the full text and the delivery state.

Charts. Requests, cost and latency over time, with a table of the same numbers.

Evaluation. Accuracy with 95% intervals, cost and latency of the four setups, how the cascade handled the examples, and calibration. It is measured offline on labeled examples, not on live traffic.

Try it. Send a text through the cascade and see the label, the confidence, the tier, the cost and the latency.

Features

• Config-driven. Categories, thresholds, models and targets live in YAML. A new use case needs labeled data and a config, including a one-line description of each label for the LLM prompt.
• Pluggable models. Every model implements one interface, so swapping providers is a config change.
• Confidence-based cascade. Two thresholds decide between accept, escalate and human review.
• Human-in-the-loop queue. Low-confidence items are stored with the model's suggestion for a person to confirm or correct.
• Integration adapters. Webhook, mock ERP (OData-style REST) and JSON Lines export out of the box. Delivery runs after the response, and its status is tracked per request.
• Dashboard. A React UI served by the API under /dashboard/: a review queue for a person to confirm or correct the model's suggestion, an overview with cost, latency and delivery status, charts over time, a searchable request history, the offline evaluation with confidence intervals, and a form to try a text.
• Built-in evaluation. Accuracy, macro-F1, cost per 1,000 requests, p50/p95 latency, calibration and escalation rate, all from one command.
• Worked examples. A bank-support scenario built on the public CLINC150 dataset, and an electric-vehicle after-sales domain on synthetic data that shows a new domain needs only a spec and a config. More are planned (see Roadmap).

Quick start

git clone https://github.com/ebubekirylmaz/routeiq.git
cd routeiq

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install -e ".[dev]"

cp .env.example .env             # add OPENROUTER_API_KEY

Prepare the benchmark data, train the baseline and run an evaluation:

python data/prepare_clinc.py
python -m routeiq.train --config configs/clinc150.yaml
python -m routeiq.evaluate --config configs/clinc150.yaml

Start the API:

uvicorn routeiq.api:app --reload

Route a request:

curl -X POST http://localhost:8000/route \
  -H "Content-Type: application/json" \
  -d '{"text": "my card got declined at the store"}'

Example response:

{
  "label": "card_declined",
  "confidence": 0.7005,
  "tier": "baseline",
  "action": "accepted",
  "cost_usd": 0.0,
  "latency_ms": 26.8,
  "degraded": false,
  "request_id": 1
}

Or run it with Docker, without a local Python setup:

cp .env.example .env             # add OPENROUTER_API_KEY
docker compose up --build

The first build takes a few minutes: it installs the dependencies, downloads CLINC150, and trains the baseline inside the image. The API is then available at http://localhost:8000. It is published on localhost only, because it has no authentication. The API key is read from .env when the container starts and is not stored in the image. The SQLite database lives in the routeiq-data volume and survives restarts; docker compose down -v deletes it. The mock ERP keeps its tickets in memory, so they reset on restart. The image also contains the built dashboard: open http://localhost:8000/dashboard/.

Look at the dashboard without calling a model

A new installation has no requests, so every screen is empty. This fills a separate database with the CLINC150 test set, replayed through the cascade from the answers the real models gave when they were evaluated. No model is called and nothing costs money. It needs the baseline trained in the steps above and the dashboard built (see Dashboard):

python scripts/replay_demo.py                    # writes demo.db
ROUTEIQ_DB=demo.db OPENROUTER_API_KEY=unused uvicorn routeiq.api:app

Open http://localhost:8000/dashboard/. With the key set to unused the language model cannot answer, so a text you send through Try it that the baseline is not sure about goes to the review queue; use your real key to see the whole cascade. ROUTEIQ_DB keeps the demo apart from your real database. More in Demo data.

API

Endpoint	Purpose
GET /health	Liveness check
GET /config	The task, the labels with their descriptions, the tiers with their thresholds, the type of the delivery target, where the data comes from (data_source) and the example sentences of the Try it screen. Target URLs, file paths, secret names and prices are never returned
POST /route	Body {"text": "..."} (1 to 5,000 characters). Runs the cascade and stores the request
GET /requests	Request history with filters, paging and a total count (see below)
GET /review?limit=50&offset=0	Requests waiting for a person, oldest first, with the model's suggestion. The header X-Total-Count holds the number waiting
POST /review/{id}	Body {"label": "..."}. Records the person's decision
POST /deliveries/retry?limit=100	Resends deliveries that failed, or that have been pending for more than 5 minutes (for example after a restart), oldest first
GET /stats?since=...	Totals for everything or for a window: requests, accepted per tier, human-review, delivery and reviewer-agreement counts, cost, average and p95 latency
GET /stats/timeseries	The same kind of numbers per hour or per day, for charts (see below)
GET /evaluation	Accuracy with 95% intervals, macro-F1, cost, latency and calibration of the four setups (baseline, LLM, cascade, cascade with human review), computed from the recorded predictions of the test split. It is measured offline on labeled examples, not on live traffic. 404 when nothing is recorded for the configuration

action is accepted or human_review. tier names the tier that produced the label (or the last suggestion). degraded is true when a tier failed and the cascade fell back to what it had; the error details are logged and stored, not returned.

Resolving a review item returns 404 for an unknown id, 409 if it is already resolved or never needed review, and 422 for a label that is not in the config.

curl http://localhost:8000/review
curl -X POST http://localhost:8000/review/1 \
  -H "Content-Type: application/json" \
  -d '{"label": "out_of_scope"}'

Request history

GET /requests filters by action, tier, review_status, delivery_status, degraded and q (a case-insensitive substring of the text; % and _ are matched literally). limit (1 to 200, default 50) and offset page through the result, newest first unless order=oldest. The number of requests matching the filters, ignoring paging, is in the X-Total-Count header.

Internal error text is never returned: each item only says whether it was degraded. A failed delivery shows its reason with any URL replaced by <url>, because such messages can contain the target's address and tokens.

Statistics

GET /stats?since=2026-10-05T00:00:00Z limits every counter to requests created at or after that time. A time without a zone is read as UTC.

reviewer_agreement is the share of human-reviewed requests where the reviewer kept the model's suggestion. It is not the model's accuracy: only the requests the model was unsure about are sent to a person, so the sample is biased towards hard cases. It is null until the first review is resolved.

GET /stats/timeseries?bucket=hour|day&since=...&until=... returns one point per bucket with the request count (split into accepted and human review), cost and average latency. Buckets are aligned to the hour or day in UTC, so the first one starts at the beginning of the bucket that contains since. since is included and until is excluded. Buckets without requests are returned with zeros, so a chart gets a continuous axis. The defaults are the last 24 hours (hourly) and the last 30 days (daily); a range of more than 1,000 buckets is rejected with 422.

Typed schema

Every endpoint declares its response model, and the operation ids are the function names (route_request, list_requests, ...). The OpenAPI schema at /openapi.json can be fed to a code generator to get typed clients; tests pin the field names and types so a change to them is noticed.

Every request is stored in a SQLite file (text, result, cost, latency, and the reviewer's decision). Settings are read from the environment: ROUTEIQ_CONFIG (default configs/clinc150.yaml) and ROUTEIQ_DB (default routeiq.db in the repository root). Interactive documentation is served at /docs.

Dashboard

The dashboard is a single-page app in dashboard/ (React, TypeScript, Vite). The API serves the built files under /dashboard/, and / redirects there. The Docker image builds it in a Node stage, so docker compose up --build is enough. Without Docker, build it once and start the API:

cd dashboard && npm install && npm run build && cd ..
uvicorn routeiq.api:app

Open http://localhost:8000/dashboard/. Node 20.19 or later is needed to build. ROUTEIQ_DASHBOARD_DIR points the API at another folder of built files.

Screen	What it shows
Review queue	Requests waiting for a person, oldest first, with one button per label. The model's suggestion is marked but not preselected. The list refreshes every 10 seconds. If somebody else already decided a request, you get a notice instead of an overwritten decision
Overview	Totals for the last 24 hours, 7 days or all time: requests, cost, p95 latency, requests where a tier failed, how requests ended, and the delivery status with a button to resend failed deliveries
Charts	Requests (accepted or sent to a person), cumulative cost and average latency per hour or day, and a table of the same numbers below the charts
Evaluation	Accuracy of the baseline, the LLM, the cascade and the cascade with human review, each with a 95% interval, plus macro-F1, cost and latency, how the cascade handled the examples, and calibration. It is measured offline on labeled examples (the recorded predictions of the test split) and the screen says it is not live traffic. The last setup assumes a person is always right
History	Every request, newest first, with filters (outcome, review, delivery, tier, tier failure), text search and paging. Filters and page are kept in the address, so a view can be bookmarked. A row opens to show the full text and the delivery problem
Try it	Send a text through the cascade and see its label, confidence, tier, cost and latency. The text is stored like any other request and may call the paid model. Ctrl or Cmd + Enter sends. The example sentences of the config are buttons that fill the box without sending it

Times are shown in UTC. The overview shows reviewer agreement next to a note that it is not the model's accuracy (see Statistics).

The dashboard has no login, like the API. Publish it on localhost only, or put both behind a gateway.

Demo data

A new installation has no requests, so every screen is empty (the short version is in the Quick start). Two scripts write a separate database, which you open by starting the API on it with ROUTEIQ_DB:

• scripts/replay_demo.py replays the 300 examples of the CLINC150 test set through the real cascade, using the answers the baseline and the LLM gave when they were evaluated (demo/clinc150_test_predictions.csv, made with scripts/export_predictions.py from the saved evaluation). The decisions, confidences, costs and latencies are real, and a test checks that the replay gives the numbers of the results table. What is not real: the times (spread over the last day, so the 24-hour views are full) and the reviewers' decisions (simulated as a person who always picks the true label, the assumption of the "cascade + human review" row). No model is called.
• scripts/seed_demo.py writes synthetic requests from a few templates, with outcomes drawn at random from a fixed seed. They are not results of the models. It exists to produce states that are hard to cause on purpose, such as failed and stuck deliveries and requests where a tier failed, so that those parts of the dashboard can be tried.

Both refuse to write into a database that already has requests unless --add is given, so they cannot bury real data under made-up data. The recorded texts come from CLINC150 (CC BY 3.0), see demo/README.md.

For development, run the API and npm run dev in dashboard/ (http://localhost:5173/dashboard/); the dev server forwards the API paths, so no CORS setup is needed. After an API change, regenerate the schema and the types:

python scripts/export_openapi.py
cd dashboard && npm run api:types

A Python test fails when the committed dashboard/openapi.json is out of date. More in dashboard/README.md.

Integrations

Final decisions are sent to a target system: requests the cascade accepted are sent right away, and requests that went to human review are sent once a person resolves them (with the person's label). Delivery happens in the background after the API responds, so a slow or unavailable target never delays the caller. Each request records its delivery status (pending, sent or failed, with the error) in the database, and POST /deliveries/retry resends the failed ones.

Delivery is at-least-once: a record can occasionally arrive twice, so the receiving side should deduplicate on request_id.

The target is chosen in the config:

target:
  type: mock_erp          # webhook | mock_erp | jsonl | none (or omit the section)
  url: http://localhost:8000/mock-erp/api/tickets

Type	Settings	What it does
jsonl	path	Appends one JSON object per line to a file
webhook	url, optional secret_env	POSTs the record as JSON. With a secret, the body is signed: header X-RouteIQ-Signature: sha256=<HMAC-SHA256 of the body>. Retries network errors, 429 and 5xx
mock_erp	url	Sends the record as a ticket to the built-in mock ERP

The record sent to webhook and jsonl looks like this:

{"request_id": 3, "text": "...", "label": "report_lost_card", "confidence": 0.79,
 "tier": "baseline", "source": "cascade"}

For human-resolved requests label is the reviewer's choice, tier is human and source is human_review.

The mock ERP runs inside the same application (only when the config targets it) under /mock-erp/api/tickets. It accepts POST, lists with $top and $skip, and returns @odata.count. Posting the same ExternalID twice returns the existing ticket instead of creating a second one, so redeliveries are harmless. It keeps its data in memory. It imitates the shape of an enterprise API and does not connect to any real ERP product.

Configuration

# configs/clinc150.yaml
domain: clinc150
data_source: public   # CLINC150 (clinc_oos): subset of intents + out-of-scope

labels:               # intents chosen from the dataset, plus the out-of-scope class
  - report_lost_card
  - damaged_card
  - card_declined
  - report_fraud
  - freeze_account
  - out_of_scope

task: bank customer support messages    # what is being classified, used in the LLM prompt

label_descriptions:                     # one line per label, used in the LLM prompt
  report_lost_card: the customer lost their card or cannot find it.
  damaged_card: the card is physically damaged (cracked, burned, bent).
  card_declined: a payment was refused or the card was not accepted.
  report_fraud: unauthorized or suspicious charges on the account.
  freeze_account: the customer wants the account blocked or locked.
  out_of_scope: anything else, including messages unrelated to banking.

examples:                               # sentences offered as buttons on the Try it screen
  - I lost my debit card yesterday
  - my card got declined at the store
  - there is a charge on my statement that I did not make
  - please freeze my account
  - my card is cracked and will not read
  - what is the weather like today

tiers:
  - name: baseline
    model: sklearn_tfidf_logreg
    accept_threshold: 0.7   # chosen on the validation set; the baseline's scores are
                            # underconfident, so thresholds are rank cut-offs, not probabilities

  - name: llm
    model: openrouter
    model_id: mistralai/mistral-small-3.2-24b-instruct
    price_in_per_m: 0.09    # USD per 1M input tokens, fallback only: the real cost
    price_out_per_m: 0.30   # USD per 1M output tokens, is read from the API response
    accept_threshold: 0.90

target:
  type: mock_erp
  url: http://localhost:8000/mock-erp/api/tickets

data_source says where the labeled data comes from: public, synthetic or private. For synthetic the dashboard shows a note on every screen, because accuracy on generated text says nothing about real data. examples (at most 12 short sentences) are offered as buttons on the Try it screen. Both are optional, and the config is validated at startup with mistakes reported at their location.

Adding a model

Implement the interface and register it:

# routeiq/models/base.py
from dataclasses import dataclass
from typing import Protocol

@dataclass
class Prediction:
    label: str
    confidence: float
    cost_usd: float = 0.0

class Classifier(Protocol):
    def classify(self, text: str, labels: list[str]) -> Prediction: ...

Included adapters:

Adapter	Purpose	Notes
sklearn_tfidf_logreg	Cheap, fast baseline	Trained locally
openrouter	LLM tier with structured output, any model available on OpenRouter	Needs OPENROUTER_API_KEY

Datasets

Domain	Source	Notes
clinc150	CLINC150 (clinc_oos on Hugging Face), public intent-classification dataset	Main benchmark, real human-written text, includes out-of-scope queries
ev_after_sales	Synthetic text generated from templates by routeiq/synthetic.py	An example of a second domain: it needs only a spec and a config. The templates of each label are divided between train, validation and test, so a test pattern was never seen in training. Accuracy on it says nothing about real data

No real company data is included.

Main benchmark: CLINC150

CLINC150 contains user queries covering 150 intents across 10 domains, plus an out-of-scope label for queries that match none of them. RouteIQ uses a subset: a handful of intents chosen from the dataset (listed in configs/clinc150.yaml) plus the out-of-scope class.

The out-of-scope class is the reason for choosing this dataset. A routing system must also recognise requests it should not route automatically, so these examples exercise the escalation and human-review path directly instead of only measuring easy, in-scope cases.

data/prepare_clinc.py downloads the dataset, keeps the chosen intents, maps the out-of-scope examples to out_of_scope, and writes train, validation and test files under data/.

Chosen scenario: bank support requests. The five intents are report_lost_card, damaged_card, card_declined, report_fraud and freeze_account. They are semantically close (for example lost vs. damaged card), so the baseline makes real mistakes and the cascade has something to fix. The other 145 CLINC150 intents are dropped; only the original oos examples become out_of_scope.

Split	Per intent	out_of_scope	Total
train	100	250	750
validation	20	100	200
test	30	150	300

The raw test split has 1,000 out-of-scope examples against 30 per intent, which would make accuracy mostly a measure of the out-of-scope class. The script therefore samples 150 of them (random_state=42). Train and validation are left as they are.

Baseline (validation only). TF-IDF (unigrams and bigrams) with logistic regression reaches accuracy 0.94 and macro-F1 0.915. The weakest classes are damaged_card and report_fraud. Its confidence is lower than its accuracy suggests: the mean confidence is 0.68 on correct predictions and 0.39 on wrong ones. Test results are reported only in the evaluation section below.

CLINC150 is released under CC BY 3.0 (see the dataset card on Hugging Face). If you use it, cite: Larson et al., "An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction", EMNLP-IJCNLP 2019 (https://aclanthology.org/D19-1131).

Adding a domain

The cascade, the evaluation and the integrations do not depend on the CLINC150 data. The bank-support scenario is the one worked example; a new domain takes these steps:

1. Put labeled train.csv, val.csv and test.csv files (columns text,label) in data/<domain>/, where <domain> is the domain of the config. data/prepare_clinc.py shows how it is done for CLINC150. Without data of your own, generate synthetic data: write the sentence templates of each label in data/synthetic/<domain>.yaml and run python -m routeiq.synthetic --config configs/<domain>.yaml (see data/synthetic/ev_after_sales.yaml).
2. Create a config under configs/ with the labels, thresholds and a target. The config is validated at startup and mistakes are reported with their location.
3. Describe what is being classified (task) and each label (label_descriptions) in the config. They are placed in the system prompt of the LLM tier. Every label needs a description, and the descriptions matter: they decide how the LLM treats the confusable labels and the out-of-scope class, so check them on the validation split, not the test split.
4. Run python -m routeiq.train and python -m routeiq.evaluate with the new config.

configs/ev_after_sales.yaml is a second domain built this way: the after-sales service of an electric-vehicle maker, on synthetic data. Its thresholds were copied from the CLINC150 config and are not tuned. The generator divides the templates of every label between train, validation and test, so a baseline does not get credit for patterns it saw in training; the score it reaches still says nothing about real text, which is why this README gives no results table for it. In Docker only the CLINC150 baseline is trained; other domains run locally.

Evaluation

python -m routeiq.evaluate --config configs/clinc150.yaml --split test

Tier predictions are cached under reports/, so repeated runs do not call the LLM again. Useful flags: --split val|test (default val), --refresh to recompute the cached predictions, and --sweep to also print the full threshold grid.

The report covers:

• Accuracy and macro-F1 per tier and for the full cascade
• Cost per 1,000 requests
• Latency (p50, p95)
• Escalation rate and human-review rate
• Accuracy vs. cost across threshold settings (--sweep prints the grid)
• Calibration: when the system says 90% confident, is it right about 90% of the time?

Results: CLINC150

Measured on the held-out test split (300 examples: 30 per intent plus 150 out-of-scope). The thresholds (baseline 0.7, LLM 0.9) were chosen on the validation split and not changed after seeing the test results. The LLM tier is mistralai/mistral-small-3.2-24b-instruct via OpenRouter.

Setup	Accuracy	Macro-F1	Cost / 1k req	p50 latency	p95 latency
Baseline only	0.917	0.899	$0	0.2 ms	0.2 ms
LLM only	0.983	0.976	$0.0118	600 ms	2.17 s
Cascade (baseline then LLM)	0.983	0.976	$0.0064	559 ms	1.93 s
Cascade + human review	0.993	0.989	$0.0064	559 ms	1.93 s

In the cascade, 139 requests (46%) were accepted by the baseline (all correct), 161 (54%) were escalated to the LLM, and 6 (2%) went to human review. The "+ human review" row assumes the reviewer is always right and does not count human time in cost or latency. With 300 examples, a 95% confidence interval on accuracy is roughly ±1.5 points, so the difference between "LLM only" and "Cascade" is not meaningful.

Key findings:

• The LLM is clearly more accurate than the baseline (0.983 vs 0.917 accuracy, 0.976 vs 0.899 macro-F1).
• The cascade matches the LLM's accuracy at about 54% of its cost. Because only 46% of requests skip the LLM, the median latency barely improves (559 ms vs 600 ms). A lower baseline threshold cuts cost and latency further, at the price of accuracy on the validation set.
• The baseline's confidence is poorly calibrated (ECE 0.27, it underestimates itself) but ranks well: every request it accepted above 0.7 was correct. The LLM's confidence is well calibrated overall (ECE 0.012).
• Human review on 2% of requests removed 3 of the cascade's 5 errors. The remaining 2 are lost-or-stolen-card messages that the LLM labeled report_fraud with confidence above 0.99, a genuinely ambiguous boundary that a confidence threshold cannot catch.
• With this inexpensive LLM the absolute saving is small (about $5 per million requests). The cascade pays off more with a pricier LLM tier or when most requests can be answered without waiting for one.

Project structure

routeiq/
├── configs/                 # YAML use-case definitions (one per domain)
├── dashboard/               # React UI (see dashboard/README.md)
├── data/
│   ├── prepare_clinc.py     # builds the CLINC150 train, validation and test files in data/clinc150/
│   └── synthetic/           # specs of generated domains: the sentence templates of every label
├── demo/                    # recorded predictions of the CLINC150 test set (see demo/README.md)
├── docs/images/             # screenshots of the dashboard
├── routeiq/
│   ├── api.py               # FastAPI app and its endpoints
│   ├── cascade.py           # threshold-based routing logic
│   ├── config.py            # config loading and validation
│   ├── dashboard.py         # serves the built UI under /dashboard/
│   ├── delivery.py          # sends decisions to the target and tracks the status
│   ├── evaluate.py          # metrics, calibration and threshold sweep
│   ├── evaluation.py        # the report behind GET /evaluation, from the recorded predictions
│   ├── redact.py            # masks URLs in error messages before they are shown
│   ├── replay.py            # replays recorded model answers through the cascade into a database
│   ├── schemas.py           # request and response models of the API
│   ├── store.py             # SQLite store: requests, review queue, deliveries
│   ├── synthetic.py         # generates synthetic labeled data from a spec
│   ├── timeutil.py          # UTC and time bucket helpers
│   ├── train.py             # baseline training
│   ├── views.py             # builds the public views (config, request items)
│   ├── models/              # classifier adapters and registry
│   └── integrations/        # JSONL export, webhook, mock ERP
├── scripts/                 # export_openapi.py (schema for the UI types); export_predictions.py, replay_demo.py and seed_demo.py (demo data)
├── tests/
├── Dockerfile
├── docker-compose.yml
├── LICENSE
├── pyproject.toml
└── README.md

Testing

pytest

Tests cover the cascade decisions at threshold boundaries (and that the threshold simulation in evaluate.py, which produces the results table, makes the same decisions as the live cascade), the OpenRouter adapter (response parsing, retries), the API routes and their OpenAPI contract, the review queue, request history and statistics, delivery tracking, the integrations, the SQLite store with its schema migration and indexes, and config validation. LLM and network calls are mocked in tests.

Several tests tie this README to the code: the replay of the recorded predictions and the report behind the Evaluation screen give the figures of the results table, its calibration values and the 139 requests the baseline accepted, and the generator of the synthetic domain is checked to give the same data every time, to give every split templates of its own, and never to repeat a text.

The dashboard has its own tests and type check (cd dashboard && npm test && npm run typecheck). They run in Vitest with a mocked API, so no server is needed.

Limitations

• Confidence scores from different model types are not directly comparable. Calibration is evaluated, and thresholds should be tuned per domain.
• Benchmarks on public data do not guarantee the same results on a real company's data.
• The prompt wording was written once and not tuned. Different label descriptions can change the LLM's accuracy, especially on the out-of-scope class, and data/prepare_clinc.py is specific to CLINC150.
• The mock ERP only imitates the shape of an enterprise API. A production integration needs authentication, retries and error handling for the real system.
• LLM cost figures depend on current provider pricing and should be rechecked.
• The CLINC150 setup uses a subset of intents plus out-of-scope. Results do not transfer to the full 150-intent task.
• CLINC150 queries are short, single-turn utterances. Real business emails are longer and messier.
• Many card_declined examples are one template sentence with small variations, so this intent is easy to learn and can make results look better than they would on varied text.
• The validation set has only 20 examples per intent, so one mistake moves a class's recall by 5 points. Small differences between classes may be noise.
• The API has no authentication. Anyone who can reach /review, /requests or /stats can read the stored texts, and anyone who can reach /review can resolve items. Put it behind a gateway or add authentication before exposing it.
• GET /requests pages with offset, so while a client pages through the newest-first list, new requests shift the pages and a row can show up twice or be skipped. The text search scans the table; it has no full-text index.
• Reviewer agreement measures how often a person keeps the model's suggestion on requests the model was unsure about. It says little about accuracy on the requests the cascade accepted.
• Request texts are stored unencrypted in a local SQLite file, with no retention limit or anonymization. A real deployment needs a retention policy and access control.
• Requests that reach the LLM wait for it. With the configured budget (10 s timeout, 2 attempts) a request can take about 21 s before it falls back to human review.
• POST /deliveries/retry takes the oldest failed deliveries first, so a record that keeps failing can hold back newer ones when the limit is smaller than the backlog.
• The mock ERP keeps its tickets in memory and the application is designed for a single process; neither is meant for production use.
• The dashboard does not record who made a decision: a review is stored with the label and the time only, so there is no audit trail per person.
• The history screen does not refresh by itself, because new requests would move the rows while somebody reads. It has a Refresh button; the review queue and the overview poll.
• The thresholds of the electric-vehicle example domain were copied from CLINC150 and not tuned, and its data is synthetic.
• There is no hosted demo: the project is meant to be run locally (see the Quick start). The screenshots above were made from a local run.

Roadmap

• Baseline training and evaluation
• OpenRouter adapter with structured output
• Cascade routing and threshold sweep
• FastAPI service and human-review queue
• Mock ERP integration
• Docker setup
• Automated test that the threshold simulation in evaluate.py matches the live cascade
• Example domain with synthetic data, labeled as synthetic: electric-vehicle after-sales
• More example domains (supplier communication, internal requests)
• Dashboard (React + TypeScript): review queue, overview, charts, request history and a form to try a text
• Dashboard: offline evaluation screen (accuracy needs labeled ground truth, which production requests do not have, so it is measured on labeled examples and not on live traffic)
• Threshold auto-tuning from review feedback
• Drift monitoring

License

MIT, see LICENSE. The CLINC150 data keeps its own license (CC BY 3.0).