jev-measured

Measured cost, latency and raw output from the live Jev API (TypeSafe AI's System One decision model) across twelve realistic use cases. Reproducible: bring your own key and re-run it.

Published because most numbers circulating about Jev are either the vendor's own or arithmetic from the rate card. These are what the API actually returned.

• Measured: 2026-09-18 (cost/latency/shape; four more use cases on 2026-10-11), 2026-09-20 (accuracy)
• Model: typesafe/jev-1.13 via OpenRouter; §8 adds twelve other decision models (2026-10-08)
• Raw data: data/measured.json
• Cost of the whole run: under one cent

Quick answers

Each answer links to the section with the data behind it.

Question	Answer (measured)
How much does one Jev decision cost?	$0.000015–0.000030 across twelve real use cases (364–724 input tokens at $0.042 per 1M; output is not billed). §1
How fast is Jev?	458–563 ms raw from a laptop, of which ~199 ms is the network floor; 184 ms median from a US-East server through OpenRouter. §2, §8
Is Jev more accurate than chat models?	It ties, it does not win: 27/27 labelled tickets, the same as mistral-small-3.2 (1.4× the price, 4× slower); gemini-2.5-flash-lite and gpt-5-nano scored 26/27. Only Jev also returned a confidence (0.979 clear, 0.841 ambiguous). §7
Which decision model is most accurate?	On 633 items: d1 96.5% and Decider V1.1 96.0% on simple choices; Clef Flash 92.0% and Clef 89.0% with 59–77 options; Jev 94.0% / 79.0%; GPT-6 Luna Decisions 94.1% / 79.8%. §8
What does Score return?	The probability-weighted mean of the rubric indices (0..n−1), not 0–1. Noul has no confidence field. §3
Why does typesafe/jev fail on OpenRouter chat completions?	Jev is a decisions model: use POST /api/alpha/decisions, not /chat/completions. Gateway gotcha
Does asking more questions cost more?	Only their own text: each question is billed as input tokens, but the state is billed once per request, and five questions return as fast as one. §6

---

Findings

1. A real decision costs ~20× less than the figure in circulation

The number you see quoted for Jev is ~$0.0004 per decision. Across twelve real use cases, the API reported (the four marked * were added on 2026-10-11):

Use case	Input tokens	Cost reported by the API
rag-reranking	364	$0.0000153
llm-model-routing	369	$0.0000155
agent-tool-selection	403	$0.0000169
content-moderation	420	$0.0000176
lead-scoring	472	$0.0000198
agent-output-guardrails	474	$0.0000199
phishing-detection	533	$0.0000224
support-ticket-triage	539	$0.0000226
news-filtering*	540	$0.0000227
csv-data-validation*	576	$0.0000242
product-categorization*	706	$0.0000297
invoice-classification*	724	$0.0000304

$0.0000153 – $0.0000304, i.e. 13× to 26× below $0.0004, and about 20× at the median.

This is not a discount and not a vendor error. Pricing is $0.042 per 1M input tokens with no output tokens billed, and cost therefore scales with one thing only: your state plus your questions. A real support ticket is a few hundred tokens, not a few thousand. The $0.0004 figure describes a much larger state than most workloads send.

The practical consequence: the only number that matters for your budget is your own average state length. Multiply it by $0.042/1M. Nothing else moves the bill.

And since questions share one state and run in parallel, ask everything in one call — three separate calls pay for the state three times.

2. Quote latency twice, or don't quote it

Raw wall-clock medians here are 458–563 ms. That number is close to meaningless on its own: measured from this machine, the TCP handshake alone to the host had a median of 198.8 ms (7 samples, 191.2–266.4 ms).

	median
Raw round trip	458–563 ms
Measured network floor	198.8 ms
Remainder	259–364 ms

The four cases added on 2026-10-11 ran at 595–634 ms raw against a floor of 259.9 ms that day: a remainder of 335–374 ms, close to the first run's band once the floor is taken out.

So this run says little about Jev's speed and a lot about the distance between this machine and the host. bench/net_floor.py measures the floor so the two can be separated. Anyone publishing a Jev latency number without doing this is publishing their own geography.

(Method borrowed from anisselbd/jev-phishing-bench, which measures a floor before quoting anything.)

3. Score is not 0–1, and Noul has no confidence field

Two response shapes that are easy to get wrong, both confirmed across every call in this dataset.

Score returns the probability-weighted mean of the rubric indices, 0..n-1 — not a normalised float. From support-ticket-triage:

{
  "type": "score",
  "score": 2.68,
  "confidence": 0.68,
  "legend": {"0": "Low", "1": "Normal", "2": "High", "3": "Urgent"},
  "probabilities": {"0": 0, "1": 0, "2": 0.32, "3": 0.68}
}

2×0.32 + 3×0.68 = 2.68. A four-level rubric can return 2.68, meaning "between High and Urgent, leaning Urgent". Round it for the band; the fraction tells you which way the mass leans.

Recomputing the score from the returned probabilities reproduces it to within ±0.02 — the probabilities are rounded to two decimals in the response, while score is computed from the full-precision distribution. Don't treat a small mismatch as a bug.

Note also that probabilities is keyed "0", "1", "2"… — by index, never by label. Use legend to map back.

Noul ships no confidence field. All 13 Noul answers in this dataset:

{"type": "noul", "noul": 0.99}

That is the entire object. If you need a confidence gate on a Noul, distance from 0.5 is what you have.

---

4. We measured Jev against chat models, published a wrong conclusion, and corrected it

bench/headtohead.py puts the same fixtures to Jev and to three chat models through one gateway. Across eight fixtures, five runs each:

Model	Median	Mean cost	vs Jev
typesafe/jev-1.13	352 ms	$0.0000188	1.0x
mistral-small-3.2-24b	1,343 ms	$0.0000255	1.4x
gemini-2.5-flash-lite	877 ms	$0.0000323	1.7x
gpt-5-nano	7,504 ms	$0.0003434	18.3x

Two things worth taking from that before the correction. A cheap chat model is 1.4–1.7x the cost, not two orders of magnitude — on a single-question fixture mistral-small came out cheaper than Jev. And the large multiple belongs to a reasoning model spending output tokens on thinking, which is a different claim from "decision models are cheaper".

The correction. We first ran the chat models with a plain prompt and found they returned the wrong type — asked for a probability between 0 and 1, gemini-2.5-flash-lite answered true on three runs of five, gpt-5-nano returned probabilities as strings, mistral dropped a field. We published that as the argument for a typed model.

It was an artifact of how we asked. Every major chat API supports structured output and we had not switched it on. Same prompt, same fixtures, one extra field in the request body:

                        plain   json_schema
  gemini-2.5-flash-lite   2/5          5/5
  gpt-5-nano              4/5          5/5
  mistral-small-3.2       5/5          5/5

response_format: {"type": "json_schema", "strict": true} removes every violation we found, on every fixture. So type reliability is not a reason to choose Jev — it is one line in your request body. Check both halves:

python3 -m bench.headtohead --repeats 5            # plain prompt
python3 -m bench.headtohead --repeats 5 --schema   # the way you would actually do it
python3 -m bench.headtohead --repeats 5 --frontier # adds GPT-5.6 Sol/Terra and Opus 5

Measuring a model without the feature built for the job tests your prompt, not the model. The wrong version is left in the history rather than deleted, because a benchmark that only ever confirms its author is not worth reading.

5. The gateway is not free, and its tail is worse than its median

Same state and questions, alternating between routes twelve times so both see the same network at the same minute:

Route	p50	p90	Reports cost
api.typesafe.ai	313 ms	423 ms	no
OpenRouter /api/alpha/decisions	734 ms	1,739 ms	yes

A first attempt ran the routes in sequence rather than interleaved and put the gap at 497 ms; interleaving corrected it to 421 ms, and benchmark runs a few hours earlier had OpenRouter at 339–453 ms. The gateway tax is not one number — measure your own route at your own hour.

Two undocumented differences, both worth knowing before you pick:

• The model id differs. First-party answers as jev-1.13.0, OpenRouter as typesafe/jev-1.13-20260917. GET /v1/models lists only jev-latest and jev-preview, yet jev-1.13.0 is accepted — while jev-1.13 without the patch number is rejected as an unknown model.
• The first-party API reports input_tokens and output_tokens but no cost. That field is OpenRouter's addition.

6. Five questions in one request take the same time as one

From x-envoy-upstream-service-time, the server's own clock, rather than by subtracting an estimated network floor:

Questions in one request	Server time
1	70 ms
3	72 ms
4	81 ms
5	74 ms

"Questions are evaluated in parallel" is a vendor claim everywhere else. This is it read off the server's own header. Same practical advice the cost numbers give: ask everything you want to know in one call.

7. Accuracy: Jev does not win, it ties

Everything above is cost, latency and output shape. None of it says whether an answer is right. bench/labelled.py adds 27 support tickets with a correct label, plus 6 deliberately ambiguous ones carrying no label at all. Every model runs the whole set three times.

Model	Correct	Per pass	Confidence, clear	Confidence, ambiguous
typesafe/jev-1.13	27/27	27 · 27 · 27	0.979	0.841
mistral-small-3.2-24b	27/27	27 · 27 · 27	—	—
gemini-2.5-flash-lite	26/27	26 · 26 · 26	—	—
gpt-5-nano	26/27	26 · 26 · 26	—	—

"Jev is more accurate" is not supported by this. It ties mistral-small, a general chat model that costs 1.4x as much and answers four times slower — but answers just as correctly, on every pass. Both misses across every model were the same ticket: a 419 advance-fee scam, filed as sales by Gemini and billing by GPT-5 nano rather than spam, on all three passes. That is a reproducible disagreement about one ticket, not a general accuracy gap.

The confidence columns are the part only Jev can produce. Certainty falls from 0.979 on the clear set to 0.841 on the ambiguous one — calibrated in the right direction, though by a smaller margin than you might want if you intend to route on a threshold. The chat models return no confidence field at all.

7b. We nearly published a second artifact

The first version of this ran one pass and put gemini at 26/27; re-running it gave 25/27 on the same tickets. Widening to three passes at default sampling produced this, which looked like a finding:

Model	Per pass, default sampling	With temperature: 0
gemini-2.5-flash-lite	26 · 25 · 26	26 · 26 · 26
gpt-5-nano	24 · 26 · 26	26 · 26 · 26
mistral-small-3.2	27 · 27 · 27	27 · 27 · 27
typesafe/jev-1.13	27 · 27 · 27	27 · 27 · 27

An answer that changes between identical calls is a serious problem for anything you route on, and "chat models are inconsistent, Jev is not" was a finished paragraph before we checked it. temperature: 0 removes it entirely.

That is the second time in this repo that the same mistake produced a flattering result — see finding 4 for the first. Both times the fix was a field in the request body. Both runs are kept: data/accuracy.json is the published one, data/accuracy-sampling-default.json is the comparison.

python3 -m bench.accuracy                     # 3 passes, temperature 0
python3 -m bench.accuracy --sampling-default  # send no temperature at all
python3 -m bench.accuracy --only jev --repeats 5

Limits. Twenty-seven tickets written by us, on one task, in English, three passes each. Enough to retire the claim that Jev is more accurate; not enough to establish that anything else is. For a real accuracy study on independent data, see anisselbd/jev-phishing-bench (2,000 labelled emails).

---

8. Thirteen decision models on the same 633 items (2026-10-08)

OpenAI put its Decisions API (GPT-6 Luna Decisions) into public beta on 2026-10-06, and by 2026-10-08 OpenRouter's Decisions API served thirteen decision models from ten makers. All thirteen, one question per request, on the same items:

• Simple choices (column average): AG News (4 labels, 100 items) and 27 labelled support tickets (5 queues).
• Many options: MASSIVE intents (59 labels) and Banking77 (77 labels), 100 items each.
• Yes/no: WikiToxic and Amazon polarity, 100 items each, asked as a statement.
• Also in the raw data: an emotion set (6 labels) and 6 deliberately ambiguous tickets with no gold label.
• Items are a fixed-seed sample (20261008) of the BTZSC test splits; the tickets come from bench/labelled.py.

Model	Maker	Simple choices	59–77 options	Yes/no	Wrong at ≥90% conf.	p50 latency	$ per 1k decisions
d1 (liquid/d1)	Liquid AI	96.5%	86.0%	96.5%	3.2%	322 ms	$0.0078
Decider V1.1 27B (perplexity/pplx-decider-v1.1-27b)	Perplexity	96.0%	85.0%	95.5%	4.6%	269 ms	$0.0072
Mercury Decide (inception/mercury-decide:free)	Inception	95.6%	85.0%	94.0%	5.4%	untimed	free tier
Clef (cloudflare/clef)	Cloudflare	95.0%	89.0%	97.0%	2.3%	313 ms	$0.1541
Kev 4B (jaredpalmer/kev-4b)	Jared Palmer	95.0%	78.5%	91.0%	2.6%	359 ms	$0.0115
Solar Decide (upstage/solar-decide)	Upstage	94.6%	refused	93.0%	13.7%	559 ms	$0.0216
GPT-6 Luna Decisions (openai/gpt-6-luna-decisions)	OpenAI	94.1%	79.8%	93.5%	9.2%	138 ms	$0.0401
Solar Decide Flash (upstage/solar-decide-flash)	Upstage	94.1%	refused	91.5%	9.5%	597 ms	$0.0216
Jev 1.13 (typesafe/jev-1.13)	TypeSafe	94.0%	79.0%	95.0%	11.5%	184 ms	$0.0304
Clef Flash (cloudflare/clef-flash)	Cloudflare	92.6%	92.0%	97.0%	4.2%	369 ms	$0.0578
Tev1 4B Experimental (togethercomputer/tev1-4b-experimental)	Together AI	90.4%	refused	93.5%	5.1%	136 ms	$0.0072
Span-01 (respan/span-01)	Respan	yes/no only	—	93.5%	—	128 ms	$0.0029
Span-01 Lite (respan/span-01-lite:free)	Respan	yes/no only	—	93.5%	—	untimed	free tier

What the table says:

• On a simple gate the models barely differ (90–96.5%); pick on latency and price there.
• With dozens of options they do. Solar Decide takes at most 26 options and Tev1 at most 20, so they refused the 59- and 77-option sets outright; Jev and GPT-6 Luna Decisions fell to ~79–80%, while the two Clef models held 89–92%.
• Read the confident-and-wrong column before the accuracy column if a threshold triggers an action: Jev was wrong on 11.5% of the answers it gave ≥90% confidence, d1 on 3.2%, Clef on 2.3%.
• Images reach the image models only as chat-style image_url parts. OpenAI-style input_image parts are read as text, without an error.
• Latency: 20 sequential calls after one warm-up, from a server in Washington, D.C. (Vercel iad1), all through OpenRouter's Decisions API. Free-tier models were rate-limited too hard to time.

Raw answers: data/decision-models/run.jsonl (7,363 rows: model, item, gold label, answer, confidence, tokens, cost — no item text). Summary: data/decision-models/summary.json. Scripts: bench/decision-models/. Per-model profiles and the same tables with charts: jev-agent.com/decisions-api and jev-agent.com/decision-models.

9. Laya, the open-weight model, on the same items (2026-10-09)

Laya (Convai Innovations, Apache 2.0, first published 2026-09-18) is the most-starred open decision model and speaks the same /v1/systemone request shape. Its own benchmark compares against Jev's published figures, since its authors had no TypeSafe access, so we ran it on the §8 items. Laya 0.4.1, English checkpoint (convaiinnovations/laya, ModernBERT-large, 421M), zero-shot, the same questions, one per call, on an Apple M2 (PyTorch MPS). The rebuilt item sample matched all 627 gold labels of the §8 run.

	Laya	Jev 1.13
AG News (4 labels)	91.0%	88.0%
Emotion (6)	60.0%	63.0%
Support tickets (5 queues)	74.1%	100%
MASSIVE (59)	45.0%	76.0%
Banking77 (77)	44.0%	82.0%
WikiToxic, yes/no	91.0%	92.0%
Amazon polarity, yes/no	67.0%	98.0%
Wrong at ≥90% confidence (choice)	26.8%	11.5%
Ticket confidence, clear / ambiguous	0.53 / 0.45	0.98 / 0.82
Median latency	163 ms on an M2 laptop	184 ms via OpenRouter from iad1

• Laya's own numbers agree with ours: it reports 0.950 on AG News and 0.425 on Banking77. Its README explains the many-option drop (all options share a 192-token budget) and advises keeping choices under ~20.
• Confidence is converted. Laya's confidence is 1 − normalised entropy; Jev's is (n·p_max − 1)/(n − 1). data/laya/run.jsonl keeps Laya's raw entropy_confidence and pmax, and confidence is recomputed on Jev's definition so the two columns compare like with like.
• The yes/no wording was not the problem. Reviews scored 67% as asked, 71% without the true/false criteria and 56% as a question in Laya's style; toxicity 91%, 88%, 88% (data/laya/variants.json).
• Batched on the M2: 13.0, 16.5 and 17.9 decisions/s at batch sizes 1, 16 and 32 (AG News).

Scripts: bench/laya/ (run_laya.py reads the same btzsc-sample.json and tickets.json as bench/decision-models/). Write-up: jev-agent.com/jev-vs-laya.

10. Chat models on the same items (2026-10-11/12)

Same 627 labelled items as §8 (the six ambiguous tickets have no label and are left out), same instructions and options, through OpenRouter's chat endpoint at temperature 0 with reasoning off. The model is told to reply with one option; a reply naming no option counts as wrong. Grouped as in §8:

Model	News + tickets	59–77 options	Yes/no	Emotions	$ per 1,000 items
Claude Haiku 5.5	95.5%	78.5%	96.0%	67%	$0.0336
Qwen3.8 Flash	95.0%	77.0%	95.5%	63%	$0.0231
DeepSeek V4.1 Flash	93.6%	80.5%	95.5%	65%	$0.0220
GPT-6 Luna (chat)	92.5%	78.5%	95.0%	60%	$0.0237
Nemotron 3 Ultra (free)	89.4%	78.0%	92.5%	59%	free

• Chat models tie decision models on a few options and trail them on many. With 59 and 77 intents the best chat model here scored 80.5%; Clef Flash scored 92.0% and d1 86.0% (§8).
• Same family, two doors: GPT-6 Luna through chat scored 92.5% / 78.5%; GPT-6 Luna Decisions 94.1% / 79.8% (§8). The chat route billed $0.024 per 1,000 items against $0.040, and returns no probabilities.
• Most many-option misses are neighbouring labels: order_physical_card answered get_physical_card, calendar_set answered alarm_set, transfer_not_received_by_recipient answered transfer_timing.
• Nemotron 3 Ultra (free tier) answered 19 items with no option at all; they count as wrong. Gemma 4 31B's free tier was rate-limited too hard to finish.
• Not measured: GPT-6.1 Sol, Claude Sonnet 5.5 and Gemini 3.8 Flash refuse to run with reasoning off; a 21-item pilot at minimal effort is in chat-run.jsonl but the full set was not run.

Scripts: bench/decision-models/chat.mjs and chat-summarize.mjs, run from a directory holding the btzsc-sample.json and tickets.json that prep.mjs and bench/labelled.py produce (the regenerated sample matched all 600 golds in run.jsonl). Raw answers: data/decision-models/chat-run.jsonl. Write-up: jev-agent.com/best-llm-for-text-classification.

Reproduce it

git clone https://github.com/WallerChen/jev-measured
cd jev-measured
cp .env.example .env     # add one key — see below
python -m bench.net_floor              # measure your own network floor first
python -m bench.run                    # all 8 cases -> data/measured.json

Python 3.10+. The OpenRouter path needs no dependencies at all (stdlib urllib); pip install -r requirements.txt is only needed for the first-party TypeSafe SDK path.

Which transport runs is decided by which key is present:

Key set	Transport	Model
TYPESAFE_API_KEY	First-party API	jev-latest
OPENROUTER_API_KEY	OpenRouter	typesafe/jev-1.13

Other useful invocations:

python -m bench.run --limit 1                          # smoke test, prints raw answers
python -m bench.run --only phishing-detection --repeats 5
python -m bench.run --only invoice-classification,news-filtering --merge   # re-measure some, keep the rest of data/measured.json

⚠️ Gateway gotcha

If you go through OpenRouter: Jev will not appear in the model catalogue, and /api/v1/chat/completions will reject it. Its modality is text->decisions rather than text->text, so listings built for chat models filter it out, and it reports an empty supported_parameters. Searching a gateway for "jev" can return nothing while the model is live and serving traffic.

Address it by exact id, and don't send temperature or max_tokens — none apply.

# Confirm it is live even though the catalogue omits it:
curl -s https://openrouter.ai/api/v1/models/typesafe/jev-1.13/endpoints

---

How the fixtures are built

Twelve use cases in bench/fixtures.py (the last four added on 2026-10-11), each a plain dict: a state and the typed questions asked about it.

• Several states are deliberately ambiguous. A fixture that returns 1.00 confidence every time teaches nothing about what the probabilities are for. Please don't "improve" them into easy cases.
• Failures are recorded, not hidden. An errored call keeps its error in the output rather than vanishing.
• Latency is always reported raw and floor-adjusted.

What this is not

Not a broad accuracy benchmark. The twelve main fixtures have no ground-truth labels at all — they measure cost, latency and response shape only. bench/accuracy.py adds 27 labelled tickets on one task, three passes each, which is enough to show Jev ties rather than wins, and not enough to rank anything. For accuracy at scale see anisselbd/jev-phishing-bench (2,000 labelled emails, Jev vs Claude Haiku 4.5).

Not affiliated with TypeSafe AI. Where this disagrees with docs.typesafe.ai, the official docs are correct — please open an issue so it can be fixed here.

Longer write-ups of each use case, with the code, are at jev-agent.com/use-cases.

Licence

MIT. Quote the figures freely; attribution appreciated but not required.