Reranker is ~98% of search latency: jina-reranker-v3 is architecturally incompatible, and a 7-12x smaller cross-encoder is available

[citron07r]

Reranking is essentially all of Vera's search cost: roughly 6.4 s of a 6.5 s search on a
4790-chunk index, against 39 ms for embedding and 16 ms for BM25. Under CoreML it is CPU-only by
design (reranker_execution_provider), so that cost is structural rather than a misconfiguration.

This is a survey of whether the current default can be improved. All model facts below come from
each repo's config.json and file list via the Hugging Face API, not from model cards or
third-party summaries.

Current default: jinaai/jina-reranker-v2-base-multilingual, onnx/model_quantized.onnx,
279,577,152 bytes (~280 MB), XLMRobertaForSequenceClassification, num_labels: 1.

jina-reranker-v3 / v3.5 are not an upgrade path

Worth recording so nobody spends time on it. The blocker is architecture, not packaging.

jina-reranker-v3 reports architectures: ["JinaForRanking"], model_type: "qwen3", built on
Qwen3-0.6B with a custom modeling.py. Its card describes a "last but not late interaction"
listwise design: the query and up to 64 documents share one context window, and per-document
embeddings are taken from each document's last token through an MLP projector (1024 → 512 → 256).
The public API is model.rerank(query, documents) over the whole candidate set.

Vera's call site is a per-pair cross-encoder: (query, doc) in, one logit out. Adopting v3 would
mean reimplementing the projector head and restructuring the calling convention from a per-pair
loop to a batch-of-candidates-in-one-context call. Neither official repo ships ONNX at all, and the
two unofficial exports have 11 and 19 downloads, one of them self-flagged NG by its author.

bge-reranker-v2-m3 is a regression on the axis that matters

Architecture-compatible — XLMRobertaForSequenceClassification with a single-logit head, same shape
as the current model — but:

• onnx/model_quantized.onnx is 570,727,094 bytes (~570 MB) against the current ~280 MB, so
  roughly 2x larger on the component that is already 98% of search latency.
• It is not a clean quality win either. The BEIR table published in jina-reranker-v3's own README
  puts bge-reranker-v2-m3 at 56.51 against jina-reranker-v2's 57.06, though it leads on
  MIRACL (69.32 vs 63.65).

Slower, and behind on the general retrieval benchmark. Not recommended.

The available win is a smaller cross-encoder

reranker	quantized ONNX	vs current	official ONNX	languages
jina-reranker-v2-base-multilingual (current)	280 MB	baseline	yes	multilingual
onnx-community/bge-reranker-v2-m3-ONNX	570 MB	2.0x larger	community	multilingual
jinaai/jina-reranker-v1-turbo-en	38 MB	7.3x smaller	yes	English
jinaai/jina-reranker-v1-tiny-en	33 MB	8.4x smaller	yes	English
cross-encoder/ms-marco-MiniLM-L6-v2	23 MB	12x smaller	yes	English

All three small candidates are ordinary cross-encoders with single-logit heads, so they fit the
existing interface without code changes beyond the constants in local_models/mod.rs.
ms-marco-MiniLM-L6-v2 ships official ONNX in its own repo and has 89M downloads;
jina-reranker-v1-turbo-en is the conservative option if staying in the Jina family matters.

What is not established

• The latency figures are extrapolated from file size, not measured. Size tracks FLOPs
  reasonably for same-family quantized BERT-style encoders, but that is an assumption, not a
  benchmark. A real before/after on an indexed repo is the thing that should decide this.
• The English-only tradeoff is untested. Plausibly fine for code, where identifiers and
  comments skew ASCII/English, but nobody has checked it against a realistic query mix, and the
  current model is multilingual.

Suggested outcome

Keep jina-reranker-v2-base-multilingual as the quality default — nothing surveyed beats it without
a regression elsewhere. Prototype ms-marco-MiniLM-L6-v2 or jina-reranker-v1-turbo-en behind the
existing VERA_LOCAL_* override mechanism and measure, since a 7-12x smaller reranker on a
component that is 98% of search latency is the single largest speed lever available.

Happy to do the prototype and post before/after numbers if that is wanted.

[citron07r]

Re-ran this against the HF API and, this time, against ONNX Runtime. The size numbers hold exactly. The latency conclusion drawn from them does not, and there are four blockers that only appear when you call session.run.

Everything below is either a file-list fact (HF API ?blobs=true, exact bytes) or a measurement (Apple M4, onnxruntime 1.29.0, CPU EP, which is what reranker_execution_provider pins to on macOS anyway).

The premise holds on disk

Ratio against the incumbent's onnx/model_quantized.onnx, 279,577,152 bytes, confirmed byte-identical to the file already cached in ~/.vera/models.

repo	params	quantized ONNX	vs incumbent	licence
jina-reranker-v2-base-multilingual (current)	278,437,633	279,577,152	1.00x	cc-by-nc-4.0
jina-reranker-v1-turbo-en	37,771,777	38,295,104	7.30x smaller	apache-2.0
jina-reranker-v1-tiny-en	33,046,273	33,424,854	8.36x smaller	apache-2.0
ms-marco-MiniLM-L6-v2	22,714,113	23,200,716	12.05x smaller	apache-2.0
ms-marco-TinyBERT-L2-v2	4,386,561	4,518,071	61.88x smaller	apache-2.0
mxbai-rerank-xsmall-v1	70,830,337	87,245,802	3.20x smaller	apache-2.0
gte-reranker-modernbert-base	149,605,633	150,871,837	1.85x smaller	apache-2.0
onnx-community/bge-reranker-v2-m3-ONNX	567,755,777	570,727,094	2.04x LARGER	none declared
onnx-community/Qwen3-Reranker-0.6B-ONNX	n/a	1,219,344,796	4.36x LARGER	apache-2.0

The 7.30x and 12.05x here are exact. Note bge-reranker-v2-m3 is twice the size of the incumbent, so the "it now has ONNX" update is true and does not help this issue. BAAI/bge-reranker-base is also out: same size as the incumbent, and fp32 only at 1.1 GB.

Size is a poor proxy for latency, and that is the load-bearing error

Measurement, replaying what local_reranker.rs actually does: 50 candidates (config.rs:114) in batches of 8 (local_reranker.rs:13), pairs tokenized as one sequence, padded to batch max, capped at 512, against 50 real 40-line slices of crates/vera-core/src. Five reps, models interleaved, median.

model	50-candidate pass	measured speedup	disk ratio
current	6.27 s	1.0x	1.00x
jina-reranker-v1-turbo-en	2.06 s	3.0x	7.30x
jina-reranker-v1-tiny-en	1.28 s	4.9x	8.36x
ms-marco-MiniLM-L6-v2	1.47 s	4.3x	12.05x
ms-marco-TinyBERT-L2-v2	0.20 s	31.9x	61.88x

The incumbent's 6.27 s independently reproduces the 6.4 s in the issue body from a separate process, so the baseline is sound.

12.05x smaller on disk buys 4.3x on latency. 7.30x buys 3.0x. The reason is structural, not noise: 192,001,536 of the incumbent's 278,437,633 params, 69%, are XLM-R's 250k-token embedding table. That is a gather. It costs download and memory, not FLOPs. What differs at inference is encoder geometry, 12 layers at hidden 768 against 6 at 384.

Two candidates are refuted outright. Fixed-shape [8, 512], 30 reps: mxbai-rerank-xsmall-v1 is only 1.4x, and gte-reranker-modernbert-base is 0.8x, slower than the incumbent despite being 1.85x smaller, because it is 22 layers at hidden 768.

Four blockers that only fail at session.run

1. Every cross-encoder/ms-marco-* export requires token_type_ids. local_reranker.rs:142-145 feeds exactly two tensors. The incumbent is XLM-R and has none, so it works. Every BERT-based candidate declares it required:

Required inputs (['token_type_ids']) are missing from input feed
(['input_ids', 'attention_mask']).

Fixable, but not free: the second segment must come from encoding.get_type_ids(), not zeros, because scoring is segment-sensitive. Zeros would silently degrade ranking rather than error, which is the worse failure. So the MiniLM family is not a constants-only change.

2. onnx/model_quantized.onnx does not exist in those repos. They ship model_qint8_arm64.onnx, model_qint8_avx512_vnni.onnx, model_quint8_avx2.onnx and so on. RERANKER_ONNX_FILE is one hardcoded path, so the download 404s. This matters beyond naming, measured on identical weights:

export	batch of 8
ms-marco-MiniLM-L6-v2 model_qint8_arm64.onnx	88.3 ms
Xenova/ms-marco-MiniLM-L-6-v2 model_quantized.onnx	121.8 ms

1.38x for the same model, purely from arch-matched quantization. That is worth more than the gap between two adjacent MiniLM depths, and a single constant cannot express it.

3. ettin-reranker-32m/68m-v1 have no classification head. Not in the earlier survey and ideal on paper (31.9M params, ModernBERT, apache-2.0, no token_type_ids). The export emits last_hidden_state [batch, seq, 384], so Vera hits anyhow::bail!("Unexpected tensor shape for reranker") at local_reranker.rs:173. Clean file list, unusable.

4. gte-reranker-modernbert-base's tokenizer sets padding: {"strategy": {"Fixed": 8000}}. Under local_reranker.rs:110-122 every batch would pad to the full 512 regardless of content, paying worst-case compute on every call.

Only two candidates are drop-in today

jina-reranker-v1-turbo-en and jina-reranker-v1-tiny-en. Both take only input_ids and attention_mask, emit [batch, 1], and ship onnx/model_quantized.onnx at exactly the path RERANKER_ONNX_FILE already points at. Both apache-2.0, so either also resolves the reranker half of #66.

The prototype this issue proposes cannot be run today

There is no VERA_LOCAL_RERANKER_* override. The VERA_LOCAL_* mechanism covers embeddings only. The reranker repo is a hardcoded const in two places, local_models/mod.rs:43 and again at local_reranker.rs:11, which is also two sources of truth for the same string.

A mirrored VERA_LOCAL_RERANKER_{REPO,ONNX_FILE,TOKENIZER_FILE} is a prerequisite for anyone measuring this on their own index, and is worth doing regardless of which model wins.

The quality evidence is thin exactly where it matters

The 57.06 quoted for the incumbent and the 49.60 for jina-reranker-v1-turbo-en come from different tables on different cards over different suites. They cannot be subtracted. I could not find any published table scoring the incumbent and the small candidates on the same benchmark.

The one code-relevant number that exists is CoIR on jina-reranker-v3's card, where jina-reranker-v2 scores 56.14, behind mxbai-rerank-base-v2 at 65.71. No small English cross-encoder has a CoIR figure at all. For a code search tool that is the gap, not BEIR.

Top-5 agreement with the incumbent on one query over 50 real chunks: turbo-en 4/5, MiniLM-L6 4/5, TinyBERT-L2 4/5. n=1, and I am reporting it as a non-result. TinyBERT-L2 matching 4/5 while being 31.9x faster is evidence that 50 post-fusion candidates are already well separated, not that it is good.

Suggested revision

1. Add VERA_LOCAL_RERANKER_{REPO,ONNX_FILE,TOKENIZER_FILE} and collapse the duplicated const. Unblocks measurement, commits to no model.
2. Prototype jina-reranker-v1-turbo-en first rather than ms-marco-MiniLM-L6-v2. It is the only genuine drop-in, apache-2.0, and leads the one internally comparable BEIR table. 6.27 s to 2.06 s.
3. Treat MiniLM-L6 as second, gated on token_type_ids and arch-aware export selection.
4. Drop bge-reranker-v2-m3, Qwen3-Reranker-0.6B, gte-reranker-modernbert-base, bge-reranker-base and both ettin-reranker sizes, each refuted above on size, architecture or measured latency.
5. Decide on code-retrieval quality, not BEIR.

I would keep the incumbent as the quality default until step 5 is answered. The speed lever is real and it is the largest one in search latency, but the headline should read 3-4x faster, in a package 7-12x smaller, not 7-12x faster.

Caveat on the absolute figures: M4 P/E scheduling gives p90/median around 1.5-2.5x, so treat the millisecond numbers as +/- 20%. Medians were stable across three runs and the ordering never changed. CoreML was not tested.