Embeddings are truncated at 512 tokens while chunks are allowed to be ~6-7K tokens and the model supports 8192

[citron07r]

Three numbers in the local embedding path disagree with each other:

setting	value	source
max_chunk_bytes	24576	config.rs, commented "~6K-7K tokens, safe for any embedding model"
EMBEDDING_MAX_LENGTH	512	local_models/mod.rs
model max sequence length	8192	jina-embeddings-v5-text-nano-retrieval model card

load_tokenizer applies the middle one as a hard truncation:

tokenizer.with_truncation(Some(tokenizers::TruncationParams {
    max_length,                       // 512
    strategy: tokenizers::TruncationStrategy::LongestFirst,
    ..Default::default()
}))

So the chunker is allowed to emit chunks of ~6-7K tokens, the model can encode 8192, and everything
past token 512 is discarded before embedding. The comment on max_chunk_bytes says the size is
chosen to be safe for the embedding model, but the embedding model never sees most of it.

Measured impact

Indexed two repositories from the eval corpus at their pinned commits and measured chunk sizes
against a ~2KB proxy for 512 tokens (code runs roughly 3-4 characters per token):

repo	chunks	over ~512 tokens	largest chunk	content past the window
ripgrep	5495	505 (9.2%)	24558 B	1.5 MB
fastify	8986	185 (2.1%)	24554 B	529 KB

A minority of chunks by count, but they are the largest ones — whole functions and tier-0 line
chunks — and 1.5 MB of ripgrep's indexed content contributes nothing to its own embedding. The
tail of a long function is unreachable by vector search, though BM25 still matches it, which is
probably why this has not been obvious.

Notes

• The 2KB proxy is rough. The precise figure needs tokenizing the corpus, but the direction does
  not depend on the constant: the largest chunk is 12x the window whatever the exact ratio.
• Raising EMBEDDING_MAX_LENGTH is not free. Attention cost grows with sequence length and the
  local backend is already the slow path, so the sensible fix might instead be lowering
  max_chunk_bytes to match the window, or splitting large chunks for embedding and keeping them
  whole for display.
• Whichever way it goes, the two constants should be derived from one another rather than set
  independently, so they cannot drift apart again.

Raising this as an issue rather than a PR because the resolution is a quality-versus-latency call
that wants benchmark evidence, and this touches the indexing path for every local-backend user.

[citron07r]

Took the measurement this issue was missing, and mapped what else EMBEDDING_MAX_LENGTH is wired into. Line references are upstream master at e3d79b3.

Measured token distribution

The issue argued from a byte proxy. These are real tokenizations: chunk boundaries and chunk text taken from an actual .vera/metadata.db built by parse_file, the embedding text rebuilt by a line-for-line port of chunk_text.rs:43 build_structured_text that reproduces all 14 assertions in that file's test module, and the real tokenizer.json for jina-embeddings-v5-text-nano-retrieval with truncation off.

	Vera's own source	vendored tree-sitter	all of crates/
chunks	3138	538	3676
median	118	256	129
p95	637	2495	944
p99	1609	3243	2302
max	2188	7391	7391
over 512	7.1%	21.0%	9.2%
over 1024	2.7%	17.3%	4.8%
over 2048	0.1%	11.2%	1.7%
tokens discarded past 512	17.2%	54.5%	29.5%

Three corrections to the issue as written.

The 2KB proxy was about right. 4.27 bytes per token here, so 512 tokens is ~2187 bytes and the 24576-byte cap is ~5754 tokens. The "~6K-7K tokens" note at config.rs:38 holds.

max_chunk_bytes is rarely the binding constraint. max_chunk_lines 200 splits first (parsing/chunker.rs:71), and 200 lines of Rust is ~2200 tokens, so split_oversized_chunks (chunker.rs:438) seldom fires on hand-written code. The 6-7K chunks are real but come from whole-file and generated-data shapes, which is why the vendored column looks nothing like the others. The two constants that actually disagree are max_chunk_lines and EMBEDDING_MAX_LENGTH, not max_chunk_bytes.

The header spends part of the window. The metadata block is prepended and truncation is right-directional, so the header always survives and the code tail is what disappears: median 59 tokens, p99 131, max 217. Of the 224 over-budget chunks the median keeps 56.8% of its code body, the worst 19.9%.

So 7.1% of chunks lose a median 43% of their body. Worth fixing, and it does not need 8192: 2048 captures 99.9% of chunks and 100% of tokens in this corpus, and 4096 and 8192 buy nothing here at the same cost.

One caveat: 3134 of 3138 chunks are Rust, and this is one repository. Do not read 7.1% as a cross-language figure.

Why the cap cannot move on its own

bucket_for is seq_len.div_ceil(max_length / 8) * (max_length / 8), and the tokenizer truncates at max_length, so the scaler has exactly 8 buckets at any cap. Raising it does not add resolution, it just widens each bucket while real chunk lengths stay put. At 8192, bucket_for(128) and bucket_for(1024) both return 1024, so 97.3% of this corpus lands in one learned window. embed_chunks_concurrent sorts by content length before batching (embedding/provider.rs:1033), so both batch shapes reliably occur and both write to that window: a success at the short end raises the cap the long end reuses, and an OOM at the long end sets max_safe_batch = batch_len - 1 for the short end too.

I ran the existing scaler tests at max_length = 8192 rather than reasoning about them. All four fail:

test	at 512	at 8192
adaptive_batch_scaler_shrinks_long_batches_quadratically	32	128
adaptive_batch_scaler_keeps_short_and_long_buckets_independent	128	24
adaptive_batch_scaler_learns_safe_windows_per_length_bucket	passes	fails on the cold-start assertion
adaptive_batch_scaler_dampens_cold_start_for_large_models	passes	fails

The second one is the invariant named in its own title: a failure recorded at seq_len 512 now suppresses batches at seq_len 128.

The fourth is the one I would not have predicted. At 8192 the 50MB and the 400MB model both return 128, because sequence_guided_batch_len computes 2048 before dampening and 512 after, and both clamp back to the requested 128. The model-size dampener is silently dead above a certain cap. The CoreML test breaks the same way: at 8192 both the CUDA and CoreML scalers return 16 where the test asserts 1 and 4.

What has to change first

Make the two derived values absolute token counts:

• BATCH_BUCKET_WIDTH_TOKENS = 64 replacing max_length / 8 at local_provider.rs:104
• BATCH_REFERENCE_SEQ_LEN = 256 replacing max_length / 2 at local_provider.rs:109

At 512 these are numerically identical to today's expressions, so it is a no-op at the current cap. Verified by running the suite: all four scaler tests pass with the constants at both 512 and 8192, and all 16 tests in the local_provider module pass at 512. Bucket count then becomes max_length / 64, sparse in the BTreeMap, which is the resolution the design assumed it had.

The index will not be invalidated, and that is the dangerous part

model_identity() for a preset returns display_name() plus |pooling=<mode> and nothing else (local_models/mod.rs:313-331). max_length is absent.

The subtlety is worse than an omission. The preset branch fires on full struct equality, and max_length is a struct field populated from EMBEDDING_MAX_LENGTH in preset(). So raising the constant also raises what Self::jina() builds, the config still equals the preset, and the identity string stays byte-identical by construction. An index built at 512 passes the guard at indexing/update.rs:324, and vera update then mixes 512-truncated vectors with longer ones in the same table.

Only users whose max_length already differs from the preset fall into the generic branch that does encode it, which is the opposite of the population that needs invalidating.

Whatever cap is chosen, max_length needs to be in the preset identity too, or the guard needs a separate check. This is the same silent-mismatch shape as the pooling problem in #94.

Separately, the persisted scaler profile is keyed on max_length (local_provider.rs:630-641) and rejects on mismatch at :668, so every GPU user cold-starts at exactly the moment the cold-start dampener stops dampening. save_persisted_batch_scaler inserts by key and never prunes, so orphaned entries stay in the file.

Cost

Attention is quadratic in sequence length and the FFN linear, so the real multiplier sits between these columns. Relative to 512, over Vera's own source:

cap	chunks fully embedded	tokens embedded	linear	quadratic
512	92.9%	82.8%	1.00x	1.00x
1024	97.3%	93.5%	1.13x	1.65x
2048	99.9%	100.0%	1.21x	2.38x
8192	100%	100%	1.21x	2.38x

These are arithmetic over measured token counts, not wall-clock. I have not run a timed reindex.

Suggested split

Two PRs. First the bucket and reference constants: self-contained, no behaviour change at 512, covered by the four existing tests. Then the cap raise plus index invalidation, where the number to argue about is 2048 rather than 8192.

Happy to send the first one now if you want it, since it is a strict prerequisite either way.