Investigate slimming the RAG source-block delimiter to save prompt context

[kfaracik]

Context

Every retrieved chunk — on-device documents and web results alike — is wrapped by sourceBlock (utils/contextUtils.ts:88) as:

--- Source 1: <name> ---
<passage>
--- End of Source 1 ---

This delimiter grammar is load-bearing, not decorative:

• the system prompt tells the model sources arrive as "Source N: <name>" and to cite by it (utils/promptUtils.ts:34-37);
• SOURCE_HEADER = /--- [^:]+: (.+?) ---/g (constants/retrieval.ts:60) parses these headers to build the present-in-context source set (sourcesPresentInContext), which drives citation attribution;
• the truncation fallback re-closes the last open block using the --- markers (utils/promptUtils.ts:171-180).

Idea

Slim the per-block boilerplate to save prompt tokens, which matters on the small on-device model's tight context budget.

Why the naive fix does NOT work

• Swapping the --- character saves ~0 tokens. --- is ~1 BPE token in the executorch/Llama-Qwen tokenizer; a single replacement char is also ~1 token, and a rarer Unicode char (§, ¶, private-use) can tokenize to 2-3 tokens — i.e. worse.
• The real overhead is the repeated boilerplate per block, above all the explicit --- End of Source N --- closer: ~4-6 tokens/block, so ~20-48 tokens across a typical 5-8 block prompt.

Touch points that must stay consistent

• SOURCE_HEADER regex — constants/retrieval.ts:60
• truncation re-close — utils/promptUtils.ts:171-180
• system-prompt instruction ("Source N: <name>") — utils/promptUtils.ts:34-37
• neutralizeDelimiters — utils/web/webResultsToContext.ts (the prompt-injection surface moves to the new delimiter, it does not disappear)
• affects doc blocks too, not only web results

How to decide (acceptance)

1. Measure tokens before/after with the actual on-device tokenizer on a representative multi-source prompt (current format vs. no-explicit-closer vs. shortened tag).
2. Verify citation/grounding attribution does not regress on the small on-device model — that reliability is the whole point of the feature.
3. Consider a rarer delimiter (e.g. ⟦Source 1⟧) purely for robustness — page text rarely contains it, shrinking the forgery surface — independent of any token savings.

Non-goals

Not part of the current web-search review-fix PR. The delimiter format spans the whole RAG path (docs + web), so it should ship separately and only on measurements.

[kfaracik]

Other prompt-context savings to evaluate together

While the delimiter is the headline, the same audit surfaced several other levers on the prompt-assembly path. All are tokenizer-dependent, so each needs a before/after measurement with the on-device tokenizer and a grounding/attribution check — grouping them here so they can be measured in one pass.

1. Drop or dedupe the explicit block closer (biggest lever). sourceBlock (utils/contextUtils.ts:93) and formatFirstChunks (utils/contextUtils.ts:128) emit --- End of Source N --- per block. It also re-encodes the index a second time. ~4-6 tokens/block, ~20-48 across a 5-8 block prompt. A single trailing sentinel or "next opener = previous close" is enough for the model and the SOURCE_HEADER parser.

2. Trim per-line leading whitespace in the block frame (free, zero risk). The frame is \n --- ... --- \n ${passage} \n --- ... — every line starts \n (newline + space) and markers are padded ---. \n frequently tokenizes worse than a bare \n, and the padding spaces are pure waste. Normalizing to \n--- ... ---\n${passage}\n--- ... costs nothing in comprehension. Applies to sourceBlock and formatFirstChunks.

3. Dedupe the SERP snippet against the page excerpt (web path). webResultsToContext (utils/web/webResultsToContext.ts:111-115) prepends snippet then relevant, but the SERP snippet is usually a substring of the page body, so selectRelevantContent often re-includes the same sentences — we pay for them twice. The doc path already dedupes overlap via stripLeadingOverlap (utils/contextUtils.ts:64); reuse it here, or drop the snippet when relevant is present.

4. Condense the per-turn instruction blocks (largest fixed cost after the context itself). getContextInstruction (utils/promptUtils.ts:20-56) and getPreferredSourceInstruction (utils/promptUtils.ts:58-66) add a multi-sentence preamble to every grounded turn (~500-800 chars combined), partly overlapping in intent. Candidate for tightening and/or merging into one block. Highest attribution-regression risk of the set — the small on-device model leans on this guidance, so measure grounding carefully.

5. Normalize internal whitespace of scraped passages. Article content can retain runs of blank lines / repeated spaces from the page; the SERP script already collapses whitespace but the article extractor path does not fully. Collapsing \n{2,} and repeated spaces inside passages before they enter sourceBlock trims tokens with no meaning loss.

6. (Minor) CUSTOM_PROMPT_GUARD. constants/prompts.ts:1 is prepended verbatim before every custom system prompt (~110 chars each turn a custom prompt is set). Low priority, listed for completeness.

Suggested order by value/risk: 2 and 3 first (free / near-free), then 1 (needs parser + truncation + re-close updates, tracked in the issue body), then 5, then 4 (measure comprehension), 6 last.