When a chunk's token count exceeds the embedder's cap, encode in src/engine/parallel_embedder.rs prints a line and moves on:
if token_count > MAX_LENGTH {
eprintln!(
"Warning: embedding input truncated from {} to {} tokens ({} discarded)",
token_count,
MAX_LENGTH,
token_count - MAX_LENGTH
);
}It is the only warning in the crawl that is not a CrawlWarning, which costs three things. It prints once per chunk from parallel workers, so a file that produces forty oversized chunks produces forty lines interleaved with everything else. It names no file, so a reader cannot tell which file to look at. And it is invisible to the crawl summary, so a crawl that truncated ten thousand chunks reports the same summary as one that truncated none.
David Huang (@davidh233)'s team ran into the practical version of this on a 39-hour crawl of a 230k-file monorepo: the messages arrive from parallel workers across the whole run, and the handful of cases anyone would want to investigate are buried in thousands of repetitions of cases nobody cares about.
A chunk that hits the cap is usually minified output, generated code, or a file with very long single lines, and the fix is normally a patternsToExclude entry rather than anything in the code. That decision requires knowing which files, and today the output does not say.
Proposal
Add a variant to CrawlWarning in src/engine/warning.rs carrying the file path and the count, emitted once per file rather than once per chunk:
/// Embedding input exceeded the token cap and was truncated.
EmbeddingInputTruncated {
relative_path: String,
chunk_count: usize,
max_tokens: usize,
},That puts it under the same rendering and counting as the existing four variants, and the crawl summary gains a line naming the affected files. The stored text is unaffected either way, so FTS still covers the truncated tail; what is lost is only the vector's coverage of it.
Two implementation notes. encode takes &self and worker_index and returns a vector, with no warning sink in scope, so the truncation has to be reported back to the caller in src/app/crawl/pipeline.rs rather than emitted where it is detected. And per-file aggregation means the count accumulates across workers before anything is emitted, since chunks of one file are spread across the pool.
Not in this issue
Whether truncation warnings can be muted. Chunk-level warnings like chunker-fallback are mutable because a stored column identifies which chunks are affected, so a mute verb can ask the database which paths are currently degraded. Truncation has no such marker: the embedder truncates and moves on, and nothing about it is written down. Making it mutable would need either a column recorded at chunk-write time or a second way for the mute verb to derive its candidates, and that belongs with the mute design rather than here.
Whether these failures should affect crawl_complete. Truncation is a degradation rather than a failure, so it does not, but the broader question of which per-file problems should prevent a crawl from reporting success is open and separate.
When a chunk's token count exceeds the embedder's cap,
encodein src/engine/parallel_embedder.rs prints a line and moves on:It is the only warning in the crawl that is not a
CrawlWarning, which costs three things. It prints once per chunk from parallel workers, so a file that produces forty oversized chunks produces forty lines interleaved with everything else. It names no file, so a reader cannot tell which file to look at. And it is invisible to the crawl summary, so a crawl that truncated ten thousand chunks reports the same summary as one that truncated none.David Huang (@davidh233)'s team ran into the practical version of this on a 39-hour crawl of a 230k-file monorepo: the messages arrive from parallel workers across the whole run, and the handful of cases anyone would want to investigate are buried in thousands of repetitions of cases nobody cares about.
A chunk that hits the cap is usually minified output, generated code, or a file with very long single lines, and the fix is normally a
patternsToExcludeentry rather than anything in the code. That decision requires knowing which files, and today the output does not say.Proposal
Add a variant to
CrawlWarningin src/engine/warning.rs carrying the file path and the count, emitted once per file rather than once per chunk:That puts it under the same rendering and counting as the existing four variants, and the crawl summary gains a line naming the affected files. The stored text is unaffected either way, so FTS still covers the truncated tail; what is lost is only the vector's coverage of it.
Two implementation notes.
encodetakes&selfandworker_indexand returns a vector, with no warning sink in scope, so the truncation has to be reported back to the caller in src/app/crawl/pipeline.rs rather than emitted where it is detected. And per-file aggregation means the count accumulates across workers before anything is emitted, since chunks of one file are spread across the pool.Not in this issue
Whether truncation warnings can be muted. Chunk-level warnings like
chunker-fallbackare mutable because a stored column identifies which chunks are affected, so a mute verb can ask the database which paths are currently degraded. Truncation has no such marker: the embedder truncates and moves on, and nothing about it is written down. Making it mutable would need either a column recorded at chunk-write time or a second way for the mute verb to derive its candidates, and that belongs with the mute design rather than here.Whether these failures should affect
crawl_complete. Truncation is a degradation rather than a failure, so it does not, but the broader question of which per-file problems should prevent a crawl from reporting success is open and separate.