Fix markdown/one-line resume chunking (issue #1) - #3
Open
Divergent-Code wants to merge 1 commit into
Open
Conversation
chunk_source_material() split on raw newlines only, so a markdown resume pasted as one block (what Colab's input() produces) collapsed into a single giant chunk. Then every requirement's retrieval matched that one chunk regardless of relevance, silently defeating the RAG grounding and reintroducing fabrication. - Split on newlines, markdown headers (#..######), and bullet markers (-, *, •), so inline bullets in a single-line paste break apart correctly. - Strip markdown formatting (#, *, _, `, >) and list punctuation per chunk. - Warn loudly when the resume parses into fewer than `warn_below` (5) chunks, so a degenerate parse fails visibly instead of silently. Verified on the sample resume: 1 -> 13 chunks, and the one-line and multi-line markdown forms of the same content produce an identical chunk count (13 == 13), per the issue's equivalence expectation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes camunity issue #1 — the chunking root cause behind the retrieval problems (and referenced by #4).
Stacked PR: base is
feat/llm-injection-screen, so this diff shows only the chunking change. Stack order on your fork:main← #1 (multi-provider) ← #2 (injection) ← this.The bug
chunk_source_material()split on raw newlines only. A markdown resume pasted as one block — which is exactly what Colab'sinput()produces from a paste — collapsed into a single giant chunk. Every requirement's retrieval then matched that one chunk regardless of relevance, silently defeating the RAG grounding and letting fabrication back in.The fix
#..######), and bullet markers (-,*,•), so inline bullets in a one-line paste break apart.warn_below(5) chunks — a degenerate parse now fails visibly instead of silently (the issue's "fail loudly" ask).Verification (on the sample resume)
The one-line and multi-line forms of the same content now produce an identical chunk count, matching the equivalence the issue asks for. En-dash date ranges ("Jun 2023 – Present") are correctly not split (en-dash ≠ hyphen).
All code cells compile; changed-cell outputs cleared.
🤖 Generated with Claude Code