docs(ingestion): investigate issue #149 — superseded by #927 - #242
Closed
folusho-adeyemi wants to merge 3 commits into
Closed
Conversation
…less docs
Add a root-cause note at the content-collection guard in
_extract_sections. A document with no markdown headings never populates
heading_stack, so no section is emitted and chunk() returns [], silently
excluding the whole document from the RAG index.
Reproduction:
StructuralChunker().chunk('plain text ' * 20, {}) -> 0 chunks
pytest test_document_with_no_headings -> FAILED (assert 0 >= 1)
Bypassing pre-commit: ruff/mypy flag pre-existing issues in this file
(unused current_level, missing annotations) that are out of scope for a
doc-only reproduction commit; they will be addressed with the Week 9 fix.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
5 tasks
folusho-adeyemi
changed the title
Author
|
Closing in favour of #927. The branch was renamed to |
folusho-adeyemi
added a commit
to folusho-adeyemi/pathreview
that referenced
this pull request
Aug 12, 2026
Records that no reviewer feedback arrived on ascherj#927 or ascherj#242, and what I self-reviewed against CONTRIBUTING.md and the repo tooling in its absence: the branch rename, the two existing tests that computed an assertion without ever checking it, and the pre-existing-failure baseline. Reflects on the four-week cycle - that the fix was small relative to establishing a baseline in a repo where make check already fails, that deciding what not to fix was harder than fixing, where AI assistance helped versus where it produced a confident but unnecessary recommendation, and why the root-cause fix was chosen over the SemanticChunker fallback other claimants proposed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
Superseded by #927 — please review that PR instead.
This PR was opened from the branch
149-structural-chunker-silently-drops-documents-that-contain-no-headings, which does not follow the<type>/<issue-number>-<short-description>branch naming convention required by CONTRIBUTING.md. The work was moved tofix/149-structural-chunker-drops-heading-less-docsand continues in #927. No commits are lost — #927 contains every commit from this branch plus the completed fix, tests and journal entries.This description is filled in below for the record.
Summary
StructuralChunkersilently dropped any markdown document that contained no headings._extract_sections()only collected content lines after a heading had been seen, and only emitted a section when the heading stack was non-empty, so a heading-less document produced zero sections andchunk()returned[]. BecauseStrategySelectorroutes everyreadmesource type to this chunker, a README with no headings never entered the RAG index and no error was raised anywhere. The same guard also discarded any preamble appearing before the first heading.At the point this PR was opened it contained only the Week 8 investigation — a failing reproduction test and
PLAN.md. The implemented fix, its unit tests and the Week 9 check-ins are in #927.Issue
Relates to #149 — closed by #927.
Changes
Commits that were on this branch when it was superseded:
396bdf4started working on issue 149e422a33docs: reproduce issue Structural chunker silently drops documents that contain no headings #149 — structural chunker drops heading-less docs3641e76docs: add PLAN.md and Week 8 journal entry for issue Structural chunker silently drops documents that contain no headings #149The fix itself (
fix(ingestion): emit sections for markdown without headings) and its tests (test(ingestion): cover heading-less documents and preamble chunking) landed on the renamed branch and are reviewable in #927.Testing
Testing is documented in full in #927, which reports:
mainmake test-unittest_document_with_no_headingsmake lintmake typecheckmake test-unit) — verified on fix(ingestion): emit sections for markdown without headings #927make test-integration) — not run, requires Docker servicesmake lint) — verified on fix(ingestion): emit sections for markdown without headings #927make typecheck) — verified on fix(ingestion): emit sections for markdown without headings #927Screenshots / Demo
Not applicable — backend chunking logic with no user-facing surface.
Notes for Reviewers
Nothing to review here — please use #927. Closing this to avoid two open PRs for issue #149.