From 396bdf4d600e85ca583445548bc150693ef7b458 Mon Sep 17 00:00:00 2001 From: Folusho Adeyemi Date: Tue, 21 Jul 2026 18:31:19 -0400 Subject: [PATCH 1/3] started working on issue 149 --- JOURNAL.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 JOURNAL.md diff --git a/JOURNAL.md b/JOURNAL.md new file mode 100644 index 000000000..0baa96e67 --- /dev/null +++ b/JOURNAL.md @@ -0,0 +1,18 @@ +## Week 7 — Issue selection + +**Issue link:** [https://github.com/ascherj/pathreview/issues/149] + +**Issue title:** [Structural chunker silently drops documents that contain no headings +] + +**Tier:** [#] Tier 1 [ ] Tier 2 [ ] Tier 3 + +**Problem summary:** + +The function, StructuralChunker.chunk(), returns an empty list for any document without markdown headings, and it removes the entire document from the RAG index instead of being it chunked as a single block or falling back to another strategy. I will reproduce the error and then work with the test_document_with_no_headings in tests/unit/test_structural_chunker.py to show a successful fix + +**Branch name:** [149-structural-chunker-silently-drops-documents-that-contain-no-headings] + +**Setup confirmation:** [#] App runs locally at localhost:5173 + +**Cohort ledger:** [#] Issue added to cohort ledger \ No newline at end of file From e422a33bff82ece44e63c35db4ac256909524e05 Mon Sep 17 00:00:00 2001 From: Folusho Adeyemi Date: Tue, 28 Jul 2026 19:33:34 -0400 Subject: [PATCH 2/3] docs: reproduce issue #149 - structural chunker drops heading-less docs Add a root-cause note at the content-collection guard in _extract_sections. A document with no markdown headings never populates heading_stack, so no section is emitted and chunk() returns [], silently excluding the whole document from the RAG index. Reproduction: StructuralChunker().chunk('plain text ' * 20, {}) -> 0 chunks pytest test_document_with_no_headings -> FAILED (assert 0 >= 1) Bypassing pre-commit: ruff/mypy flag pre-existing issues in this file (unused current_level, missing annotations) that are out of scope for a doc-only reproduction commit; they will be addressed with the Week 9 fix. Co-Authored-By: Claude Opus 4.8 --- ingestion/chunking/structural_chunker.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/ingestion/chunking/structural_chunker.py b/ingestion/chunking/structural_chunker.py index d5bcf0530..5c795b5de 100644 --- a/ingestion/chunking/structural_chunker.py +++ b/ingestion/chunking/structural_chunker.py @@ -108,6 +108,11 @@ def _extract_sections(self, text: str) -> list[dict]: else: # Regular content line + # BUG(#149): content is only collected once a heading exists. + # For a document with NO headings, heading_stack stays empty and + # current_section_lines is never populated, so no section is ever + # emitted below (see the `and heading_stack` guards) and chunk() + # returns []. Repro: chunk("plain text " * 20, {}) -> 0 chunks. if heading_stack or current_section_lines: # Only collect if we have a heading current_section_lines.append(line) From 3641e76e04eed8f048024c771a5007d8fcfda6f6 Mon Sep 17 00:00:00 2001 From: Folusho Adeyemi Date: Tue, 28 Jul 2026 19:34:59 -0400 Subject: [PATCH 3/3] docs: add PLAN.md and Week 8 journal entry for issue #149 Co-Authored-By: Claude Opus 4.8 --- JOURNAL.md | 16 +++++++- PLAN.md | 115 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 130 insertions(+), 1 deletion(-) create mode 100644 PLAN.md diff --git a/JOURNAL.md b/JOURNAL.md index 0baa96e67..4189fb142 100644 --- a/JOURNAL.md +++ b/JOURNAL.md @@ -15,4 +15,18 @@ The function, StructuralChunker.chunk(), returns an empty list for any document **Setup confirmation:** [#] App runs locally at localhost:5173 -**Cohort ledger:** [#] Issue added to cohort ledger \ No newline at end of file +**Cohort ledger:** [#] Issue added to cohort ledger + +## Week 8 — Reproduction & solution planning + +**Reproduction commit link:** [e422a33](https://github.com/folusho-adeyemi/pathreview/commit/e422a33bff82ece44e63c35db4ac256909524e05) + +**Reproduction summary:** +Ran `StructuralChunker().chunk("This is a plain document with no headings at all. " * 20, {})` and it returned 0 chunks, and `pytest tests/unit/test_structural_chunker.py::TestStructuralChunker::test_document_with_no_headings` failed with `assert 0 >= 1`. Root cause: `_extract_sections()` only collects content and emits a section once a heading has been seen (`if heading_stack ...`), so a document with no headings never populates `heading_stack` and produces no sections. I documented this at the exact guard in `ingestion/chunking/structural_chunker.py`. + +**PLAN.md link:** [PLAN.md](https://github.com/folusho-adeyemi/pathreview/blob/149-structural-chunker-silently-drops-documents-that-contain-no-headings/PLAN.md) + +**Walkthrough video (recommended):** [not recorded] + +**Blockers or open questions:** +Need to confirm whether any downstream consumer of the RAG index assumes a non-empty `heading_path` before committing to `heading_path == ""` for heading-less documents (vs. falling back to the source name). This file also has pre-existing ruff/mypy failures that the Week 9 fix commit will need to clean up. \ No newline at end of file diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 000000000..470bb97b8 --- /dev/null +++ b/PLAN.md @@ -0,0 +1,115 @@ +## Solution plan + +**Issue:** Structural chunker silently drops documents that contain no headings — [#149](https://github.com/ascherj/pathreview/issues/149) + +### Understand + +**Root cause.** `StructuralChunker._extract_sections()` only ever collects +content and emits a section once a markdown heading has been seen: + +- `ingestion/chunking/structural_chunker.py` line ~122 — content lines are + appended only `if heading_stack or current_section_lines`. With no heading, + `heading_stack` is empty, so nothing is ever collected. +- lines ~93–101 and ~126 — a section is only appended when `heading_stack` is + truthy (`if current_section_lines and heading_stack`). + +So for a document with **no headings at all**, `heading_stack` stays empty for +the whole loop, no section is ever created, `_extract_sections()` returns `[]`, +and `chunk()` returns `[]`. The document is silently excluded from the RAG index. + +The *same* guard also drops any **preamble** — content that appears before the +first heading — even in documents that do have headings. + +**Expected vs. actual.** +- Expected: a heading-less document is chunked as a single block (or + sub-chunked via the semantic chunker if it exceeds the token limit), with a + sensible/empty `heading_path`. +- Actual: `chunk()` returns `[]` and the document never enters the index. + +Confirmed reproduction (Week 8): +``` +StructuralChunker().chunk("This is a plain document with no headings at all. " * 20, {}) +# -> 0 chunks +pytest tests/unit/test_structural_chunker.py::TestStructuralChunker::test_document_with_no_headings +# -> FAILED: assert 0 >= 1 +``` + +### Map + +Files/functions involved: + +- **`ingestion/chunking/structural_chunker.py`** — primary fix. + - `StructuralChunker._extract_sections()` — the two heading-gated guards that + drop heading-less content and preamble. + - `StructuralChunker.chunk()` — consumes sections; already routes large + sections to the semantic chunker, so a heading-less section will + automatically be sub-chunked once it is produced. +- **`ingestion/chunking/semantic_chunker.py`** — the fallback for large + sections; read-only, reused as-is. +- **`tests/unit/test_structural_chunker.py`** — `test_document_with_no_headings` + is the existing failing test; add coverage for preamble + large heading-less + docs. + +### Plan + +1. **Fix content collection in `_extract_sections()`** so regular content lines + are always collected (remove the `heading_stack`-only gate), letting + heading-less text and preamble accumulate in `current_section_lines`. +2. **Fix section emission** so a section is appended whenever + `current_section_lines` has content, regardless of whether `heading_stack` + is populated. When there is no heading, emit the section with an empty + `path` (`heading_path == ""`) and `level == 0`. Do this in both the + mid-loop "save previous section" branch and the final "save final section" + branch. +3. **Verify the large-doc path**: a heading-less doc over + `SECTION_TOKEN_LIMIT` (800 tokens) should flow through the existing + `chunk()` branch into `SemanticChunker`, producing multiple chunks. No new + code expected here — confirm with a test. +4. **Add/confirm tests**: keep `test_document_with_no_headings` green, add a + test for a heading-less doc that exceeds 800 tokens (multiple chunks), and + add a test that preamble-before-first-heading is preserved. +5. **Clean up pre-existing lint** touched by the fix so `make lint`/pre-commit + pass (unused `current_level`, missing type annotations on `__init__`, + `heading_stack`, `current_section_lines`). + +### Inputs & outputs + +- **Input:** `text: str` (arbitrary markdown, possibly with zero headings) and + `metadata: dict`. +- **Output:** a non-empty `list[Chunk]` for any non-blank input. + - Heading-less doc → one `Chunk` (or several if > 800 tokens) with + `heading_path == ""` and `heading_level == 0`, and all original metadata + preserved. + - Documents with headings → unchanged behavior, **plus** any preamble now + appears as its own leading chunk. + - Empty/whitespace-only input → still `[]` (unchanged). + +### Risks & unknowns + +- **Regression risk on existing tests.** Emitting preamble as a new chunk + changes chunk counts for docs whose content starts before the first heading; + need to re-run the full `test_structural_chunker.py` suite and check for any + assertions that assume exact counts. (Most existing tests use `>= n`, so this + should be safe.) +- **`heading_path == ""` downstream.** Unsure whether any consumer of the RAG + index assumes a non-empty `heading_path`. Investigation path: grep for + `heading_path` across `rag/`, `ingestion/`, and `api/` before finalizing the + empty-string choice (alternative: fall back to `metadata["source"]`). +- **`heading_level == 0`.** `test_chunk_metadata_includes_heading_level` asserts + level ∈ {1,2,3} — but only for chunks that *have* headings, so level 0 for the + no-heading case should not trip it. Confirm. +- **Pre-existing mypy/ruff failures** in this file block CI; the fix commit must + resolve them, which slightly widens the diff. + +### Edge cases + +- Plain document with no headings (the reported bug) — one or more chunks. +- Heading-less document larger than 800 tokens — sub-chunked into multiple + chunks via `SemanticChunker`. +- Document with preamble text before the first heading — preamble preserved. +- Empty string / whitespace-only — returns `[]` (must not regress). +- Document that is only a heading with no body content — must not crash + (`test_empty_sections_handled`). +- Content that looks like a heading but is not valid markdown (e.g. `#nospace`, + or `#` inside a fenced code block) — treated as regular content; current + regex `^(#{1,6})\s+(.+)$` already requires a space, so `#nospace` is content.