fix: preserve summaries across incremental update()

update() computed summaries and then discarded them, so every tree it
wrote back had no summaries at all. Two independent causes:

1. build_tree_from_nodes() rebuilt each node dict from scratch and never
   copied `summary`. md_to_tree() was unaffected because it summarizes
   after building the tree; update() summarizes before, so its results
   were dropped. Fixed at the shared function, which both paths use.

2. The cached-summary lookup keyed the old tree by bare `title` but read
   it by full `title_path`. title_path exists only on the flat node list,
   never on the persisted tree, so every clean section missed the cache
   and fell through to ''. Rebuilt the map by walking the tree with the
   same ' > ' join.

Also adds split_summary_fields() so update() matches index()'s
convention (parents -> prefix_summary, leaves -> summary), and drops the
now-unused structure_to_list import.

This mattered because get_document_structure() strips `text` and hands
the model titles + summaries only. After an update the payload was bare
titles, so retrieval had nothing to reason over -- it would misroute
silently rather than error.

sample.md is rewritten with longer sections so the demo actually crosses
the 200-token summarization threshold; the previous version was short
enough that every node returned raw text and no summary was generated.

Tests: 2 new deterministic cases (no API key needed) covering summary
survival through build_tree_from_nodes and the tree-walk/title_path key
agreement. 7 passing.

Author: harshrathod0585

examples/documents/sample.md

@@ -2,37 +2,101 @@
 
 PageIndex turns long documents into a navigable tree of sections, each with a
 summary, so agents can reason over structure instead of flat chunks. This
-sample doc is used by the incremental update demo.
+sample document is used by the incremental update demo, and its sections are
+deliberately long enough that section summaries are generated by a model rather
+than passed through as raw text, which is what makes the incremental behaviour
+observable when only part of the document is edited.
 
 ## 1. What PageIndex Does
 
 PageIndex parses a PDF or Markdown file into a hierarchical structure of nodes.
-Each node holds a title, its text, and a generated summary. The tree lets a
-retrieval agent walk from the document root down to the exact section that
-answers a question, without embedding every chunk into a vector store.
+Each node holds a title, its own text span, and a generated summary. The tree
+lets a retrieval agent walk from the document root down to the exact section
+that answers a question, without embedding every chunk into a vector store and
+without relying on nearest-neighbour similarity to decide relevance.
+
+The practical consequence is that retrieval becomes an act of navigation rather
+than an act of matching. An agent reads the root description, decides which
+branch is plausible, reads that branch's summary, and descends. At every step
+the decision is legible: there is a title and a summary that explain why the
+branch was taken. When the agent lands on a leaf it has the full text of that
+section, not a windowed fragment that may have been cut mid-argument.
+
+This matters most for documents where meaning depends on position. A clause in
+a contract, a subsection of a policy manual, or a numbered requirement in a
+regulatory filing all derive part of their meaning from where they sit in the
+document. Flat chunking discards that placement. A tree preserves it, and the
+path from root to leaf is itself a piece of evidence the agent can cite.
 
 ## 2. Indexing
 
-Indexing builds the tree once. For Markdown, headings define the hierarchy; for
-PDFs, the table of contents and page layout are used. Every section is
-summarized, and the whole document gets a short description. The result is
-persisted in a workspace as JSON keyed by a document id.
+Indexing builds the tree once. For Markdown, headings define the hierarchy
+directly: each heading opens a node, and the heading level determines where
+that node attaches to its parent. For PDFs, the table of contents and the page
+layout are used instead, with a series of checks that verify the extracted
+table of contents actually corresponds to the physical pages of the document.
+
+Every section is then summarized, and the whole document is given a short
+description derived from the structure. Summarization is conditional: a section
+whose text falls below a token threshold is stored verbatim, on the grounds
+that a summary of a short passage costs a model call and returns something no
+more useful than the passage itself. Longer sections are sent to the model.
+
+The result is persisted in a workspace directory as JSON, keyed by a document
+identifier. Alongside the tree, the record stores a hash of the whole file and
+a map of per-section hashes. Those hashes are what make the next run cheap:
+they are the record of what the tree was built from, so a later run can compare
+against them instead of re-deriving the tree from scratch to find out whether
+anything moved.
 
 ## 3. Incremental Update
 
-When a document changes, PageIndex avoids rebuilding everything. It hashes the
-file and each section: if the file hash is unchanged the update is skipped
-entirely, and if only some sections changed, only those (plus their ancestors)
-are re-summarized. Unchanged sections reuse their cached summary.
+When a document changes, PageIndex avoids rebuilding everything. The update
+path applies two gates in sequence, and each gate that passes eliminates a
+larger amount of work than the one before it.
+
+The first gate is the file hash. If the hash of the file's current contents
+matches the hash recorded at index time, nothing in the document has changed,
+the update returns immediately with a status of unchanged, and no model call is
+made at all. This is the common case for a scheduled re-ingest over a corpus
+where most documents are static between runs.
+
+The second gate is the section diff. The file is re-parsed into sections, each
+section is hashed, and the new hash map is compared against the stored one.
+That comparison yields three sets: sections that are new, sections that were
+removed, and sections whose text changed in place. The changed and added
+sections are marked dirty, and the ancestors of each dirty section are added to
+the set as well, on the assumption that a parent's roll-up may be affected by
+what happened underneath it.
+
+Everything in that set is re-summarized. Everything outside it reuses the
+summary already stored on the previous tree. A two-page revision to a five
+hundred page manual therefore costs a handful of model calls rather than a
+full rebuild, and the cost scales with the size of the edit rather than with
+the size of the document.
 
 ## 4. Vectorless Retrieval
 
 Because the tree carries summaries at every level, an agent can retrieve by
-traversing the structure instead of doing nearest-neighbor search over
-embeddings. This keeps retrieval explainable and cheap to maintain.
+traversing the structure instead of doing nearest-neighbour search over
+embeddings. There is no index to build beyond the tree itself, no embedding
+model to keep consistent between ingest and query time, and no drift when the
+embedding model is upgraded underneath a corpus that was embedded with an
+older version.
+
+It also keeps retrieval explainable. A vector search returns a ranked list with
+a similarity score, and the score is not an explanation: it does not say why
+one passage outranked another, and it cannot be audited after the fact. A
+traversal returns a path, and the path is an explanation: this section, inside
+this chapter, inside this document, chosen because its summary matched what was
+asked. For compliance and audit workflows that difference is the point.
 
 ## Appendix: Key Methods
 
-`client.index(path)` builds the tree. `client.update(doc_id)` refreshes it
-incrementally. `client.get_doc_id_by_path(path)` resolves an existing document
-so the same file is never indexed twice.
+`client.index(path)` builds the tree for a document that has not been seen
+before and returns its document identifier. `client.update(doc_id)` refreshes
+an existing tree incrementally, applying the two gates described above and
+returning a dictionary describing which sections were updated, added, or
+deleted. `client.get_doc_id_by_path(path)` resolves an existing document by its
+source path, so that re-ingesting the same file finds the tree that already
+exists instead of minting a second identifier and orphaning the first.

pageindex/client.py

@@ -14,6 +14,7 @@
     extract_node_text_content,
     get_node_summary,
     build_tree_from_nodes,
+    split_summary_fields,
 )
 from .retrieve import get_document, get_document_structure, get_page_content
 from .utils import (
@@ -22,7 +23,6 @@
     hash_text,
     compute_section_hashes,
     find_ancestors,
-    structure_to_list,
     write_node_id,
     format_structure,
 )
@@ -292,12 +292,17 @@ def update(self, doc_id: str) -> dict:
         for path in dirty:
             to_summarize.update(find_ancestors(path))
 
-        # Reuse cached summaries for clean sections.
-        old_structure_flat = structure_to_list(doc.get('structure', []))
-        old_summary_map = {
-            n.get('title_path', n.get('title')): n.get('summary') or n.get('prefix_summary', '')
-            for n in old_structure_flat
-        }
+        # Reuse cached summaries for clean sections. The persisted tree has no
+        # title_path, so rebuild it to match the keys used by new_nodes.
+        old_summary_map = {}
+
+        def _collect_summaries(nodes, prefix=''):
+            for n in nodes:
+                path = f"{prefix} > {n['title']}" if prefix else n['title']
+                old_summary_map[path] = n.get('summary') or n.get('prefix_summary', '')
+                _collect_summaries(n.get('nodes', []), path)
+
+        _collect_summaries(doc.get('structure', []))
 
         async def _identity(val):
             return val
@@ -323,6 +328,7 @@ async def _regenerate():
 
         # Rebuild the tree with fresh node ids.
         new_structure = build_tree_from_nodes(new_nodes)
+        split_summary_fields(new_structure)
         write_node_id(new_structure)
         new_structure = format_structure(
             new_structure,

pageindex/page_index_md.py

@@ -29,6 +29,19 @@ async def generate_summaries_for_structure_md(structure, summary_token_threshold
     return structure
 
 
+def split_summary_fields(tree_nodes):
+    """Apply the same leaf/parent convention as generate_summaries_for_structure_md.
+
+    For callers that attach `summary` to a flat node list before the tree is
+    built, so parent nodes end up with `prefix_summary` as index() produces.
+    """
+    for node in tree_nodes:
+        if node.get('nodes'):
+            node['prefix_summary'] = node.pop('summary', '')
+            split_summary_fields(node['nodes'])
+    return tree_nodes
+
+
 def extract_nodes_from_markdown(markdown_content):
     header_pattern = r'^(#{1,6})\s+(.+)$'
     code_block_pattern = r'^```'
@@ -217,6 +230,10 @@ def build_tree_from_nodes(node_list):
             'line_num': node['line_num'],
             'nodes': []
         }
+        # Callers that summarize before building the tree (e.g. incremental
+        # update) would otherwise have their summaries dropped here.
+        if 'summary' in node:
+            tree_node['summary'] = node['summary']
         node_counter += 1
 
         while stack and stack[-1][1] >= current_level:

tests/test_incremental_update.py

@@ -10,7 +10,12 @@
 
 sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
 
-from pageindex.page_index_md import extract_nodes_from_markdown, extract_node_text_content
+from pageindex.page_index_md import (
+    extract_nodes_from_markdown,
+    extract_node_text_content,
+    build_tree_from_nodes,
+    split_summary_fields,
+)
 from pageindex.utils import compute_section_hashes, find_ancestors
 
 
@@ -65,6 +70,42 @@ def test_dirty_set_includes_ancestors():
     assert to_summarize == {"Root", "Root > A", "Root > A > A1"}, to_summarize
 
 
+def test_build_tree_preserves_summaries():
+    """update() attaches summaries before building the tree; they must survive."""
+    md = "# Root\nintro\n## A\nalpha\n## B\nbeta\n"
+    node_list, lines = extract_nodes_from_markdown(md)
+    nodes = extract_node_text_content(node_list, lines)
+    for n in nodes:
+        n["summary"] = f"S:{n['title_path']}"
+
+    tree = split_summary_fields(build_tree_from_nodes(nodes))
+    root = tree[0]
+    # Parents carry prefix_summary, leaves carry summary — as index() produces.
+    assert root["prefix_summary"] == "S:Root", root
+    assert "summary" not in root, root
+    assert [c["summary"] for c in root["nodes"]] == ["S:Root > A", "S:Root > B"]
+
+
+def test_tree_walk_paths_match_flat_title_paths():
+    """The cached-summary lookup keys the old tree by walking it; those paths
+    must equal the title_paths the new flat node list is keyed by."""
+    md = "# Root\nintro\n## A\nalpha\n### A1\nsub\n## B\nbeta\n"
+    node_list, lines = extract_nodes_from_markdown(md)
+    nodes = extract_node_text_content(node_list, lines)
+    tree = build_tree_from_nodes(nodes)
+
+    walked = []
+
+    def collect(ns, prefix=""):
+        for n in ns:
+            path = f"{prefix} > {n['title']}" if prefix else n["title"]
+            walked.append(path)
+            collect(n.get("nodes", []), path)
+
+    collect(tree)
+    assert walked == [n["title_path"] for n in nodes], walked
+
+
 if __name__ == "__main__":
     fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")]
     for fn in fns: