Repository navigation
fix(search): iterative vscode field walk — deep content nesting no longer aborts index - #83
Conversation
github-actions
Bot
added
the
documentation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| if any(r.is_dir() for r in copilot_roots): | ||
| _index_copilot_dirs(con, copilot_roots, stats) | ||
| if codex_dir is not None and Path(codex_dir).is_dir(): | ||
| _index_codex_root(con, Path(codex_dir), stats) | ||
| vscode_dirs = ( | ||
| [Path(d).expanduser() for d in vscode_chat_dirs] | ||
| if vscode_chat_dirs is not None | ||
| else [] | ||
| ) | ||
| if any(d.is_dir() for d in vscode_dirs): | ||
| _index_vscode_chat_dirs(con, vscode_dirs, stats) |
There was a problem hiding this comment.
🔴 Disabled sources lose indexed history
When build_index receives a disabled source, it deletes that source's existing docs instead of leaving them untouched. The CLI passes None for each --no-* flag, so later queries lose previously indexed history.
Learn more
A source argument of None means the caller did not select that collector. The CLI source selection also uses None for explicit --no-* flags, while empty default directory lists mean auto-detection found no roots. The new branches call _drop_vanished_files with an empty present set for both situations. That removes all documents and watermarks for the source, even if its underlying stores still exist. The same behavior applies to Copilot, Codex and VS Code.
Example: Index Copilot and Codex into one database. Run devin-search index --sessions-db sessions.db --no-copilot --no-codex against it. Both sources' search hits disappear although neither store changed.
Recommended fix: Distinguish a disabled source (None) from an enabled source with no existing roots ([] or a missing explicit path). Call _drop_vanished_files only for enabled sources whose roots truly vanished. Preserve existing docs and markers for disabled sources; cover both transitions with tests.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Fixed in 5d6db3d. build_index now removes stale docs and markers for a configured source whose roots all vanished, even though its collector is skipped. Covered by test_vanished_source_root_drops_stale_docs.
| except (json.JSONDecodeError, RecursionError): | ||
| bad_lines += 1 # truncated tail or malformed mid-line |
There was a problem hiding this comment.
🔍 Deep JSON recovery lacks regression tests
No new test exercises RecursionError from JSON parsing, despite handlers across the collectors. Add deep-JSON fixtures to cover the recovery paths.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Done in 426dcf0 — three fixtures now drive json.loads past the recursion limit (200k-deep JSON as a copilot JSONL line, a codex rollout line, and vscode snapshot + mutations files) and assert sibling records still index. test_copilot_deep_json_line_skipped, test_codex_deep_rollout_line_skipped, test_vscode_deep_json_files_skipped.
The previous fix made deep-JSON records fail soft, but no fixture drove json.loads past the recursion limit, so the recovery paths were untested (devin-review analysis on #83). New tests inject a 200k-deep JSON value as a copilot JSONL line, a codex rollout line and vscode snapshot/mutations files, and assert sibling records still index. Generated with [Devin](https://devin.ai) & QA Engineer Icaro Galvao Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Rebase-port of the stale #83 branch: main landed the collector rewrite (#82), content digests and --tool while the PR was open, so its net delta was re-extracted rather than merged literally. - _vscode_field_text is iterative — deeply nested content fields no longer RecursionError the index run - json.loads RecursionError treated as JSONDecodeError across vscode snapshot/mutations, copilot and codex rollout parsing - copilot non-string `type` records skipped instead of TypeError - codex threads docs re-index on cwd/created_at metadata changes - managed-but-gone copilot/codex/vscode roots sweep stale docs — completes the vanished-source contract documented in 0.2.1 (`--no-<surface>` still preserves: None means unmanaged, not vanished) Generated with [Devin](https://devin.ai) & QA Engineer Icaro Galvao Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
426dcf0 to
5151280
Compare
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Rebase-port of the stale #83 branch: main landed the collector rewrite (#82), content digests and --tool while the PR was open, so its net delta was re-extracted rather than merged literally. - _vscode_field_text is iterative — deeply nested content fields no longer RecursionError the index run - json.loads RecursionError treated as JSONDecodeError across vscode snapshot/mutations, copilot and codex rollout parsing - copilot non-string `type` records skipped instead of TypeError - codex threads docs re-index on cwd/created_at metadata changes - managed-but-gone copilot/codex/vscode roots sweep stale docs — completes the vanished-source contract documented in 0.2.1 (`--no-<surface>` still preserves: None means unmanaged, not vanished) Generated with [Devin](https://devin.ai) & QA Engineer Icaro Galvao Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
5151280 to
6c6891f
Compare
json.loads of a 5000-deep doc RecursionErrors on 3.11 before the walk ever runs, so the file-level variant could only pass on 3.12+. The unit-level call tests the iterative fix on every supported Python; the deep-JSON file path is covered by test_vscode_deep_json_files_skipped. Generated with [Devin](https://devin.ai) & QA Engineer Icaro Galvao Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Summary
_vscode_field_textrewritten iterative — a chat doc with hundreds of nestedcontentfields hitRecursionErrorand aborted the whole index run (devin-review finding on feat(search): multi-source collectors — copilot-cli, codex-cli, vscode-chat #82, which merged while the fix was being prepared).\n-join is associative over the flat leaf sequence, traversal order preserved (lists in order, dicts in_VSCODE_CONTENT_KEYSorder, empties dropped).contentdoc indexes and searches fine.Test plan
pytest tests/test_collectors.py— 29 passed incl.test_vscode_deep_content_nesting_indexesGenerated with Devin