Summary
index_personal_documents walks the target directory with no hidden-directory exclusion, and DEFAULT_FILE_EXTENSIONS includes .js, .json, .css, .html. Result: pointing RAG at any real-world folder silently indexes tool internals — an Obsidian vault's .obsidian/ (minified plugin JS), a git repo's .git/ (hooks, packed-refs samples), .vscode/, node_modules-adjacent caches, etc. This both multiplies indexing time and pollutes retrieval with junk chunks.
Where
src/rag_vector.py::index_personal_documents — for root, _, files in os.walk(directory) never prunes directories; every file passing the extension filter is chunked and embedded into all lanes.
Impact observed
Mounting an Obsidian vault (159 markdown files, ~840KB of actual content) also swept in .obsidian/: 2.6MB of minified plugin JavaScript, which chunks into thousands of embeddings (×2 lanes). Real content was a rounding error next to the junk: the index run went from an expected ~3 minutes to 40+, and semantic search over "my notes" would have returned fragments of minified plugin code. (Compounded by the companion issue that indexing blocks the event loop, this presented as a 40-minute app freeze.)
Our workaround: tmpfs-mount empty directories over <dir>/.git and <dir>/.obsidian in the container so the walker can't see them — functional, but not something users should need.
Suggested fix
Prune hidden directories in the walk by default:
for root, dirs, files in os.walk(directory):
dirs[:] = [d for d in dirs if not d.startswith('.')]Optionally also skip hidden files, and expose an include_hidden flag on POST /api/personal/add_directory for the rare user who wants them. A default exclusion list for well-known junk dirs (node_modules, __pycache__, .venv) would be a nice bonus but the dot-dir prune alone fixes the common cases.
Environment
Odysseus v1.0.2 (docker compose, main); os.walk loop verified unchanged on dev HEAD 2026-07-15.
Summary
index_personal_documentswalks the target directory with no hidden-directory exclusion, andDEFAULT_FILE_EXTENSIONSincludes.js,.json,.css,.html. Result: pointing RAG at any real-world folder silently indexes tool internals — an Obsidian vault's.obsidian/(minified plugin JS), a git repo's.git/(hooks, packed-refs samples),.vscode/,node_modules-adjacent caches, etc. This both multiplies indexing time and pollutes retrieval with junk chunks.Where
src/rag_vector.py::index_personal_documents—for root, _, files in os.walk(directory)never prunes directories; every file passing the extension filter is chunked and embedded into all lanes.Impact observed
Mounting an Obsidian vault (159 markdown files, ~840KB of actual content) also swept in
.obsidian/: 2.6MB of minified plugin JavaScript, which chunks into thousands of embeddings (×2 lanes). Real content was a rounding error next to the junk: the index run went from an expected ~3 minutes to 40+, and semantic search over "my notes" would have returned fragments of minified plugin code. (Compounded by the companion issue that indexing blocks the event loop, this presented as a 40-minute app freeze.)Our workaround: tmpfs-mount empty directories over
<dir>/.gitand<dir>/.obsidianin the container so the walker can't see them — functional, but not something users should need.Suggested fix
Prune hidden directories in the walk by default:
Optionally also skip hidden files, and expose an
include_hiddenflag onPOST /api/personal/add_directoryfor the rare user who wants them. A default exclusion list for well-known junk dirs (node_modules,__pycache__,.venv) would be a nice bonus but the dot-dir prune alone fixes the common cases.Environment
Odysseus v1.0.2 (docker compose, main);
os.walkloop verified unchanged ondevHEAD 2026-07-15.