fix(rag): skip hidden and junk directories when indexing

[StressTestor]

Summary

index_personal_documents walks the target directory with no pruning, and DEFAULT_FILE_EXTENSIONS includes .js, .json, .css, .html. Point RAG at a real-world folder and it indexes tool internals: an Obsidian vault's .obsidian/ (2.6MB of minified plugin JS in the reporter's case), a repo's .git/, node_modules/, __pycache__/. Indexing time multiplies and retrieval fills with junk chunks.

This PR prunes hidden directories and a small junk list (node_modules, __pycache__, venv) from the walk in place, and skips hidden files. The explicitly passed root is exempt, so a user who deliberately indexes a hidden directory still gets its contents. Plus 3 hermetic regression tests (same VectorRAG.__new__ pattern as test_rag_remove_directory_scope.py); all 3 fail on dev without the fix.

Target branch

• This PR targets dev, not main.

Linked Issue

Fixes #5559

Type of Change

• Bug fix (non-breaking — fixes a confirmed issue)

Checklist

• I searched open issues and open PRs — this is not a duplicate.
• This PR targets dev
• My changes are limited to the scope described above — no unrelated refactors or whitespace changes mixed in.
• I actually ran the app (uvicorn app:app + a local ChromaDB at localhost:8100) and verified the change works end-to-end.

How to Test

1. python -m pytest tests/test_rag_index_hidden_dirs.py: 3 tests covering hidden dirs, hidden files, junk dirs, and the explicit-hidden-root exemption. They fail on dev without the fix.

2. End-to-end (what i ran): start ChromaDB and the app, then create a directory under data/personal_docs/ containing real content plus junk dirs:

   scratch-vault/
     readme.md                      <- real
     notes/meeting.md               <- real
     .obsidian/plugins/main.js      <- junk
     .git/config.json               <- junk
     node_modules/lib/index.js      <- junk
     __pycache__/cached.py          <- junk
     venv/lib/site.py               <- junk
   

3. POST /api/personal/add_directory with that path. Response on this branch: indexed_count: 2, failed_count: 0.

4. Confirm in the vector store that only the two real files landed. Querying the odysseus_rag_fastembed collection's metadatas after the call returns exactly notes/meeting.md and readme.md as sources, none of the junk paths. On dev the same tree also embeds the .obsidian/, .git/, node_modules/, __pycache__/, and venv/ files.

Also ran: full python -m pytest (4673 passed, 3 skipped; the 5 failures are the docker-socket tests, which fail identically on clean dev on this machine, macOS with no docker socket) and python -m compileall app.py core routes src services scripts tests.

Scope notes

• Chunks already indexed from junk dirs are not cleaned up by this change. Remove and re-add the directory to rebuild it clean.
• Hidden files are now skipped with no opt-out. The issue marks include_hidden as optional; adding it means plumbing a flag through the route, PersonalDocsManager, RAGManager, and the UI, so i left it out to keep this small. Can follow up if wanted.
• The junk list is deliberately minimal. dist/, build/, target/ could join it, but those names collide with legitimate content folders more often.