fix(personal): run directory indexing off the event loop

[StressTestor]

Summary

POST /api/personal/add_directory is an async def handler that called the fully synchronous rag.index_personal_documents inline, so the entire indexing job (os.walk, file reads, per-chunk embedding, Chroma inserts) ran on the event loop. Every other request queued behind it: indexing a real directory froze the UI and API for 25+ minutes with no sign of life (the reporter's users concluded the app crashed).

This PR moves the blocking section into the threadpool via fastapi.concurrency.run_in_threadpool. personal_docs_manager.add_directory stays inside the moved section because it triggers refresh_index(), which re-extracts text across tracked directories, also blocking work. A module-level lock serializes index jobs: without it the threadpool move would let concurrent requests run parallel jobs that race PersonalDocsManager's unsynchronized list mutations and plain open('w') file writes. Jobs previously serialized on the blocked loop anyway, so one-at-a-time is behavior parity, minus the freeze.

Target branch

• This PR targets dev, not main.

Linked Issue

Fixes #5558

Type of Change

• Bug fix (non-breaking — fixes a confirmed issue)

Checklist

• I searched open issues and open PRs — this is not a duplicate.
• This PR targets dev
• My changes are limited to the scope described above — no unrelated refactors or whitespace changes mixed in.
• I actually ran the app (uvicorn app:app + a local ChromaDB at localhost:8100) and verified the change works end-to-end.

How to Test

1. python -m pytest tests/test_add_directory_event_loop.py: 4 tests. The thread-identity test asserts indexing and bookkeeping run off the event loop thread (fails on dev, where both run on it); the serialization test asserts concurrent requests never run parallel index jobs; the other two pin response shape and the error path.
2. End-to-end (what i ran): start ChromaDB and the app, seed data/personal_docs/bigvault/ with 150 markdown files, POST /api/personal/add_directory, and while the job is running, time a GET /api/personal from a second terminal.
3. Results on this branch: the GET answers in 0.002s while indexing is in flight. Same probe on unpatched dev: the GET takes 3.2s, because it queues behind the whole remaining index job (with the reporter's 25-minute job, that GET waits minutes).
4. The index job itself completes identically on both: indexed_count: 150, failed_count: 0.

Also ran: full python -m pytest (4674 passed, 3 skipped; the 5 failures are the docker-socket tests, which fail identically on clean dev on this machine, macOS with no docker socket) and python -m compileall app.py core routes src services scripts tests.

Scope notes

• This is the issue's option 2 (threadpool), not option 3 (background job with id + progress). The request still blocks the client for the duration, but the app stays responsive for everyone. Option 3 is a feature-sized change (new endpoints + Knowledge UI work); happy to take it as a follow-up if wanted.
• A long index job still holds the HTTP request open, so aggressive proxy timeouts can drop the response while the job completes server-side. Pre-existing behavior, unchanged here.
• The lock covers add_directory jobs only. Now that the app stays responsive during indexing, an admin hitting POST /reload or DELETE /remove_directory mid-job could race the indexing thread's bookkeeping, a window that could not occur before because the blocked loop prevented any dispatch. Admin-only and narrow; the clean fix is a lock inside PersonalDocsManager itself, which i kept out of this PR to keep the diff reviewable. Can follow up.