Summary
The Google Drive sync path still processes each file in a batch serially after embeddings are computed.
Evidence
- downloads files concurrently, but then loops through each file and performs , thumbnail generation, thumbnail upload, DB insert, and Google Drive move one file at a time.
- Other ingestion paths already use concurrent download helpers, so this path is now the clearest remaining storage-bound bottleneck.
Why this matters
For larger Google Drive batches, the worker spends a disproportionate amount of time in serial object-storage and Drive API operations.
Suggested direction
- Profile the post-embedding loop in
- Introduce bounded concurrency for storage uploads and/or post-upload moves where correctness allows
- Preserve duplicate handling and per-file error isolation
Summary
The Google Drive sync path still processes each file in a batch serially after embeddings are computed.
Evidence
Why this matters
For larger Google Drive batches, the worker spends a disproportionate amount of time in serial object-storage and Drive API operations.
Suggested direction