Optimize blob download and caching with hash verification#14
Merged
Conversation
The GET proxy path hashed every downloaded chunk but never read the digest, so all that SHA-256 work was wasted CPU, and the HEAD proxy path cached blobs without checking them at all. Both now verify the hash in the same pass as the download (no second read from flash) and refuse to cache mismatching content, which also avoids serving a bad blob forever and re-downloading it. Copies also move from 8 KiB to 64 KiB buffers, cutting loop iterations, stream syscalls and digest calls per blob by ~8x, and the blocking OkHttp calls now run on Dispatchers.IO instead of parking a Ktor CIO event-loop thread for the whole download. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY7qfJK7UMnQuLyb3jjeQ2
Every downloaded blob triggered two full directory scans (a stat per cached file each) via pruneIfNeeded + updateSize. The cache size is now tracked incrementally under a lock, scanned once at startup, and the directory is only listed when the tracked size actually crosses the 1 GB prune limit (which also re-syncs the counter). Also reuse a single Tika instance instead of loading the MIME-type registry on every detectMimeType call. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY7qfJK7UMnQuLyb3jjeQ2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR improves the performance and reliability of blob downloads and caching by implementing single-pass hash verification during download, increasing buffer sizes, and optimizing the file store's size tracking and pruning logic.
Key Changes
CustomHttpServer.kt:
Dispatchers.IOcontext for blocking I/O operations (fetchAndSaveandstreamFromServer) to prevent blocking Ktor's event-loop threadstoHex()utility function for efficient byte array to hex string conversionstreamFromServerto only cache blobs if their hash matches the requested hashAndroidFileStore.kt:
Tikainstance creation to class initialization (single instance) instead of creating it per MIME type detectionpruneIfNeeded()to only scan the directory when the size limit is actually exceededMAX_CACHE_SIZE_BYTES,PRUNE_TARGET_SIZE_BYTES)moveFile()to properly track size changes when replacing existing filesNotable Implementation Details
https://claude.ai/code/session_01EY7qfJK7UMnQuLyb3jjeQ2