feat: add HEAD prefetch cache, and some cleanup - #91
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces a HEAD-prefetch-based cache to improve “smart-cache” routing decisions (notably returning 404 for genuinely missing objects, per #72) while also cleaning up error variants, clippy directives, E2E fixtures, and Nix build plumbing.
Changes:
- Add a Moka-backed prefetch cache that probes S3 + upstream via HEAD and caches positive/negative existence results.
- Update routing/tests/E2E fixtures to validate 404 behavior and upstream request counts.
- Simplify Rust error variants and refactor Nix flake packaging (removing
nix/crane.nix).
Reviewed changes
Copilot reviewed 15 out of 17 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/e2e/simple.sh | Improves E2E assertions (better failure messages) and adds scenarios validating HEAD-prefetch behavior + upstream call counts. |
| tests/e2e/fake_services.py | Adds request counting endpoint and upstream fixtures to support the new E2E validations. |
| src/s3_cache.rs | New prefetch cache module (Moka) that performs HEAD probes and caches existence results. |
| src/repos.rs | Integrates prefetch cache into RouteAction::Cache handling to return 404 when probes confirm missing. |
| src/common.rs | Extends IntelMission and Config to carry prefetch client/cache and TTL configuration. |
| src/main.rs | Wires new prefetch client/cache into runtime initialization and adds TTL handling. |
| src/utils.rs | Updates error variant usage and adjusts tests for the new IntelMission fields. |
| src/queue.rs | Updates tests to construct IntelMission with prefetch client/cache. |
| src/storage.rs | Updates simplified error variant usage. |
| src/browse.rs | Updates simplified error variant usage. |
| src/artifacts.rs | Updates simplified error variant usage and related matching logic. |
| nix/crane.nix | Removed (flake now inlines crane args). |
| flake.nix | Inlines crane args and tweaks build inputs/check behavior; minor comment typo noted. |
| Cargo.toml | Adds moka dependency for the prefetch cache. |
| Cargo.lock | Locks new transitive dependencies for moka. |
| .gitignore | Ignores Python __pycache__/ for E2E helpers. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- add mocking test for smart cache following upstream 404 resp - clean unnecessary FnMut and clippy directives
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Close #72