Skip to content

feat: add HEAD prefetch cache, and some cleanup - #91

Merged
definfo merged 7 commits into
sjtug:masterfrom
definfo:s3-cache
Jun 14, 2026
Merged

feat: add HEAD prefetch cache, and some cleanup#91
definfo merged 7 commits into
sjtug:masterfrom
definfo:s3-cache

Conversation

@definfo

@definfo definfo commented Jun 12, 2026

Copy link
Copy Markdown
Contributor
  • add mocking test for smart cache following upstream 404 resp
  • clean unnecessary FnMut and clippy directives
  • simplify error variants, remove unused async allow directives
  • add HEAD prefetch cache

Close #72

@definfo definfo changed the title chore: cleanup feat: add HEAD prefetch cache, and some cleanup Jun 12, 2026
@definfo
definfo requested a review from Copilot June 14, 2026 03:34

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a HEAD-prefetch-based cache to improve “smart-cache” routing decisions (notably returning 404 for genuinely missing objects, per #72) while also cleaning up error variants, clippy directives, E2E fixtures, and Nix build plumbing.

Changes:

  • Add a Moka-backed prefetch cache that probes S3 + upstream via HEAD and caches positive/negative existence results.
  • Update routing/tests/E2E fixtures to validate 404 behavior and upstream request counts.
  • Simplify Rust error variants and refactor Nix flake packaging (removing nix/crane.nix).

Reviewed changes

Copilot reviewed 15 out of 17 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
tests/e2e/simple.sh Improves E2E assertions (better failure messages) and adds scenarios validating HEAD-prefetch behavior + upstream call counts.
tests/e2e/fake_services.py Adds request counting endpoint and upstream fixtures to support the new E2E validations.
src/s3_cache.rs New prefetch cache module (Moka) that performs HEAD probes and caches existence results.
src/repos.rs Integrates prefetch cache into RouteAction::Cache handling to return 404 when probes confirm missing.
src/common.rs Extends IntelMission and Config to carry prefetch client/cache and TTL configuration.
src/main.rs Wires new prefetch client/cache into runtime initialization and adds TTL handling.
src/utils.rs Updates error variant usage and adjusts tests for the new IntelMission fields.
src/queue.rs Updates tests to construct IntelMission with prefetch client/cache.
src/storage.rs Updates simplified error variant usage.
src/browse.rs Updates simplified error variant usage.
src/artifacts.rs Updates simplified error variant usage and related matching logic.
nix/crane.nix Removed (flake now inlines crane args).
flake.nix Inlines crane args and tweaks build inputs/check behavior; minor comment typo noted.
Cargo.toml Adds moka dependency for the prefetch cache.
Cargo.lock Locks new transitive dependencies for moka.
.gitignore Ignores Python __pycache__/ for E2E helpers.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/s3_cache.rs
Comment thread src/repos.rs
Comment thread src/repos.rs
Comment thread flake.nix
@definfo
definfo merged commit 53af9b0 into sjtug:master Jun 14, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Return 404 for non-existent files instead of 403

2 participants