Query extensions: HTTP cache, value-carrying persistence, server-wins policy - #5
Merged
Conversation
Introduce Fetched<T> in the core layer (data + an optional server-derived
CachePolicy) and a use_query_with_policy / fetch_query_with_policy fetcher
variant that lets a fetcher override the caller's per-query CachePolicy on
success ("server wins"), per Core Change 1 of docs/features.md.
Generalizes the single retry loop via a private FetchedLike<T> adapter
(impls for T and Fetched<T>) so the existing Result<T,E> fetch paths are
unchanged (server_policy = None) while the new variant applies the server
policy after complete_success. Non-breaking and additive.
Tests: 4 new with_policy tests (override, none-keeps-caller, SWR override,
refetch override). All 751+7+21 existing tests still pass; clippy clean.
New companion crate crates/gpui-query-http (Phase C1 of docs/features.md): cache_policy_from_headers(&http::HeaderMap) -> Result<CachePolicy, ParseError> parses Cache-Control (no-store/no-cache/max-age/s-maxage/stale-while-revalidate, case-insensitive, quoted values, s-maxage precedence) into gpui-query's CachePolicy. Plus CacheMeta (serde) for future ETag/Last-Modified persistence, and ParseError (thiserror). Depends on gpui-query core-only (no GPUI), so the core crate stays free of reqwest/http/bytes (Guiding Principle 1). Completes the server-wins arc alongside Phase A's Fetched::with_policy: a fetcher parses response headers into a CachePolicy and returns Fetched::with_policy(data, policy). 13 unit tests + 1 doctest; #![deny(missing_docs)]; clippy + fmt clean.
Phase C2 of the gpui-query HTTP layer. HttpCache is generic over a
new HttpBackend trait that abstracts a single conditional GET, so the
cache is not hardcoded to any request library.
New public API (gpui-query-http, default features):
- HttpBackend trait (non-object-safe; -> impl Future + Send) with
associated Error type and fetch(url, conditionals) method.
- Conditionals { if_none_match, if_modified_since } with a
from_meta(cached) constructor.
- BackendResponse { status: u16, headers: http::HeaderMap,
body: bytes::Bytes } — owned, library-agnostic.
- HttpCache<B: HttpBackend> { new, async fetch } keyed by URL string,
with std::sync::Mutex-guarded meta + bodies maps (guards never held
across .await). Fresh entries short-circuit the backend; 304 serves
the cached body; 200 parses cache_policy_from_headers and stores.
- HttpError (thiserror): Backend { #[source] }, InvalidPolicy(#[from]
ParseError), NotModifiedWithoutCachedBody { url }.
- Policy <-> CacheMeta Duration conversion helpers
(fresh_for/stale_for_from_policy, policy_from_meta).
Optional reqwest backend behind the new opt-in 'reqwest' feature:
- ReqwestBackend(reqwest::Client) implementing HttpBackend.
- reqwest added as optional dep (rustls-tls, default-features off).
- bytes = "1" always; tokio dev-dep for #[tokio::test].
Tests (MockBackend returning canned BackendResponse queue): 200 stores
body+meta and returns server policy; fresh second fetch short-circuits
(no backend call); 304 returns cached body; no-store returns NoCache
and stores nothing. 13 existing C1 tests still pass (17 total).
cargo test (default), cargo build --features reqwest, cargo clippy
--all-targets --features reqwest, and cargo fmt --check all clean.
…ePersister
Phase B of the query-extensions effort (docs/features.md). Adds an opt-in
`persist` feature gating the shipped metadata-only skeleton and layers a
richer async, value-carrying persistence surface on top, plus a reference
disk adapter.
B1 — feature gate:
- new `persist = ["client", "dep:serde_json", "dep:thiserror"]` feature
- extract `current_time_ms` into `client/time.rs` (ungated; GC uses it)
- gate `QueryPersister`, `DehydratedEntry`/`DehydratedState`,
`collect_key_status_into` (all 3 erased-bucket traits + impls), and
`dehydrate`/`hydrate`/`persist`/`restore` behind the feature
- re-gate the dehydrate/persist-using tests so the default build stays green
B2 — precise dirty signal (OQ2 -> Option B):
- new `client/mutation_signal.rs` with `CacheMutation` marker Global
(`Default`, infallible bump via `default_global`)
- bump it at the 3 completion sites (fetch_retry, mutation internals,
infinite fetch_runners) + `QueryClient::set_query_data`
B3 — async persister (`client/persist.rs`):
- `Persister` trait (non-object-safe, `impl Future + Send`)
- `PersistedEntry` (value: serde_json::Value), `PersistSnapshot`,
`PersistFilter` (owned), `PersistOptions`, `PersistHandle`,
`PersistError` (thiserror), `PERSIST_VERSION`
- typed serializer + deserializer registries on `QueryClient` (TypeId-keyed,
no `T: Serialize` bound leak); `collect_persistable_into` on the erased
buckets serializes only Success entries with a registered serializer
- `persist_with` debounces via `observe_global::<CacheMutation>` +
`background_executor().timer`, coalesces via a shared snapshot slot
- `hydrate` async fn re-primes via the deserializer registry + set_query_data
B4 — `gpui-query-persist` crate:
- `FilePersister` with atomic writes (NamedTempFile -> fsync -> F_FULLFSYNC on
macOS -> persist/rename -> parent-dir fsync on POSIX), tolerant load
(missing=empty, corrupt=empty+log, version=typed VersionMismatch), Mutex
- `PersistFormat::{Json, Bincode}` (bincode adapts serde_json::Value via
JSON-string wrapping since bincode can't drive deserialize_any)
- `in_cache_dir` (dirs::cache_dir, OQ8), explicit-path, NoopPersister re-export
- 7 tests: json/bincode round-trip, corrupt tolerance, version reject,
concurrent saves, noop, format choice
Verification (all green): build/test default (532), persist (535), core-only
(417), gpui-query-persist (7); clippy clean for new code; cargo fmt --all clean.
The `cargo fmt --all` pass also normalized pre-existing whitespace
violations repo-wide (the tree was not fmt-clean before this work).
Fixes surfaced by building under the combined `hook persist` feature set (which the initial Phase B work had not exercised) and by running the persist_with/hydrate integration tests: - Dirty-signal bump sites referenced the private `mutation_signal` module path; use the re-exported `crate::client::CacheMutation` (E0603). - `persist_with` now creates the CacheMutation marker before registering the observer, so the dirty-signal notifications are guaranteed to wake it. - Serializer-registry lookup keyed on `TypeId::of::<(T,E)>()` but `register_serializer::<T,E>` stores at `TypeId::of::<T>()`; align the bucket lookups (`erased_ops.rs`, `infinite_bucket.rs`) on `T`, since serialization depends on the data type, not the error type. - persist_with_hydrate tests: use a harness holding the `Entity` across `run_until_parked` (the bucket stores only WeakEntity, so an unheld entity is dropped before the async observer/hydrate-read runs); fix `apply_success` epoch-ms (was 1_000 = 1970, so entries were filtered by max_age); correct the max_age=0 assumption (0 = disabled, not 'all too old'); zero debounce (the TestAppContext mock clock doesn't advance wall-clock timers); drive async hydrate via `cx.update` not the non-existent `AsyncApp::run`. The implementation was correct throughout; these were build/test-structure issues. All 7 persist_with_hydrate tests pass; full workspace green across default / core / hook / persist / all-features (758+ tests, 0 failures).
- Surface mutex poisoning as HttpError::Poisoned instead of panicking - Short-circuit on no-store/no-cache to avoid masking parse errors - Propagate per-key meta through Fetched into persisted snapshots - Add large snapshot and corrupt bincode round-trip tests
imperative fetch completion
After Audit + adversarial Verify, sequentially fix each verified-real
issue per track, then have an independent agent re-read the source to
confirm the fix landed and introduced no regressions.
- New FIX_SCHEMA / REVERIFY_SCHEMA and fix/reverify prompt builders.
- Fix agents run sequentially, not in parallel or worktrees: tracks
overlap on shared files (persist.rs, fetched.rs, Cargo.toml, lib.rs),
so concurrent edits would be lost (each Edit re-reads state) and
worktree changes don't merge back into the working tree.
- Pass { fix: false } via Workflow args to run audit/verify only.
- Workflow now returns { audit, fixes, fixApplied }.
The features design doc and the website planning doc were working notes, not user-facing documentation. The extensions they described have shipped and are covered by the tests and the docs under website/.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges the query-extensions work into master.
Crate changes
gpui-query-httpcrate: a library-agnosticHttpCacheover anHttpBackendtrait, pluscache_policy_from_headersfor Cache-Control,ETag, and Last-Modified parsing. reqwest is an optional backend.
Fetched<T>and the*_with_policyhooks, so a fetcher can return theserver's cache policy and have the resource adopt it (server wins).
persistfeature: an asyncPersister, aPersistedEntrythat carries the cached value and metadata,a debounced
persist_withsubscription driven by theCacheMutationsignal, typed
hydrate, and snapshot versioning. The referenceFilePersistermoves into a newgpui-query-persistcrate.serializereturnsOption, and imperative fetchcompletion now notifies the cache-mutation signal.
Web
Docs
Tests:
cargo test --workspace --all-featurespasses (820).