Add optional liquid cache for indexed parquet scans - #211
Closed
alchemist51 wants to merge 1 commit into
Closed
Conversation
alchemist51
force-pushed
the
liquid-cache-seam
branch
3 times, most recently
from
July 28, 2026 09:47
a2a45b0 to
87d59f6
Compare
alchemist51
force-pushed
the
liquid-cache-seam
branch
from
July 28, 2026 09:52
87d59f6 to
180b0ea
Compare
Adds an opt-in, in-memory decoded-batch cache (liquid cache) for the DataFusion analytics backend, wired behind a cargo feature so it is absent from the default build. When the `liquid_cache` feature is off (the default), the integration is an inlined no-op: eligible scans use the plain ParquetSource unchanged. When on, an eligible indexed row-group scan — all projected columns numeric/date/timestamp/boolean, within a column budget, and no string or binary predicate column — is wrapped with LiquidParquetSource so decoded batches are served from and populated into the process-global cache. Enabling is a single-file edit plus a build flag: the liquid cache dependency is a commented placeholder in the analytics-backend-datafusion crate (uncomment it, set the git repo/branch, and uncomment the matching `dep:` feature entries), then build with -PliquidCache (or LIQUID_CACHE=1). The default build declares no liquid cache source, so it resolves and compiles exactly as before. Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>
alchemist51
force-pushed
the
liquid-cache-seam
branch
from
July 28, 2026 09:56
180b0ea to
c51e11c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an opt-in in-memory decoded-batch cache (liquid cache) for the DataFusion analytics backend, wired behind a cargo feature so it is completely absent from the default build. Liquid cache lives outside this repo — it's pulled in only when a user explicitly points the build at a liquid cache source.
Behaviour
ParquetSourceunchanged. No liquid cache dependency is declared, resolved, fetched, or compiled. The build behaves exactly as before.LiquidParquetSource, so decoded batches are served from and populated into the process-global cache.How to enable
Two steps — a single-file edit plus a build flag:
sandbox/plugins/analytics-backend-datafusion/rust/Cargo.toml, uncomment the liquid cache dependency lines, set the git repo + branch to your liquid cache source, and uncomment the matchingdep:entries on theliquid_cachefeature (all three edits are adjacent, with inline instructions).-PliquidCache(orLIQUID_CACHE=1).For local development against a checkout, redirect the git dep with a workspace
[patch].Why a commented placeholder instead of a committed dependency
Cargo resolves (and fetches) every declared optional dependency even when its feature is off — only compilation is feature-gated. A committed git URL would therefore be fetched on every default build, and a placeholder/unreachable URL would break it (an invalid URL even fails manifest parsing). Leaving the dependency commented out is what keeps the default build free of any liquid cache source while still making enablement a one-file edit.
Changes
6 files, additive (
Cargo.lockintentionally not committed; the repo's Rust build runs unlocked and regenerates it):analytics-backend-datafusion/rust/src/liquid_cache.rs— feature-gated module.maybe_wrap_parquet_sourceis a no-op when off; when on, applies the eligibility gate and wraps the source. Holds the process-global cache reference.analytics-backend-datafusion/rust/src/indexed_table/parquet_bridge.rs— 12-line hunk routing the parquet source throughmaybe_wrap_parquet_source.analytics-backend-datafusion/rust/Cargo.toml— commented placeholder dependency +liquid_cachefeature; all liquid-cache config consolidated here.dataformat-native/rust/Cargo.toml— pointer note (config lives in the plugin crate).dataformat-native/build.gradle—-PliquidCache/LIQUID_CACHE=1toggles the cargo feature.analytics-backend-datafusion/rust/src/lib.rs— module declaration.Testing
cargo check -p opensearch-datafusionclean,cargo fmt --checkclean, no liquid cache source resolved.cargo metadataresolves and pulls the liquid cache crates.Notes / follow-ups