[Analytics] Wire liquid cache into the analytics engine (plugin + integration) - #22595
Open
cocosz wants to merge 3 commits into
Open
[Analytics] Wire liquid cache into the analytics engine (plugin + integration)#22595cocosz wants to merge 3 commits into
cocosz wants to merge 3 commits into
Conversation
Contributor
PR Code Analyzer ❗AI-powered 'Code-Diff-Analyzer' found issues on commit 33567f0. ⛔ Hard block: Issues at Medium severity or above will block this PR from merging. 'Diff too large, requires skip by maintainers after manual review' Pull Requests Author(s): Please update your Pull Request according to the report above. Repository Maintainer(s): You can Thanks. |
cocosz
force-pushed
the
liquid-cache-plugin-integration
branch
from
July 28, 2026 23:58
8a397b0 to
e8ee755
Compare
This was referenced Jul 29, 2026
cocosz
force-pushed
the
liquid-cache-plugin-integration
branch
6 times, most recently
from
July 29, 2026 00:59
fda62fc to
eb2ff19
Compare
added 3 commits
July 29, 2026 06:33
Adds an opt-in, in-memory decoded-batch cache (liquid cache) for the DataFusion analytics backend, wired behind a cargo feature so it is absent from the default build. When the `liquid_cache` feature is off (the default), the integration is an inlined no-op: eligible scans use the plain ParquetSource unchanged. When on, an eligible indexed row-group scan — all projected columns numeric/date/timestamp/boolean, within a column budget, and no string or binary predicate column — is wrapped with LiquidParquetSource so decoded batches are served from and populated into the process-global cache. Enabling is a single-file edit plus a build flag: the liquid cache dependency is a commented placeholder in the analytics-backend-datafusion crate (uncomment it, set the git repo/branch, and uncomment the matching `dep:` feature entries), then build with -PliquidCache (or LIQUID_CACHE=1). The default build declares no liquid cache source, so it resolves and compiles exactly as before. Signed-off-by: Tanvir Alam <tanvralm@amazon.com>
Adds the in-memory decoded-batch cache (opensearch-liquid-cache-core) and its DataFusion parquet reader (opensearch-liquid-cache-datafusion) as a self-contained Cargo workspace under sandbox/plugins/liquid-cache/src/main/rust. These crates build and unit-test on their own and are not yet wired into the engine; the analytics backend compiles them in only when built with the liquid_cache cargo feature, which a follow-up change enables. Kept as its own PR so the vendored library can be reviewed in isolation. Signed-off-by: Tanvir Alam <tanvralm@amazon.com>
Adds the Java plugin (settings, stats, clear, native bridge) and the top crate (runtime + lc_* control surface), activates the optional liquid-cache dependency behind the liquid_cache feature, and wires engagement on both scan paths: the listing path via the cache's optimizer on the session, and the indexed path via the maybe_wrap hook. Also marks the analytics plugin ExtensiblePlugin so this plugin can attach, adds the feature flag, and the end-to-end LiquidCacheIT. Signed-off-by: Tanvir Alam <tanvralm@amazon.com>
cocosz
force-pushed
the
liquid-cache-plugin-integration
branch
from
July 29, 2026 01:05
eb2ff19 to
33567f0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Turns the liquid cache on: wires the vendored library into the analytics engine and exposes it as an OpenSearch plugin, gated behind the
liquid_cachecargo feature and thedatafusion.liquid_cachefeature flag.Because the earlier PRs already landed their files in final form, this PR is purely additive plus two small wiring flips:
opensearch-liquid-cachepath dependency in the analytics backendCargo.toml, and adds the top crate as a member of the liquid-cache workspace. The engine hook (liquid_cache.rs) is not touched here — it already shipped in its final form in [Analytics] Add optional liquid cache seam for indexed parquet scans (cargo feature) #22553.ffi.rs,lib.rs,runtime.rs— the process-global runtime, engagement policy, and thelc_*C FFI.LiquidCachePluginbootstraps the runtime,LiquidCacheBridgebinds thelc_*symbols,LiquidCacheSettingsexposes configuration, andLiquidCacheClearAction/LiquidCacheStatsActionprovide admin/observability endpoints.build.gradlebuilds the engine with-PliquidCache;LiquidCacheITverifies an indexed match+predicate query engages the cache (total_entries > 0);FeatureFlags.javaregisters the flag.This is PR 3 of a 3-part stack:
Changes
Cargo.toml(uncomment dep), liquid-cachesrc/main/rust/Cargo.toml(add top-crate member)session_context.rs,DataFusionPlugin.javaliquid-cache/src/main/rust/src/{ffi.rs, lib.rs, runtime.rs}LiquidCachePlugin,LiquidCacheBridge,LiquidCacheSettings,action/LiquidCacheClearAction,action/LiquidCacheStatsActionbuild.gradle,README.md, qabuild.gradleserver/.../FeatureFlags.javaLiquidCacheSettingsTests,LiquidCacheIT18 files changed, +1,606 / -18.
Check List
-PliquidCache.LiquidCacheITpasses — indexed match+predicate query engages the cache (total_entries > 0).