Skip to content

[Analytics] Wire liquid cache into the analytics engine (plugin + integration) - #22595

Open
cocosz wants to merge 3 commits into
opensearch-project:mainfrom
cocosz:liquid-cache-plugin-integration
Open

[Analytics] Wire liquid cache into the analytics engine (plugin + integration)#22595
cocosz wants to merge 3 commits into
opensearch-project:mainfrom
cocosz:liquid-cache-plugin-integration

Conversation

@cocosz

@cocosz cocosz commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Description

Turns the liquid cache on: wires the vendored library into the analytics engine and exposes it as an OpenSearch plugin, gated behind the liquid_cache cargo feature and the datafusion.liquid_cache feature flag.

Because the earlier PRs already landed their files in final form, this PR is purely additive plus two small wiring flips:

  • Enable the dependency: uncomments the optional opensearch-liquid-cache path dependency in the analytics backend Cargo.toml, and adds the top crate as a member of the liquid-cache workspace. The engine hook (liquid_cache.rs) is not touched here — it already shipped in its final form in [Analytics] Add optional liquid cache seam for indexed parquet scans (cargo feature) #22553.
  • Top crate (Rust): adds ffi.rs, lib.rs, runtime.rs — the process-global runtime, engagement policy, and the lc_* C FFI.
  • Plugin (Java): LiquidCachePlugin bootstraps the runtime, LiquidCacheBridge binds the lc_* symbols, LiquidCacheSettings exposes configuration, and LiquidCacheClearAction / LiquidCacheStatsAction provide admin/observability endpoints.
  • Build + tests: plugin build.gradle builds the engine with -PliquidCache; LiquidCacheIT verifies an indexed match+predicate query engages the cache (total_entries > 0); FeatureFlags.java registers the flag.

This is PR 3 of a 3-part stack:

  1. Seam ([Analytics] Add optional liquid cache seam for indexed parquet scans (cargo feature) #22553): feature-gated in-process hook (no-op by default).
  2. Library ([Analytics] Vendor the liquid-cache library crates #22569): vendors the liquid-cache crates in isolation.
  3. This PR — integration: turns the feature on + Java plugin + integration test.

Changes

Area Files
Wiring flips analytics Cargo.toml (uncomment dep), liquid-cache src/main/rust/Cargo.toml (add top-crate member)
Engine analytics session_context.rs, DataFusionPlugin.java
Top crate liquid-cache/src/main/rust/src/{ffi.rs, lib.rs, runtime.rs}
Java plugin LiquidCachePlugin, LiquidCacheBridge, LiquidCacheSettings, action/LiquidCacheClearAction, action/LiquidCacheStatsAction
Build plugin build.gradle, README.md, qa build.gradle
Feature flag server/.../FeatureFlags.java
Tests LiquidCacheSettingsTests, LiquidCacheIT

18 files changed, +1,606 / -18.

Check List

  • Builds with -PliquidCache.
  • LiquidCacheIT passes — indexed match+predicate query engages the cache (total_entries > 0).
  • Default build (feature off) unaffected.
  • API changes companion PR raised (n/a).
  • Public documentation issue/PR created (n/a — sandbox).

Note: PRs target main, so until the parents merge this PR's GitHub diff includes the seam + library commits. Review/merge order: #22553#22569 → this PR.

@cocosz
cocosz requested a review from a team as a code owner July 28, 2026 23:52
@github-actions

github-actions Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

PR Code Analyzer ❗

AI-powered 'Code-Diff-Analyzer' found issues on commit 33567f0.

Hard block: Issues at Medium severity or above will block this PR from merging.

'Diff too large, requires skip by maintainers after manual review'


Pull Requests Author(s): Please update your Pull Request according to the report above.

Repository Maintainer(s): You can bypass diff analyzer by adding label skip-diff-analyzer after reviewing the changes carefully, then re-run failed actions. To re-enable the analyzer, remove the label, then re-run all actions.


⚠️ Note: The Code-Diff-Analyzer helps protect against potentially harmful code patterns. Please ensure you have thoroughly reviewed the changes beforehand.

Thanks.

Tanvir Alam added 3 commits July 29, 2026 06:33
Adds an opt-in, in-memory decoded-batch cache (liquid cache) for the
DataFusion analytics backend, wired behind a cargo feature so it is
absent from the default build.

When the `liquid_cache` feature is off (the default), the integration is
an inlined no-op: eligible scans use the plain ParquetSource unchanged.
When on, an eligible indexed row-group scan — all projected columns
numeric/date/timestamp/boolean, within a column budget, and no string or
binary predicate column — is wrapped with LiquidParquetSource so decoded
batches are served from and populated into the process-global cache.

Enabling is a single-file edit plus a build flag: the liquid cache
dependency is a commented placeholder in the analytics-backend-datafusion
crate (uncomment it, set the git repo/branch, and uncomment the matching
`dep:` feature entries), then build with -PliquidCache (or LIQUID_CACHE=1).
The default build declares no liquid cache source, so it resolves and
compiles exactly as before.

Signed-off-by: Tanvir Alam <tanvralm@amazon.com>
Adds the in-memory decoded-batch cache (opensearch-liquid-cache-core) and its DataFusion parquet reader (opensearch-liquid-cache-datafusion) as a self-contained Cargo workspace under sandbox/plugins/liquid-cache/src/main/rust.

These crates build and unit-test on their own and are not yet wired into the engine; the analytics backend compiles them in only when built with the liquid_cache cargo feature, which a follow-up change enables. Kept as its own PR so the vendored library can be reviewed in isolation.

Signed-off-by: Tanvir Alam <tanvralm@amazon.com>
Adds the Java plugin (settings, stats, clear, native bridge) and the top crate (runtime + lc_* control surface), activates the optional liquid-cache dependency behind the liquid_cache feature, and wires engagement on both scan paths: the listing path via the cache's optimizer on the session, and the indexed path via the maybe_wrap hook. Also marks the analytics plugin ExtensiblePlugin so this plugin can attach, adds the feature flag, and the end-to-end LiquidCacheIT.

Signed-off-by: Tanvir Alam <tanvralm@amazon.com>
@cocosz
cocosz force-pushed the liquid-cache-plugin-integration branch from eb2ff19 to 33567f0 Compare July 29, 2026 01:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant