Skip to content

Add optional liquid cache for indexed parquet scans - #211

Closed
alchemist51 wants to merge 1 commit into
mainfrom
liquid-cache-seam
Closed

Add optional liquid cache for indexed parquet scans#211
alchemist51 wants to merge 1 commit into
mainfrom
liquid-cache-seam

Conversation

@alchemist51

@alchemist51 alchemist51 commented Jul 27, 2026

Copy link
Copy Markdown
Owner

What

Adds an opt-in in-memory decoded-batch cache (liquid cache) for the DataFusion analytics backend, wired behind a cargo feature so it is completely absent from the default build. Liquid cache lives outside this repo — it's pulled in only when a user explicitly points the build at a liquid cache source.

Behaviour

  • Feature off (default): the integration is an inlined no-op — eligible scans use the plain ParquetSource unchanged. No liquid cache dependency is declared, resolved, fetched, or compiled. The build behaves exactly as before.
  • Feature on: an eligible indexed row-group scan — all projected columns numeric/date/timestamp/boolean, within a column budget, and no string/binary predicate column — is wrapped with LiquidParquetSource, so decoded batches are served from and populated into the process-global cache.

How to enable

Two steps — a single-file edit plus a build flag:

  1. In sandbox/plugins/analytics-backend-datafusion/rust/Cargo.toml, uncomment the liquid cache dependency lines, set the git repo + branch to your liquid cache source, and uncomment the matching dep: entries on the liquid_cache feature (all three edits are adjacent, with inline instructions).
  2. Build with -PliquidCache (or LIQUID_CACHE=1).
./gradlew ... -PliquidCache

For local development against a checkout, redirect the git dep with a workspace [patch].

Why a commented placeholder instead of a committed dependency

Cargo resolves (and fetches) every declared optional dependency even when its feature is off — only compilation is feature-gated. A committed git URL would therefore be fetched on every default build, and a placeholder/unreachable URL would break it (an invalid URL even fails manifest parsing). Leaving the dependency commented out is what keeps the default build free of any liquid cache source while still making enablement a one-file edit.

Changes

6 files, additive (Cargo.lock intentionally not committed; the repo's Rust build runs unlocked and regenerates it):

  • analytics-backend-datafusion/rust/src/liquid_cache.rs — feature-gated module. maybe_wrap_parquet_source is a no-op when off; when on, applies the eligibility gate and wraps the source. Holds the process-global cache reference.
  • analytics-backend-datafusion/rust/src/indexed_table/parquet_bridge.rs — 12-line hunk routing the parquet source through maybe_wrap_parquet_source.
  • analytics-backend-datafusion/rust/Cargo.toml — commented placeholder dependency + liquid_cache feature; all liquid-cache config consolidated here.
  • dataformat-native/rust/Cargo.toml — pointer note (config lives in the plugin crate).
  • dataformat-native/build.gradle-PliquidCache / LIQUID_CACHE=1 toggles the cargo feature.
  • analytics-backend-datafusion/rust/src/lib.rs — module declaration.

Testing

  • Feature off: cargo check -p opensearch-datafusion clean, cargo fmt --check clean, no liquid cache source resolved.
  • Enable path: simulated the one-file edit against a real repo/branch — cargo metadata resolves and pulls the liquid cache crates.

Notes / follow-ups

  • Feature-on currently compiles only on Linux (the upstream liquid cache disk tier uses io-uring).
  • The runtime bootstrap that populates the cache reference is not wired yet; the wrap is inert until that lands.

@alchemist51
alchemist51 changed the base branch from main to 2.x July 28, 2026 08:29
@alchemist51
alchemist51 changed the base branch from 2.x to main July 28, 2026 08:29
@alchemist51
alchemist51 force-pushed the liquid-cache-seam branch 3 times, most recently from a2a45b0 to 87d59f6 Compare July 28, 2026 09:47
@alchemist51 alchemist51 changed the title Add seam to plug in an external native query optimizer Pluggable liquid cache to OpenSearch Jul 28, 2026
Adds an opt-in, in-memory decoded-batch cache (liquid cache) for the
DataFusion analytics backend, wired behind a cargo feature so it is
absent from the default build.

When the `liquid_cache` feature is off (the default), the integration is
an inlined no-op: eligible scans use the plain ParquetSource unchanged.
When on, an eligible indexed row-group scan — all projected columns
numeric/date/timestamp/boolean, within a column budget, and no string or
binary predicate column — is wrapped with LiquidParquetSource so decoded
batches are served from and populated into the process-global cache.

Enabling is a single-file edit plus a build flag: the liquid cache
dependency is a commented placeholder in the analytics-backend-datafusion
crate (uncomment it, set the git repo/branch, and uncomment the matching
`dep:` feature entries), then build with -PliquidCache (or LIQUID_CACHE=1).
The default build declares no liquid cache source, so it resolves and
compiles exactly as before.

Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>
@alchemist51 alchemist51 changed the title Pluggable liquid cache to OpenSearch Add optional liquid cache for indexed parquet scans Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant