Expose row-group/cell-level statistics (min/max zone map, null_count, bloom filter, Diva) via the v3 manifest reader for predicate pushdown / data skipping

[xiaofan-luan]

Summary

Storage v3 (loon manifest) segments currently provide no data-skipping metadata to the query engine. milvus-storage already stores/collects most of the raw statistics (parquet footer min/max, per-field null_counts, bloom filters; Vortex zone maps internally), but the manifest / FFI reader does not expose them to consumers, and there is no predicate-pushdown surface. As a result a v3 sealed segment in Milvus segcore builds an empty chunk-level skip index and prunes 0 row-groups/cells for any predicate.

Request: expose row-group / cell-level statistics through the v3 reader — min/max zone map, null_count, bloom filter, and a Diva-style range index — plus a pushdown surface so the engine can skip row-groups/cells without materializing column data. In particular, make IS NULL / IS NOT NULL answerable from metadata alone.

Current state (measured)

• v3 load path ChunkedSegmentSealedImpl::LoadColumnGroup calls load_field_data_common(..., statistics = std::nullopt, ...); the in-code comment is literally "manifest cannot provide parquet skip index directly."

• v2 (packed parquet, non-manifest) reads per-row-group parquet::Statistics and builds a skip index behind common.parquetStatsSkipIndex.enabled.

• Unit-test measurement on sealed segments, selective range predicate:

  format / config	cells pruned
  v2 + flag ON	59 / 79 (~75%)
  v2 + flag OFF (default)	0 / 79
  v3 (manifest)	0 / 1 — even with the flag ON

The common.parquetStatsSkipIndex.enabled flag does not reach the manifest path, so v3 is the only format with no way — flag on or off — to skip at row-group/cell granularity.

Requested capabilities (per row-group / cell, per column)

1. min / max (zone map) — range & equality pruning. Vortex already computes zone maps internally (RowGroupZoneMapCount / ZoneMapSegmentIds); Parquet carries footer stats. Please surface both uniformly through one reader API.
2. null_count — enables a fast metadata-only IS NULL / IS NOT NULL and null-aware pruning:
   • null_count == 0 → no nulls → IS NULL prunes the whole group; IS NOT NULL selects all.
   • null_count == num_rows → all null → IS NOT NULL prunes; IS NULL selects all.
   • For partial-null groups, ideally allow reading validity / definition levels only (without decoding the value column) so IS NULL stays cheap.
3. bloom filter — equality / IN pruning for high-cardinality columns.
4. Diva range-filter index (VLDB 2025) — compact per-block range index for selective range predicates where zone maps are too coarse (unsorted / overlapping ranges).

Proposed API shape (either works for the engine)

• (a) Stats blob — the reader returns, per column group, a per-row-group stats structure (min/max/null_count/bloom handle) that the engine turns into its own skip index, mirroring how the v2 path consumes parquet::Statistics. The engine aggregates row-group stats up to its cell granularity itself.
• (b) Pushdown callback — can_skip(column, row_group_or_cell, predicate) -> bool and/or prune(predicate) -> surviving row-group set, evaluated inside the reader so bloom/Diva stay encapsulated. Non-monoid indexes like Diva are easier to keep on the storage side this way.

Granularity note

Milvus packs several parquet row-groups into a cell (cell_target_size_bytes, default ~4 MiB) as its materialization / skip unit. Please expose per-row-group stats (finest) and let the engine aggregate to cells, or expose per-cell aggregates. Aggregation is a simple monoid for min/max (min-of-mins / max-of-maxes) and bloom (OR-union); Diva is not mergeable, which is a further reason option (b) may be preferable for it.

Consumer

Milvus segcore will consume this to build FieldChunkMetrics for v3 (replacing the std::nullopt at LoadColumnGroup) and to add a metadata-only IS NULL / IS NOT NULL fast path. A tracking issue / PR on the Milvus side will follow.