[BUG] EOFException "read past EOF" in cardinality aggregation dynamic pruning when combined with a hybrid query

[gudata]

Describe the bug

A cardinality aggregation fails on individual shards with EOFException: read past EOF
when it accompanies a hybrid query that has two or more subqueries. The failure comes
from the dynamic pruning optimization in CardinalityAggregator: it seeks to an offset past
the end of the doc-values slice and the shard returns a status_exception.

status_exception: Failed when performing dynamic pruning in cardinality aggregation.
  You can set cluster setting
  [search.dynamic_pruning.cardinality_aggregation.max_allowed_cardinality] to 0 to disable.
caused_by: runtime_exception: java.io.EOFException: read past EOF (pos=31):
  MemorySegmentIndexInput(path=".../index/_32gc.cfs")
  [slice=_32gc_Lucene90_0.dvd] [slice=randomaccess]

The request still returns HTTP 200 with _shards.failed > 0, so results are silently
partial: fewer hits than requested and an understated (or zero) cardinality value. Clients
that only check the HTTP status treat the truncated response as correct.

The pos values are consistently tiny (observed 11, 31, 293), which suggests a wrongly
computed base offset rather than an off-by-a-little read near the true end of the slice.

This is not data corruption

Verified on an affected shard copy:

• Primary and replica of the shard report an identical document count.

• _shard_stores reports no store_exception; the shard is STARTED (Lucene checksums
  pass at recovery).

• On the same shard copy, terms, value_count, and a match_all cardinality all read
  the same field successfully.

• Most telling: keeping the query, the shard copy, and the matched document set fixed, and
  changing only the aggregation:

  aggregation on the same field	outcome
  terms	succeeds
  value_count	succeeds
  cardinality (pruning engaged)	fails with read past EOF
  no aggregation	succeeds

So the doc-values file is readable; only the dynamic pruning path misreads it.

Trigger conditions

Isolated by holding everything else constant and varying one ingredient at a time:

variant	result
hybrid with 2 subqueries (query_string + knn) + cardinality	fails
hybrid with 2 subqueries (query_string + query_string) + cardinality	fails
hybrid with 1 subquery (query_string) + cardinality	passes
hybrid with 1 subquery (knn) + cardinality	passes
plain bool/knn/query_string (no hybrid) + cardinality	passes
hybrid with 2 subqueries, no cardinality aggregation	passes

Notes:

• A search_pipeline (normalization processor) is not required — the failure is
  identical with and without one.
• A knn/vector subquery is not required — two text subqueries reproduce it.
• Low result cardinality alone is not sufficient: a single-subquery variant returning a
  distinct count of 2 (well under the default threshold of 100) succeeds.
• The failure is per-segment and per-shard-copy. A primary and its replica hold different
  segments because they merge independently, so the same query fails on one copy and
  succeeds on the other. Running without a preference round-robins between copies and
  therefore reproduces only intermittently. Use preference=_replica / preference=_primary
  to pin a copy.
• Affected segments are transient: a merge clears them and later ingest produces new ones.
  Over three days we observed four distinct segments across three indexes, so this is a
  recurring state, not one damaged file.

Suspected cause

This looks like the same class of bug as #17739 / #17775, where dynamic pruning in
CardinalityAggregator broke because BooleanScorer behaves differently from
DefaultBulkScorer. A hybrid query with two or more subqueries is executed by the
neural-search plugin's own scorer rather than the default one, whereas a single-subquery
hybrid degenerates to the standard path — which matches the pass/fail split in the table
above exactly. The pruning path appears to assume scorer/iterator semantics that the hybrid
scorer does not provide, and derives a doc-values offset from that assumption.

Related component

Search:Aggregations

To Reproduce

We have not reduced this to a synthetic dataset, because the failure depends on the segment
layout a shard happens to have, and merges clear the affected segments within a day or two.
The following reproduces it reliably against an index that currently has an affected segment.

1. Leave search.dynamic_pruning.cardinality_aggregation.max_allowed_cardinality at its
   default of 100.
2. Have an index with a high-cardinality keyword field (hundreds of thousands of distinct
   values) and enough ingest activity to keep producing small compound (.cfs) segments.
3. Run a hybrid query with two subqueries plus a cardinality aggregation on that field,
   with a filter selective enough that the matched set falls under the pruning threshold.
   Pin the shard copy with preference:

POST /my-index-*/_search?preference=_replica
{
  "size": 0,
  "query": {
    "hybrid": {
      "queries": [
        { "query_string": { "query": "alpha", "fields": ["content"], "default_operator": "AND" } },
        { "query_string": { "query": "beta",  "fields": ["content"], "default_operator": "AND" } }
      ],
      "filter": {
        "range": { "timestamp": { "gte": "2026-06-16T00:00:00Z", "lte": "2026-07-16T00:00:00Z" } }
      }
    }
  },
  "aggs": {
    "distinct": { "cardinality": { "field": "entity_id", "precision_threshold": 40000 } }
  }
}

4. Repeat with preference=_primary and with the aggregation swapped for terms — both
   succeed on the same data, which isolates the pruning path.

Expected behavior

The cardinality aggregation should either compute correctly under dynamic pruning, or fall
back to the non-pruning path. It should not read past the end of a doc-values slice, and a
failure in an optimization should not silently truncate results behind an HTTP 200.

Additional Details

Plugins: neural-search (required for the hybrid query), k-NN. The managed service
masks plugin versions as x.x.x.x; bundled core plugins report 3.5.0.

Host/Environment:

• OpenSearch 3.5.0 on every node, no mixed-version cluster
  (from _cat/nodes?h=version; note that on a managed service GET / reports
  "number": "7.10.2" because of the Elasticsearch compatibility-mode override —
  the real version is in _cat/nodes, and the lucene_version: 10.3.2 plus
  minimum_wire_compatibility_version: 2.19.0 in the same response give it away)
• Lucene 10.3.2
• Managed service deployment, Linux
• Default search.dynamic_pruning.cardinality_aggregation.max_allowed_cardinality (100)
• search.concurrent_segment_search.mode: none

Workaround: disabling the optimization cluster-wide removes the failure:

PUT /_cluster/settings
{ "persistent": { "search.dynamic_pruning.cardinality_aggregation.max_allowed_cardinality": "0" } }

Additional context

Two secondary issues that may be worth separate consideration:

1. A failure inside an optional performance optimization degrades a correct result into a
   silently partial one. Falling back to the non-pruning path on error would keep the
   response correct at the cost of latency.
2. The error message names the setting to disable the optimization, which is good, but the
   response is still HTTP 200 — callers that check only the status code never see it.