[ES|QL] Replace an IN subquery evaluated as a false filter with an empty local relation

[fang-xing-esql]

Fixes: #155563

Steps to reproduce the bug

curl -sS --fail-with-body -u elastic-admin:elastic-password -H 'Content-Type: application/json' -X PUT http://localhost:9200/_query/data_source/s3_public -d '{
  "type": "s3",
  "settings": {
    "auth": "none",
    "region": "us-east-1"
  }
}'

curl -sS --fail-with-body -u elastic-admin:elastic-password -H 'Content-Type: application/json' -X PUT http://localhost:9200/_query/dataset/noaa_weather -d '{
  "data_source": "s3_public",
  "resource": "s3://noaa-ghcn-pds/parquet/by_station/STATION=USW00094728/ELEMENT=TMAX/*.parquet"
}'

curl -u elastic:password -X POST "localhost:9200/_query?format=txt&pretty" -H 'Content-Type: application/json' -d'
{
  "query": "FROM noaa_weather | WHERE STATION IN (FROM noaa_weather | STATS c = COUNT(*) BY STATION | WHERE c > 999999 | KEEP STATION) | STATS count = COUNT(*)"
}
'

Root cause analysis

Query text:

FROM noaa_weather
| WHERE STATION IN (FROM noaa_weather   ===> this subquery returns empty results
                    | STATS c = COUNT(*) BY STATION
                    | WHERE c > 999999
                    | KEEP STATION)
| STATS count = COUNT(*)

The plan after planners on the coordinator:

Limit[1000]
\_Aggregate[[], COUNT(*) AS count]
  \_SemiJoin[STATION]
    |_ExternalRelation[...parquet]            ← main scan (left)
    \_Project[STATION]                        ← the subquery (right)
      \_Filter[c > 999999]
        \_Aggregate[BY STATION]
          \_ExternalRelation[...parquet]

The plan after the in subquery is evaluated, the subquery returns empty results, so the SemiJoin in subquery is replaced by filter(false)

Limit[1000]
\_Aggregate[[], COUNT(*) AS count]
  \_Filter[false]                       ← replace SemiJoin
    \_ExternalRelation[...parquet][STATION]

LimitExec[1000]
\_AggregateExec[COUNT(*), FINAL, [$$count$count, $$count$seen]]   ← expects 2 intermediate cols
  \_ExchangeExec[[$$count$count, $$count$seen]]
    \_FragmentExec[ Aggregate ← Filter[false] ← ExternalRelation ]  ← logical, untouched

The plan after applyExternalDistributionStrategy in ComputeService removed Exchange from the plan

LimitExec[1000]
\_AggregateExec[COUNT(*), FINAL, [$$count$count, $$count$seen]]   ← expects 2 intermediate cols
    \_FragmentExec[ Aggregate ← Filter[false] ← ExternalRelation ]  ← logical, untouched

The plan after LocalLogicalPlanOptimizer replaced filter(false) with empty local relation
OutputExec
\_LimitExec[1000]
  \_AggregateExec[COUNT(*), FINAL, [$$count$count, $$count$seen]]   <- expect 2 attributes
    \_LocalSourceExec[[count], Page{[Long(0)], [Boolean(true)]}]   <- only 1 attribute left, 2 blocks

The ExchangeExec is removed from the plan by applyExternalDistributionStrategy in ComputeService. In non-external dataset path, the exchange is not removed, so we don't hit this issue with regular indices.

The fix
If an in subquery returns empty results, the SemiJoin can be replaced by an empty LocalRelation, instead of putting a Filter(false) on top of its LHS child, then it won't cause applyExternalDistributionStrategy in ComputeService to remove the ExchangeExec. It is a better choice to replace Filter(false) with an empty LocalRelation in AbstractSubqueryJoin and SemiJoin from performance perspective, as this plan is turned into a coordinator only plan, and it doesn't need to reach data nodes.

With the empty LocalRelation in place, the rewritten main plan is:

Limit
\_Aggregate[COUNT(*)]
  \_LocalRelation[empty, output = dataset's columns]     ← no ExternalRelation anywhere

If an in subquery returns empty results, the AntiJoin can be replaced by its LHS child, instead of putting a Filter(true) on top of its LHS child.

[elasticsearchmachine]

Hi @fang-xing-esql, I've created a changelog YAML for you.

[github-actions]

🔍 Preview links for changed docs

⏳ Building and deploying preview... View progress

This comment will be updated with preview links when the build is complete.

[github-actions]

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

• Check out the cumulative docs guidelines
• Reach out in the #docs Slack channel

[elasticsearchmachine]

Hi @fang-xing-esql, I've created a changelog YAML for you.

[Copilot]

Pull request overview

This PR fixes an ES|QL failure mode where WHERE <field> IN (subquery) (and related NULL/empty-result short-circuit cases) could fold a plan into a coordinator-only path that then crashed during physical planning (reported as IndexOutOfBoundsException: toIndex = 2) when the subquery produced zero rows, particularly for external datasets / data federation and CCS scenarios.

Changes:

• Rewrite SEMI join’s “empty right side” and shared NULL short-circuit plans to return an empty LocalRelation (instead of Filter(FALSE)), and simplify ANTI join’s “empty right side” plan to return the left child.
• Preserve prior shard accounting for clusters that were already searched by subplans when the main plan becomes coordinator-only.
• Add regression tests for empty-result IN subqueries over datasets, and update existing unit/CCS tests to reflect the new rewrite shape.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/join/AbstractSubqueryJoin.java	Changes NULL short-circuit plan to return empty LocalRelation and updates related docstrings.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/join/SemiJoin.java	Makes empty-right SEMI join collapse to an empty LocalRelation via EmptyLocalSupplier.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/join/AntiJoin.java	Makes empty-right ANTI join collapse directly to the left child plan.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/ComputeService.java	Avoids overwriting shard stats when coordinator-only main plan follows subplans that already searched shards.
x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/plan/logical/join/SubqueryJoinTests.java	Updates unit tests to assert the new LocalRelation collapse behavior for empty/NULL short-circuit paths.
x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/FromDatasetSubqueryIT.java	Adds regression coverage for empty IN-subquery followed by STATS/SORT over external datasets; includes ANTI/NULL variants.
x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/CrossClusterInSubqueryIT.java	Updates CCS shard-count expectations for cases where the main plan no longer searches remotes.
docs/changelog/155648.yaml	Adds changelog entry for the bug fix.

Suppressed comments (2)

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/plan/logical/join/SubqueryJoinTests.java:156

• This Javadoc claims the empty-result path is equivalent to pruning a constant-false filter, but in this test the ANTI empty-right case rewrites to the LHS child (not a FALSE filter). The helper is really asserting that inlineData returned a LocalRelation with the left output and optionally an EmptyLocalSupplier.

     * The empty-result and NULL short-circuit paths collapse the join to an empty {@link LocalRelation} — the same
     * rewrite {@code PruneFilters} applies to a constant-false filter, inlined into the join because subquery
     * substitution runs after the logical optimizer and a {@code Filter(FALSE)} would otherwise survive to physical
     * planning. The relation must carry the join's output (the left side's attributes) so references above resolve.
     */

x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/FromDatasetSubqueryIT.java:1238

• AntiJoin#buildEmptyRightSidePlan no longer substitutes a constant-true filter; it returns the left child directly. The Javadoc should be updated so it matches the implementation.

     * ANTI join, empty right side: {@code x NOT IN ()} is TRUE for every row, so {@code AntiJoin#buildEmptyRightSidePlan} substitutes
     * a constant-true filter and the whole dataset survives. The trailing STATS pins that the surviving scan still composes with the
     * external aggregation split (the counterpart of {@link #testEmptyInSubqueryThenStatsOnDataset}).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.