Skip to content

Add partial result detection to metadata queries when shard limit is exceeded (Metadata limits PR2) - #2214

Open
bhavanap1131-cmd wants to merge 8 commits into
filodb:developfrom
bhavanap1131-cmd:metadata_limits_P2
Open

Add partial result detection to metadata queries when shard limit is exceeded (Metadata limits PR2)#2214
bhavanap1131-cmd wants to merge 8 commits into
filodb:developfrom
bhavanap1131-cmd:metadata_limits_P2

Conversation

@bhavanap1131-cmd

@bhavanap1131-cmd bhavanap1131-cmd commented Apr 28, 2026

Copy link
Copy Markdown
Contributor

Summary:
Metadata queries (/series, /labels, /label/values, label cardinality) previously had no way to signal that results were truncated when a shard hit its enforced limit. This PR propagates QuerySession into the metadata query paths so shards can flag resultCouldBePartial = true when the result count equals the limit, allowing callers to send a partial results warning to the user.

New Changes:

  • Added querySession: QuerySession parameter to TimeSeriesStore.partKeysWithFilters and its implementations in TimeSeriesShard and DownsampledTimeSeriesShard
  • Both shard implementations now set querySession.resultCouldBePartial = true with a reason when result.length == limit across all fetch paths (with/without first-last sample times, series scan, label values scan)
  • LabelValuesExec (no-filter path) sets the partial flag when returned label count equals the limit
  • LabelCardinalityExec replaces the hardcoded 1000000 cap with queryContext.plannerParams.enforcedLimits.execPlanLeafSamples to be consistent with the per-query limits.

@bhavanap1131-cmd bhavanap1131-cmd changed the title Metadata limits p2 Add partial result detection to metadata queries when shard limit is exceeded (Metadata limits PR2) May 21, 2026
Comment thread core/src/main/scala/filodb.core/downsample/DownsampledTimeSeriesShard.scala Outdated

@alextheimer alextheimer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one non-blocking comment 👍

Comment thread query/src/main/scala/filodb/query/exec/MetadataExecPlan.scala Outdated
bhavanap1131-cmd and others added 8 commits June 1, 2026 16:13
…f hardcoded 1M limit

Replace the hardcoded 1,000,000 limit in LabelCardinalityExec with
queryContext.plannerParams.enforcedLimits.execPlanLeafSamples so the
limit is configurable per query. Add test verifying the limit is respected.
…ed shard limits

Add querySession param to partKeysWithFilters across store interface and impls to
detect truncation. Set resultCouldBePartial in DownsampledTimeSeriesShard label
value iterators, TimeSeriesShard/DownsampledTimeSeriesShard partKeysWithFilters,
and MetadataExecPlan LabelValuesExec/PartKeysExec when shard limits are hit.
…nd iterator checks

- Strengthen partial result reason messages from vague "may be partial" to
  actionable s"Some shards returned a result size greater than $limit;
  apply more filters or reduce the query time-range." across all locations
- Add resultCouldBePartial checks to SingleLabelValuesResultIterator and
  LabelValueResultIterator in TimeSeriesShard (in-memory shard path was
  missing the flag, only DownsampledTimeSeriesShard had it)
- Update existing test assertions to match new message format
- Add tests for single-label and multi-label filter paths hitting
  execPlanSamples limit in LabelValuesExec

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…partial result check

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ar scalastyle limit

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

@alextheimer alextheimer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you restore this PR with git reflog, as well?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants