Skip to content

Ingester: re-validate ownership before evicting non-owned series - #16534

Draft
duricanikolic wants to merge 2 commits into
mainfrom
yuri/repro-non-owned-series-eviction-race
Draft

Ingester: re-validate ownership before evicting non-owned series#16534
duricanikolic wants to merge 2 commits into
mainfrom
yuri/repro-non-owned-series-eviction-race

Conversation

@duricanikolic

@duricanikolic duricanikolic commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

What this PR does

Fixes a race condition in the ingester's early compaction of non-owned series (-ingester.early-compaction-non-owned-series-enabled).

When a series is no longer owned by an ingester, it is queued for eviction and eventually flushed to a block once its grace period expires. If the ingester regains ownership of the series before the grace period expires, the eviction path does not re-check ownership before flushing it. Instead, it relies on the potentially stale ownership state from when the series was originally queued.

As a result, a series that has become owned again and received new samples in the meantime could still be evicted from the head, causing those samples to be silently dropped.

The race exists because ownership reconciliation and early compaction run independently. The owned-series service periodically reconciles ownership, while the compaction loop processes queued series on a separate schedule. There is no guarantee that ownership will be reconciled before a queued series becomes eligible for eviction.

This PR fixes the race by making compactBlocksDueToNonOwnedSeries re-evaluate ownership immediately before processing series queued for eviction. For each tenant with queued series, it refreshes their ownership status against the ingester's current view before consuming the queue:

  • Series that are owned again are removed from the eviction queue and preserved.
  • Series that are still non-owned are evicted as before.

The additional ownership check is performed only for tenants with series queued for eviction, so tenants without pending early compaction incur no additional overhead.

A new unit test reproduces the race and verifies that a series that becomes owned again is not evicted. Existing tests covering early compaction continue to pass unchanged.

Which issue(s) this PR fixes or relates to

Fixes #

Checklist

  • Tests updated.
  • Documentation added.
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]. If changelog entry is not needed, please add the changelog-not-needed label to the PR.
  • about-versioning.md updated with experimental features.

Signed-off-by: Yuri Nikolic <durica.nikolic@grafana.com>
@duricanikolic duricanikolic self-assigned this Sep 7, 2026
Signed-off-by: Yuri Nikolic <durica.nikolic@grafana.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant