Invalidate exact bitset filter cache keys on reader close

[jwils]

Description

The node-level bitset filter cache introduced in #21179 defers cleanup of entries from closed readers to a periodic sweep over Cache#keys(). That sweep is unreliable: the live keys() iterator walks the LRU linked list without holding the LRU lock, and iteration under concurrent mutation is undefined. Every cache hit relinks the accessed entry to the head of the LRU list, so an entry promoted mid-sweep can end up behind the cursor and never be visited. Because purgeStaleEntries() unconditionally drains staleCacheKeys after each pass and a reader closes only once, a skipped entry is never revisited: it stays in the cache until size-based eviction reaches it — potentially forever — pinning its bitset, its query, and the Value.listener → IndexService graph on heap while being invisible to shard bitset stats.

This applies the same fix as the fielddata cache in #22491: invalidate the exact keys synchronously when the reader closes, instead of marking the reader stale and relying on a sweep.

• getAndLoadIfNotPresent() records every key cached for a reader in a per-reader registry (keysByReader); the closed listener is registered at most once per reader via the registry's atomic computeIfAbsent. Keys are only unregistered on reader close: trimming on eviction would race with a concurrent reload of the same key re-registering it, and a lost registration would leave the entry with no cleanup path. A registered key whose entry was evicted costs two references, is bounded by the reader's lifetime, and invalidating it on close is a no-op.
• onClose() invalidates exactly the keys registered for the closed reader, synchronously. A reader closes only once and exact-key invalidation is O(#queries cached for the reader), so there is no need to batch it.
• With no sweep left, the periodic BitsetCacheCleaner, the indices.cache.bitset.cleanup_interval setting, purgeStaleEntries(), and the stale-mark bookkeeping are removed, and the constructor no longer needs a ThreadPool.
• New regression tests cover exact per-reader invalidation across multiple cached queries and the eviction-then-reload case that motivates never unregistering keys on eviction; existing purge-based tests now assert entries are gone immediately after reader close.

Related Issues

Related to #21179. Companion to #22491, which made the same change for the fielddata cache. No longer depends on #22499 (Cache#keysSnapshot()): with exact-key invalidation there is no sweep left in this cache.

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 9452a6c)

Here are some key observations to aid the review process:

🧪 PR contains tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ Recommended focus areas for review Setting Registration The INDICES_BITSET_FILTER_CACHE_CLEAN_INTERVAL_SETTING (indices.cache.bitset.cleanup_interval) is removed but if it was previously registered in ClusterSettings/SettingsModule, its registration also needs to be removed, otherwise the code still references a deleted symbol. Additionally, removing this public setting is a breaking change for users who may have set it in opensearch.yml; nodes with that setting configured could fail to start with an "unknown setting" error. ); public static final Setting<ByteSizeValue> INDICES_BITSET_FILTER_CACHE_SIZE_SETTING = Setting.memorySizeSetting( "indices.cache.bitset.size", "5%", Property.NodeScope ); private final Cache<BitsetCacheKey, Value> cache; /** * Every key cached for a reader, so {@link #onClose(IndexReader.CacheKey)} can invalidate the * exact keys when the reader closes. Keys are only removed on reader close: trimming on * eviction would race with a concurrent reload of the same key re-registering it, and a lost * registration would leave the entry with no cleanup path. A registered key whose entry was * evicted costs two references, is bounded by the reader's lifetime, and invalidating it on * close is a no-op. */ private final ConcurrentMap<IndexReader.CacheKey, Set<BitsetCacheKey>> keysByReader = ConcurrentCollections.newConcurrentMap(); public IndicesBitsetFilterCache(Settings settings) { long sizeInBytes = INDICES_BITSET_FILTER_CACHE_SIZE_SETTING.get(settings).getBytes(); CacheBuilder<BitsetCacheKey, Value> cacheBuilder = CacheBuilder.<BitsetCacheKey, Value>builder().removalListener(this); if (sizeInBytes > 0) { cacheBuilder.setMaximumWeight(sizeInBytes).weigher(new BitsetWeigher()); } this.cache = cacheBuilder.build(); } Potential Race In getAndLoadIfNotPresent, keysByReader.computeIfAbsent(...).add(cacheKey) and cache.computeIfAbsent(cacheKey, ...) are not atomic together. If the reader closes between the add and the cache.computeIfAbsent call (possible because onClose can fire from any thread as soon as the reader is closed—there is no guarantee the caller holds the reader open beyond Lucene's contract), onClose may remove the reader entry from keysByReader before the new cache entry is inserted, leaving the entry orphaned in the cache with no cleanup path until size-based eviction. Consider inserting into the cache first and confirming registration afterward, or documenting the caller's reader-lifetime requirement more strongly. final BitsetCacheKey cacheKey = new BitsetCacheKey(coreCacheReader, query); // Register the closed listener at most once per reader (the mapping function runs // atomically) and record the key so onClose can invalidate it exactly. The caller holds // the reader open, so onClose cannot fire before the key is recorded. keysByReader.computeIfAbsent(coreCacheReader, k -> { cacheHelper.addClosedListener(this); return ConcurrentCollections.newConcurrentSet(); }).add(cacheKey); return cache.computeIfAbsent(cacheKey, key -> { final BitSet bitSet = bitsetFromQuery(query, context); Value value = new Value(bitSet, shardId, listener); listener.onCache(shardId, value.bitset); return value; }).bitset;

[github-actions]

PR Code Suggestions ✨

Latest suggestions up to 9452a6c

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Fix race between key registration and reader close There is a race where onClose can run between computeIfAbsent returning the set and .add(cacheKey) executing: if the reader closes at that instant, keysByReader.remove sees an empty (or already-removed) set, and the subsequent add re-inserts the key with no cleanup path. Recheck after adding and invalidate immediately if the reader-key mapping is gone. server/src/main/java/org/opensearch/indices/IndicesBitsetFilterCache.java [141-145] -keysByReader.computeIfAbsent(coreCacheReader, k -> { +Set<BitsetCacheKey> keys = keysByReader.computeIfAbsent(coreCacheReader, k -> { cacheHelper.addClosedListener(this); return ConcurrentCollections.newConcurrentSet(); -}).add(cacheKey); +}); +keys.add(cacheKey); +if (keysByReader.get(coreCacheReader) != keys) { + // Reader closed concurrently; ensure we don't leak this key. + cache.invalidate(cacheKey); +} return cache.computeIfAbsent(cacheKey, key -> { Suggestion importance[1-10]: 6 __ Why: The comment in the PR notes "The caller holds the reader open, so onClose cannot fire before the key is recorded," which suggests the race may not occur in practice. However, if the invariant does not hold universally, the suggestion identifies a legitimate concern about a potential leak. The improved code's recheck logic is a reasonable defensive measure but its necessity depends on the caller contract.	Low

---

Previous suggestions

Suggestions up to commit a2036c8

Category	Suggestion	Impact
Possible issue	Fix race between key registration and reader close There is a race between recording the key in keysByReader and the reader's onClose firing: if onClose runs after computeIfAbsent completes but the closed-listener registration was already pending, keysByReader.remove may return the set before add(cacheKey) executes, leaving the loaded cache entry orphaned. Register the key set first, then add the cacheKey, and finally check that the reader hasn't been closed in the meantime (e.g., re-check keysByReader.get(coreCacheReader) contains the key after cache.computeIfAbsent, and invalidate if missing). server/src/main/java/org/opensearch/indices/IndicesBitsetFilterCache.java [141-145] -keysByReader.computeIfAbsent(coreCacheReader, k -> { +Set<BitsetCacheKey> keySet = keysByReader.computeIfAbsent(coreCacheReader, k -> { cacheHelper.addClosedListener(this); return ConcurrentCollections.newConcurrentSet(); -}).add(cacheKey); -return cache.computeIfAbsent(cacheKey, key -> { +}); +keySet.add(cacheKey); +Value v = cache.computeIfAbsent(cacheKey, key -> { final BitSet bitSet = bitsetFromQuery(query, context); Value value = new Value(bitSet, shardId, listener); listener.onCache(shardId, value.bitset); + return value; +}); +// If onClose fired concurrently after we added the key but before caching, ensure cleanup. +if (keysByReader.get(coreCacheReader) == null) { + cache.invalidate(cacheKey); +} +return v.bitset; Suggestion importance[1-10]: 3 __ Why: The claimed race is questionable: the caller holds the reader open (as noted in the PR comment), so onClose cannot fire before the caching completes. The suggestion's reasoning about a race is not clearly correct, and the proposed fix adds complexity for a scenario the PR explicitly addresses.	Low

Suggestions up to commit d3c5ed6

Category	Suggestion	Impact
General	Guard initial capacity against overflow The field count is a long but ArrayList constructor requires an int; if count exceeds Integer.MAX_VALUE this will fail to compile, and if it is a long that gets implicitly narrowed elsewhere it could produce a negative capacity. Cast explicitly and clamp to a non-negative int to avoid IllegalArgumentException on new ArrayList<>(negative). server/src/main/java/org/opensearch/common/cache/Cache.java [754] public List<K> keysSnapshot() { - List<K> keys = new ArrayList<>(count); + List<K> keys = new ArrayList<>((int) Math.min(Math.max(count, 0L), Integer.MAX_VALUE)); for (CacheSegment<K, V> segment : segments) { try (ReleasableLock ignored = segment.readLock.acquire()) { keys.addAll(segment.map.keySet()); } } return keys; } Suggestion importance[1-10]: 2 __ Why: The count field in Cache is likely a long, but the concern about overflow is largely theoretical for an in-memory cache. Existing code elsewhere (e.g., keys() iterator) uses the same pattern without issue, and this is a minor defensive measure at best.	Low

Suggestions up to commit 511426b

Category	Suggestion	Impact
General	Avoid leaking empty per-reader key sets When entries are evicted by weight (not by reader close), the per-reader set can become empty but is never removed from keysByReader, causing a slow memory leak of empty sets keyed by still-open readers. Consider removing the set entry when it becomes empty, using an atomic compute to avoid racing with a concurrent load that just re-added a key. server/src/main/java/org/opensearch/indices/IndicesBitsetFilterCache.java [179-188] @Override public void onRemoval(RemovalNotification<BitsetCacheKey, Value> notification) { final BitsetCacheKey key = notification.getKey(); if (key != null) { - // Keep the per-reader key index in sync on evictions and explicit invalidations. - // The set itself is removed when the reader closes. - final Set<BitsetCacheKey> keys = keysByReader.get(key.readerCacheKey); - if (keys != null) { - keys.remove(key); - } + keysByReader.computeIfPresent(key.readerCacheKey, (k, set) -> { + set.remove(key); + return set.isEmpty() ? null : set; + }); } Suggestion importance[1-10]: 6 __ Why: Valid concern: when entries are evicted by weight while the reader remains open, empty sets accumulate in keysByReader. The suggested computeIfPresent fix is correct and prevents a slow memory leak, though the impact is minor since sets are small.	Low
Possible issue	Guard against reader-close race during load There is a race between computeIfAbsent inserting the key into keysByReader and a concurrent onClose that has already removed the reader's set. If onClose runs after registeredKeys.add (which returned true earlier for another thread) but before this loader completes, the newly created set in keysByReader will never be cleaned up, and the cache entry will not be invalidated. Consider checking that the reader is still registered (or re-adding to keysByReader after verifying) and/or invalidating the just-loaded entry if the reader has closed. server/src/main/java/org/opensearch/indices/IndicesBitsetFilterCache.java [141-149] return cache.computeIfAbsent(cacheKey, key -> { - // Record the key before loading so onClose can find it. The reader is held open - // by the caller for the duration of the load, so it cannot close concurrently. - keysByReader.computeIfAbsent(coreCacheReader, k -> ConcurrentCollections.newConcurrentSet()).add(key); final BitSet bitSet = bitsetFromQuery(query, context); Value value = new Value(bitSet, shardId, listener); listener.onCache(shardId, value.bitset); + keysByReader.computeIfAbsent(coreCacheReader, k -> ConcurrentCollections.newConcurrentSet()).add(key); + if (registeredKeys.contains(coreCacheReader) == false) { + // Reader closed concurrently; ensure we don't leak this entry. + keysByReader.remove(coreCacheReader); + cache.invalidate(key); + } return value; }).bitset; Suggestion importance[1-10]: 5 __ Why: The PR comment notes the reader is held open by the caller during load, which likely mitigates this race. However, the concern about ordering between onClose and the loader is plausible, and a defensive check could improve robustness. Moderate impact but not clearly a bug.	Low

Suggestions up to commit b79ef1e

Category	Suggestion	Impact
Possible issue	Fix race between load and reader close There is a race between computeIfAbsent populating keysByReader and onClose removing the reader's set. If onClose runs after the computeIfAbsent load path added the entry to the cache but before/while it inserts into keysByReader, the removal may miss the key and leak the cache entry. Insert into keysByReader after the cache insertion, and re-check the reader's liveness (or add defensively after with a second onClose-safe check) to avoid orphaned entries. server/src/main/java/org/opensearch/indices/IndicesBitsetFilterCache.java [141-149] -return cache.computeIfAbsent(cacheKey, key -> { - // Record the key before loading so onClose can find it. The reader is held open - // by the caller for the duration of the load, so it cannot close concurrently. - keysByReader.computeIfAbsent(coreCacheReader, k -> ConcurrentCollections.newConcurrentSet()).add(key); +final Value cached = cache.computeIfAbsent(cacheKey, key -> { final BitSet bitSet = bitsetFromQuery(query, context); Value value = new Value(bitSet, shardId, listener); listener.onCache(shardId, value.bitset); return value; -}).bitset; +}); +// Track after insertion; if the reader closed concurrently, invalidate to avoid a leak. +final Set<BitsetCacheKey> tracked = keysByReader.computeIfAbsent(coreCacheReader, k -> ConcurrentCollections.newConcurrentSet()); +tracked.add(cacheKey); +if (keysByReader.get(coreCacheReader) != tracked) { + cache.invalidate(cacheKey); +} +return cached.bitset; Suggestion importance[1-10]: 6 __ Why: There is a plausible race: onClose could execute after the cache insert but before keysByReader is populated, leaking the entry. However, the PR comment claims the reader is held open by the caller during load, which mitigates this. The suggestion raises a legitimate concern worth verifying.	Low
General	Prevent per-reader index growth on evictions When entries are evicted by weight (not by reader close), the per-reader set may become empty but is never removed from keysByReader, leaving a permanent entry keyed by the still-live reader. Over time this accumulates. Remove the set from keysByReader when it becomes empty, using an atomic check to avoid races with concurrent inserts in getAndLoadIfNotPresent. server/src/main/java/org/opensearch/indices/IndicesBitsetFilterCache.java [179-194] @Override public void onRemoval(RemovalNotification<BitsetCacheKey, Value> notification) { final BitsetCacheKey key = notification.getKey(); if (key != null) { - // Keep the per-reader key index in sync on evictions and explicit invalidations. - // The set itself is removed when the reader closes. final Set<BitsetCacheKey> keys = keysByReader.get(key.readerCacheKey); if (keys != null) { keys.remove(key); + if (keys.isEmpty()) { + keysByReader.remove(key.readerCacheKey, keys); + } } } Suggestion importance[1-10]: 6 __ Why: Valid observation: when weight-based eviction empties the set but the reader is still live, the empty set remains in keysByReader indefinitely, causing minor memory growth. The atomic remove(key, value) fix is appropriate.	Low
	Handle removed setting for backward compatibility The setting INDICES_BITSET_FILTER_CACHE_CLEAN_INTERVAL_SETTING was removed but if it was previously registered in a settings module/plugin registry, removing it may break rolling upgrades or existing configurations that reference it (nodes will fail to start with "unknown setting"). Consider deprecating it (archive/ignore) rather than deleting outright, or ensure it is registered as an archived/removed setting to avoid startup failures on clusters that had it set. server/src/main/java/org/opensearch/indices/IndicesBitsetFilterCache.java [81-87] +public static final Setting<ByteSizeValue> INDICES_BITSET_FILTER_CACHE_SIZE_SETTING = Setting.memorySizeSetting( + "indices.cache.bitset.size", + "5%", + Property.NodeScope +); +private final Cache<BitsetCacheKey, Value> cache; Suggestion importance[1-10]: 5 __ Why: Removing a public setting can cause node startup failures on upgrades if the setting was previously configured. However, this is an experimental API and the improved_code is identical to existing_code, only providing advice.	Low

[github-actions]

❌ Gradle check result for b79ef1e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

Persistent review updated to latest commit 511426b

[github-actions]

❌ Gradle check result for 511426b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 511426b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[jwils]

@sgup432 please review this as well.

[github-actions]

Persistent review updated to latest commit d3c5ed6

[github-actions]

❌ Gradle check result for d3c5ed6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

Persistent review updated to latest commit a2036c8

[jwils]

@andrross I believe this is another concerning case that can cause permeant misses (until the cache is full) if hit. Since these also link to unaccounted for objects like Query, ValueWrapper, etc I think it may also be able to grow larger than the max cache size.

[github-actions]

✅ Gradle check result for a2036c8: SUCCESS

[codecov]

Codecov Report

❌ Patch coverage is 92.30769% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 71.43%. Comparing base (08eb405) to head (a2036c8).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...g/opensearch/indices/IndicesBitsetFilterCache.java	91.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #22498      +/-   ##
============================================
- Coverage     71.45%   71.43%   -0.02%     
+ Complexity    76760    76741      -19     
============================================
  Files          6136     6136              
  Lines        357635   357601      -34     
  Branches      52128    52125       -3     
============================================
- Hits         255532   255442      -90     
- Misses        81755    81824      +69     
+ Partials      20348    20335      -13     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[kkhatua]

Thanks for the PR, @jwils

We recently did some minor improvements ( #21179 ) to contain this cache size limit, but a rewrite of the cache (like field data cache) is probably what is needed but that's not going to be trivial.

We'll need some time to review this PR as there are competing priorities at the moment.

[jwils]

@kkhatua our clusters are not on 3.7 yet so I can't say for sure of how large the impact is, but #21179 introduced a bug that this attempts to fix. I am concerned the cache size is no longer bounded without either a fix like this or reverting #21179

[jwils]

Similar to the other pr. I believe removing the async BitsetCacheCleaner flow is safe, but I think the fix is important either way, so more than happy to refactor this to still retain the BitsetCacheCleaner if that makes it easier to review.

[github-actions]

Persistent review updated to latest commit 9452a6c

[github-actions]

❌ Gradle check result for 9452a6c: TIMEOUT

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?