Repository navigation
perf(cleaner): walk only metadata rows in the TTL sweep and reconcile - #278
Merged
Merged
Conversation
Both the per-minute TTL sweep and the startup and hourly size
reconciliation iterated from the first key of the database and skipped
every row outside the metadata prefix. The eviction indexes,
back-references, compaction and deletion queues share the database, and
under LRU those rows outnumber the metadata rows two to one, so each pass
read three rows for every one it could act on, on a row count that grows
with the cache.
Seek to the metadata prefix and stop when it ends. A new counter,
ocache_cleaner_rows_scanned_total{scan="ttl"|"reconcile"}, and a
"scanned" field on each pass's summary log expose the rows walked; the new
test seeds rows on both sides of the prefix and pins the count to the
metadata rows alone.
Perfloop cases case_hb73egdr2s and case_456vm3wag5 (the prefix bound is
the useful part of the latter; an expiry index is not worth its write
amplification).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YCoN4Him5ZY4hZovVp37ra
Signed-off-by: Ovais Tariq <ot@tigrisdata.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
cleanupExpiredKeys(every cleaner tick, 1 min by default) andreconcileFromMetadata(startup, then hourly) both started atSeekToFirstand skipped every row that is not under!meta/. The eviction indexes, back-references, compaction and deletion queues live in the same RocksDB, and under LRU there are two index rows per key, so each pass read three rows for every one it could act on. The cost scales with the number of keys, which for TAG's 1 MiB blocks is the cache size in MiB.Change
keys.MetadataPrefixand stop withValidForPrefix. The per-row prefix check is gone.ocache_cleaner_rows_scanned_total{scan="ttl"|"reconcile"}and ascannedfield on each pass's completion log, so the walked-row count is visible in production.TestCleanerScans_WalkOnlyMetadataRowsseeds 1000 rows in keyspaces sorting on both sides of the prefix plus five metadata rows, and asserts the sweep walks exactly 5 and the reconcile exactly 3 (after two expiries are deleted), leaving the other rows intact.make lintandmake test-storagepass.Closes perfloop cases
case_hb73egdr2sandcase_456vm3wag5(the prefix bound is the useful half of the latter; a separate expiry index is not worth its write amplification).🤖 Generated with Claude Code
https://claude.ai/code/session_01YCoN4Him5ZY4hZovVp37ra
Note
Medium Risk
Changes core cleaner iteration for TTL deletion and size/eviction-index reconciliation; behavior should be equivalent but mis-scoped prefix iteration could miss rows or skew totals.
Overview
Limits cleaner scans to metadata keys so periodic TTL expiry and startup/hourly size reconciliation no longer walk the full RocksDB and skip eviction indexes, back-references, and queues. Both paths now
Seektokeys.MetadataPrefixand iterate withValidForPrefix, removing per-rowIsMetadataKeychecks.Adds
ocache_cleaner_rows_scanned_total{scan="ttl"|"reconcile"}and ascannedfield on the TTL and reconcile completion logs.TestCleanerScans_WalkOnlyMetadataRowsseeds foreign keyspaces and asserts only metadata rows are counted and touched.Reviewed by Cursor Bugbot for commit 769fcdf. Bugbot is set up for automated code reviews on this repo. Configure here.