fix: paginate summary aggregation to avoid 500-item truncation (#337)

[matthyx]

Summary

Fixes #337 — on-the-fly summary aggregations silently truncated to the first 500 objects on larger clusters.

ConfigurationScanSummaryStorage.GetList and VulnerabilitySummaryStorage.GetList build cluster-wide, per-namespace summaries by listing the underlying workloadconfigurationscansummaries / vulnerabilitymanifestsummaries. The storage layer (storage.go) applies a hard-coded default Limit = 500 when a client lists without an explicit limit and returns a continuation token — but these two aggregators issued a single realStore.GetList call and never consumed the token.

Result on clusters with >500 stored objects:

• namespaces beyond the first page silently disappear from the list results;
• per-namespace severity counters are wrong (a namespace can report high: 0 via LIST while a direct GET returns the real count);
• nothing warns that the data is partial.

Fix

Both GetList methods now follow the continuation token in a loop, accumulating every page before aggregating. The token is a monotonic SQLite rowid (rowid > :cont ORDER BY rowid), so paging is duplicate- and skip-free, and the loop terminates when the store returns an empty token (only set when a page is full).

Tests

Added ..._GetList_Pagination regression tests for both storages: create 500 objects in one namespace + 200 in a second (so the second namespace lives entirely on page 2), then assert both namespaces are present and all 700 objects are aggregated. Verified these fail against the pre-fix single-call code (1 namespace / count 500) and pass with the fix.

Scope

• kubescape/prometheus-exporter (the other consumer named in the issue) is handled separately in fix(api): follow continuation token when listing summaries (kubescape/storage#337) prometheus-exporter#52.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes

  • Fixed incomplete result retrieval for configuration and vulnerability summaries when query results exceed page limits.
  • Results now correctly aggregate across multiple pages.

• Tests

  • Added regression tests to verify pagination handling in configuration and vulnerability summary queries.

[coderabbitai]

Warning

Review limit reached

@matthyx, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 50 minutes and 25 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: d9f40401-cc2c-4d39-90f6-7c84b0ba5169

📥 Commits

Reviewing files that changed from the base of the PR and between 0a83362 and ab84de3.

📒 Files selected for processing (4)

• pkg/registry/file/configurationscansummarystorage.go
• pkg/registry/file/configurationscansummarystorage_test.go
• pkg/registry/file/vulnerabilitysummarystorage.go
• pkg/registry/file/vulnerabilitysummarystorage_test.go

📝 Walkthrough

Walkthrough

ConfigurationScanSummaryStorage.GetList and VulnerabilitySummaryStorage.GetList each replace a single realStore.GetList call with a pagination loop that clears the continuation token, fetches each page, appends items, and advances the token until exhausted. Regression tests verify multi-page aggregation for both storage types.

Changes

Pagination fix for scan summary aggregation

Layer / File(s)	Summary
ConfigurationScanSummary pagination loop and regression test pkg/registry/file/configurationscansummarystorage.go, pkg/registry/file/configurationscansummarystorage_test.go	GetList now copies opts into listOpts, clears the continuation token, and loops calling realStore.GetList while appending Items and advancing listOpts.Predicate.Continue until the token is empty. The regression test seeds 500 "alpha" and 200 "beta" summaries across two namespaces and asserts full cross-page aggregation.
VulnerabilitySummary pagination loop and regression test pkg/registry/file/vulnerabilitysummarystorage.go, pkg/registry/file/vulnerabilitysummarystorage_test.go	GetList replaces its single realStore.GetList call with the identical pagination loop pattern. The regression test creates enough VulnerabilityManifestSummary objects in two namespaces to span two pages, then asserts both namespaces appear and per-namespace scope counts match the created object counts.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐇 Hop, hop, hop through every page,
No more stopping at five hundred's cage!
Each namespace found, each summary whole,
The continuation token plays its role.
From alpha through beta, we gather them all —
No silent truncation, no datum shall fall! 🌟

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'fix: paginate summary aggregation to avoid 500-item truncation (#337)' accurately summarizes the main change—implementing pagination in summary aggregation to fix truncation at 500 items.
Linked Issues check	✅ Passed	The PR directly addresses issue #337 by implementing pagination in ConfigurationScanSummaryStorage.GetList and VulnerabilitySummaryStorage.GetList, eliminating silent data truncation and ensuring all objects are aggregated.
Out of Scope Changes check	✅ Passed	All changes are directly scoped to fixing the pagination bug: storage implementation changes for ConfigurationScanSummary and VulnerabilitySummary, plus corresponding regression tests.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/337-paginate-summary-aggregation

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Summary:

• License scan: failure
• Credentials scan: failure
• Vulnerabilities scan: failure
• Unit test: success
• Go linting: failure

[matthyx]

Review — no blockers

I reviewed the pagination fix against the underlying store semantics and ran the tests. No blocking issues found.

Correctness ✅

The loop matches the store's continuation contract exactly:

• storage.go sets a continuation token only when a page is full (len(list) == limit), and the token is the last SQLite rowid; listMetadata/listMetadataKeys query rowid > :cont ORDER BY rowid. So paging is strictly monotonic — duplicate-free, skip-free, and guaranteed to terminate.
• Resetting listOpts.Predicate.Continue = "" up front correctly discards any caller-supplied token (the aggregated GetList never emits one, so clients can't legitimately page it anyway).
• Accumulating page.Items across pages before aggregating is the right fix for the truncation.

Tests ✅

Verified locally with Go 1.25:

• Both ..._GetList_Pagination tests pass on this branch.
• Reverting only the two source files to main (keeping the new tests) makes both fail with exactly the truncation symptom — 1 namespace and High: 500 instead of 2 namespaces / 700. So they are genuine regression tests, not tautologies. 👍

Non-blocking note (no change required)

This trades "bounded-but-wrong" for "correct-but-unbounded": on a cluster with N summaries the aggregation now issues ⌈N/500⌉ sequential list pages and, in the ResourceVersionFullSpec path, one get per object, holding all N in memory before aggregating. That's inherent to a correct cluster-wide aggregation and fine for now, but if these endpoints ever get hit on very large clusters it'd be worth revisiting (e.g. aggregate incrementally per page rather than buffering all items). Not a blocker for this fix.

LGTM.

[github-actions]

Summary:

• License scan: failure
• Credentials scan: failure
• Vulnerabilities scan: failure
• Unit test: success
• Go linting: failure