fix(api): follow continuation token when listing summaries (kubescape/storage#337)

[matthyx]

Problem

The storage backend (kubescape/storage) applies a hard-coded default page size of 500 when a client lists objects without an explicit limit, and returns a metadata.continue token for the rest:

// pkg/registry/file/storage.go
if opts.Predicate.Limit == 0 {
    opts.Predicate.Limit = 500
}

api/api.go lists summaries with ResourceVersionFullSpec but never consumed that continue token. On any cluster with more than 500 scanned objects (common in production), the exporter aggregated metrics from only the first page:

• whole namespaces beyond the first page silently disappeared from kubescape_controls_total_* / vulnerability metrics;
• per-namespace severity counters were wrong (e.g. a namespace reporting high: 0 via LIST while a direct GET returned high: 91);
• no warning anywhere — dashboards looked healthy while being incorrect.

Reported in kubescape/storage#337.

Fix

Add a generic listAllPages helper that follows the continuation token across pages, accumulating every object, and route all four Get* list methods through it.

• ResourceVersionFullSpec is preserved on every page request so each page still carries the full object spec. It is set via ResourceVersion (not ResourceVersionMatch), which is the only one allowed alongside a continue token — ValidateListOptions in apimachinery forbids ResourceVersionMatch + Continue, but not ResourceVersion + Continue.
• The two virtual resources (VulnerabilitySummaries, ConfigurationScanSummaries) ignore list options server-side and never return a continue token, so they resolve in a single iteration — unaffected.
• The aggregated list's continue token is cleared so downstream consumers don't mistake the merged result for a truncated page.

Scope

ConfigurationScanSummaryStorage.GetList (in the storage repo) is the server-side aggregation and is out of scope for this PR — it is tracked separately in kubescape/storage#337.

Tests

Added api/api_test.go covering:

• multi-page accumulation with correct continue-token threading and ResourceVersionFullSpec re-sent on every page;
• single-page (virtual-resource) behavior;
• error propagation.

go build ./..., go vet ./..., and go test ./... all pass.

Fixes part of kubescape/storage#337.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes

  • Improved reliability and completeness of list operations by ensuring vulnerability manifests, vulnerability summaries, workload configuration scans, and configuration scans return all available results across all pages.

• Tests

  • Added comprehensive test coverage for multi-page list operations, including continuation token handling, item aggregation, and error scenarios.

[coderabbitai]

Caution

Review failed

Pull request was closed or merged during review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: ca31a5be-bfec-490d-b0f7-875c4b7e41ec

📥 Commits

Reviewing files that changed from the base of the PR and between 8a87816 and 7a3d00d.

📒 Files selected for processing (2)

• api/api.go
• api/api_test.go

---

📝 Walkthrough

Walkthrough

Adds a generic pagedList constraint and listAllPages helper in api/api.go that follows Kubernetes continuation tokens to aggregate all pages into a single list. Four existing list methods (GetVulnerabilityManifestSummaries, GetVulnerabilitySummaries, GetWorkloadConfigurationScanSummaries, GetConfigScanSummaries) are updated to use this helper. A new test file covers multi-page aggregation, single-page, and error propagation cases.

Changes

Paginated list aggregation

Layer / File(s)	Summary
pagedList constraint and listAllPages implementation api/api.go	Defines the pagedList[T] generic constraint over Kubernetes list types (requiring GetContinue/SetContinue) and implements listAllPages, which loops List calls seeded with ResourceVersionFullSpec, threads Continue tokens between pages, accumulates items via a caller-supplied append callback, and clears the final Continue field before returning. Updates GetVulnerabilityManifestSummaries, GetVulnerabilitySummaries, GetWorkloadConfigurationScanSummaries, and GetConfigScanSummaries to delegate to listAllPages instead of a single List call.
Unit tests for listAllPages api/api_test.go	Adds summaryList and appendSummaries test helpers and three tests: multi-page follow (asserts ResourceVersion, Continue threading, item aggregation, and empty final Continue), single-page (asserts exactly one request), and error propagation (asserts nil result and errors.Is match on a later-page failure).

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related issues

• GetList silently truncates to 500 items — ConfigurationScanSummary aggregation and prometheus-exporter report wrong per-namespace data on larger clusters storage#337: The listAllPages helper added in this PR directly implements pagination via continuation tokens, fixing the silent truncation to the first 500 items that the issue reports for prometheus-exporter and other consumers.

Poem

🐇 Hop, hop, hop through pages we go,
No more stopping at item five-oh-oh!
Continue tokens we faithfully chase,
Appending each list with elegant grace.
The rabbit collects every scan in the land —
All pages in paw, none left in the sand! 🌿

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 14.29% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title directly addresses the main change: implementing continuation token following in list operations to fix pagination for summaries, which is the core fix described in the PR objectives.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/list-pagination-resourceversion-fullspec

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

Summary:

• License scan: success
• Credentials scan: failure
• Vulnerabilities scan: failure
• Unit test: success
• Go linting: failure

[matthyx]

Review: no blockers

I traced the fix against the storage backend and apimachinery and it holds up. Summary of what I verified:

The load-bearing claim is correct. ValidateListOptions (apimachinery v0.35.0, pkg/apis/meta/internalversion/validation/validation.go) only forbids resourceVersionMatch + continue. Plain resourceVersion + continue is not rejected, so re-sending ResourceVersion: fullSpec on every page is accepted at the handler layer. The etcd3 store's "resource version is not allowed when using continue" guard does not apply here because kubescape/storage uses a custom StorageImpl, not etcd3.

The server contract matches the fix. In kubescape/storage pkg/registry/file/storage.go, GetListWithConn honors opts.Predicate.Continue and Limit, recognizes ResourceVersionFullSpec to return full specs on every page, and sets the continue token only when a page is exactly full (len(list) == Limit). So the boundary case where the total is an exact multiple of 500 terminates safely after one extra empty request.

No infinite-loop risk on the virtual resources. VulnerabilitySummaryStorage.GetList and ConfigurationScanSummaryStorage.GetList build a fresh aggregated list with no continue token, so GetContinue() returns "" and listAllPages resolves in a single iteration — consistent with the PR description.

Helper logic is sound. Token threading reads page.GetContinue() (current page, not the accumulating list), the first page is adopted by reference and later pages appended, and the aggregate's stale token is cleared before return. The unit tests cover multi-page threading, single-page, and error propagation.

Non-blocking notes (optional)

1. Only GetVulnerabilityManifestSummaries and GetWorkloadConfigurationScanSummaries (the real resources) actually benefit from this client-side pagination. GetVulnerabilitySummaries / GetConfigScanSummaries (virtual) still receive a server-side-truncated aggregation until kubescape/storage#337 lands — the PR says this, but a one-line code comment on those two methods pointing at #337 would save the next reader the trip.
2. listAllPages is intentionally unbounded (accumulates all objects in memory, which is the goal). Consider a debug-level log of the page count / total items on large clusters so a future "why is this slow / using memory" question has a breadcrumb. Not required.
3. Tests exercise the generic helper directly; the wiring of each Get* method (correct append func + fullSpec) is type-checked but not asserted. Trivial to leave as-is.

LGTM — safe to merge.

[github-actions]

Summary:

• License scan: success
• Credentials scan: failure
• Vulnerabilities scan: failure
• Unit test: success
• Go linting: failure