Page datastore queries that fetched all applications - #7234
Open
srinivasr wants to merge 3 commits into
Open
Conversation
ListApplications for piped, ListApplications for web, and the application lookup in CreateDeploymentChain ran one query with no limit and returned every matching row. A project with thousands of applications pays for all of them on every sync even though the RPC responses have no cursor field and cannot page. These now fetch 100 rows at a time and combine the pages before responding. Paging by cursor needs a fixed sort order, so the queries sort by Id. ListEvents also pages now. It defaults to newest first when a request leaves the order unset, which previously errored on the second page. Added the composite indexes these query shapes need to firestoreindexensurer; Firestore does not create them automatically. Fixes pipe-cd#7051 Signed-off-by: srinivasr <sriniv4sreddy@gmail.com>
srinivasr
requested review from
armistcxy,
khanhtc1202 and
t-kikuc
and
a lite review from Copilot
August 24, 2026 20:10
✅ Deploy Preview for pipecd-site ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR addresses unbounded datastore reads in PipeCD’s control-plane gRPC APIs by switching several “list everything” queries to internal cursor-based pagination, aggregating results server-side to preserve existing RPC contracts while avoiding single large queries (notably improving piped sync behavior for large projects).
Changes:
- Page and aggregate results for
ListApplications(piped + web) andCreateDeploymentChainapplication lookups, using stable ordering and a fixed per-page limit. - Page and aggregate
ListEvents, including a default stable ordering when the request does not specify one. - Add Firestore composite indexes required by the new query shapes and update index parsing tests; add a unit test for paged
ListApplications.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/app/server/grpcapi/web_api.go | Switch WebAPI ListApplications to internally page via listAllApplications with a fixed limit. |
| pkg/app/server/grpcapi/piped_api.go | Add paging + stable ordering for piped ListApplications, ListEvents, and application listing used by CreateDeploymentChain. |
| pkg/app/server/grpcapi/list_applications_test.go | Add unit tests verifying multi-page aggregation behavior for piped ListApplications. |
| pkg/app/server/grpcapi/application_lister.go | Introduce a shared helper to page/aggregate application lists via datastore cursor. |
| pkg/app/server/grpcapi/event_lister.go | Introduce a shared helper to page/aggregate event lists via datastore cursor. |
| pkg/app/ops/firestoreindexensurer/indexes.json | Add composite indexes for the new ordered/paged Firestore query patterns. |
| pkg/app/ops/firestoreindexensurer/indexes_test.go | Update expected parsed index list to include the new indexes. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…ering - Move listApplicationsPageSize to application_lister.go for colocation - Explicitly handle ListOrder_NONE in ListEvents to avoid masking client bugs - Add unit tests for ListEvents pagination and default ordering behavior Signed-off-by: srinivasr <sriniv4sreddy@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
Fixes unbounded datastore queries in
ListApplications(piped and web) andCreateDeploymentChain. These endpoints ran one query with no limit and returned every matching row; now they fetch pages of 100 and combine them before responding. Cursor paging needs a stable sort, so these queries order byId.ListEventsgets the same treatment, with a default of newest-first when a request leaves the order unset — without that default, paging would error on the second page.Adds the composite indexes these query shapes need to
firestoreindexensurer.Why we need it:
A project with thousands of applications pays for all of them on every piped sync. The
TODOinListApplicationsflagged this; until the reads are chunked, the query grows with the project and eventually risks datastore timeouts.Which issue(s) this PR fixes:
Fixes #7051
Does this PR introduce a user-facing change?:
No breaking changes. Responses still contain the complete aggregated list — they're just fetched from the backend in chunks.
firestoreindexensurercreates the new composite indexes automatically on control-plane startup, but GCP index builds take time — calls fail withFAILED_PRECONDITIONuntil each index finishes building. MySQL users need no action.