Improve performance of pagination queries - #1207
Merged
InfiniteStash merged 4 commits intoAug 26, 2026
Merged
Conversation
Paginated entity queries projected full rows, so LIMIT/OFFSET made Postgres fetch a heap tuple for every row before the offset only to throw it away. Page 27,477 of the scene list produced 549,560 rows to return 20, at ~4.3GB of buffer traffic. Project ids instead and hydrate the page via LoadIds, which is the shape SearchScenesWithCount and Studio.Search already use. The id-only scan is covered by the existing partial indexes, so it stays index-only and the discarded rows never touch the heap. Heap fetches drop from 549,560 to ~700 on scenes, and 800,020 to 798 on edits: edits, created_at, offset 800k: 11,334ms -> 78ms scenes, date, offset 550k: 1,037ms -> 54ms Row multiplicity is unchanged, since ApplyMultiIDCriterion uses semi-joins and pre-grouped subqueries, and every LoadIds rebuilds results in the caller's id order. Sites are left alone: the query has no pagination and returns 113 rows, so the extra round trip would be a regression. Also renames Performer.LoadByIds to LoadIds, which was the only one of the eight services not using that name.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Paginating through queries is incredibly db intensive at high page counts when the queries load the entire objects, since the whole table has to be loaded and discarded.
This changes the queries to instead just paginate ids which can be quickly paginated, and then have the ids loaded separately like we already do for search queries.
Drafted with Claude.