Skip to content

Postgres: entity list/query paging without a sort has no ORDER BY, so offset pages duplicate and omit rows #142

Description

@rrrodzilla

What happens

crates/schema-forge-postgres/src/query.rs only emits ORDER BY when query.sort is non-empty. A paged walk with limit/offset and no sort therefore runs SELECT ... FROM "T" LIMIT 500 OFFSET N with no total order. Postgres does not guarantee a stable order across separate statements, so consecutive pages overlap and skip.

Measured on a 67,649-row Outlet table (dispatch-dev, Postgres backend, v0.37.x):

walk rows returned distinct ids duplicates
POST /entities/query with sort: [{field: id, direction: asc}] 67,649 67,649 0
GET /entities?limit=500&offset=N (no sort), walk 1 67,649 44,443 23,206
same, walk 2 67,649 46,788 20,861

Each unsorted walk returns the right total but about 30% of the rows are served twice and 30% never, and the missing set differs per walk. A client that dedupes by id silently loses a third of the table; one that does not gets a third duplicated. The Dispatch crawler loaded a random ~70% subset of its corpus on every pass until this was caught.

Expected

When the caller supplies no sort, paging should still be deterministic. Suggested: append ORDER BY "id" ASC as a trailing tiebreaker whenever limit or offset is present (and arguably always), on every backend. Callers that do sort should also get id appended as the final key so equal sort values page stably.

Related observation

Through the HTTP API, sort: [{field: "id", direction: "desc"}] came back ascending for both id and a timestamp field. The Postgres builder handles SortOrder::Descending correctly, so the direction appears to be dropped between the request body and the Query. Worth a test at the acton layer that a desc request produces DESC in the SQL.

Also: a filter on the id field returns 400 "unknown field 'id'", which forces id-range paging clients onto offset paging, where this bug bites.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions