What happens
crates/schema-forge-postgres/src/query.rs only emits ORDER BY when query.sort is non-empty. A paged walk with limit/offset and no sort therefore runs SELECT ... FROM "T" LIMIT 500 OFFSET N with no total order. Postgres does not guarantee a stable order across separate statements, so consecutive pages overlap and skip.
Measured on a 67,649-row Outlet table (dispatch-dev, Postgres backend, v0.37.x):
| walk |
rows returned |
distinct ids |
duplicates |
POST /entities/query with sort: [{field: id, direction: asc}] |
67,649 |
67,649 |
0 |
GET /entities?limit=500&offset=N (no sort), walk 1 |
67,649 |
44,443 |
23,206 |
| same, walk 2 |
67,649 |
46,788 |
20,861 |
Each unsorted walk returns the right total but about 30% of the rows are served twice and 30% never, and the missing set differs per walk. A client that dedupes by id silently loses a third of the table; one that does not gets a third duplicated. The Dispatch crawler loaded a random ~70% subset of its corpus on every pass until this was caught.
Expected
When the caller supplies no sort, paging should still be deterministic. Suggested: append ORDER BY "id" ASC as a trailing tiebreaker whenever limit or offset is present (and arguably always), on every backend. Callers that do sort should also get id appended as the final key so equal sort values page stably.
Related observation
Through the HTTP API, sort: [{field: "id", direction: "desc"}] came back ascending for both id and a timestamp field. The Postgres builder handles SortOrder::Descending correctly, so the direction appears to be dropped between the request body and the Query. Worth a test at the acton layer that a desc request produces DESC in the SQL.
Also: a filter on the id field returns 400 "unknown field 'id'", which forces id-range paging clients onto offset paging, where this bug bites.
What happens
crates/schema-forge-postgres/src/query.rsonly emitsORDER BYwhenquery.sortis non-empty. A paged walk withlimit/offsetand no sort therefore runsSELECT ... FROM "T" LIMIT 500 OFFSET Nwith no total order. Postgres does not guarantee a stable order across separate statements, so consecutive pages overlap and skip.Measured on a 67,649-row
Outlettable (dispatch-dev, Postgres backend, v0.37.x):POST /entities/querywithsort: [{field: id, direction: asc}]GET /entities?limit=500&offset=N(no sort), walk 1Each unsorted walk returns the right total but about 30% of the rows are served twice and 30% never, and the missing set differs per walk. A client that dedupes by id silently loses a third of the table; one that does not gets a third duplicated. The Dispatch crawler loaded a random ~70% subset of its corpus on every pass until this was caught.
Expected
When the caller supplies no sort, paging should still be deterministic. Suggested: append
ORDER BY "id" ASCas a trailing tiebreaker wheneverlimitoroffsetis present (and arguably always), on every backend. Callers that do sort should also getidappended as the final key so equal sort values page stably.Related observation
Through the HTTP API,
sort: [{field: "id", direction: "desc"}]came back ascending for bothidand a timestamp field. The Postgres builder handlesSortOrder::Descendingcorrectly, so the direction appears to be dropped between the request body and theQuery. Worth a test at the acton layer that adescrequest producesDESCin the SQL.Also: a
filteron theidfield returns 400 "unknown field 'id'", which forces id-range paging clients onto offset paging, where this bug bites.