perf(query): DISTINCT+LIMIT early termination via lazy var-length expansion - #210
Merged
Merged
Conversation
…ansion RETURN DISTINCT ... LIMIT k with no ORDER BY and no aggregation may legally stop as soon as k (plus SKIP) distinct projected rows exist, but two things prevented it: the final-clause early-stop only handled plain row counts (a distinct cap is data-dependent, not a take(n)), and expand_variable_row materialized every path for an input row before anything downstream saw one — for [:KNOWS*1..3] at avg degree ~100 that Vec of ~1M paths WAS the query's cost, so no consumer-side stop could help. Two halves: - VarExpandIter replaces the eager BFS: an explicit-stack depth-first enumeration yielding each qualifying path's row as discovered. Same path set, same edge-isomorphism exclusions, same guards (relationship_expansion per edge, count_stream downstream); emission order becomes depth-first (unspecified without ORDER BY), and the unbounded *0.. depth-cap error is discovered lazily — identical outcome for any fully-drained query, but a satisfied DISTINCT+LIMIT may now finish before reaching the offending branch (the eager version did all that work and then discarded it). - collect_rows_until_distinct, final_stream_limit's DISTINCT counterpart: pulls the pipeline while projecting each row through the RETURN items and deduping with exactly dedup_rows' key (eval_return_expr -> value_hash_key), stopping at SKIP+LIMIT distinct rows, so materialize_return's semantics decide what counts as distinct. Measured (LDBC-style SF 0.1, ldbc_style_ops): IC1 friends *1..3 DISTINCT LIMIT 20: 643 ms -> 1.84 ms; IC5 fixed-2-hop DISTINCT LIMIT: 71 ms -> 714 us. All other suite rows unchanged against main measured under identical conditions. Laziness is regression-tested with a relationship-expansion budget that full enumeration exhausts but the early stop stays under. TCK: 3880/3880, unchanged.
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
RETURN DISTINCT ... LIMIT k(no ORDER BY, no aggregation) may legally stop once k distinct projected rows exist. Two blockers: the final-clause early stop only understood plain row counts, andexpand_variable_rowmaterialized every var-length path for an input row before anything downstream ran —[:KNOWS*1..3]at avg degree ~100 builds ~1M paths in that Vec, which is the entire cost of the benchmark's IC1. BENCHMARKS.md documented this as legal-but-unimplemented.What
VarExpandIterreplaces the eager per-row BFS with an explicit-stack DFS yielding rows as paths are discovered. Identical path set and edge-isomorphism exclusions;relationship_expansionstill meters every edge andcount_streamstill counts emitted rows. Two documented behavior notes: emission order within a hop is now depth-first (row order without ORDER BY is unspecified — full workspace + TCK pass unchanged), and the unbounded*0..depth-cap error is discovered lazily (identical for fully-drained queries; a satisfied DISTINCT+LIMIT may now succeed where the eager version errored after doing all the work anyway).collect_rows_until_distinct— the DISTINCT counterpart offinal_stream_limit: pulls the pipeline while projecting rows through the RETURN items and deduping with exactlydedup_rows's key, stopping at SKIP+LIMIT distinct rows.materialize_returnthen re-applies the same dedup to the ≤cap kept rows, so semantics are decided by the existing machinery.Measured (LDBC-style SF 0.1)
*1..3DISTINCT LIMIT 20Verification
ResourceLimit) but the DISTINCT+LIMIT run stays under — plus a correctness test asserting the LIMITed rows are a duplicate-free subset of the full DISTINCT set, with SKIP composition and LIMIT-larger-than-set casescargo test --workspacegreen; SF 0.1--ignoredworkload test green; TCK 3880/3880 unchanged; fmt + both clippy configs clean