feat: q2 filter change + parallelized gridding for stats - #16
Merged
Conversation
pranav-walimbe
force-pushed
the
feat/fast-sampler
branch
3 times, most recently
from
August 25, 2026 02:17
857873b to
7d9dff3
Compare
collect_points made two serial passes over every point. estimate_distribution scattered into a sqrt(n) cell grid, which is n cells and so 12 MB at SF1 and 120 MB at SF10, then build_histogram scattered again into 32x32. Both now come from one parallel pass over a fixed 256x256 grid. At 256 KiB it stays in cache and is cheap enough to hold per thread, so the fill is a rayon fold and reduce with no atomics. Folding blocks of 8 gives the histogram exactly, since flooring at 256 then dividing by 8 lands on the same cell as flooring at 32. Distribution now uses Morisita's index of dispersion rather than a coefficient of variation. The old test needed a sqrt(n) grid to pin mean occupancy at 1.0, because raw CV depends on how crowded a cell is. Morisita's index reads 1.0 for a Poisson field at any grid size or point count, which is what frees the grid to be fixed and small. Statistics on real pickup points, best of 7: points before after delta 100,000 0.0006 0.0006 -5.9% 1,000,000 0.0061 0.0021 -66.5% 3,000,000 0.0224 0.0039 -82.7% The histogram drives early exits that return an empty result, so a cell that wrongly reads empty is a wrong answer and not a slow one. A differential run of 1000 queries over 3M real points, 515 of them landing in empty space, returns identical answers on both builds, and folded_histogram_matches_direct_binning holds the exactness in a test.
q2 scanned z_name and z_boundary for every zone, collected them, then filtered to Coconino County in Python and kept one row. The pinned reference SQL already puts the filter inside the subquery, so pushing it into scan_parquet matches the query rather than diverging from it. Polars prefilters, reading the predicate column and decoding z_boundary only for surviving rows. Measured on a fixture built to SF1's zone layout, six files of one row group each: 0.512s to 0.155s and a 2273 MiB peak to 1275 MiB, so -70% wall and -44% peak.
…ause 53 comments across the codebase followed the shape 'fact, so consequence' or 'fact, which explains it'. The clause after the comma restated what the code already showed or padded a claim the reader did not need. Each one now states its fact and stops. Where the trailing clause carried real information it moves into the sentence or onto its own line rather than hanging off a comma. Comments only. No code changed.
pranav-walimbe
force-pushed
the
feat/fast-sampler
branch
from
August 25, 2026 02:25
7d9dff3 to
49da8c1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.