-
Notifications
You must be signed in to change notification settings - Fork 1
refresh README benchmarks + cross-order table (issue #65) #131
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
cfb663f
020c5f4
bf2665c
d979666
ad10c6f
4037726
8c270dd
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,107 @@ | ||
| """Cross-order throughput benchmark — writes the table straight into | ||
| docs/benchmarks.md between the BENCH_CROSS_ORDER markers (and prints it). | ||
|
|
||
| python bench_cross_order.py | ||
|
|
||
| Self-contained: fixed-seed coordinate arrays, no external data. Reports raw | ||
| throughput (millions of morton indices per second) for encode (`geo2mort`) and | ||
| decode (`mort2geo`), plus mixed-order coverage (`morton_coverage_moc`) timing, | ||
| at representative orders. Throughput/timings are machine/run dependent; cell | ||
| counts are deterministic (fixed seed / fixed polygon). | ||
|
|
||
| Note on the coverage column: coverage cost scales with the polygon's *boundary | ||
| length measured in cells* (~2**order per boundary edge for the MOC), so a fixed | ||
| polygon covered at order 29 explodes. The coverage input is therefore a small | ||
| fixed ~0.01 degree (~1 km) box, chosen so order 29 stays tractable; see | ||
| docs/coverage_methods.md for the flat-vs-MOC / precision / budget trade-offs on | ||
| real-world polygons. Flat `morton_coverage` is not benchmarked cross-order | ||
| because it scales as ~4**order along the boundary and exhausts memory well | ||
| before order 29. | ||
| """ | ||
| import time | ||
| from pathlib import Path | ||
|
|
||
| import numpy as np | ||
|
|
||
| import mortie | ||
|
|
||
| DOC = Path("docs/benchmarks.md") | ||
| START, END = "<!-- BENCH_CROSS_ORDER:START -->", "<!-- BENCH_CROSS_ORDER:END -->" | ||
|
|
||
| SEED = 20260722 | ||
| N = 1_000_000 # coordinates encoded/decoded per order | ||
| ORDERS = [4, 12, 18, 29] | ||
|
|
||
| # Small fixed box (~0.01 degree ~ 1 km) for the coverage column; deliberately | ||
| # small so order-29 MOC coverage stays tractable (see module docstring). | ||
| BOX_LAT = np.array([10.0, 10.0, 10.01, 10.01]) | ||
| BOX_LON = np.array([20.0, 20.01, 20.01, 20.0]) | ||
|
|
||
|
|
||
| def coords(n=N, seed=SEED): | ||
| """Fixed-seed lat/lon arrays covering the whole sphere.""" | ||
| rng = np.random.default_rng(seed) | ||
| lat = rng.uniform(-89.0, 89.0, n) | ||
| lon = rng.uniform(-180.0, 180.0, n) | ||
| return lat, lon | ||
|
|
||
|
|
||
| def timed(fn, rep): | ||
| ts, out = [], None | ||
| for _ in range(rep): | ||
| t0 = time.perf_counter() | ||
| out = fn() | ||
| ts.append(time.perf_counter() - t0) | ||
| return float(np.median(ts)), out | ||
|
|
||
|
|
||
| def build_table(): | ||
| lat, lon = coords() | ||
| # Warm the extension once before timing anything: the first parallel call | ||
| # spins up the rayon threadpool and touches cold caches -- a one-time cost | ||
| # that is ~5x the warm time on a small cover (and negligible on a large | ||
| # batch). We report steady-state, so a throwaway call absorbs it here. | ||
| mortie.morton_coverage_moc(BOX_LAT, BOX_LON, order=6) | ||
| rows = ["| order | encode (M idx/s) | decode (M idx/s) | coverage (cells / ms) |", | ||
| "|--:|--:|--:|--:|"] | ||
| for order in ORDERS: | ||
| # encode: geo2mort(lat, lon, order) -> uint64 morton array | ||
| morton = mortie.geo2mort(lat, lon, order=order) # warmup + decode input | ||
| t_enc, _ = timed(lambda: mortie.geo2mort(lat, lon, order=order), 5) | ||
|
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🤖 from Claude (review) The lambdas close over the loop variable Generated by Claude Code |
||
| enc_mps = N / t_enc / 1e6 | ||
|
|
||
| # decode: mort2geo(morton) -> (lat, lon) | ||
| mortie.mort2geo(morton) # warmup | ||
| t_dec, _ = timed(lambda: mortie.mort2geo(morton), 5) | ||
| dec_mps = N / t_dec / 1e6 | ||
|
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🤖 from Claude (review) Throughput uses Generated by Claude Code |
||
|
|
||
| # coverage: warmed above; median of a few reps at cheap orders, a | ||
| # single timed call at order 29 (seconds-scale, one sample is plenty). | ||
| reps = 1 if order >= 24 else 5 | ||
| t_cov, cov = timed( | ||
| lambda: mortie.morton_coverage_moc(BOX_LAT, BOX_LON, order=order), | ||
| reps) | ||
|
|
||
| rows.append(f"| {order} | {enc_mps:,.1f} | {dec_mps:,.1f} | " | ||
| f"{len(cov):,}c / {t_cov * 1e3:.0f}ms |") | ||
| return "\n".join(rows) | ||
|
|
||
|
|
||
| def main(): | ||
| table = build_table() | ||
| print(table) | ||
| doc = DOC.read_text() | ||
| if START not in doc or END not in doc: | ||
| raise SystemExit(f"markers {START} / {END} not found in {DOC}") | ||
| pre = doc[: doc.index(START) + len(START)] | ||
| post = doc[doc.index(END):] | ||
| note = (f"Encode/decode throughput measured over {N:,} fixed-seed coordinates; " | ||
| "coverage over a fixed ~0.01 degree box. `M idx/s` = millions of morton " | ||
| "indices per second, `c` = cell count, `ms` = milliseconds. Throughput " | ||
| "and timings are machine/run dependent; cell counts are deterministic.") | ||
| DOC.write_text(f"{pre}\n\n{table}\n\n{note}\n\n{post}") | ||
| print(f"\nwrote table into {DOC}") | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| main() | ||
|
espg marked this conversation as resolved.
|
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| # Benchmarks | ||
|
|
||
| Absolute throughput for the core morton operations across representative HEALPix | ||
| orders. `mortie`'s morton core is Rust-only — there is no Python fallback to | ||
| compare against (the Rust extension is the sole runtime path), so these are raw | ||
| numbers, not a speedup ratio. | ||
|
|
||
| The two headline operations are the coordinate <-> morton conversions: | ||
|
|
||
| - **encode** — `geo2mort(lats, lons, order)`: lat/lon -> `uint64` morton indices. | ||
| - **decode** — `mort2geo(morton)`: morton indices -> lat/lon of the cell centre. | ||
|
|
||
| Both are reported as **millions of morton indices per second** (`M idx/s`), | ||
| measured over a fixed-seed array of one million coordinates spanning the sphere. | ||
| Higher is better; the order-encoded morton means throughput is nearly flat | ||
| across orders (the work is the same per element). | ||
|
|
||
| The **coverage** column times `morton_coverage_moc` — the compact mixed-order | ||
| polygon cover — on a small fixed box. It scales with the polygon's *boundary | ||
| length in cells* rather than element count, so unlike encode/decode it grows | ||
| steeply with order; see [coverage_methods.md](coverage_methods.md) for the | ||
| flat-vs-MOC / precision / budget trade-offs and a per-method matrix. | ||
|
|
||
| ## Cross-order throughput | ||
|
|
||
| The table below is regenerated by `bench_cross_order.py` (run from the repo | ||
| root) — it writes itself in place between the markers: | ||
|
|
||
| <!-- BENCH_CROSS_ORDER:START --> | ||
|
|
||
| | order | encode (M idx/s) | decode (M idx/s) | coverage (cells / ms) | | ||
| |--:|--:|--:|--:| | ||
| | 4 | 47.9 | 13.1 | 1c / 0ms | | ||
| | 12 | 38.9 | 10.5 | 4c / 0ms | | ||
| | 18 | 29.9 | 9.3 | 186c / 1ms | | ||
| | 29 | 26.9 | 7.6 | 419,292c / 4306ms | | ||
|
|
||
| Encode/decode throughput measured over 1,000,000 fixed-seed coordinates; coverage over a fixed ~0.01 degree box. `M idx/s` = millions of morton indices per second, `c` = cell count, `ms` = milliseconds. Throughput and timings are machine/run dependent; cell counts are deterministic. | ||
|
|
||
| <!-- BENCH_CROSS_ORDER:END --> | ||
|
|
||
| ### Notes on the numbers | ||
|
|
||
| - **Cell/coverage counts are deterministic** (a fixed seed and a fixed polygon); | ||
| **throughput and timings are machine/run dependent** — treat them as | ||
| order-of-magnitude, not reproducible constants. | ||
| - **encode/decode are element-wise**, so `M idx/s` is roughly order-independent; | ||
| the coordinate array is one million points and each order re-encodes the same | ||
| points. Each throughput figure is the warm median of five timed runs. | ||
| - **The coverage `ms` is a warm median** (five reps at cheap orders; a single | ||
| timed call at order 29, which is seconds-scale) taken after a throwaway warm-up | ||
| call — see *First-call warm-up* below. The cell count beside it is exact and | ||
| reproducible. | ||
| - **Coverage is deliberately scoped small.** The coverage input is a fixed | ||
| ~0.01 degree (~1 km) box, chosen so order 29 stays tractable. Coverage cost | ||
| grows with boundary length measured in cells (~2\*\*order per boundary edge for | ||
| the MOC), so a large real-world polygon covered at order 29 would be far more | ||
| expensive — cover it at a working order or with a `max_cells=` budget instead | ||
| (see [coverage_methods.md](coverage_methods.md)). Flat `morton_coverage` is not | ||
| benchmarked cross-order because it scales as ~4\*\*order along the boundary and | ||
| exhausts memory well before order 29. | ||
|
|
||
| ### First-call warm-up | ||
|
|
||
| The **first** call into the extension pays a one-time cost — the `rayon` | ||
| threadpool spins up and caches are cold — that later calls do not. The size of | ||
| the penalty is inversely proportional to the per-call work, so it matters most | ||
| for small, latency-sensitive calls: | ||
|
|
||
| | operation (measured) | cold (first call) | warm (steady state) | ratio | | ||
| |---|--:|--:|--:| | ||
| | `morton_coverage_moc`, ~1 km box, order 11 | ~1.1 ms | ~0.2 ms | ~5x | | ||
| | `mort2geo`, 1M indices, order 12 | ~196 ms | ~132 ms | ~1.5x | | ||
| | `geo2mort`, 1M indices, order 12 | ~47 ms | ~44 ms | ~1.05x | | ||
|
|
||
| So a big batch encode/decode barely notices it, but a small cover (or a single | ||
| point, or a tight per-call loop) can run several times slower on its first | ||
| invocation. Two practical consequences: | ||
|
|
||
| - **When benchmarking**, make one throwaway call before timing (this page's | ||
| generator warms the extension once up front, then reports the steady state). | ||
| - **In production**, if first-call latency matters (a request path, an | ||
| interactive tool), warm the functions you use once at startup with a small | ||
| throwaway call; steady-state throughput is what the table above reports. | ||
|
|
||
| ## Regenerating benchmarks | ||
|
|
||
| Both benchmark tables regenerate themselves in place, between HTML-comment | ||
| markers, on the current build. Rebuild the Rust extension first | ||
| (`maturin develop --release`, see [../BUILDING.md](../BUILDING.md)), then run | ||
| from the repo root: | ||
|
|
||
| ```bash | ||
| python bench_cross_order.py # this page: cross-order encode/decode/coverage | ||
| python bench_matrix.py # docs/coverage_methods.md: coverage-method matrix | ||
| ``` | ||
|
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🤖 from Claude (review) The reproducibility section mirrors how Generated by Claude Code |
||
|
|
||
| Each script prints its table and rewrites its target doc between the markers, so | ||
| committing the doc after a run keeps the published numbers current. Cell counts | ||
| are deterministic; timings are machine dependent — regenerate on the machine you | ||
| want to quote. | ||
Uh oh!
There was an error while loading. Please reload this page.