feat(rust): encode v2 float columns as a dictionary, behind a flag by CommanderStorm · Pull Request #1602 · maplibre/maplibre-tile-spec

CommanderStorm · 2026-09-01T08:38:07Z

The float encoder can now emit a dictionary, chosen only when the column actually gets smaller once both layouts are costed in full with their headers. Off by default behind with_float_codecs, which is gated on unstable-v2.

github-actions · 2026-09-01T08:57:39Z

Performance Comparison

Performance Comparison `main` → `stack-v2-float-dict-encode`

Total Elapsed Time: 2.23s → 2.20s (-1.2%)
CPU Baseline: 81.29µs → 81.71µs (+0.5%)
Benchmark ID: mlt-convert

timing - Function execution time metrics.

+--------------------------------------+------------------------------+---------------------------------+---------------------------------+--------------------------------+------------------------------+
| Function                             | Calls                        | Avg                             | P95                             | Total                          | % Total                      |
+--------------------------------------+------------------------------+---------------------------------+---------------------------------+--------------------------------+------------------------------+
| optimizer::encode                    | 14096 → 14096 (+0.0%)        | 542.68µs → 537.25µs (-1.0%)     | 2.37ms → 2.33ms (-1.7%)         | 7.65s → 7.57s (-1.0%)          | 343.50% → 344.03% (+0.2%)    |
+--------------------------------------+------------------------------+---------------------------------+---------------------------------+--------------------------------+------------------------------+
| optimizer::encode_into               | 28192 → 28192 (+0.0%)        | 174.03µs → 172.53µs (-0.9%)     | 745.47µs → 744.45µs (-0.1%)     | 4.91s → 4.86s (-1.0%)          | 220.32% → 220.96% (+0.3%)    |
+--------------------------------------+------------------------------+---------------------------------+---------------------------------+--------------------------------+------------------------------+
| encode01::write_to                   | 28192 → 28192 (+0.0%)        | 137.95µs → 136.61µs (-1.0%)     | 656.89µs → 657.92µs (+0.2%)     | 3.89s → 3.85s (-1.0%)          | 174.64% → 174.96% (+0.2%)    |
+--------------------------------------+------------------------------+---------------------------------+---------------------------------+--------------------------------+------------------------------+
| writer::with                         | 1573034 → 1573034 (+0.0%)    | 1.70µs → 1.70µs (+0.0%)         | 2.31µs → 2.31µs (+0.0%)         | 2.67s → 2.68s (+0.4%)          | 120.09% → 121.59% (+1.2%)    |
+--------------------------------------+------------------------------+---------------------------------+---------------------------------+--------------------------------+------------------------------+
| mlt::main                            | 1 → 1 (+0.0%)                | 2.23s → 2.20s (-1.3%)           | 2.23s → 2.20s (-1.3%)           | 2.23s → 2.20s (-1.3%)          | 100.00% → 100.00% (+0.0%)    |
+--------------------------------------+------------------------------+---------------------------------+---------------------------------+--------------------------------+------------------------------+
| optimizer::analyze                   | 14096 → 14096 (+0.0%)        | 124.34µs → 123.74µs (-0.5%)     | 629.76µs → 632.83µs (+0.5%)     | 1.75s → 1.74s (-0.6%)          | 78.71% → 79.24% (+0.7%)      |
+--------------------------------------+------------------------------+---------------------------------+---------------------------------+--------------------------------+------------------------------+
| shared_dict::group_string_properties | 14096 → 14096 (+0.0%)        | 119.78µs → 119.42µs (-0.3%)     | 608.25µs → 609.28µs (+0.2%)     | 1.69s → 1.68s (-0.6%)          | 75.82% → 76.47% (+0.9%)      |
+--------------------------------------+------------------------------+---------------------------------+---------------------------------+--------------------------------+------------------------------+
| streams::dict_may_be_beneficial      | 28192 → 28192 (+0.0%)        | 49.30µs → 48.50µs (-1.6%)       | 220.29µs → 220.80µs (+0.2%)     | 1.39s → 1.37s (-1.4%)          | 62.41% → 62.11% (-0.5%)      |
+--------------------------------------+------------------------------+---------------------------------+---------------------------------+--------------------------------+------------------------------+
| encode::write_prop                   | 116572 → 116572 (+0.0%)      | 8.07µs → 7.97µs (-1.2%)         | 18.09µs → 18.24µs (+0.8%)       | 941.11ms → 929.76ms (-1.2%)    | 42.26% → 42.24% (-0.0%)      |
+--------------------------------------+------------------------------+---------------------------------+---------------------------------+--------------------------------+------------------------------+
| tile::from_tile                      | 28192 → 28192 (+0.0%)        | 20.18µs → 19.55µs (-3.1%)       | 78.40µs → 78.14µs (-0.3%)       | 569.04ms → 551.32ms (-3.1%)    | 25.55% → 25.05% (-2.0%)      |
+--------------------------------------+------------------------------+---------------------------------+---------------------------------+--------------------------------+------------------------------+
| shared_dict::write_shared_dict       | 24002 → 24002 (+0.0%)        | 23.46µs → 22.70µs (-3.2%)       | 53.18µs → 54.17µs (+1.9%)       | 563.08ms → 544.81ms (-3.2%)    | 25.29% → 24.75% (-2.1%)      |
+--------------------------------------+------------------------------+---------------------------------+---------------------------------+--------------------------------+------------------------------+
| strings::write_str_col               | 57070 → 57070 (+0.0%)        | 5.28µs → 5.47µs (+3.6%)         | 12.89µs → 12.93µs (+0.3%)       | 301.06ms → 312.17ms (+3.7%)    | 13.52% → 14.18% (+4.9%)      |
+--------------------------------------+------------------------------+---------------------------------+---------------------------------+--------------------------------+------------------------------+
| sort::curve_params                   | 14096 → 14096 (+0.0%)        | 16.91µs → 16.70µs (-1.2%)       | 79.36µs → 77.12µs (-2.8%)       | 238.43ms → 235.39ms (-1.3%)    | 10.71% → 10.69% (-0.2%)      |
+--------------------------------------+------------------------------+---------------------------------+---------------------------------+--------------------------------+------------------------------+
| strings::fsst_try_train              | 40536 → 40536 (+0.0%)        | 3.74µs → 3.51µs (-6.1%)         | 80.00ns → 80.00ns (+0.0%)       | 151.60ms → 142.24ms (-6.2%)    | 6.81% → 6.46% (-5.1%)        |
+--------------------------------------+------------------------------+---------------------------------+---------------------------------+--------------------------------+------------------------------+
| 🆕 streams::build_hilbert_dict       | 0 → 362 (+100.0%) ⚠️         | 0.00ns → 231.84µs (+100.0%) ⚠️  | 0.00ns → 407.81µs (+100.0%) ⚠️  | 0.00ns → 83.93ms (+100.0%) ⚠️  | 0.00% → 3.81% (+100.0%) ⚠️   |
+--------------------------------------+------------------------------+---------------------------------+---------------------------------+--------------------------------+------------------------------+
| 🗑️ strings::write_str_plain          | 57070 → 0 (-100.0%) 🚀       | 1.40µs → 0.00ns (-100.0%) 🚀    | 3.59µs → 0.00ns (-100.0%) 🚀    | 79.98ms → 0.00ns (-100.0%) 🚀  | 3.59% → 0.00% (-100.0%) 🚀   |
+--------------------------------------+------------------------------+---------------------------------+---------------------------------+--------------------------------+------------------------------+

alloc-bytes - Exclusive allocation bytes by each function (excluding nested calls).

+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| Function                             | Calls                        | Avg                          | P95                            | Total                          | % Total                    |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| tile::from_tile                      | 28192 → 28192 (+0.0%)        | 44.1 KB → 44.1 KB (+0.0%)    | 238.5 KB → 238.5 KB (+0.0%)    | 1.2 GB → 1.2 GB (+0.0%)        | 23.23% → 23.23% (+0.0%)    |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| optimizer::encode                    | 14096 → 14096 (+0.0%)        | 86.2 KB → 86.2 KB (+0.0%)    | 159.2 KB → 159.2 KB (+0.0%)    | 1.2 GB → 1.2 GB (+0.0%)        | 22.69% → 22.69% (+0.0%)    |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| writer::with                         | 1573034 → 1573034 (+0.0%)    | 723 B → 723 B (+0.0%)        | 25.0 KB → 25.0 KB (+0.0%)      | 1.1 GB → 1.1 GB (+0.0%)        | 20.74% → 20.74% (+0.0%)    |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| strings::fsst_try_train              | 40536 → 40536 (+0.0%)        | 16.8 KB → 16.8 KB (+0.0%)    | 3.6 MB → 3.6 MB (+0.0%)        | 664.5 MB → 664.5 MB (+0.0%)    | 12.70% → 12.70% (+0.0%)    |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| shared_dict::group_string_properties | 14096 → 14096 (+0.0%)        | 30.3 KB → 30.3 KB (+0.0%)    | 171.6 KB → 171.6 KB (+0.0%)    | 416.9 MB → 416.9 MB (+0.0%)    | 7.97% → 7.97% (+0.0%)      |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| strings::write_str_col               | 57070 → 57070 (+0.0%)        | 4.1 KB → 4.1 KB (+0.0%)      | 11.0 KB → 11.0 KB (+0.0%)      | 227.7 MB → 227.7 MB (+0.0%)    | 4.35% → 4.35% (+0.0%)      |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| encode01::write_to                   | 28192 → 28192 (+0.0%)        | 7.5 KB → 7.5 KB (+0.0%)      | 35.7 KB → 35.7 KB (+0.0%)      | 206.0 MB → 206.0 MB (+0.0%)    | 3.94% → 3.94% (+0.0%)      |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| shared_dict::write_shared_dict       | 24002 → 24002 (+0.0%)        | 6.6 KB → 6.6 KB (+0.0%)      | 37.4 KB → 37.4 KB (+0.0%)      | 154.9 MB → 154.9 MB (+0.0%)    | 2.96% → 2.96% (+0.0%)      |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| strings::write_raw_str_data          | 137744 → 137744 (+0.0%)      | 122 B → 122 B (+0.0%)        | 9.0 KB → 9.0 KB (+0.0%)        | 16.1 MB → 16.1 MB (+0.0%)      | 0.31% → 0.31% (+0.0%)      |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| streams::build_morton_dict           | 362 → 362 (+0.0%)            | 44.6 KB → 44.6 KB (+0.0%)    | 89.9 KB → 89.9 KB (+0.0%)      | 15.8 MB → 15.8 MB (+0.0%)      | 0.30% → 0.30% (+0.0%)      |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| optimizer::analyze                   | 14096 → 14096 (+0.0%)        | 1.0 KB → 1.0 KB (+0.0%)      | 6.5 KB → 6.5 KB (+0.0%)        | 14.0 MB → 14.0 MB (+0.0%)      | 0.27% → 0.27% (+0.0%)      |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| streams::build_hilbert_dict          | 362 → 362 (+0.0%)            | 25.9 KB → 25.9 KB (+0.0%)    | 90.7 KB → 90.7 KB (+0.0%)      | 9.2 MB → 9.2 MB (+0.0%)        | 0.18% → 0.18% (+0.0%)      |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| streams::dict_may_be_beneficial      | 28192 → 28192 (+0.0%)        | 256 B → 256 B (+0.0%)        | 256 B → 256 B (+0.0%)          | 6.9 MB → 6.9 MB (+0.0%)        | 0.13% → 0.13% (+0.0%)      |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| strings::write_str_plain             | 57070 → 57070 (+0.0%)        | 69 B → 69 B (+0.0%)          | 240 B → 240 B (+0.0%)          | 3.8 MB → 3.8 MB (+0.0%)        | 0.07% → 0.07% (+0.0%)      |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| encode::write_prop                   | 116572 → 116572 (+0.0%)      | 27 B → 27 B (+0.0%)          | 256 B → 256 B (+0.0%)          | 3.1 MB → 3.1 MB (+0.0%)        | 0.06% → 0.06% (+0.0%)      |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+

Threads

No threads to compare

Generated with hotpath-rs

Per-tile encoded size diff

zoom	tiles	prev_total	curr_total
0	1	71471	71471
1	4	220986	220986
2	16	615003	615003
3	64	1653851	1653851
4	256	3828904	3828904
5	1024	9716821	9716821
6	4096	32401914	32401914
ALL	5461	48508950	48508950

Top 3 improvements

Top 3 degradations

codecov · 2026-09-01T12:24:15Z

Codecov Report

❌ Patch coverage is 97.43590% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.42%. Comparing base (cafe9a3) to head (aa71b42).

Files with missing lines	Patch %	Lines
rust/mlt-core/src/encoder/stream/codecs.rs	92.59%	2 Missing ⚠️

Additional details and impacted files

@@                      Coverage Diff                       @@
##           stack-v2-float-dict-decode    #1602      +/-   ##
==============================================================
+ Coverage                       70.15%   70.42%   +0.26%     
==============================================================
  Files                             108      109       +1     
  Lines                           19219    19296      +77     
==============================================================
+ Hits                            13484    13589     +105     
+ Misses                           5735     5707      -28

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

`write_float_stream` wrote raw little-endian bytes and nothing else; it can now emit a codes stream followed by the distinct values those codes index. Keys are compared as bit patterns rather than as floats, so `0.0` and `-0.0` stay separate entries and every NaN with the same bits shares one. Both layouts are costed in full with their headers and the shorter wins, so a dictionary is chosen only when the column really does get smaller. Off by default behind `with_float_dict`, which is gated on `unstable-v2` because only v2 can express the encoding. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

CommanderStorm force-pushed the stack-v2-float-dict-encode branch 2 times, most recently from 187fe96 to 4d039dd Compare September 1, 2026 11:35

CommanderStorm force-pushed the stack-v2-float-dict-encode branch from 4d039dd to 11521c6 Compare September 1, 2026 12:04

CommanderStorm force-pushed the stack-v2-float-dict-encode branch from 11521c6 to bf1e885 Compare September 1, 2026 14:09

CommanderStorm force-pushed the stack-v2-float-dict-encode branch from 884bb23 to 25fe6d8 Compare September 1, 2026 14:14

CommanderStorm force-pushed the stack-v2-float-dict-encode branch from 25fe6d8 to f1c6a81 Compare September 2, 2026 13:20

CommanderStorm marked this pull request as ready for review September 2, 2026 16:42

Copilot AI lite review requested due to automatic review settings September 2, 2026 16:42

Copilot AI reviewed Sep 2, 2026

CommanderStorm force-pushed the stack-v2-float-dict-encode branch 2 times, most recently from 38cc32c to 44e5a2f Compare September 3, 2026 15:16

CommanderStorm force-pushed the stack-v2-float-dict-encode branch 2 times, most recently from ead1156 to 4c17da0 Compare September 3, 2026 21:13

CommanderStorm force-pushed the stack-v2-float-dict-encode branch from 4c17da0 to 3ca9c29 Compare September 4, 2026 08:53

CommanderStorm force-pushed the stack-v2-float-dict-encode branch 2 times, most recently from aa71b42 to bec7b9a Compare September 4, 2026 12:00

CommanderStorm force-pushed the stack-v2-float-dict-encode branch from 3150d73 to 8713224 Compare September 4, 2026 12:25

CommanderStorm force-pushed the stack-v2-float-dict-encode branch from 8713224 to b268f06 Compare September 4, 2026 13:43

CommanderStorm force-pushed the stack-v2-float-dict-encode branch 2 times, most recently from 8a59e54 to 978e670 Compare September 4, 2026 14:15

CommanderStorm force-pushed the stack-v2-float-dict-encode branch from 978e670 to 3a4412c Compare September 4, 2026 23:59

CommanderStorm force-pushed the stack-v2-float-dict-encode branch 2 times, most recently from 0eaaec7 to b56d8a0 Compare September 5, 2026 13:42

CommanderStorm force-pushed the stack-v2-float-dict-encode branch from b56d8a0 to 5b65344 Compare September 7, 2026 11:06

Navigation Menu

Uh oh!

feat(rust): encode v2 float columns as a dictionary, behind a flag - #1602

feat(rust): encode v2 float columns as a dictionary, behind a flag#1602
CommanderStorm wants to merge 1 commit into
stack-v2-float-dict-decodefrom
stack-v2-float-dict-encode

CommanderStorm commented Sep 1, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 1, 2026 •

edited

Loading

Performance Comparison `main` → `stack-v2-float-dict-encode`

timing - Function execution time metrics.

alloc-bytes - Exclusive allocation bytes by each function (excluding nested calls).

Threads

Top 3 improvements

Top 3 degradations

Uh oh!

codecov Bot commented Sep 1, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

CommanderStorm commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Performance Comparison main → stack-v2-float-dict-encode

timing - Function execution time metrics.

alloc-bytes - Exclusive allocation bytes by each function (excluding nested calls).

Threads

Top 3 improvements

Top 3 degradations

Uh oh!

codecov Bot commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

CommanderStorm commented Sep 1, 2026 •

edited

Loading

github-actions Bot commented Sep 1, 2026 •

edited

Loading

Performance Comparison `main` → `stack-v2-float-dict-encode`

codecov Bot commented Sep 1, 2026 •

edited

Loading