Distinguish duplicate feature locations - #349
Open
e-n-f wants to merge 13 commits into
Open
Conversation
Make tmpdir a global to keep from needing to pass it around
e-n-f
marked this pull request as ready for review
May 21, 2025 21:27
There was a problem hiding this comment.
Pull request overview
This PR introduces a new --distinguish-duplicates mode to reduce unintended dropping/coalescing of intentionally duplicated geometries by treating duplicate feature locations as up to 50 “sub-layers” during feature sequencing, along with docs, changelog, and a regression test fixture.
Changes:
- Add
--distinguish-duplicatesoption and plumb it through serialization/tiling so duplicate indices can be preserved and processed per sub-layer sequence. - Implement duplicate deferral/rehydration logic during the first zoom traversal to avoid treating duplicates as “infinite density” in the global sequence.
- Add a dedicated test input/output pair plus documentation and version/changelog updates.
Reviewed changes
Copilot reviewed 11 out of 12 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| version.hpp | Bump version to v2.81.0. |
| options.hpp | Add A_DISTINGUISH_DUPLICATES option constant. |
| serial.cpp | Ensure sf.index is computed when --distinguish-duplicates is enabled. |
| main.hpp | Expose tmpdir as a global for use outside main.cpp. |
| main.cpp | Add long option parsing for --distinguish-duplicates and refactor tempdir plumbing. |
| tile.hpp | Update traverse_zooms signature (remove tmpdir parameter). |
| tile.cpp | Implement duplicate deferral using temporary files; thread/plumbing changes for first_zoom. |
| README.md | Document --distinguish-duplicates and update/extend option documentation. |
| man/tippecanoe.1 | Update manpage for new/clarified options and fix JSON examples. |
| CHANGELOG.md | Add 2.81.0 entry referencing the new option. |
| tests/distinguish-duplicates/in.json | New test input with repeated duplicate geometries and varying attributes. |
| tests/distinguish-duplicates/out/-z1_-r1_-b0_--distinguish-duplicates.json | New expected output fixture for the duplicates behavior. |
Suppressed comments (1)
tile.cpp:1176
- This reads into a
std::stringbuffer vias.c_str()(cast tovoid *), which is undefined behavior becausec_str()returns a pointer to const data. Use a writable buffer like&s[0]afterresize().
s.resize(len);
size_t n = geoms->fread((void *) s.c_str(), sizeof(char), s.size(), geompos_in);
if (n != s.size()) {
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+1144
to
+1146
| s.resize(len); | ||
| if (fread((void *) s.c_str(), 1, len, next_feature_state.deferrals[next_feature_state.which_deferral]) != (size_t) len) { | ||
| fprintf(stderr, "short read in deferred deserialization: %s\n", strerror(errno)); |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tippecanoe's previous behavior with as-needed dropping or coalescing has been to treat features that share a representative point with some other feature as having infinite density, therefore being the first to be dropped or coalesced when the tile size needs to be reduced.
This interacts badly with some datasets that intentionally repeat geometries with different sets of attributes, expecting that the client will filter the features to show one set or another. In this case, only the first instance of each feature will be available if any features had to be dropped, leaving visible gaps if a filter instead tries to select one of the other copies.
This PR adds a new option,
--distinguish-duplicates, which causes tippecanoe to recognize and to attempt to preserve duplicate feature locations. It will create up to 50 sub-layers of duplicate features within each layer, and features will be dropped or coalesced in their sub-layer's feature sequence instead of in the global feature sequence.The new test's source file contains 60 copies apiece of three features A, B, and C. The output tiles contain 50 repeated blocks alternating A, B, and C (the detected duplicates), followed by blocks of the 10 remaining A features, the 10 remaining B features, and the 10 remaining C features.