Skip to content

Distinguish duplicate feature locations - #349

Open
e-n-f wants to merge 13 commits into
mainfrom
distiguish-duplicates
Open

Distinguish duplicate feature locations#349
e-n-f wants to merge 13 commits into
mainfrom
distiguish-duplicates

Conversation

@e-n-f

@e-n-f e-n-f commented May 20, 2025

Copy link
Copy Markdown
Collaborator

Tippecanoe's previous behavior with as-needed dropping or coalescing has been to treat features that share a representative point with some other feature as having infinite density, therefore being the first to be dropped or coalesced when the tile size needs to be reduced.

This interacts badly with some datasets that intentionally repeat geometries with different sets of attributes, expecting that the client will filter the features to show one set or another. In this case, only the first instance of each feature will be available if any features had to be dropped, leaving visible gaps if a filter instead tries to select one of the other copies.

This PR adds a new option, --distinguish-duplicates, which causes tippecanoe to recognize and to attempt to preserve duplicate feature locations. It will create up to 50 sub-layers of duplicate features within each layer, and features will be dropped or coalesced in their sub-layer's feature sequence instead of in the global feature sequence.

The new test's source file contains 60 copies apiece of three features A, B, and C. The output tiles contain 50 repeated blocks alternating A, B, and C (the detected duplicates), followed by blocks of the 10 remaining A features, the 10 remaining B features, and the 10 remaining C features.

@e-n-f
e-n-f marked this pull request as ready for review May 21, 2025 21:27
@e-n-f
e-n-f requested review from Copilot and removed request for arredond July 31, 2026 19:18

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a new --distinguish-duplicates mode to reduce unintended dropping/coalescing of intentionally duplicated geometries by treating duplicate feature locations as up to 50 “sub-layers” during feature sequencing, along with docs, changelog, and a regression test fixture.

Changes:

  • Add --distinguish-duplicates option and plumb it through serialization/tiling so duplicate indices can be preserved and processed per sub-layer sequence.
  • Implement duplicate deferral/rehydration logic during the first zoom traversal to avoid treating duplicates as “infinite density” in the global sequence.
  • Add a dedicated test input/output pair plus documentation and version/changelog updates.

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
version.hpp Bump version to v2.81.0.
options.hpp Add A_DISTINGUISH_DUPLICATES option constant.
serial.cpp Ensure sf.index is computed when --distinguish-duplicates is enabled.
main.hpp Expose tmpdir as a global for use outside main.cpp.
main.cpp Add long option parsing for --distinguish-duplicates and refactor tempdir plumbing.
tile.hpp Update traverse_zooms signature (remove tmpdir parameter).
tile.cpp Implement duplicate deferral using temporary files; thread/plumbing changes for first_zoom.
README.md Document --distinguish-duplicates and update/extend option documentation.
man/tippecanoe.1 Update manpage for new/clarified options and fix JSON examples.
CHANGELOG.md Add 2.81.0 entry referencing the new option.
tests/distinguish-duplicates/in.json New test input with repeated duplicate geometries and varying attributes.
tests/distinguish-duplicates/out/-z1_-r1_-b0_--distinguish-duplicates.json New expected output fixture for the duplicates behavior.
Suppressed comments (1)

tile.cpp:1176

  • This reads into a std::string buffer via s.c_str() (cast to void *), which is undefined behavior because c_str() returns a pointer to const data. Use a writable buffer like &s[0] after resize().
			s.resize(len);
			size_t n = geoms->fread((void *) s.c_str(), sizeof(char), s.size(), geompos_in);
			if (n != s.size()) {

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tile.cpp Outdated
Comment thread tile.cpp Outdated
Comment thread tile.cpp
Comment on lines +1144 to +1146
s.resize(len);
if (fread((void *) s.c_str(), 1, len, next_feature_state.deferrals[next_feature_state.which_deferral]) != (size_t) len) {
fprintf(stderr, "short read in deferred deserialization: %s\n", strerror(errno));
e-n-f and others added 3 commits July 31, 2026 12:29
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants