Distinguish duplicate feature locations during the index sort

[e-n-f]

Reimplements #349 (--distinguish-duplicates) by deferring duplicate locations in the index sort in main.cpp instead of buffering them into temporary files per tile in tile.cpp.

Background

Tippecanoe's behavior with as-needed dropping or coalescing has been to treat features that share a representative point with some other feature as having infinite density, therefore being the first to be dropped or coalesced when the tile size needs to be reduced. That interacts badly with datasets that intentionally repeat geometries with different sets of attributes, expecting the client to filter for one set or another, since only the first instance of each location survives.

--distinguish-duplicates defers the features that share a location to later passes through the feature sequence: all the distinct locations are written first, then the first duplicate of each location, then the second, and so on, so that each duplicate is dropped or coalesced within its own pass instead of into its twin.

Approach

The features are already being put into index order by radix1()/merge(), so the deferral happens there. A feature whose index matches the previous feature's is set aside in a temporary file for its pass, and replay_deferrals() appends the passes in order once the whole radix traversal has finished. tile.cpp is not touched, and the deferral applies uniformly to every zoom level rather than only to the first one.

Up to 50 duplicates of each location are distinguished. Beyond that they are all deferred to one final pass, where some of them will still have to be dropped or coalesced in some sequence, since otherwise a location with a huge number of duplicates would need an unbounded number of passes.

The index stays in index order

The geometry ends up in deferred order, but the index is still written in index order, because three later passes over it assume quadkey order:

• -zg maxzoom guessing, which measures the gaps between consecutive distinct indices
• -Bg/-rg base zoom and drop rate guessing, which groups consecutive features into tiles — this runs whenever the base zoom is unspecified, so it is the common path, and writing the index in deferred order silently spoils it, since tile[z] resets on every pass and a tile's count gets measured per pass instead of per tile
• --drop-denser

Keeping index order also means calc_feature_minzoom() still assigns feature minzooms in index order, matching #349.

The cost is that a deferred feature's index record is written before its geometry has a location, so replay_deferrals() corrects start/end with a pwrite at a remembered offset. The Mismatched index diagnostic in read_input() is suppressed under the option, since features are legitimately no longer consecutive in the geometry.

Verification

The test input from #349 decodes byte-for-byte identically to the expected output on that branch, including the structure of the final pass. (The limit has to be "50 distinguished plus one overflow pass" rather than 50 total to match.)

Separately, on a mixed input of 3000 scattered points plus 3 locations with 5 copies each, forced to drop by tile size limit:

	duplicate copies surviving at z0
--drop-densest-as-needed	0 of 15
--drop-densest-as-needed --distinguish-duplicates	12 of 15
--coalesce-densest-as-needed	0 of 15
--coalesce-densest-as-needed --distinguish-duplicates	12 of 15

Also checked: make test passes; -P parallel reading matches serial; --temporary-directory is respected; -zg, -Bg, -rg, and -aC all behave with duplicate-heavy input; 4000 features sharing one location complete without running out of files.

Incidental bug fix

Folding the two feature-writing paths in radix1() into one write_feature() fixes a latent bug in the path taken for a single oversized feature or maximum radix recursion depth. It wrote ix.end - ix.start bytes — the feature's whole serialized length, including the trailing minzoom byte — and then appended another minzoom byte, one byte more than merge() writes, which desynchronizes the length-prefixed geometry stream.

Before this change, --prefer-radix-sort fails with wrong length decoding feature on tests/ne_110m_ocean, tests/border, tests/epsg-3857, tests/loop, and tests/tl_2022_11_tract. All five now succeed and produce output identical to the non-radix path.

There is no test for that, because --prefer-radix-sort has a second, unrelated crash (unbounded radix1() recursion once splits collapses to 1) and the number of splits depends on the machine's file descriptor limit, so a test on that flag would risk being flaky. That crash is fixed separately.

---

Generated by Claude Code

[Copilot]

Pull request overview

This PR reintroduces --distinguish-duplicates by deferring duplicate feature locations during the radix/index sort so duplicate geometries are processed in later passes, improving “as-needed” dropping/coalescing behavior for intentionally duplicated geometries. It also updates docs/tests and bumps the release version.

Changes:

• Implement duplicate-location deferral during radix1()/merge() sorting in main.cpp, with deferred-pass replay that preserves index order.
• Add the --distinguish-duplicates option plumbing (options.hpp, serial.cpp) and update documentation (README.md, man/tippecanoe.1, CHANGELOG.md).
• Add a dedicated regression test fixture (tests/distinguish-duplicates/*) and bump version to 2.81.0.

Reviewed changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
main.cpp	Adds duplicate deferral/replay mechanism during index sort and suppresses “Mismatched index” diagnostic under the option.
serial.cpp	Ensures feature index is preserved when --distinguish-duplicates is enabled.
options.hpp	Introduces A_DISTINGUISH_DUPLICATES flag constant.
README.md	Documents the new --distinguish-duplicates option.
man/tippecanoe.1	Adds manpage entry for --distinguish-duplicates.
tests/distinguish-duplicates/in.json	Adds new test input with repeated point geometries.
tests/distinguish-duplicates/out/-z1_-r1_-b0_--distinguish-duplicates.json	Adds expected output for the new test.
CHANGELOG.md	Adds 2.81.0 entry noting the new option.
version.hpp	Bumps version string to v2.81.0.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.