Convert jsonpull to C++ with shared_ptr and std::vector/std::string

[e-n-f]

Modernize jsonpull: C++ rewrite with unique_ptr ownership

Summary

Replaces the hand-rolled malloc / realloc / free memory management
in jsonpull with a modern C++ object model: smart-pointer ownership,
std::vector for arrays and hash entries, std::string for string
values, and a small inheritance hierarchy for the value-bearing types.

The motivation was to make jsonpull less of a footgun (no more
json_free mistakes, no more reading o->value.string.string for a
node that turned out to be JSON_HASH) without giving up the ability
to inspect a partial tree, splice out subtrees, or hand a tree off to
outlive its parser. After several rounds of iteration, the final
shape is also a touch faster than the original C code on a real
workload.

What changed

Memory model

• json_object_ptr is now a std::unique_ptr<json_object, json_object_deleter>
  where json_object_deleter is a stateless deleter that dispatches on
  the discriminator and static_casts to the right subclass before
  delete. No virtual destructor, no per-node vptr; subclass dtors
  still run, so the std::vector / std::string members get freed.
• Every container owns its children through std::vector<json_object_ptr>
  (arrays) or std::vector<json_entry> (hashes, where json_entry is
  a {key, value} aggregate that preserves insertion order).
• The parser keeps a unique_ptr to the most recent top-level value in
  jp->root; json_read_tree / json_disconnect are the supported
  ways to move ownership out of the parser.
• Strings are std::string. Numbers store one of three representations
  (double, unsigned long long, long long) in a discriminated union.

Type-tagged subclasses

The original C json_object used a union for its value payload,
which kept primitive nodes small. The first cut of the C++ port
couldn't reuse that union (the new members -- std::string,
std::vector<json_object_ptr>, etc. -- are non-trivial and can't sit
in an untagged union), so it widened every node to one struct
containing all possible fields. That ballooned the per-node cost to
168 bytes, even for JSON_NULL / JSON_TRUE / JSON_FALSE.

Splitting json_object into a base class plus json_number /
json_string / json_array / json_hash subclasses recovers most
of what the union was buying in C, plus a bit more via hoisting
parser-only expect state out of the data model and packing
json_number's representation into a discriminated union:

Type	Size
json_object (TRUE/FALSE/NULL)	24 bytes
json_number	32 bytes
json_string	48 bytes
json_array	48 bytes
json_hash	48 bytes

Callers reach the typed payload through asserting accessors:
o->string(), o->number(), o->array(), o->entries(), etc.
The accessors assert the type matches before downcasting, so the
class of bug where a JSON_HASH is read as a string surfaces
immediately at debug time rather than as a silent garbage read.

Public API

Ownership is now explicit in the type signature:

Function	Returns	Meaning
json_read / json_read_separators	json_object *	Borrowed view into the parser-owned tree
json_hash_get	json_object *	Borrowed view into the containing hash
json_read_tree	json_object_ptr	Caller takes ownership; back-pointers cleared
json_disconnect	json_object_ptr	Splice subtree out and hand it to caller
json_free(json_object *)	void	Destroy subtree, removing it from its owner
json_stringify(const json_object *)	std::string	Serialize to a string

json_read still returns each parser token as the tree is being built
up (so callers can keep their existing "watch for a JSON_HASH whose
type field is "Feature"" pattern), and json_read_tree now clears
parent / parser back-pointers on the way out so detached trees can
outlive the json_pull they came from.

Per-call-site changes

jsonpull is used in a lot of places, so the migration is broad
(24 files, ~1800 added / ~560 deleted), but mechanical:

• Streaming parsers (geojson-loop.cpp, read_json.cpp::parse_layers,
  plugin.cpp::parse_feature) iterate one feature at a time, hand it
  to add_feature / build a serial_feature, then call json_free(j)
  to release it before the next iteration so already-serialized
  features don't pile up in memory.
• Filter loaders (evaluator.cpp::read_filter / parse_filter,
  main.cpp) hold filters as json_object_ptr; everywhere downstream
  that just needs to evaluate the filter takes a borrowed
  json_object *.
• Tree-mutating consumers (jsontool.cpp::join_csv) use
  json_read_tree to take ownership and then mutate the resulting
  tree directly through the new subclass accessors.
• Ephemeral-tree consumers (pmtiles_file.cpp, geobuf.cpp,
  dirtiles.cpp, attribute.cpp) just read attribute hashes through
  the accessors; no ownership transfer needed.

Bug fixes flagged in code review

Bundled in as part of the port:

• jsontool.cpp::out: route non-string values through
  json_stringify instead of o->string(), which now asserts on a
  non-string type. Previously --extract would crash on numeric
  attributes.
• geojson.{hpp,cpp}::json_end_map: take json_pull_ptr & so the
  caller's shared_ptr is actually released, null-check before
  touching jp->source, and clear jp->source after delete.
• jsonpull.cpp string parser: the low-surrogate range check was
  comparing the outer-loop byte c instead of the parsed code unit
  ch, breaking decoding when a high surrogate was followed by a
  non-low-surrogate BMP code point. Covered by a new regression test
  ("\uD83D\uE000" used to combine into U+1F400; now correctly emits
  the stale high surrogate as standalone CESU-8 then U+E000 normally).
• tile-join.cpp::handle_vector_layers: require JSON_STRING (and a
  non-null key) before calling string() so a non-string value
  doesn't trip an assert.

Performance

The first cut of the port (with shared_ptr ownership and per-token
std::string construction) was a real regression. Subsequent
commits brought it back:

Build	Median of 5 runs	Δ vs main
main (pre-refactor C)	8.69 s	--
C++ port, shared_ptr	10.57 s	+22 %
shared_ptr + cheap perf wins	9.33 s	+7 %
unique_ptr (this PR's tip)	8.52 s	-2 %

(time ./tippecanoe -z0 -f -o /tmp/foo.mbtiles --extend-zooms-if-still-dropping tl_2022_us_county.json, Apple Silicon, 238 MB input.)

The shared_ptr → unique_ptr switch alone was worth the largest
chunk; the rest came from pre-reserving small std::vectors, reusing
a parser-wide std::string buffer for string tokens to skip
SSO-promotion / capacity-doubling on each token, std::moveing
container pointers into and out of the parser's frame stack, and
packing json_number's representation into a single 8-byte slot via
the discriminator.

Tests

• New unit tests in unit.cpp:
  • Surrogate-pair regression ("\uD83D\uE000").
  • json_free pruning: parsing "[[1, 2], [3, 4], [5, 6]]" token by
    token, calling json_free on the [3, 4] node, and asserting the
    outer array ends up with [1, 2] and [5, 6].
  • json_free of the parser's current root really destroys the tree
    (verified by re-reading and checking jp->root is empty).
• Full make test passes (10488 assertions across 9 test cases) plus
  the entire tests/ integration suite, including the prefilter test
  (tests/ne_110m_admin_0_countries/out/--coalesce_-z2_-Ccat.json.check)
  that turned out to be a useful canary for "we're freeing partial
  parser state too early."

Commit-by-commit

1. Convert jsonpull to C++ with shared_ptr and std::vector /
   std::string.
2. Subclass json_object so primitives shrink from 168 to 24 bytes.
3. Store hash key/value pairs in one ordered vector.
4. Move parser-only expect state out of json_object.
5. Discriminate json_number's three numeric slots into one union.
6. Fix bugs flagged in code review of the C++ port.
7. Add jsonpull regression test for surrogate-pair decoding.
8. Cheap perf wins in jsonpull C++ port.
9. Make json_free actually free the subtree.
10. Migrate jsonpull to unique_ptr ownership.

[Copilot]

Pull request overview

This PR migrates the in-repo jsonpull JSON parser from C/manual allocation to C++ using std::shared_ptr ownership and STL containers, and updates the rest of the codebase to use the new json_object_ptr / json_pull_ptr APIs.

Changes:

• Replaces jsonpull.c with jsonpull.cpp and updates jsonpull.h to expose C++ types (std::shared_ptr, std::vector, std::string) and accessors.
• Updates GeoJSON parsing/serialization, filtering/evaluation, and metadata handling to use json_object_ptr and json_stringify() returning std::string.
• Propagates new pointer types through tile generation/join/overzoom tooling and related utilities.

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tile.hpp	Updates traverse_zooms() filter parameter to json_object_ptr.
tile.cpp	Updates filter/parser pointer types (json_object_ptr, json_pull_ptr) through tiling pipeline.
tile-join.cpp	Updates filter plumbing and JSON traversal to json_object_ptr and STL-backed accessors.
read_json.hpp	Updates function signatures to accept json_object_ptr / json_pull_ptr.
read_json.cpp	Switches JSON stringify usage to std::string and updates JSON navigation to new accessors.
pmtiles_file.cpp	Updates metadata JSON parsing/serialization to new JSON APIs and string return types.
plugin.hpp	Updates parse_feature() signature to accept json_pull_ptr.
plugin.cpp	Updates filter output parsing to json_object_ptr / json_pull_ptr and removes manual frees.
overzoom.cpp	Updates filter pointer type to json_object_ptr.
Makefile	Updates indent target to format jsonpull/jsonpull.cpp instead of jsonpull/*.[ch].
main.cpp	Updates stream JSON begin API and input parsing call sites to json_pull_ptr / json_object_ptr.
jsontool.cpp	Updates JSON plumbing to shared_ptr, STL, and json_stringify() returning std::string.
jsonpull/jsonpull.h	Introduces C++ shared_ptr-based API, STL-backed value storage, and accessor methods.
jsonpull/jsonpull.cpp	New C++ implementation of jsonpull parser/stringifier using shared ownership.
jsonpull/jsonpull.c	Removes the legacy C implementation.
geometry.hpp	Updates overzoom() signatures to accept json_object_ptr filter.
geojson.hpp	Updates JSON parse argument types and map-backed parser API types.
geojson.cpp	Updates GeoJSON serialization/parsing to new JSON APIs and shared ownership.
geojson-loop.hpp	Updates json_feature_action interface and parse entrypoint to shared_ptr types.
geojson-loop.cpp	Updates feature/geometry detection loop to new JSON APIs and std::string stringify.
geobuf.cpp	Updates parsing of embedded tippecanoe JSON to shared_ptr types.
evaluator.hpp	Updates filter/evaluation API to accept json_object_ptr and returns shared_ptr filters.
evaluator.cpp	Updates filter evaluation and error reporting to new JSON APIs and std::string stringify.
dirtiles.cpp	Updates directory metadata JSON parsing to shared_ptr types and STL traversal.
clip.cpp	Updates overzoom() signatures to accept json_object_ptr filter.
attribute.cpp	Updates JSON parsing for -E accumulation config to shared_ptr types and STL traversal.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 2 comments.