Skip to content

geospatial-api/geoparquet-fastapi-serializer

BranchesTags

Repository files navigation

geoparquet-fastapi-serializer

One FastAPI endpoint. Four spatial representations. The client picks.

GET /features                                           -> GeoJSON FeatureCollection
GET /features   Accept: application/vnd.apache.parquet  -> GeoParquet 1.1
GET /features   Accept: application/geo+json-seq        -> newline-delimited GeoJSON
GET /features   Accept: text/csv                        -> CSV with WKT geometry
GET /features?format=geoparquet                         -> explicit override

Same query, same route, same handler. No duplicated endpoints, no /features.parquet twin, no result set held in memory.

What it does

geoparquet-fastapi-serializer is a response layer for FastAPI that turns rows of spatial data into whichever representation the caller asked for:

  • Content negotiation to RFC 9110 — real media ranges, q values, wildcards, specificity-based tie-breaking, and 406 Not Acceptable with a body that tells the client what it could have asked for. A ?format= query parameter overrides the header when you need a shareable link.
  • GeoParquet 1.1 output — ISO WKB geometry column plus a conformant geo metadata block carrying version, primary_column, encoding, geometry_types, a full PROJJSON crs, bbox, edges, and optionally the 1.1 covering.bbox struct column that makes cloud range-reads fast.
  • Genuine streaming — Parquet is written one row group at a time through a rolling buffer, GeoJSON is framed incrementally by hand, NDGeoJSON and CSV emit per row. Peak memory tracks your row-group size, not your result size. A million features serialise in the same ~30 MiB as a hundred thousand.
  • Permissive input — rows may carry geometry as Shapely objects, WKB bytes, hex WKB, WKT, GeoJSON mappings, or anything exposing __geo_interface__. Rows themselves may be dicts, SQLAlchemy Rows, named tuples, dataclasses or Pydantic models, from a list or an async cursor.
  • Correct OpenAPI — a spatial_responses() helper documents every negotiated media type on the route, with a real GeoJSON FeatureCollection JSON Schema, so /docs and generated clients stop pretending the endpoint only speaks application/json.

Why it exists

A spatial API that only speaks GeoJSON is quietly expensive for everyone downstream.

GeoJSON is text. Every coordinate becomes a decimal string, every feature repeats every property name, and nothing compresses columnar-wise because nothing is columnar. In the benchmark below, 100,000 modest point features cost 20.9 MiB as GeoJSON and 3.8 MiB as GeoParquet — 5.5x — and the GeoJSON has to be parsed by a JSON parser on the other end before it becomes usable, while the Parquet can be memory-mapped, column- projected and row-group-skipped.

But you cannot simply switch to GeoParquet: a browser cannot render it, and a curl user debugging your API wants to read the response. Both formats are correct answers to different questions, which is exactly the problem HTTP content negotiation was designed to solve — and which most spatial APIs solve instead by bolting a second endpoint onto the first, doubling the surface area that has to stay in sync.

The other half of the problem is memory. The obvious implementation builds a list of features and hands it to a serialiser, which means peak memory scales with result size and one large export can take down the process. Parquet in particular looks un-streamable, because its footer holds metadata — including the bounding box of every geometry — that you only know once you have seen all the data. It turns out that footer is written last, so the metadata can be assembled during a single streaming pass and committed at close. That is what this library does.

Install

Not published to any package index. Clone and install in editable mode:

git clone https://github.com/geospatial-api/geoparquet-fastapi-serializer.git
cd geoparquet-fastapi-serializer
python -m venv .venv && source .venv/bin/activate
pip install -e .

Or vendor the package directly into your project — it is self-contained, with no imports outside its own tree:

cp -r src/geoparquet_fastapi /path/to/your/project/

Then ensure fastapi, pyarrow>=14, shapely>=2 and pydantic>=2 are in your own dependencies.

Requires Python 3.10+.

Quickstart

from fastapi import FastAPI, Request

from geoparquet_fastapi import SerializationOptions, SpatialResponse, spatial_responses

app = FastAPI()

ROWS = [
    {"id": 1, "name": "Charing Cross", "geometry": "POINT (-0.1247 51.5074)"},
    {"id": 2, "name": "Tower of London", "geometry": "POINT (-0.0759 51.5081)"},
]


@app.get("/features", responses=spatial_responses(), response_class=SpatialResponse)
async def features(request: Request) -> SpatialResponse:
    return SpatialResponse(
        ROWS,
        request=request,
        options=SerializationOptions(id_column="id"),
        filename="landmarks",
        cache_control="public, max-age=300",
    )

That is the whole integration. SpatialResponse reads Accept and ?format= off the request, picks a representation, sets Content-Type, Content-Disposition and Vary, and streams.

A runnable version with bbox filtering over 12 in-memory London landmarks lives in examples/app.py:

uvicorn examples.app:app --reload

Streaming from a database

Nothing needs to be in memory. Hand it an async generator:

from sqlalchemy import select
from geoparquet_fastapi import from_sqlalchemy_result

@app.get("/features", responses=spatial_responses(), response_class=SpatialResponse)
async def features(request: Request, session: AsyncSession = Depends(get_session)):
    stmt = select(Feature.id, Feature.name, Feature.geom.label("geometry"))
    async with session.stream(stmt) as result:
        return SpatialResponse(from_sqlalchemy_result(result), request=request)

SQLAlchemy is not a dependency of this package; from_sqlalchemy_result duck-types the Result/AsyncResult API.

Content negotiation rules

Implemented per RFC 9110 §12.5.1, which is subtler than the "split on comma, take the first" approach most APIs ship.

Recognised media types (canonical type first; aliases select the same representation but are never echoed back as Content-Type):

Representation ?format= Canonical media type Aliases
GeoJSON geojson application/geo+json application/json, application/vnd.geo+json
GeoParquet geoparquet application/vnd.apache.parquet application/x-parquet, application/parquet
NDGeoJSON ndjson application/geo+json-seq application/x-ndjson, application/geojson-seq
CSV csv text/csv —

The rules, in order:

  1. A non-empty ?format= wins outright. An explicit override in the URL is a stronger signal of intent than a header the client may not control, and it makes responses shareable as plain links. An unknown value is 400, not 406 — it is a mistake in the URL, not a negotiation failure.

  2. A missing, empty or wholly unparseable Accept is treated as */*.

  3. Each media range gets a quality from its q parameter, defaulting to 1, clamped to [0, 1]. Malformed q values fall back to 1 rather than turning a routine request into a 406.

  4. The most specific matching range wins and supplies the quality. Specificity runs type/subtype;param > type/subtype > type/* > */*. This is the rule that surprises people:

    Accept: */*;q=0.9, application/geo+json;q=0.1   ->  GeoParquet, not GeoJSON

    The client said "anything at 0.9, but GeoJSON specifically at 0.1", so GeoJSON is deprioritised even though */* is listed first and scores higher.

  5. q=0 means explicitly unacceptable, not merely least-preferred.

  6. The canonical media type is authoritative, because it is what will be sent. Aliases are consulted only when the client said nothing about the canonical type — so Accept: application/json yields GeoJSON, but Accept: application/*;q=0.9, application/geo+json;q=0.1 does not let application/json smuggle the 0.9 back in.

  7. Remaining ties break by specificity, then by server preference order (GeoJSON, GeoParquet, NDGeoJSON, CSV). GeoJSON leads deliberately: it is what a browser sending */* should get, being the only one of the four that renders in a tab.

  8. If nothing is acceptable: 406, with a body listing what is on offer.

Individual malformed entries (json, text/, ;;;) are skipped rather than failing the whole header.

$ curl -s localhost:8000/features -H 'Accept: application/xml' | jq

{
  "detail": "no acceptable representation for Accept: 'application/xml'; server offers application/geo+json, application/vnd.apache.parquet, application/geo+json-seq, text/csv",
  "supported_media_types": [
    "application/geo+json",
    "application/vnd.apache.parquet",
    "application/geo+json-seq",
    "text/csv"
  ],
  "supported_formats": ["geojson", "geoparquet", "ndjson", "csv"]
}

Every negotiated response carries Vary: Accept, since the same URL yields four different bodies and caches must key on it.

Worked examples

All output below is real, produced by examples/app.py.

GeoJSON

$ curl -s 'localhost:8000/features?bbox=-0.13,51.50,-0.09,51.52' \
       -H 'Accept: application/geo+json' -D-

content-type: application/geo+json; charset=utf-8
content-disposition: attachment; filename="landmarks.geojson"
cache-control: public, max-age=300
vary: Accept

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "id": 1,
      "geometry": { "type": "Point", "coordinates": [-0.1247, 51.5074] },
      "properties": {
        "name": "Charing Cross",
        "category": "transport",
        "visitors_2023": 34000000,
        "rating": 4.1,
        "opened": "1864-01-11",
        "tags": { "zone": 1, "step_free": true }
      }
    },
    {
      "type": "Feature",
      "id": 3,
      "geometry": { "type": "Point", "coordinates": [-0.127, 51.5194] },
      "properties": {
        "name": "British Museum",
        "category": "museum",
        "visitors_2023": 5800000,
        "rating": 4.7,
        "opened": "1759-01-15",
        "tags": { "free": true }
      }
    }
  ],
  "bbox": [-0.127, 51.5007, -0.0759, 51.5194]
}

Note the bbox member sits after features. RFC 7946 places no ordering constraint on members, and deferring it is what allows the extent to be accumulated in the same single streaming pass that writes the features.

NDGeoJSON

$ curl -s 'localhost:8000/features?bbox=-0.13,51.50,-0.09,51.52&format=ndjson'

{"type":"Feature","id":1,"geometry":{"type":"Point","coordinates":[-0.1247,51.5074]},"properties":{"name":"Charing Cross","category":"transport","visitors_2023":34000000,"rating":4.1,"opened":"1864-01-11","tags":{"zone":1,"step_free":true}}}
{"type":"Feature","id":3,"geometry":{"type":"Point","coordinates":[-0.127,51.5194]},"properties":{"name":"British Museum","category":"museum","visitors_2023":5800000,"rating":4.7,"opened":"1759-01-15","tags":{"free":true}}}

One complete Feature per line, no enclosing collection. A consumer starts work on feature 1 without waiting for feature 1,000,000, a truncated transfer still yields every whole line before the cut, and wc -l, split and jq -c all work directly.

CSV

$ curl -s 'localhost:8000/features?bbox=-0.13,51.50,-0.09,51.52&format=csv'

id,name,category,visitors_2023,rating,opened,tags,geometry
1,Charing Cross,transport,34000000,4.1,1864-01-11,"{""zone"":1,""step_free"":true}",POINT (-0.1247 51.5074)
3,British Museum,museum,5800000,4.7,1759-01-15,"{""free"":true}",POINT (-0.127 51.5194)
5,St Paul's Cathedral,landmark,1300000,4.6,1710-10-20,,POINT (-0.0983 51.5138)
6,Tate Modern,museum,4700000,4.5,2000-05-11,"{""free"":true}",POINT (-0.0995 51.5076)
11,Thames Path (central),route,,,,"{""length_km"":6.4}","LINESTRING (-0.1246 51.5007, -0.0983 51.5079, -0.0759 51.5081)"

Geometry as WKT in the final column; nested values as compact JSON; nulls as empty cells.

GeoParquet

$ curl -s 'localhost:8000/features?format=geoparquet&covering_bbox=true' -o landmarks.parquet
$ python -c "import pyarrow.parquet as pq, json; \
  print(json.dumps(json.loads(pq.ParquetFile('landmarks.parquet').metadata.metadata[b'geo']), indent=2))"

{
  "version": "1.1.0",
  "primary_column": "geometry",
  "columns": {
    "geometry": {
      "encoding": "WKB",
      "geometry_types": ["LineString", "Point", "Polygon"],
      "crs": { "...": "full OGC:CRS84 PROJJSON" },
      "edges": "planar",
      "bbox": [-0.2944, 51.4769, -0.0005, 51.5313],
      "covering": {
        "bbox": {
          "xmin": ["bbox", "xmin"],
          "ymin": ["bbox", "ymin"],
          "xmax": ["bbox", "xmax"],
          "ymax": ["bbox", "ymax"]
        }
      }
    }
  }
}

Arrow schema: ['id', 'name', 'category', 'visitors_2023', 'rating', 'opened', 'tags', 'bbox', 'geometry']. The geometry column is placed last so a projected scan of the attributes never has to skip a wide binary column first.

API reference

SpatialResponse

The one you usually want.

SpatialResponse(
    content,                    # rows: list / iterable / async iterable
    *,
    request=None,               # FastAPI Request; supplies accept + ?format=
    accept=None,                # or pass the header directly
    format_override=None,       # or the format token directly
    representations=DEFAULT_REPRESENTATIONS,
    options=None,               # SerializationOptions
    status_code=200,
    headers=None,
    filename=None,              # sets Content-Disposition, extension auto-appended
    disposition="attachment",   # or "inline"
    cache_control=None,
    etag=None,                  # None | str | True (see below)
    background=None,
)

Passing accept=/format_override= instead of request= makes the response unit-testable with no HTTP layer at all.

Fixed-representation responses

GeoJSONResponse, GeoParquetResponse, NDGeoJSONResponse and CSVResponse take the same arguments minus the negotiation ones, and always emit their own format.

SerializationOptions

Option Default Meaning
geometry_column "geometry" Key in each row holding the geometry.
id_column None Key promoted to the GeoJSON Feature id. Removed from properties in GeoJSON, retained as a normal column in Parquet/CSV.
crs OGC_CRS84_PROJJSON PROJJSON object. None writes an explicit "unknown CRS".
edges "planar" "planar" or "spherical".
covering_bbox False Emit the GeoParquet 1.1 covering.bbox struct column.
bbox_column "bbox" Name of that column.
schema None Explicit pyarrow.Schema; bypasses inference.
sample_size 100 Rows sampled before the schema is frozen.
row_group_size 2048 Rows per Parquet row group, and the streaming chunk size.
compression "snappy" Parquet codec.
include_collection_bbox True Emit the GeoJSON collection bbox.

negotiate()

from geoparquet_fastapi import negotiate

negotiate("text/csv;q=0.9, application/geo+json;q=0.8").name   # 'csv'
negotiate("*/*").name                                          # 'geojson'
negotiate(None, "geoparquet").name                             # 'geoparquet'

Raises NotAcceptableError or UnsupportedFormatError. parse_accept() is exposed separately if you want the parsed media ranges.

Exceptions

All inherit GeoSerializationError, so one handler catches everything:

Exception Raised when
GeometryDecodeError A geometry value matches no supported input encoding.
InvalidCRSError crs= is not a plausible PROJJSON object (e.g. "EPSG:4326").
SchemaInferenceError Column types cannot be unified, or a row is not a mapping.
UnsupportedFormatError ?format= named an unknown representation.
NotAcceptableError Nothing on offer satisfies Accept.

OpenAPI

from geoparquet_fastapi import spatial_responses, format_query_parameter

@app.get(
    "/features",
    responses=spatial_responses(),
    response_class=SpatialResponse,
    openapi_extra={"parameters": [format_query_parameter()]},
)

spatial_responses() lists every media type and alias under 200 — with a full RFC 7946 FeatureCollection JSON Schema (all seven geometry types, including recursive GeometryCollection) for the GeoJSON variant, format: binary for Parquet, and worked examples for the text formats — plus a documented 406. Pair it with response_class=SpatialResponse so FastAPI stops advertising a default application/json schema inferred from the return annotation.

Streaming

Every representation streams, but they stream differently.

GeoParquet. The trick is that Parquet's footer — schema, row-group index and all key/value metadata — is written last. A ParquetWriter writes into a small non-seekable sink; after each row group the sink is drained and those bytes are yielded to the client. Because the footer has not been written yet, the geo metadata (which must carry the bbox of every geometry in the file) can be accumulated during the pass and committed with add_key_value_metadata() just before close. Peak memory is one row group in, one row group out.

GeoJSON. The FeatureCollection is framed by hand — {"type":"FeatureCollection","features":[, then one encoded Feature at a time, then ], the accumulated bbox, and }. Peak memory is proportional to the largest single feature, not the collection.

NDGeoJSON / CSV. One chunk per row. CSV samples the head of the stream to fix its header, since CSV cannot grow a column mid-file.

The schema for Parquet is frozen from the first sample_size rows, because Parquet is strongly typed and needs the schema before the first row group. Rows after the sample window that introduce a new type for an existing column will raise rather than silently corrupt the file — pass an explicit schema= when you know better than the sample.

row_group_size is the main tuning knob: it is both the Parquet row-group size and the streaming chunk size. The default 2048 favours time-to-first-byte; production exports usually want 50,000–150,000.

ETag and buffering

Streaming and strong ETags are genuinely in tension — an entity tag is a hash of the body, and the body does not exist until the stream ends, long after headers are sent. Rather than fake it:

  • etag=None (default) — stream, no ETag.
  • etag="..." — send that literal tag (use when you can derive one from a max(updated_at)).
  • etag=True — buffer the payload, hash it, and send ETag plus Content-Length.

etag=True is stable across processes and runs because the serialisers order everything deterministically — notably geometry_types is sorted, since an unstable ordering there would destabilise every Parquet ETag.

CRS and metadata

GeoParquet requires the crs field to be a PROJJSON object, not an EPSG:4326 string. Passing a string raises InvalidCRSError with a pointer to the right approach — this is the single most common GeoParquet conformance mistake.

Two correct PROJJSON documents ship with the package:

  • OGC_CRS84_PROJJSON — WGS 84, longitude first. The GeoParquet default, what GeoJSON mandates, and the default here.
  • EPSG_4326_PROJJSON — WGS 84, latitude first. Only use this if your coordinates really are stored latitude-first; emitting it over longitude-first data produces a file that reads back transposed.

For anything else, generate it with pyproj (not a dependency):

from pyproj import CRS
options = SerializationOptions(crs=CRS.from_epsg(27700).to_json_dict())

crs=None writes an explicit JSON null, which in GeoParquet means "the producer does not know the CRS" — meaningfully different from omitting the key, which would imply the OGC:CRS84 default.

The crs is written out in full even for the default, because omitting it survives less well through tools that rewrite metadata without preserving defaults.

covering.bbox

Enabling covering_bbox=True adds a bbox struct column (xmin, ymin, xmax, ymax as float32) and the matching covering entry in the geo metadata. This is what makes cloud-native range reads fast: a reader consults Parquet's per-row-group statistics on that column and skips entire row groups without fetching them.

Float32 is the spec's recommendation — the column is an index, and single precision halves its footprint. The values are rounded outward to the next representable float32, never to nearest. A covering box is a filter, so a box even one ULP inside its geometry would make a genuinely matching feature invisible.

Performance notes

Measured with benchmarks/serialize_bench.py on Python 3.12.13, pyarrow 25.0.0, shapely 2.1.2, Linux x86-64. Synthetic point features over Great Britain with seven mixed-type attributes each (int id, string name, string category, int, float, bool, Point) — deliberately not geometry-only rows, which would flatter Parquet's columnar compression relative to real feature data. Best of 3 runs, row_group_size=50,000.

$ python benchmarks/serialize_bench.py

repeat=3  row_group_size=50,000

10,000 features (7 attributes + Point geometry)
------------------------------------------------------------------------------
format            time          size       gzipped   bytes/feat   vs GeoJSON
GeoJSON            84ms       2.1 MiB     372.0 KiB         218B        1.00x
NDGeoJSON          61ms       2.1 MiB     371.9 KiB         218B        1.00x
CSV                52ms     894.4 KiB     318.0 KiB          92B        0.42x
GeoParquet         89ms     388.7 KiB     283.0 KiB          40B        0.18x

100,000 features (7 attributes + Point geometry)
------------------------------------------------------------------------------
format            time          size       gzipped   bytes/feat   vs GeoJSON
GeoJSON           777ms      20.9 MiB       3.6 MiB         219B        1.00x
NDGeoJSON         608ms      20.9 MiB       3.6 MiB         219B        1.00x
CSV               520ms       8.8 MiB       3.1 MiB          93B        0.42x
GeoParquet        924ms       3.8 MiB       2.7 MiB          40B        0.18x

Reading these honestly:

  • GeoParquet is 5.5x smaller uncompressed (20.9 MiB vs 3.8 MiB at 100k) — 40 bytes per feature against 219. That is the number that matters for egress cost and for clients on slow links.
  • GeoParquet is slightly slower to serialise (924ms vs 777ms at 100k). Snappy compression and Arrow array construction are not free. You are trading a modest amount of server CPU for a large amount of bandwidth, and for a client that can consume the result without a JSON parse. If your bottleneck is CPU and your clients are on a LAN, GeoJSON is a defensible choice.
  • After gzip the gap narrows sharply — 3.6 MiB vs 2.7 MiB, only 1.3x. If you already gzip everything at the edge, GeoParquet's wire-size win is much smaller than the raw numbers suggest. Its remaining advantages are parse cost, column projection and row-group skipping, not bytes on the wire.
  • NDGeoJSON is the same size as GeoJSON but ~20% faster, because it never builds collection framing and each line is encoded independently.
  • CSV is the fastest and smallest of the text formats, at the cost of losing types and nesting.

Memory

Peak resident memory while streaming GeoParquet, measured with tracemalloc, rows from a generator and output chunks discarded as they arrive (row_group_size=50,000):

Features Peak
10,000 5.6 MiB
100,000 29.7 MiB
1,000,000 29.7 MiB

Memory is bounded by row-group size, not result size — 1M features cost exactly what 100k does. A buffering implementation would be roughly an order of magnitude higher at the last row.

Limitations

Known and deliberate:

  • Parquet schema is frozen from the first sample_size rows. A column whose type changes after the sample window raises rather than silently coercing. Pass schema= when you know the shape up front.
  • CSV cannot grow columns mid-file. Keys first seen after the sample window are dropped, not appended, because appending would misalign every subsequent row.
  • etag=True disables streaming. By construction — see above.
  • No reprojection. Coordinates are written exactly as supplied; the crs field describes them, it does not transform them. Reproject in the database (ST_Transform) or with pyproj before serialising.
  • Only the WKB GeoParquet encoding. The 1.1 native Arrow geometry encodings (point, linestring, …) are not written. WKB is universally readable; the native encodings are not yet.
  • 2D bounding boxes. Z ordinates round-trip correctly in the geometry column and are reported in geometry_types ("Point Z"), but bbox and covering.bbox are XY only.
  • GeometryCollection in covering.bbox uses the collection's overall envelope, which is correct but loose.
  • Decimal columns become decimal128(38, scale) with the scale taken from the widest sampled value. A later value with more decimal places will fail rather than silently truncate.
  • No response compression. Use your ASGI server or reverse proxy — and note the gzip numbers above before deciding it is redundant.
  • Streaming responses have no Content-Length, so clients cannot show a progress bar. Use etag=True if you need one and can afford to buffer.

Testing

451 tests, no database, no network, no fixtures beyond in-memory data.

python -m venv .venv && source .venv/bin/activate
pip install -e '.[dev]'
python -m pytest -q
451 passed, 1 warning in 4.58s

ruff check .
ruff format --check .
All checks passed!
26 files already formatted

Coverage of note: ~90 table-driven negotiation cases (q-values, wildcards, specificity conflicts, malformed headers, real browser Accept strings), GeoParquet 1.1 metadata conformance, write/read round-trips asserting geometry equality via Shapely, PROJJSON presence and axis order, bbox correctness across row groups, covering.bbox containment under float32 rounding, bounded-memory streaming, lazy generator consumption, every geometry input encoding, empty result sets, null geometries, mixed geometry types, schema-inference edge cases, ETag stability, and end-to-end TestClient tests against examples/app.py.

CI runs ruff plus the full suite on Python 3.10, 3.11, 3.12 and 3.13.

Further reading

Background on the design decisions this library encodes:

License

MIT — see LICENSE.

About

FastAPI response layer that content-negotiates GeoParquet, GeoJSON, NDGeoJSON and CSV from a single endpoint, with streaming writers and GeoParquet 1.1 PROJJSON CRS metadata.

www.geospatial-api.com/core-geospatial-api-architecture-with-fastapi-postgis/geojson-vs-geoparquet-serialization/

Topics

apiasgicontent-negotiationfastapigeojsongeoparquetgeospatialgisparquetpyarrowpythonserializationspatialstreaming

Resources

Readme
MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages