From c8668a35e4333c5f053186b86aa5c90ba8a10f78 Mon Sep 17 00:00:00 2001 From: Richard Hull Date: Sat, 19 Jul 2025 12:20:04 +0100 Subject: [PATCH 1/3] Update README --- README.md | 126 +++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 116 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 26127a15..1fb75a41 100644 --- a/README.md +++ b/README.md @@ -1,19 +1,125 @@ # Postcode Polygons -## Data extraction and reprocessing +A high-performance Go API for spatial search and retrieval of UK postcode polygons and codepoints. It provides endpoints for querying postcode unit/district boundaries and codepoint locations, using open government and Royal Mail datasets. -First download (or otherwise generate) the **gb-postcodes-v5.tar.bz2** data file. See https://longair.net/blog/2021/08/23/open-data-gb-postcode-unit-boundaries/. +## Features -```console -$ curl https://postcodes-mapit-static.s3.eu-west-2.amazonaws.com/data/gb-postcodes-v5.tar.bz2 -O data/gb-postcodes-v5.tar.bz2 -$ go run main.go extract-data +- Fast spatial search for postcode codepoints and polygons +- REST API with bounding box queries +- Efficient in-memory spatial index (R-tree) +- Data extraction and reprocessing utilities +- Caching for polygon retrieval +- Docker support and CI/CD workflows + +## Example Usage + +### Data Extraction + +Download the NSUL polygons archive and extract/reprocess the data: + +```bash +curl https://postcodes-mapit-static.s3.eu-west-2.amazonaws.com/data/gb-postcodes-v5.tar.bz2 -o data/gb-postcodes-v5.tar.bz2 +go run main.go extract-data +``` + +This will regenerate the data files under `./data/postcodes`. + +### Starting the API Server + +```bash +go run main.go api-server +``` + +Or with custom options: + +```bash +go run main.go api-server --codepoint ./data/codepo_gb.zip --port 8080 --debug ``` -This will regenerate the (checked-in) data files under `./data/postcodes`. There is typically no need to run this unless -the NSUL datafile got updated +### API Endpoints + +- `GET /v1/postcode/codepoints?bbox=` +- `GET /v1/postcode/polygons?bbox=` + +## Architecture Overview + +### High-Level Flow + +```mermaid +graph TD + A[Client] -->|HTTP Request| B[API Server (Gin)] + B -->|/v1/postcode/codepoints| C[Spatial Index (R-tree)] + B -->|/v1/postcode/polygons| D[Polygons Repo (Memoized)] + C -->|Search| E[CodePoint Data] + D -->|Retrieve| F[GeoJSON Polygons] + F -->|Serve| A +``` -## Starting the API server +### Data Extraction -```console -$ go run main.go api-server +```mermaid +flowchart TD + X[NSUL Tar.bz2 Archive] -->|Extract| Y[GeoJSON FeatureCollections] + Y -->|Reprocess & Compress| Z[data/postcodes/units & districts] ``` + +### Key Components + +- **main.go**: CLI entrypoint, command routing +- **cmd/api_server.go**: API server setup, routes, middleware +- **cmd/extract_data.go**: Data extraction and reprocessing +- **spatial-index/**: R-tree spatial index for codepoints +- **internal/**: Polygon repo, file operations, caching +- **routes/**: API endpoint handlers + +## Development + +### Prerequisites + +- Go 1.24+ +- Data files: `data/codepo_gb.zip`, `data/postcodes/units/`, `data/postcodes/districts/` + +### Running Locally + +1. Extract data as above +2. Start the server: + + ```bash + go run main.go api-server + ``` + +3. Query endpoints with bounding box parameters. + +### Docker + +Build and run with Docker: + +```bash +docker build -t postcode-polygons . +docker run -p 8080:8080 postcode-polygons +``` + +## Testing + +Run all tests: + +```bash +go test ./... +``` + +Test coverage and reports are generated in `test-reports/`. + +## TODO & Future Enhancements + +- [ ] Add OpenAPI/Swagger documentation +- [ ] Support for additional spatial queries (e.g., nearest, within polygon) +- [ ] More granular error handling and logging +- [ ] Automated data updates from upstream sources +- [ ] Add authentication/authorization for API endpoints +- [ ] Improve Docker image size and build speed +- [ ] Add more unit and integration tests + +## License & Attribution + +MIT License. See `LICENSE.md` and `ATTRIBUTION.md` for further details. + From 462c2f055b6c13d46ddc928627cee06db997e435 Mon Sep 17 00:00:00 2001 From: Richard Hull Date: Sat, 19 Jul 2025 12:37:40 +0100 Subject: [PATCH 2/3] Added more detail --- README.md | 100 +++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 80 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index 1fb75a41..8f66f87d 100644 --- a/README.md +++ b/README.md @@ -13,49 +13,104 @@ A high-performance Go API for spatial search and retrieval of UK postcode polygo ## Example Usage -### Data Extraction +Use the `--help` flag to show what commands are available: -Download the NSUL polygons archive and extract/reprocess the data: +```console +$ go run main.go --help +HTTP server & data extraction -```bash -curl https://postcodes-mapit-static.s3.eu-west-2.amazonaws.com/data/gb-postcodes-v5.tar.bz2 -o data/gb-postcodes-v5.tar.bz2 -go run main.go extract-data -``` +Usage: + postcode-polygons [command] -This will regenerate the data files under `./data/postcodes`. +Available Commands: + api-server Start HTTP API server + completion Generate the autocompletion script for the specified shell + extract-data Extract NSUL polygons + help Help about any command + +Flags: + -h, --help help for postcode-polygons + +Use "postcode-polygons [command] --help" for more information about a command. +``` ### Starting the API Server -```bash -go run main.go api-server +```console +$ go run main.go api-server ``` Or with custom options: -```bash -go run main.go api-server --codepoint ./data/codepo_gb.zip --port 8080 --debug +```console +$ go run main.go api-server --codepoint ./data/codepo_gb.zip --port 8080 --debug +``` + +Use the `--help` flag with the **api-server** command to see what options are available: + +```console +$ go run main.go api-server --help +Start HTTP API server + +Usage: + postcode-polygons api-server [--codepoint ] [--port ] [--debug] [flags] + +Flags: + --codepoint string Path to CodePoint Open zip file (default "./data/codepo_gb.zip") + --debug Enable debugging (pprof) - WARING: do not enable in production + -h, --help help for api-server + --port int Port to run HTTP server on (default 8080) ``` ### API Endpoints -- `GET /v1/postcode/codepoints?bbox=` -- `GET /v1/postcode/polygons?bbox=` +- `GET /v1/postcode/codepoints?bbox=` returns a list of codepoints bound by the eastings/northings region. +- `GET /v1/postcode/polygons?bbox=` returns a [GeoJSON](https://geojson.org/) structure representing the postcode polygons that have codepoints inside the bounding box represented by the eastings/northings region. + +### Regenerating Postcode Data (optional) + +**NOTE:** this is not required for standard setup, only if you wish to regenerate the polygon data. + +Download the NSUL polygons archive and extract/reprocess the data: + +```console +$ mkdir data +$ curl https://postcodes-mapit-static.s3.eu-west-2.amazonaws.com/data/gb-postcodes-v5.tar.bz2 -o data/gb-postcodes-v5.tar.bz2 +$ go run main.go extract-data +``` + +This will regenerate the data files under `./data/postcodes`. + +Use the `--help` flag with the **extract-data** command to see what options are available: + +```console +$ go run main.go extract-data --help +Extract NSUL polygons + +Usage: + postcode-polygons extract-data [--polygon ] [flags] + +Flags: + -h, --help help for extract-data + --polygon string Path to NSUL polygons tar.bz2 file (default "./data/gb-postcodes-v5.tar.bz2") +``` ## Architecture Overview ### High-Level Flow +#### API Server + ```mermaid graph TD - A[Client] -->|HTTP Request| B[API Server (Gin)] - B -->|/v1/postcode/codepoints| C[Spatial Index (R-tree)] - B -->|/v1/postcode/polygons| D[Polygons Repo (Memoized)] + A[Client] -->|HTTP Request| B[API Server] + B -->|/v1/postcode/codepoints| C[R-Tree Spatial Index] + B -->|/v1/postcode/polygons| D[Polygons Repo] C -->|Search| E[CodePoint Data] D -->|Retrieve| F[GeoJSON Polygons] - F -->|Serve| A ``` -### Data Extraction +#### Data Extraction ```mermaid flowchart TD @@ -77,7 +132,12 @@ flowchart TD ### Prerequisites - Go 1.24+ -- Data files: `data/codepo_gb.zip`, `data/postcodes/units/`, `data/postcodes/districts/` +- Data files (all these locations are checked into the git repo): + - `data/codepo_gb.zip`, + - `data/postcodes/units/`, + - `data/postcodes/districts/` + +Note that `codepo_gb.zip` was originally sourced from https://www.ordnancesurvey.co.uk/products/code-point-open, and will be periodically synced into this repo. ### Running Locally @@ -121,5 +181,5 @@ Test coverage and reports are generated in `test-reports/`. ## License & Attribution -MIT License. See `LICENSE.md` and `ATTRIBUTION.md` for further details. +See `LICENSE.md` and `ATTRIBUTION.md` for further details. From b2ec43c357225f6257d4ebb220d30319ea84c703 Mon Sep 17 00:00:00 2001 From: Richard Hull Date: Sat, 19 Jul 2025 12:47:01 +0100 Subject: [PATCH 3/3] Indent header --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 8f66f87d..bb0fd320 100644 --- a/README.md +++ b/README.md @@ -62,7 +62,7 @@ Flags: --port int Port to run HTTP server on (default 8080) ``` -### API Endpoints +#### API Endpoints - `GET /v1/postcode/codepoints?bbox=` returns a list of codepoints bound by the eastings/northings region. - `GET /v1/postcode/polygons?bbox=` returns a [GeoJSON](https://geojson.org/) structure representing the postcode polygons that have codepoints inside the bounding box represented by the eastings/northings region.