datagovuk-sandbox

A Flask prototyping app for data.gov.uk ideas. Anything merged to main is automatically tested and deployed to AWS ECS Fargate.

Live: https://sandbox.test.data.gov.uk

---

CI / CD pipeline

flowchart LR
    PR[Pull Request] --> CI

    subgraph CI["CI - ci.yml"]
        Lint[ruff check]
        Format[ruff format --check]
        Tests[pytest]
        Lint --> Format --> Tests
    end

    CI -->|all green| Merge[Merge to main]

    subgraph Deploy["Deploy - deploy.yml"]
        OIDC[Assume AWS role\nvia GitHub OIDC]
        ECR{Image already\nin ECR?}
        Build[docker build + push\ntagged with commit SHA]
        TaskDef[Download current\ntask definition]
        Render[Swap in new\nimage tag]
        ECS[Register new revision\nDeploy to ECS]
        Wait[Wait for\nhealthy tasks]

        OIDC --> ECR
        ECR -->|no| Build --> TaskDef
        ECR -->|yes - skip| TaskDef
        TaskDef --> Render --> ECS --> Wait
    end

    Merge --> Deploy

Loading

---

Running locally

Important

Everything runs through Docker. Do not run uv sync, uv add, or flask directly on your machine - use the just commands below.

Prerequisites

Tool	Install
Docker Desktop	docs.docker.com
just	brew install just

Quickstart

git clone https://github.com/alphagov/datagovuk-sandbox
cd datagovuk-sandbox
cp example.flaskenv .flaskenv
just serve

App available at: http://localhost:5050

Commands

Command	What it does
just serve	Build the image and start the full stack
just index	(Re)create the OpenSearch index and load the markdown from collections/ (stack must be running)
just shell	Open a bash shell in the running web container
just add <package>	Add a Python package via uv add (stack must be running)

---

Contributing

main is protected - no direct pushes.

Important

Fake news. Everything goes here, just wishful thinking but if this were real then this would be true

1. Branch: git checkout -b feature/DGUK-XXX-short-description
2. Make your changes
3. Push and open a PR
4. CI must pass before merge

---

CI checks

The CI workflow (.github/workflows/ci.yml) runs on every PR and every push to main:

Check	Command	What it catches
Lint	ruff check .	Code errors and style violations
Format	ruff format --check .	Inconsistent formatting
Tests	pytest tests/ -v	Regressions

Run locally before pushing:

uv run ruff check .
uv run ruff format --check .
uv run pytest tests/ -v

---

Deployment

The deploy workflow (.github/workflows/deploy.yml) runs automatically on every merge to main:

1. Authenticates to AWS using GitHub OIDC - no credentials stored in GitHub Secrets
2. Checks if the image for this commit SHA already exists in ECR - skips build if it does (ECR tags are immutable)
3. Builds and pushes the Docker image tagged with the git commit SHA
4. Downloads the current ECS task definition, swaps in the new image tag
5. Registers the updated task definition as a new revision
6. Tells ECS to deploy and waits for healthy tasks before finishing

AWS resources

Resource	Value
AWS account	gds-ndl-test - 525320085442
Region	eu-west-2 (London)
ECS cluster	test
ECS service	datagovuk-sandbox
ECR registry	525320085442.dkr.ecr.eu-west-2.amazonaws.com/datagovuk-sandbox
ALB endpoint	http://test-datagovuk-sandbox-143318717.eu-west-2.elb.amazonaws.com
RDS (PostgreSQL 16)	test-postgres.clwmgus4m1tq.eu-west-2.rds.amazonaws.com:5432
Container logs	CloudWatch - /ecs/test/datagovuk-sandbox

The app runs in a private subnet behind an Application Load Balancer. It connects to RDS using IAM authentication - no hardcoded passwords. SECRET_KEY is generated by Terraform and injected as an environment variable. All infrastructure is managed in the datagovuk-infrastructure repo.

---

Adding a visualisation

All current charts are file-based - no database required.

data/<slug>.json    <- chart config (title, CSV file, column mapping, Highcharts options)
data/<slug>.csv     <- the data

The view at /visualisations/<slug> auto-discovers every data/*.json file. Adding a new chart is:

1. Drop your CSV into data/
2. Copy data/visualisation.json.template to data/<slug>.json
3. Fill in the column names and Highcharts config
4. Push to main - the deploy pipeline does the rest

See data/README.md for the full field reference and examples.

---

Search experimentation

Keyword search over two sets of docments, using Opensearch. No database yet - both indices are built directly from files in the repo.

	Source	Index	UI	JSON
Collections	application/content/collections/<collection>/<topic>.md	collections	/search/collections/keyword	/api/collections/keyword
Data directory	data/solr/solr_docs.json	datasets	/search/directory/keyword	/api/directory/keyword

They are deliberately kept separate. The two share no fields beyond the searched text - collection and page_last_updated exist only on collections, organisation, formats and licence only on datasets so a merged index gave facets that didn't line up and results that would be hard to read if not separated. Whether and how to merge them is a decision left open, not baked in.

Once the app is running (just serve), build both indices:

just index

Both are dropped and rebuilt every time, so re-run whenever either source changes. It takes a couple of seconds. http://localhost:5050/search is the hub, listing both UIs with sample API calls.

Facets

Tailored per corpus, so every facet shown is one the documents actually populate:

• Collections - Collection, Available as (website / API / dataset)
• Data directory - Organisation, Format, Licence

"Available as" is collections-only: the has_api / has_dataset / has_website flags come from the markdown frontmatter. Datasets have no equivalent.

status is indexed on collections but not faceted: every topic is for-publication today, so it would render one link filtering nothing.

The Licence facet only lists datasets that record a licence, 575 of the 752 have no licence field at all.

Both pages show 20 results per page.

Notes

Queries use a single OpenSearch query_string over title (boosted) and body, with AND as the default operator. There is no wildcard syntax. An empty query matches everything, so both pages open on the full corpus with all the facet counts, and the facets can be browsed without searching first.

The JSON endpoints return the OpenSearch response untouched - hits, aggregations, _score and all - rather than any server side post processing, so clients see is the real query result:

curl "http://localhost:5050/api/collections/keyword?q=flooding&available_as=api"
curl "http://localhost:5050/api/directory/keyword?organisation=durham-county-council&format=CSV"
curl "http://localhost:5050/api/directory/keyword?size=0"          # facet counts only

They take the same params as their UI (q, that index's facet params, and available_as on collections), plus size and from for paging.

Results are always in relevance order. There is no sort param: nothing in the UI offered a sort control, sort can be added later if needed.

---

Site checks for data.gov.uk

Scripts in scripts/ verify URLs on data.gov.uk collection pages using Playwright. They run automatically every day at 6am via .github/workflows/check-collection-urls.yml and commit results back to the repo.

To trigger a manual run, use the Actions tab in GitHub.

Running locally

# Pull collection URLs from the datagovuk_find repo
uv run python -m scripts.cli get-collection-urls

# Check each URL is reachable and on the correct page
uv run python -m scripts.cli check-urls

# Flag URLs with missing link text
uv run python -m scripts.cli check-link-text

Results CSV columns

Results land in testing/results/collection-check-<timestamp>.csv:

Column	Description
collection	Collection name (e.g. environment)
slug	URL path segment on data.gov.uk
url	The URL being checked
link-text	Display text for the link
type	website, api, or dataset
on-page	Whether the URL appeared as a link on the collection page
reachable	Whether the URL loaded successfully in the browser