Skip to content

Feature/python api#91

Open
ctsilva wants to merge 17 commits into
urban-toolkit:mainfrom
ctsilva:feature/python-api
Open

Feature/python api#91
ctsilva wants to merge 17 commits into
urban-toolkit:mainfrom
ctsilva:feature/python-api

Conversation

@ctsilva

@ctsilva ctsilva commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

feat: add declarative Python API and runtime

Overview

This PR introduces a declarative Python API for Autark, following the Altair/Vega-Lite design pattern. It enables users to create urban visual analytics applications using high-level Python code that generates JSON specifications executed by a TypeScript runtime.

Core Components

Python API (python/autark/ - ~2k lines of code)

  • Declarative builders for data sources, transforms, views, and interactions
  • Jupyter integration via anywidget
  • Zero runtime dependencies in Python

Runtime System (autk-runtime/src/ - ~2k lines of code)

  • TypeScript runtime that executes declarative specifications
  • Data loader supporting OSM, GeoJSON, CSV, JSON, and GeoTIFF
  • Transform executor and view renderer

Schema (schema/autark-spec-v0.1.json)

  • Formal JSON Schema for validation
  • TypeScript type definitions

Key Features

  • Declarative approach - describe what, not how
  • Jupyter-native - renders as interactive widgets in notebooks
  • OSMnx compatible - tested with street network analysis
  • JSON/GeoTIFF support - new data sources with geometry support

Example Usage

import autark as ak

spec = ak.Spec(
    data=[ak.OSM("manhattan", ["buildings", "roads"])],
    views=[ak.Map(layers=[...]), ak.Histogram(...)],
    links=[ak.Link(source="map", target="histogram", action="filter")]
)
# Renders automatically in Jupyter

ctsilva and others added 17 commits June 9, 2026 23:20
Jupyter pages (e.g. :8888) load the runtime from a separate dev server
(:8000), which previously failed in two ways: browsers block cross-origin
Web Workers (DuckDB-WASM), and root-relative data/schema URLs resolved
against the notebook origin and 404ed.

- Create the DuckDB worker from a same-origin blob: URL that
  importScripts() the real worker script (DuckDB-WASM recommended pattern)
- Resolve root-relative data URLs against the runtime server origin in the
  Python to_html, and fetch the schema from the runtime module's origin
- Default runtime_url to the absolute dev-server URL
- Add CORS dev server, Jupyter quick-start script, and a Playwright
  cross-origin regression check (page on :8001, runtime on :8000)
- Consolidate Jupyter docs into python/examples/JUPYTER_TESTING.md

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Lock in the cross-origin worker fix with an automated regression test and
remove the manual-server requirements from the suite:

- Add tests/runtime/cross-origin-worker.test.ts: serves the page and the
  runtime from different origins (like Jupyter + cors_server.py) and
  asserts DuckDB initializes and the map renders without SecurityError
- Extract tests/helpers/static-server.ts and use it in jupyter-test.test.ts,
  which previously failed unless a server was started by hand on :8765
- Parametrize test-cross-origin.html via ?runtimeOrigin= and drop the
  redundant standalone check script
- Record verified Jupyter/VS Code manual testing in the implementation plan

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
spec.widget() now renders specs in Jupyter/JupyterLab/VS Code/Colab with a
fully bundled runtime shipped inside the Python package, removing the
local-dev-server requirement of _repr_html_().

To make the bundle work from the blob:/data: URLs anywidget uses:
- Bundle the spec schema inline (@autark-schema alias) instead of fetching
  it from the runtime server
- Fall back to jsDelivr CDN bundles for DuckDB worker/wasm assets when the
  module origin is not http(s)/file; the same-origin blob worker pattern
  already handles the cross-origin worker

New autk-runtime widget entry + vite.widget.config.js produce a single-file
ESM at python/autark/static/autark-widget.js (gitignored build artifact,
built by npm run build). Python side adds autark.create_widget /
Spec.widget() behind an autark[widget] extra, warning on root-relative data
URLs that have no server to resolve them.

Covered by tests/runtime/widget-bundle.test.ts, which imports the bundle
from a blob: URL like anywidget and serves jsDelivr requests locally.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Add zero-server widget demo cell (inline data) to the integration
  notebook and present spec.widget() as the primary Jupyter option in
  QUICKSTART/JUPYTER_QUICKSTART
- Replace stale plain http.server instructions with cors_server.py, which
  the dev-server display path requires
- Ignore Python build artifacts (build/, egg-info) produced by wheel builds

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- save_html() now embeds the widget runtime bundle by default, producing a
  standalone document that renders from file:// or any host with no dev
  server (DuckDB assets come from the jsDelivr CDN); pass runtime_url for
  the previous lightweight dev-server output. Adds ak.to_embedded_html().
- Add example specs 04 (geojson input), 05 (csv points + histogram), and
  06 (multi-layer OSM); 04/05 are executed in browser tests. Spec 05 omits
  the histogram-to-map link because highlight links to points layers crash
  the runtime - recorded as runtime TODO 5.
- Set up strict mypy for the Python package (dev extra, pyproject config)
  and fix the findings.
- Cover the embedded export end to end: Playwright test generates the file
  via the real Python path and asserts rendering with zero dev-server
  requests; new unit tests for display helpers; jsDelivr routing extracted
  to a shared test helper.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Implements full runtime support for JSON and GeoTIFF data sources, completing
the data loading pipeline for these formats.

Schema updates:
- Add JsonDataSource with url/values options and flatten flag
- Add GeoTiffDataSource with url, band selection, and CRS override
- Update DataSource union type to include both new sources

Runtime implementation:
- Add loadJsonSource() to handle URL and inline JSON data
- Add loadGeoTiffSource() for GeoTIFF files with band selection
- Update TypeScript types to match schema definitions
- Integrate with existing AutkDb.loadJson() and loadGeoTiff() methods

Python API updates:
- Update JSON class to support object/array values and flatten flag
- Update GeoTIFF class to include band parameter (1-based index)
- Add example scripts for both data source types

Documentation:
- Update PYTHON_API.md with usage examples and runtime support status
- Mark implementation tasks as completed in PYTHON_API_IMPLEMENTATION.md
- Add new example specs demonstrating JSON and GeoTIFF usage

The implementation enables new use cases for tabular data analysis (JSON)
and raster visualization (GeoTIFF), expanding Autark's data handling
capabilities.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
…ation

Implements complete runtime support for JSON and GeoTIFF data sources,
including geometry support for JSON and OSMnx integration examples.

JSON/GeoTIFF Runtime Support:
- Add JSON data source with geometry support (lat/lng and WKT)
- Add GeoTIFF raster data source with band selection
- Wire both into schema and runtime data loader
- Update Python builders to match schema

OSMnx Integration:
- Add osmnx_autark_workflow.py example showing OSMnx to Autark pipeline
- Document workflow for using OSMnx with Autark specs
- Successfully tested with real street network data

Updates:
- Fix example spec validation issues
- Update test notebook with working examples
- Add geometry support to JSON data source
- Fix GeoTIFF required fields in schema

Testing:
- JSON data loads and visualizes in tables/scatterplots
- OSMnx street networks successfully convert to Autark specs
- Jupyter notebook integration working
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant