nyc-eo-explorer

🚧 Work in progress. The explorer is usable today, but the corpus underneath it is still being refined — text quality, metadata, and coverage are all in motion upstream, and data shapes may change with them. Nothing here should yet be treated as a complete or authoritative record. Issues and contributions welcome.

A fully static explorer for the NYC mayoral executive orders corpus — search, filter, and browse the full corpus in the browser, with no server and no API.

Queries run in the browser: DuckDB-WASM executes real SQL over Parquet files served as static assets.

Vibe coded with Claude by BetaNYC.

Deployed to GitHub Pages from main by .github/workflows/deploy.yml: the corpus repo is checked out alongside, the Parquet and picture crops are rebuilt from it, and the whole thing is published as a static directory. Everything the page needs is served from Pages except the scanned source PDFs, which are 1.1 GB of git-LFS files and are read cross-origin instead — see "The source PDFs, side by side" below.

Requirements

• Node ≥ 20
• The data repo checked out as a sibling directory:

<parent>/
  nyc-executive-orders/   <- the corpus
  nyc-eo-explorer/        <- this repo

That layout is assumed by the data build. Override it with EO_CORPUS_DIR=/path/to/corpus.

Getting started

npm install
npm run dev          # http://localhost:5275

predev builds the Parquet artifacts automatically if they are missing or if the corpus changed upstream, so those two commands are the whole setup. No Python, no uv, no DuckDB install — the data build uses the @duckdb/node-api npm package.

Command	What it does
npm run dev	Dev server (rebuilds data first if stale)
npm run data	Force a data rebuild
npm run build / npm run preview	Production build / serve it
npm test	Vitest — pure functions and real-SQL integration
npm run lint / npm run format	ESLint / Prettier

How it works

The corpus repo publishes executive orders as Markdown and JSON. That's the right format for reading, diffing, and archiving, and the wrong one for searching 2,300 documents in a browser. This app translates it into something queryable, then does the querying client-side.

1. Build. npm run data reads the sibling corpus repo and rewrites it as Parquet — a compact columnar format DuckDB can read efficiently — in public/data/. Those files are derived artifacts: gitignored, and rebuilt whenever the corpus changes upstream.

File	Size	Contents
eo_meta.parquet	~80 KB	One row per order: dates, mayor, title, flags
eo_text.parquet	~1.7 MB	The full text of every order (~9 MB, compressed)
supersession_edges.parquet	~4 KB	Which orders revoked or amended which
pictures.json	~150 KB	Seals and signatures clipped out of the old scans
manifest.json	~1 KB	Counts, year range, checksums

Metadata and text are split because most of the app never needs the text. Browsing, filtering, and the charts run off the 80 KB file; the megabyte of text is fetched the first time you search or open an order.

2. Serve. Everything is a static file. There is no server, no API, and no database to run — npm run build produces a directory, and GitHub Pages hosts it.

3. Query. DuckDB-WASM runs in the browser and executes real SQL against those Parquet files, pulling them over HTTP a chunk at a time rather than downloading everything up front. Every search, facet count, and chart in the app is a SQL query. The engine is ~7 MB compressed, which is the largest single cost in the design and the reason it loads once and is shared across the whole app.

The one thing not served from Pages is the scanned source PDFs — 1.1 GB of git-LFS files, too large to host there. Those are read from media.githubusercontent.com, pinned to the corpus commit the site was built from.

Search

The corpus is heavily OCR'd — about a fifth of orders are flagged as noisy, and 67 have no recoverable text at all — so a word index is the wrong default: Adminstration will never match administration. Three modes are offered as equals instead.

Mode	What it does	Notes
Exact (default)	Plain substring match, case-insensitive	Always available
Regex	Full regex — e.g. admin[il1]strat[il1]on	For working around OCR
Relevance	BM25 ranking via DuckDB's full-text search	Opt-in; downloads ~480 KB

Relevance needs an extension that isn't built into duckdb-wasm, so selecting it fetches one at runtime. If that fails, search falls back to the other two modes rather than breaking.

Reading an order

The detail view shows the transcribed text next to the scan it came from. The text is Markdown-ish — the OCR emits headings, emphasis, and HTML tables — so it renders through marked and is sanitized with DOMPurify, with a Rendered / Raw toggle because anyone checking a transcription against the scan needs the characters as stored.

The scan is displayed with pdf.js's own viewer component, which brings continuous scroll, text selection, and fit-to-width along with it. Pre-1974 orders are a few pages inside a scanned bound volume, so the viewer opens on the order's first page (page 49 of 334, not page 1) and the toolbar lists the specific pages that order occupies — they aren't always contiguous, since the volumes have blank versos.

Those old volumes are also where pictures.json comes from: the OCR tags regions of each page as pictures — city seals, mayoral signatures — and upstream clips them out of the scans, since none of it survives in the text. The build matches each clip to whichever order was printed on that page. The Pictures tab turns that around and searches by text to show images: look up robert moses and you get the seals and signatures from the three orders that name him.

Gotchas worth knowing

• Column types are written explicitly in the build script. EO numbers are strings — "001", "23", "1.37" — and left to guess, DuckDB turns "001" into 1 and quietly loses data.
• The pre-1974 corpus is optional. Upstream is still OCR'ing those volumes; the build picks them up when they appear. Don't assume an eo_id is EO or EEO — that era also has 1951-EM-001, 1972-AM-007, and others.
• Cross-origin range requests need help. CORS hides Accept-Ranges from pdf.js, so without the custom transport in PdfViewer it downloads entire 49 MB volumes before showing a page.

Testing

npm test

• Unit tests cover the pure logic: the SQL builder, the query parser, hash-state round-tripping, and snippet extraction. These are the modules where a bug is silent — a wrong result set looks like a plausible result set.
• test/sql-integration.test.js runs every SQL shape the app can emit against a real DuckDB and the real Parquet. Counts are derived from the manifest, never hardcoded — the corpus grows as Phase E volumes land — so the assertions are invariants (the parts sum to the whole, a facet partitions the corpus). A unit test on a string builder can only prove it produced the string you expected — not that the string compiles. It skips itself if public/data has not been built.

Not covered: the browser worker, ?url asset resolution, and the FTS extension load. Those need a real browser; if they earn it, add Playwright as a separate test:e2e.

Conventions

Deliberately mirrors qa-ui/ in the data repo so the two read as one project: React 18.3 + Vite 6, plain .jsx (no TypeScript), one hand-written styles.css with a CSS-variable design system, no UI library, hooks only, and no router — deep links ride the URL hash.

One deliberate divergence: qa-ui serves its data through dev-server middleware, which means its npm run build produces a non-functional bundle. This app reads plain files from public/, so dev and production are the same code path. The one asset that is not a plain file — the scans, which are too large to be one — is the only place the two diverge, and src/paths.js is the single seam where that happens.

Chart colors were validated for colorblind separation and contrast rather than chosen by eye. text_quality uses a sequential ramp because it is an ordered scale, not a set of identities.

Want to help?

This is early-stage, openly built civic infrastructure — contributions to the app, its tests, or the corpus it reads are all welcome.

git clone https://github.com/BetaNYC/nyc-eo-explorer.git && cd nyc-eo-explorer
npm install && npm run dev     # builds the Parquet artifacts first if they're stale
npm test                       # unit + real-SQL integration

You need the corpus repo checked out as a sibling directory (see Requirements) — that is where the data comes from. Work on the data itself — OCR text, titles, metadata, coverage gaps — belongs upstream in that repo, which has its own review queue and open issues.

The one hard rule carries over from upstream: never invent or guess at content. This app displays a public record; an empty or flagged field is correct, a plausible substitute is a bug. Nothing in the UI may present derived or guessed data as if it came from the source document.

Questions, ideas, or a bug you've spotted: open an issue or hello@beta.nyc.

Part of BetaNYC's civic data tools

This explorer is one of a family of free, open civic data assets BetaNYC builds and stewards:

• nyc-executive-orders — the corpus this app reads: an open, machine-readable archive of NYC mayoral executive orders, 1974–present.
• ny-gov-web-registry — which NY government entities exist, and where they live on the web.
• ny-gov-web-archiver — the throttled Wayback Machine harvester behind the corpus's historical backfills.
• nyc-boundaries — NYC administrative boundaries, mapped and queryable.
• Seven MCP servers giving AI agents direct access to NYC/NYS civic data: Council legislation, City Record, Checkbook spending, 311, Charter/Code/Rules, NYS legislation, and the Council budget.

AI use in this project

BetaNYC uses AI tools openly and with human accountability. This repository was built by AI agents (Anthropic's Claude) working under the direction and review of BetaNYC staff — the app code was written by AI, and the scope and design decisions are human-made.

One commitment about the content: the executive orders this app displays are not AI-generated. Order text is served verbatim from the corpus, which retrieves it from official City sources and the Internet Archive — never summarized, paraphrased, or invented. The app renders and searches that text; it does not rewrite it. Where the corpus is incomplete or its OCR is imperfect, the UI says so rather than smoothing it over.

Questions about our approach: hello@beta.nyc.

License

• Code (this repository): MIT © 2026 BetaNYC — use and modify freely with attribution preserved.
• The data it displays — the corpus, the source scans, and the picture crops — lives in nyc-executive-orders under Creative Commons Attribution-ShareAlike 4.0: reuse and adapt with credit to BetaNYC, and share adaptations under the same license.

The underlying executive orders are public records of the City of New York; these licenses cover this tooling and the compilation it reads.

A note on sources

Documents shown here are reproductions of public records, retrieved from official City sources and the Internet Archive under its access guidelines. Text of the oldest orders is OCR output — faithful to the scan, not perfected — and is not legal advice; verify against the source PDF, which every record links to.