Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 75 additions & 0 deletions .github/workflows/pages.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
name: Playground

on:
push:
branches:
- master
paths:
- '.github/workflows/pages.yml'
- 'apps/showcase/**'
- 'docs/benchmarks/showcase-latest.json'
- 'src/**'
- 'package.json'
- 'pnpm-lock.yaml'
workflow_dispatch:

permissions:
contents: read

concurrency:
group: github-pages
cancel-in-progress: false

jobs:
build:
name: Build playground
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
with:
persist-credentials: false

- name: Setup pnpm
uses: pnpm/action-setup@f40ffcd9367d9f12939873eb1018b921a783ffaa # v4
with:
version: 10.26.2

- name: Setup Node
uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
with:
node-version: 24
cache: pnpm
cache-dependency-path: pnpm-lock.yaml

- name: Install dependencies
run: pnpm install --frozen-lockfile

- name: Test playground
run: pnpm playground:test

- name: Build playground for GitHub Pages
run: pnpm playground:build:pages

- name: Configure Pages
uses: actions/configure-pages@13742983d7736d9b0ae728b81ab479565c72886d # v5

- name: Upload Pages artifact
uses: actions/upload-pages-artifact@7b1f4a764d45c48632c6b24a0339c27f5614fb0b # v4
with:
path: apps/showcase/dist

deploy:
name: Deploy playground
needs: build
runs-on: ubuntu-latest
permissions:
pages: write
id-token: write
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e # v4
26 changes: 12 additions & 14 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ on:
required: true
type: string
publish_npm:
description: Publish to npm. Requires NPM_TOKEN repo secret.
description: Publish to npm via trusted publishing or NPM_TOKEN bootstrap.
required: false
type: boolean
default: false
Expand All @@ -24,9 +24,10 @@ jobs:
release:
name: Validate, pack, and release
runs-on: ubuntu-latest
environment: npm
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
with:
persist-credentials: false

Expand All @@ -36,12 +37,11 @@ jobs:
version: 10.26.2

- name: Setup Node
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
with:
node-version: 24
cache: pnpm
cache-dependency-path: pnpm-lock.yaml
registry-url: https://registry.npmjs.org
package-manager-cache: false

- name: Install dependencies
run: pnpm install --frozen-lockfile
Expand Down Expand Up @@ -72,6 +72,9 @@ jobs:
- name: Build package
run: pnpm build

- name: Verify published entry points
run: pnpm verify:package-exports

- name: Pack tarball
run: |
mkdir -p .release
Expand All @@ -92,18 +95,13 @@ jobs:
GH_TOKEN: ${{ github.token }}
run: |
if gh release view "$RELEASE_TAG" >/dev/null 2>&1; then
echo "Release $RELEASE_TAG already exists" >&2
exit 1
echo "Release $RELEASE_TAG already exists; reusing it for an npm-only rerun."
else
gh release create "$RELEASE_TAG" .release/*.tgz --title "$RELEASE_TAG" --notes-file RELEASE_NOTES.md
fi
gh release create "$RELEASE_TAG" .release/*.tgz --title "$RELEASE_TAG" --notes-file RELEASE_NOTES.md

- name: Publish to npm
if: ${{ github.event_name == 'workflow_dispatch' && inputs.publish_npm }}
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
run: |
if [ -z "${NODE_AUTH_TOKEN:-}" ]; then
echo "NPM_TOKEN secret missing; refusing pnpm publish." >&2
exit 1
fi
pnpm publish --provenance --access public --no-git-checks
run: npm publish --provenance --access public
4 changes: 2 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@

Initial managed release for the @tanstack/db parity milestone.

- Ships the all-green 289/289 export compatibility catalog against
`@tanstack/db@0.6.5`.
- Ships the all-green 294/294 export compatibility catalog against
`@tanstack/db@0.6.16`.
- Adds TanStack DB migration docs, release notes, and head-to-head benchmark
coverage.
- Keeps ArrowBase columnar storage, snapshot/Arrow IPC, IndexedDB persistence,
Expand Down
32 changes: 32 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Contributing to ArrowBase

Thanks for helping improve ArrowBase. Bug reports, focused fixes, compatibility
cases, and benchmark methodology improvements are welcome.

## Development setup

Requirements: Node.js 20 or newer and pnpm 10.26.2.

```sh
pnpm install --frozen-lockfile
pnpm check
pnpm playground:check
pnpm build
pnpm verify:package-exports
```

Run `pnpm playground:dev` for the browser playground. Run
`pnpm playground:e2e` after installing Playwright's Chromium dependency when
changing user-facing flows.

## Pull requests

- Keep changes focused and add behavioral tests for public API changes.
- Preserve the exact compatibility catalog and operator-manifest gates.
- Treat benchmark results as evidence: compare equal result hashes before
reporting timing or memory ratios.
- Add a changelog entry for user-visible changes.
- Do not commit generated build output, credentials, or local benchmark noise.

By contributing, you agree that your contributions are licensed under the MIT
License in this repository.
99 changes: 63 additions & 36 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,63 +1,90 @@
# ArrowBase

ArrowBase — columnar Arrow store with @tanstack/db-compatible API.
[![CI](https://github.com/natanelia/arrowbase/actions/workflows/ci.yml/badge.svg)](https://github.com/natanelia/arrowbase/actions/workflows/ci.yml)
[![npm](https://img.shields.io/npm/v/arrowbase.svg)](https://www.npmjs.com/package/arrowbase)
[![license: MIT](https://img.shields.io/badge/license-MIT-171a18.svg)](./LICENSE)
[![playground](https://img.shields.io/badge/playground-open-df542d.svg)](https://natanelia.github.io/arrowbase/)

ArrowBase is an in-memory, local-first document store built on Apache
Arrow-style column buffers inside a `SharedArrayBuffer`. It keeps the
TanStack DB mental model (`createCollection`, transactions, live queries,
local-only/local-storage helpers) while adding columnar storage,
vectorized filters, snapshot/Arrow IPC export, IndexedDB persistence,
cross-tab sync, and React hooks.
> ArrowBase — columnar Arrow store with @tanstack/db-compatible API.

**Release status**: v0.1.0 @tanstack/db parity release, upgraded to
`@tanstack/db@0.6.16`. The compatibility catalog has expanded to 294/294 green
rows, while a runtime root-export audit plus an exact 28/28 operator-manifest
test guard the new upstream surface. Documented tolerances remain for fixed-capacity storage,
BigInt/Number key normalization, and floating-point aggregates.
ArrowBase is a local-first TypeScript database built on Arrow-style column
buffers. It keeps the TanStack DB mental model—collections, transactions,
live queries, persistence, and React hooks—while adding vectorized filters,
spatial indexes, Arrow IPC, snapshots, and cross-tab sync.

**Release docs**:
**v0.1.0 release candidate:** the compatibility catalog is 294/294 green
against `@tanstack/db@0.6.16`. Exact public-export and operator-manifest tests
guard that surface. Pre-1.0 means the API can still evolve between minor
versions; documented tolerances cover fixed capacity, safe-integer key
normalization, and floating-point aggregates.

- [COMPATIBILITY.md](./COMPATIBILITY.md) — export-by-export parity catalog.
- [MIGRATING.md](./MIGRATING.md) — side-by-side TanStack DB → ArrowBase guide.
- [docs/showcase.md](./docs/showcase.md) — playable ArrowBase vs TanStack DB playground and benchmark guide.
- [RELEASE_NOTES.md](./RELEASE_NOTES.md) — v0.1.0 parity release notes.
- [CHANGELOG.md](./CHANGELOG.md) — release history.
[Open the live playground](https://natanelia.github.io/arrowbase/) ·
[Read the migration guide](./MIGRATING.md) ·
[Inspect compatibility](./COMPATIBILITY.md)

**Project docs**:

- [COMPATIBILITY.md](./COMPATIBILITY.md) — all 294 audited upstream exports.
- [MIGRATING.md](./MIGRATING.md) — side-by-side TanStack DB replacements.
- [docs/showcase.md](./docs/showcase.md) — playground and benchmark contract.
- [docs/ivm.md](./docs/ivm.md) — incremental query engine boundaries.
- [SECURITY.md](./SECURITY.md) and [CONTRIBUTING.md](./CONTRIBUTING.md).
- [RELEASE_NOTES.md](./RELEASE_NOTES.md) and [CHANGELOG.md](./CHANGELOG.md).

## Requirements

- Node ≥ 20 or a browser with `SharedArrayBuffer` available
(COOP/COEP headers required in browsers).
- Node.js 20 or newer.
- Modern browsers. ArrowBase uses `SharedArrayBuffer` when cross-origin
isolation is available and safely falls back to `ArrayBuffer` otherwise.

## Install

```bash
pnpm install
pnpm build
pnpm test # unit + integration suite
pnpm bench # regression benches (tinybench)
pnpm bench:vs-tanstack # head-to-head @tanstack/db parity bench
pnpm bench:showcase # regenerate docs/benchmarks/showcase-latest.json
pnpm playground:dev # playable ArrowBase vs TanStack DB showcase
pnpm playground:test # operation parity + UI tests
pnpm playground:build # production showcase build
pnpm playground:e2e # desktop + mobile Playwright smoke
```sh
npm install arrowbase
```

## Showcase playground
Optional integrations are installed only when you use their subpaths:

`apps/showcase` is the kitchen-sink ArrowBase vs TanStack DB demo. It seeds deterministic city data, runs each headline operation through both an ArrowBase adapter and a TanStack DB adapter, compares the correctness hash first, and only then unlocks speed and memory ratios in the UI.
```sh
npm install @tanstack/db # arrowbase/tanstack
npm install apache-arrow # arrowbase/arrow
npm install react # arrowbase/react
```

Run it locally:
## Playground

```bash
The hosted [ArrowBase City Ops Console](https://natanelia.github.io/arrowbase/)
is an interactive ArrowBase vs TanStack DB playground. It runs deterministic
spatial, reactive, transactional, and IPC workloads in browser workers. A
correctness hash must match before timing or memory ratios unlock.

See [docs/showcase.md](./docs/showcase.md) for the benchmark contract, URL
controls, operation registry, and local or tunnel workflows.

## Development

```sh
pnpm install --frozen-lockfile
pnpm check # typecheck + 700+ behavioral tests
pnpm playground:check # UI typecheck + tests + production build
pnpm playground:e2e # desktop and mobile Playwright smoke
pnpm bench:vs-tanstack # head-to-head benchmark
pnpm build
pnpm verify:package-exports
```

Use these focused playground commands during UI work:

```sh
pnpm playground:dev
pnpm playground:typecheck
pnpm playground:test
pnpm playground:build
pnpm playground:e2e
```

The playground is mobile-friendly for Telegram viewing and can be shared temporarily with Cloudflare Tunnel; see [docs/showcase.md](./docs/showcase.md) for the `trycloudflare.com` workflow, bounded benchmark URL params, generated `docs/benchmarks/showcase-latest.json` snapshot, operation registry, and CI gates.
Performance ratios are evidence, not promises: the showcase compares equal
result hashes before presenting speed or memory measurements.

## What's in the box

Expand Down
4 changes: 2 additions & 2 deletions RELEASE_NOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@

Version: **v0.1.0**

Compatibility: 289/289 exports ✅
Compatibility: 294/294 exports ✅

## Summary

This release positions ArrowBase as a columnar Arrow store with an
`@tanstack/db`-compatible API. The public surface has been reviewed against
`@tanstack/db@0.6.5`, the compatibility catalog is all green, and migration docs
`@tanstack/db@0.6.16`, the compatibility catalog is all green, and migration docs
now show side-by-side replacements for common TanStack DB usage.

## Highlights
Expand Down
17 changes: 17 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Security policy

## Supported versions

ArrowBase is pre-1.0. Security fixes are applied to the latest published minor
release only.

## Reporting a vulnerability

Please do not open a public issue for a suspected vulnerability. Use
[GitHub private vulnerability reporting](https://github.com/natanelia/arrowbase/security/advisories/new)
so the report, reproduction, and remediation can be coordinated privately.

Include the affected version, runtime, a minimal reproduction, and the impact
you believe is possible. You should receive an acknowledgement within seven
days. Confirmed issues will be tracked in a private advisory until a fix and
coordinated disclosure are ready.
1 change: 1 addition & 0 deletions apps/showcase/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"scripts": {
"dev": "vite --host 0.0.0.0",
"build": "tsc --noEmit && vite build",
"build:pages": "tsc --noEmit && vite build --base=/arrowbase/",
"typecheck": "tsc --noEmit",
"preview": "vite preview --host 0.0.0.0",
"e2e": "playwright test",
Expand Down
Loading
Loading