Add OPFS-backed browser storage and remove IndexedDB storage

[panarch]

Moved from gluesql/gluesql#1920 because gluesql-js is now maintained in this repository.

Original author: @panarch
Original issue: gluesql/gluesql#1920

---

Background

GlueSQL currently exposes async storage traits throughout the core execution layer. In practice, most existing storage backends are synchronous or blocking by nature: memory, file, redb, sled, web storage, redis, etc. MongoDB can also be supported through the Rust driver's synchronous API.

The main browser-side exception is IndexedDB. IndexedDB is asynchronous by design, and keeping it as a first-class core storage backend forces async concerns into the execution layer even though query planning, expression evaluation, joins, aggregation, and most storage implementations do not benefit from async execution.

IndexedDB also brings browser-specific complexity into the storage model, including async-only access, version upgrades for object store changes, transaction edge cases, and limitations tracked in existing issues:

• Flaw in the idb crate (dependency of gluesql-idb-storage crate) regarding transactions gluesql#1712
• Make IndexedDb storage implement index traits gluesql#1420
• Make IdbStorage storage support Transactions gluesql#1403
• storage request: absurd-sql gluesql#1383

OPFS provides FileSystemSyncAccessHandle, which allows synchronous file I/O inside a Dedicated Worker. If GlueSQL's WASM package runs inside a worker, the JavaScript API can remain Promise-based while the GlueSQL core and storage trait stay synchronous.

Proposal

Add an OPFS-backed storage backend for browser/WASM environments and remove IndexedDB storage support in the next breaking release.

High-level shape:

• Run GlueSQL WASM inside a Dedicated Worker.
• Keep the public JavaScript API asynchronous because worker RPC is asynchronous.
• Keep GlueSQL core execution and storage access synchronous.
• Use OPFS FileSystemSyncAccessHandle for synchronous read/write inside the worker.
• Remove gluesql-idb-storage and the JavaScript indexedDB engine path.

If OPFS synchronous access handles are unavailable, the OPFS backend should be unavailable. It should not silently fall back to IndexedDB or another persistent backend.

Migration stance

No dedicated IndexedDB-to-OPFS migration utility is proposed.

Users who need to migrate existing IndexedDB data can export data with the current IndexedDB-supported version, for example by selecting from existing tables and writing the result externally, then import it into a newer OPFS-backed setup.

Initial design direction

Avoid a row-per-file design, because OPFS file/directory handle acquisition is asynchronous. Instead, open the required OPFS file handles during worker initialization, then use synchronous access handles during query execution.

Possible MVP layout:

• one metadata/schema file
• one data/log file, or one file per table opened during initialization
• append-log records for schema and row changes
• in-memory offset index rebuilt at startup
• later compaction/checkpoint support

This is closer to a file storage backend than an IndexedDB key-value backend.

Why this matters

This would allow GlueSQL to simplify the core execution model:

• remove async from core storage traits and executor paths
• remove async wrappers around synchronous storages
• avoid async/blocking mismatches such as Async malpractices in the redb storage implementation? gluesql#1778
• avoid keeping IndexedDB as the design constraint for the whole core
• provide a cleaner long-term WASM storage story

Open questions

• What should the first OPFS storage format look like?
• Should OPFS storage be table-log based, page based, or reuse/adapt file-storage concepts?
• How should multi-tab locking be handled?

[juhee200]

@panarch
hi! I would like to take this issue.

Based on the design direction in the issue and the new sync baseline from #12, here's how I'd propose splitting the work:

Step 1 — opfs-storage crate (Rust only)

• New storages/opfs-storage implementing the sync Store/StoreMut traits, same shape as web-storage after Migrate to GlueSQL's synchronous API and remove IndexedDB storage #12
• Acquire OPFS file handles asynchronously during initialization, then use FileSystemSyncAccessHandle for all reads/writes during execution
• MVP file layout as suggested: one metadata/schema file + one data file per table, opened at init
• Error out (no silent fallback) when sync access handles are unavailable

Step 2 — Worker integration (JS side)

• Run the WASM module inside a Dedicated Worker; add an RPC layer so the main-thread API stays Promise-based
• Wire the opfs engine into CompositeStorage alongside memory/localStorage/sessionStorage

Questions:

• Would you prefer these as two separate PRs (storage crate first, worker integration second), or one PR?
• For testing, OPFS sync handles only exist in worker contexts — is extending the current webdriver-based CI setup the right direction?

If this plan looks reasonable, I'll start with Step 1.

[panarch]

I think we should initially expose OPFS through a separate worker-based browser entry point rather than moving the existing browser entry point into a worker.

FileSystemSyncAccessHandle requires a Dedicated Worker, while the current localStorage and sessionStorage engines depend on APIs exposed through Window. Moving the existing browser runtime entirely into a worker would therefore require us to redesign or remove the Web Storage engines at the same time.

For the initial scope, we could keep the existing browser entry point unchanged and add a separate entry point, for example gluesql/opfs. GlueSQL WASM and the OPFS storage would run inside a Dedicated Worker, with queries exposed through an asynchronous JavaScript proxy. OPFS could be the only storage provided by this entry point.

This would let us start with the Worker and OPFS path without redesigning the existing browser runtime. We can decide separately later whether the worker-based runtime should become the default.

[juhee200]

I think we should initially expose OPFS through a separate worker-based browser entry point rather than moving the existing browser entry point into a worker.

FileSystemSyncAccessHandle requires a Dedicated Worker, while the current localStorage and sessionStorage engines depend on APIs exposed through Window. Moving the existing browser runtime entirely into a worker would therefore require us to redesign or remove the Web Storage engines at the same time.

For the initial scope, we could keep the existing browser entry point unchanged and add a separate entry point, for example gluesql/opfs. GlueSQL WASM and the OPFS storage would run inside a Dedicated Worker, with queries exposed through an asynchronous JavaScript proxy. OPFS could be the only storage provided by this entry point.

This would let us start with the Worker and OPFS path without redesigning the existing browser runtime. We can decide separately later whether the worker-based runtime should become the default.

Separate worker-based entry point sounds good — that also resolves the WebStorage question cleanly (I verified in a prototype that the current composite setup panics in a worker, since localStorage is Exposed=Window).

To keep PRs small, I'd like to start with the worker entry point itself: the worker host, the async proxy (RPC) layer, the second build target, and CI coverage. The entry point stays OPFS-only as you described — during development the worker would run against the memory engine purely as internal test scaffolding (not exposed publicly), and the opfs-storage crate would follow as the second PR, replacing that scaffolding as the entry point's only storage. The existing browser entry point stays untouched. Does that ordering work for you?

[panarch]

That ordering works well for me.

Using the memory engine as internal scaffolding to validate the worker host, RPC layer, build, and CI first, then adding the OPFS storage in a separate PR, sounds like a good way to keep each change focused. Keeping the existing browser entry point unchanged also makes sense.

[niclashoyer]

@panarch do you plan to support SharedWorker in some way? I landed here, because other client side databases like DuckDB don't support shared workers yet and thus every new tab spawns a whole new database thread, which is pretty expensive. The only alternative seems to be using something like leader election or indexeddb (which is being replaced here).

[juhee200]

@niclashoyer
I think it`s good idea! I’m currently looking into adding SharedWorker support and have started working on it. The goal is to let multiple tabs share a single database worker instead of spawning a separate database thread for each tab. It’s still in the early stages, but I’ll share an update once I have something usable.

[juhee200]

@panarch @niclashoyer
I looked into this further, and directly using a SharedWorker does not seem feasible because the OPFS FileSystemSyncAccessHandle API is only available inside a DedicatedWorker.

However, I think we may be able to achieve similar behavior with a different architecture. I'm thinking to apply this.

[niclashoyer]

@juhee200 would you mind sharing your idea for the alternative architecture? It seems that it might be possible to spawn a dedicated worker from a shared worker: rhashimoto/wa-sqlite#81 (comment)

But chromium is still missing this feature. Other approaches seem to use the shared worker as a coordinator for choosing tabs to spawn the necessary workers.

[juhee200]

@niclashoyer
Thanks for sharing this — really helpful context.

The direction I had in mind was exactly the coordinator approach: keep a single database worker alive and migrate leadership between tabs as they close, using Web Locks for election (the tab holding the lock owns the
worker) and BroadcastChannel for relaying queries — essentially the SharedService pattern from wa-sqlite#81.

@panarch
Safari 27 adding nested worker support is encouraging, since a dedicated
worker living under a SharedWorker would remove the need for leader
migration entirely. What I'm weighing now is whether it's worth waiting
for Chromium to catch up, or landing the coordinator approach first and
converging to the nested-worker design once support is universal.