Skip to content

fix(sqlite-persistence): preserve expression index use - #1867

Open
KyleAMathews wants to merge 3 commits into
mainfrom
rfc-1659-ws5a-expression-index-oracle
Open

KyleAMathews wants to merge 3 commits into
mainfrom
rfc-1659-ws5a-expression-index-oracle

Conversation

@KyleAMathews

@KyleAMathews KyleAMathews commented Sep 20, 2026

Copy link
Copy Markdown
Collaborator

Compile validated JSON reference paths as canonical SQLite literals so runtime predicates match persisted expression indexes. Queries keep returning the same rows, but SQLite can now use the intended index instead of scanning the collection table.

Root cause

Persisted index DDL embedded the JSON path as a SQL literal, while runtime reference filters compiled the same path as a bound parameter. SQLite requires an indexed expression to match the predicate expression structurally; json_extract(value, ?) is semantically correct but cannot use an index defined on json_extract(value, '$.path').

Approach

  • Render the already-validated base, type-tag, and tagged-value JSON paths with the existing SQLite literal helper.
  • Keep comparison values bound as parameters.
  • Add a real Better SQLite3 driver oracle that captures production predicate SQL, independently checks rows, and replays the query through EXPLAIN QUERY PLAN to require the named expression index.
  • Register and document the focused oracle campaign.

Key invariants

  • JSON path segments are validated before literalization.
  • User comparison values remain bound; only the compiler-owned path shape is literalized.
  • Runtime predicates and persisted index DDL use the same serialized ref expression for plain and tagged values.
  • Correct rows and named-index use are asserted separately to prevent row-only false greens.

Non-goals

  • This does not broaden raw-SQL support or change null/date/bigint/tagged-value coercion semantics.
  • Native-host execution and result ordering remain outside this oracle.
  • Transient RFC review and RED/GREEN evidence stays outside the product PR.

Trade-offs

JSON path shapes become part of the generated SQL text rather than path bindings. That is required for SQLite expression-index compatibility; scalar filter values remain parameterized.

Verification

pnpm --filter @tanstack/db-sqlite-persistence-core test
pnpm --filter @tanstack/node-db-sqlite-persistence test:oracles
pnpm exec tsc -p packages/db-sqlite-persistence-core/tsconfig.json --noEmit
pnpm exec tsc -p packages/node-db-sqlite-persistence/tsconfig.json --noEmit

Local results: 96 SQLite persistence-core tests passed, all 3 expression-index oracle tests passed, both package TypeScript checks passed, and Prettier plus diff validation passed.

Files changed

  • sqlite-core-adapter.ts: compile validated ref paths as canonical SQLite literals.
  • expression-index-oracle.test.ts: verify rows and the real named-index query plan across fixed and generated cases.
  • Package/docs metadata: expose and document the oracle campaign.
  • Changeset: publish the core behavior fix as a patch.

Part of #1659

Summary by CodeRabbit

  • Bug Fixes

    • Improved SQLite query planning for persisted expression indexes by using canonical JSON path literals.
    • Queries involving nested JSON values can now correctly match and use the corresponding expression index.
  • Documentation

    • Added guidance for running SQLite expression-index validation checks.
  • Tests

    • Added automated coverage confirming indexed query plans and correct results across generated JSON path and value combinations.

@coderabbitai

coderabbitai Bot commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

Understand this PR’s impact

Explore downstream dependencies and potential security impact with Blast Radius.

View blast radius →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 7d8411cb-2dae-41df-ab39-1d8d87f4fa3d

📥 Commits

Reviewing files that changed from the base of the PR and between 35ee70d and 6720194.

📒 Files selected for processing (1)
  • packages/node-db-sqlite-persistence/tests/expression-index-oracle.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • packages/node-db-sqlite-persistence/tests/expression-index-oracle.test.ts

Included review availability: Your plan provides up to 8 included reviews per hour; 3 remain after this review.


📝 Walkthrough

Walkthrough

compileRefExpressionSql now emits JSON paths as SQL literals so runtime predicates match persisted SQLite expression indexes. A property-based oracle verifies query results and index usage through EXPLAIN QUERY PLAN.

Changes

SQLite expression-index planning

Layer / File(s) Summary
Inline reference paths
packages/db-sqlite-persistence-core/src/sqlite-core-adapter.ts, .changeset/fix-sqlite-expression-index-planning.md
compileRefExpressionSql emits validated JSON paths as SQL literals and removes the four path bindings. A patch changeset records the fix.
Validate expression-index usage
packages/node-db-sqlite-persistence/tests/expression-index-oracle.test.ts, packages/node-db-sqlite-persistence/package.json, docs/contributing/oracle-coverage.md
The property-based oracle compares indexed results with an independent scan and verifies named-index usage through EXPLAIN QUERY PLAN. A package script and coverage documentation describe how to run it.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 13 functions across 2 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description check ✅ Passed The description clearly explains the change, root cause, approach, scope, trade-offs, verification, and release changeset. It does not use the template headings or include the checklist and release-im…
Title check ✅ Passed The title clearly and concisely describes the primary change: preserving SQLite expression-index usage for persistence queries.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…ion-index-oracle

# Conflicts:
#	docs/contributing/oracle-coverage.md
@pkg-pr-new

pkg-pr-new Bot commented Sep 20, 2026

Copy link
Copy Markdown
More templates

@tanstack/angular-db

npm i https://pkg.pr.new/@tanstack/angular-db@1867

@tanstack/browser-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/browser-db-sqlite-persistence@1867

@tanstack/capacitor-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/capacitor-db-sqlite-persistence@1867

@tanstack/cloudflare-durable-objects-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/cloudflare-durable-objects-db-sqlite-persistence@1867

@tanstack/db

npm i https://pkg.pr.new/@tanstack/db@1867

@tanstack/db-ivm

npm i https://pkg.pr.new/@tanstack/db-ivm@1867

@tanstack/db-sqlite-persistence-core

npm i https://pkg.pr.new/@tanstack/db-sqlite-persistence-core@1867

@tanstack/electric-db-collection

npm i https://pkg.pr.new/@tanstack/electric-db-collection@1867

@tanstack/electron-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/electron-db-sqlite-persistence@1867

@tanstack/expo-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/expo-db-sqlite-persistence@1867

@tanstack/node-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/node-db-sqlite-persistence@1867

@tanstack/offline-transactions

npm i https://pkg.pr.new/@tanstack/offline-transactions@1867

@tanstack/powersync-db-collection

npm i https://pkg.pr.new/@tanstack/powersync-db-collection@1867

@tanstack/query-db-collection

npm i https://pkg.pr.new/@tanstack/query-db-collection@1867

@tanstack/react-db

npm i https://pkg.pr.new/@tanstack/react-db@1867

@tanstack/react-native-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/react-native-db-sqlite-persistence@1867

@tanstack/react-router-with-db

npm i https://pkg.pr.new/@tanstack/react-router-with-db@1867

@tanstack/rxdb-db-collection

npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@1867

@tanstack/solid-db

npm i https://pkg.pr.new/@tanstack/solid-db@1867

@tanstack/svelte-db

npm i https://pkg.pr.new/@tanstack/svelte-db@1867

@tanstack/tauri-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/tauri-db-sqlite-persistence@1867

@tanstack/trailbase-db-collection

npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@1867

@tanstack/vue-db

npm i https://pkg.pr.new/@tanstack/vue-db@1867

commit: 6720194

@github-actions

Copy link
Copy Markdown
Contributor

Size Change: 0 B

Total Size: 165 kB

ℹ️ View Unchanged
Filename Size
packages/db/dist/esm/client.js 3.66 kB
packages/db/dist/esm/collection-options.js 236 B
packages/db/dist/esm/collection/change-events.js 1.44 kB
packages/db/dist/esm/collection/changes.js 2.25 kB
packages/db/dist/esm/collection/cleanup-queue.js 794 B
packages/db/dist/esm/collection/events.js 481 B
packages/db/dist/esm/collection/index.js 4.63 kB
packages/db/dist/esm/collection/indexes.js 1.99 kB
packages/db/dist/esm/collection/lifecycle.js 2.15 kB
packages/db/dist/esm/collection/mutations.js 2.61 kB
packages/db/dist/esm/collection/state.js 6.51 kB
packages/db/dist/esm/collection/subscription.js 8.72 kB
packages/db/dist/esm/collection/sync.js 4.62 kB
packages/db/dist/esm/collection/transaction-metadata.js 144 B
packages/db/dist/esm/deferred.js 207 B
packages/db/dist/esm/errors.js 5.26 kB
packages/db/dist/esm/event-emitter.js 964 B
packages/db/dist/esm/index.js 3.71 kB
packages/db/dist/esm/indexes/auto-index.js 829 B
packages/db/dist/esm/indexes/base-index.js 1.14 kB
packages/db/dist/esm/indexes/basic-index.js 2.07 kB
packages/db/dist/esm/indexes/btree-index.js 2.26 kB
packages/db/dist/esm/indexes/index-registry.js 820 B
packages/db/dist/esm/indexes/reverse-index.js 376 B
packages/db/dist/esm/live-query-adapter.js 318 B
packages/db/dist/esm/live-query-observer.js 3.69 kB
packages/db/dist/esm/live-query-options.js 702 B
packages/db/dist/esm/live-query-window-controller.js 4.36 kB
packages/db/dist/esm/local-only.js 989 B
packages/db/dist/esm/local-storage.js 2.17 kB
packages/db/dist/esm/optimistic-action.js 359 B
packages/db/dist/esm/paced-mutations.js 496 B
packages/db/dist/esm/proxy.js 3.32 kB
packages/db/dist/esm/query/builder/functions.js 1.47 kB
packages/db/dist/esm/query/builder/index.js 6.69 kB
packages/db/dist/esm/query/builder/query-ir.js 116 B
packages/db/dist/esm/query/builder/ref-proxy.js 1.24 kB
packages/db/dist/esm/query/compiler/evaluators.js 1.92 kB
packages/db/dist/esm/query/compiler/expressions.js 560 B
packages/db/dist/esm/query/compiler/group-by.js 4.13 kB
packages/db/dist/esm/query/compiler/index.js 9.06 kB
packages/db/dist/esm/query/compiler/joins.js 2.95 kB
packages/db/dist/esm/query/compiler/lazy-targets.js 1.1 kB
packages/db/dist/esm/query/compiler/order-by.js 1.91 kB
packages/db/dist/esm/query/compiler/parent-routes.js 319 B
packages/db/dist/esm/query/compiler/route-metadata.js 1.24 kB
packages/db/dist/esm/query/compiler/select.js 1.58 kB
packages/db/dist/esm/query/effect.js 4.6 kB
packages/db/dist/esm/query/equality-value-identity.js 591 B
packages/db/dist/esm/query/expression-helpers.js 1.43 kB
packages/db/dist/esm/query/ir-stable-identity.js 4.04 kB
packages/db/dist/esm/query/ir.js 1.59 kB
packages/db/dist/esm/query/live-query-collection.js 391 B
packages/db/dist/esm/query/live/bucket-facade-adapter.js 2.73 kB
packages/db/dist/esm/query/live/collection-config-builder.js 6.97 kB
packages/db/dist/esm/query/live/collection-registry.js 264 B
packages/db/dist/esm/query/live/collection-subscriber.js 2.25 kB
packages/db/dist/esm/query/live/internal.js 145 B
packages/db/dist/esm/query/live/materialized-pipeline.js 2.32 kB
packages/db/dist/esm/query/live/ordered-source-loader.js 3.14 kB
packages/db/dist/esm/query/live/subset-demand-controller.js 1.26 kB
packages/db/dist/esm/query/live/utils.js 1.14 kB
packages/db/dist/esm/query/optimizer.js 2.91 kB
packages/db/dist/esm/query/query-once.js 359 B
packages/db/dist/esm/query/runtime-reference-identity.js 572 B
packages/db/dist/esm/query/subset-dedupe.js 486 B
packages/db/dist/esm/scheduler.js 1.34 kB
packages/db/dist/esm/SortedMap.js 1.3 kB
packages/db/dist/esm/strategies/debounceStrategy.js 247 B
packages/db/dist/esm/strategies/queueStrategy.js 428 B
packages/db/dist/esm/strategies/throttleStrategy.js 246 B
packages/db/dist/esm/transactions.js 3.71 kB
packages/db/dist/esm/utils.js 1.08 kB
packages/db/dist/esm/utils/array-utils.js 270 B
packages/db/dist/esm/utils/browser-polyfills.js 304 B
packages/db/dist/esm/utils/btree.js 4.51 kB
packages/db/dist/esm/utils/callbacks.js 174 B
packages/db/dist/esm/utils/comparison.js 1.49 kB
packages/db/dist/esm/utils/cursor.js 676 B
packages/db/dist/esm/utils/error.js 167 B
packages/db/dist/esm/utils/get-or-create.js 155 B
packages/db/dist/esm/utils/index-optimization.js 2.42 kB
packages/db/dist/esm/utils/type-guards.js 230 B
packages/db/dist/esm/utils/uuid.js 449 B
packages/db/dist/esm/virtual-props.js 360 B

compressed-size-action::db-package-size

@github-actions

Copy link
Copy Markdown
Contributor

Size Change: 0 B

Total Size: 7.34 kB

ℹ️ View Unchanged
Filename Size
packages/react-db/dist/esm/DbProvider.js 317 B
packages/react-db/dist/esm/HydrationBoundary.js 263 B
packages/react-db/dist/esm/index.js 330 B
packages/react-db/dist/esm/live-query-internals.js 282 B
packages/react-db/dist/esm/useLiveInfiniteQuery.js 1.9 kB
packages/react-db/dist/esm/useLiveQuery.js 2.68 kB
packages/react-db/dist/esm/useLiveQueryEffect.js 355 B
packages/react-db/dist/esm/useLiveSuspenseQuery.js 812 B
packages/react-db/dist/esm/usePacedMutations.js 401 B

compressed-size-action::react-db-package-size

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant