Skip to content

Add Database.tokenize() wrapper and re-export ScriptTokens/TokenType#52

Open
Montana wants to merge 1 commit into
motherduckdb:mainfrom
Montana:add-database-tokenize
Open

Add Database.tokenize() wrapper and re-export ScriptTokens/TokenType#52
Montana wants to merge 1 commit into
motherduckdb:mainfrom
Montana:add-database-tokenize

Conversation

@Montana

@Montana Montana commented Jul 8, 2026

Copy link
Copy Markdown

Add Database.tokenize() wrapper and re-export ScriptTokens / TokenType

Summary

The underlying duckdb Node API exposes Database.tokenize(text): ScriptTokens (added upstream in duckdb/duckdb-node for SQL syntax highlighting / editor tooling), but duckdb-async doesn't wrap it, so users of this package can't reach it without dropping down to the raw API.

This PR:

  • Adds a tokenize(text: string): duckdb.ScriptTokens method to the Database wrapper. It's a synchronous delegation following the exact pattern of the existing interrupt() method (including the uninitialized-database guard).
  • Re-exports the ScriptTokens and TokenType types/enum from duckdb, so callers can type the return value and compare token types without importing duckdb directly.
  • Adds a test mirroring duckdb-node's own tokenize test (select 1; → offsets [0, 7, 8], types [KEYWORD, NUMERIC_CONSTANT, OPERATOR]).

Notes

While auditing the API surface, Database.get() is the only other method in duckdb's type definitions not wrapped here, but it's intentionally unimplemented upstream (it throws "get() is not implemented because it's evil"), so it's deliberately excluded from this PR. With this change, the wrapper covers every implemented method on duckdb.Database.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant