perf(lexer): dispatch separators by character

[jeswr]

Avoid three separator regex attempts before every token by dispatching on the next character. Single spaces/tabs and line endings use direct checks; longer horizontal runs retain regex scanning. Complete comment lines are consumed in one match, and unfinished comments remain buffered until their line ending or EOF arrives.

The remaining _whitespace.exec() and _commentLine.exec() calls intentionally scan runs in native regex code. Comparing direct JavaScript loops with this implementation improved some short runs, but screening found about 19% more parser CPU for 128-character indentation and 31% more for 512-character comments. A bounded short-whitespace loop improved lexer-only timing in seven-round confirmations without establishing an end-to-end parser gain, so the simpler scanner is retained.

This extracts the general separator optimization from #721. Named SPACE, TAB, LF, CR, and HASH constants, explicit separatorLength, and a _commentLine pattern make the hot path readable while keeping each operation local. The scanner preserves main's existing token objects, callback timing, and line-mode behavior; the coordinate and split-CRLF corrections belong to #721.

Merge this PR before #721, then rebase #721 to combine its range bookkeeping with this scanner. The PRs both target main and touch the same separator block; their combined implementation has already passed the range tests and generated coordinate checks.

Validation: all 6,898 tests pass with 100% statement, branch, function, and line coverage, plus ESLint and Node/browser IIFE/ESM builds. Differential checks against main cover 3,600 generated documents, 29,220 two-chunk splits, and one-character chunks, including complete token objects, errors, and callback boundaries. The combined scanner and #721 range fixes pass 6,915 tests with 100% coverage, plus 5,654 independent coordinate checks, 2,231 streaming comparisons, and 28 lexer-reuse pairs.

Performance compares the published source with main 4607e09 using production Babel builds on macOS arm64 / Node 25.1.0. Seven paired rounds rotate fresh-process order, with 36 warmup and 36 measured iterations over 8,000-triple documents; streamed lexers use fresh instances and 64 KiB chunks. Full token/quad digests agree. Values are median paired CPU changes with bootstrap 95% intervals; negative means less CPU.

Workload	CPU change	95% interval
lexer: dense	-49.3%	[-50.1%, -46.7%]
lexer: comments	-37.5%	[-37.7%, -36.8%]
lexer: comments-off	-13.1%	[-14.9%, -12.8%]
lexer: multiline	-31.5%	[-32.6%, -22.3%]
parser: dense	-13.9%	[-15.6%, -12.2%]
parser: comments	-10.4%	[-11.1%, -10.0%]
parser: comments-off	-10.6%	[-11.7%, -9.6%]
parser: multiline	-22.0%	[-22.7%, -21.3%]
stream: dense	-24.8%	[-39.2%, -23.8%]
stream: comments	-25.7%	[-28.0%, -24.2%]
stream: comments-off	+3.7%	[-10.2%, +8.9%]
stream: multiline	-34.4%	[-37.9%, -28.1%]

The comments-off streamed result is inconclusive; it does not establish an improvement or regression. These synthetic timings are sensitive to runtime warmup and workload and are not application throughput guarantees. The earlier comment-body scanner showed a comment-heavy parser slowdown in one run; consuming each complete comment line in one match removed that observed slowdown in the final confirmation.

Additional seven-round checks (24 warmup and measured iterations):

Workload	CPU change	95% interval
stream: dense, 8-byte chunks	-20.6%	[-20.9%, -19.0%]
lexer: deep-indent	-5.5%	[-11.0%, -2.2%]
lexer: crlf	-39.4%	[-40.4%, -33.5%]
lexer: long-space	-27.7%	[-34.4%, -25.9%]

The current branch is rebased on main e4e2148, retaining the merged #726 direction-marker optimization. The benchmark tables above describe the original comparison against 4607e09; they have not been relabeled as measurements of the new base. The rebased source passes the full test suite at 100% coverage, lint, Node/browser builds, and all GitHub checks.

[Copilot]

🟢 Approval recommended

The changes are localized, preserve existing token shapes/ordering, and are backed by a targeted regression test covering all stream split positions for the updated separator logic.

Pull request overview

This PR optimizes the lexer’s separator scanning by dispatching on the next character (space/tab/newline/comment) instead of attempting multiple separator regexes before each token, aiming to reduce overhead on the hot path while preserving existing token emission behavior.

Changes:

• Replaces the separator-handling loop in src/N3Lexer.js with a character-dispatch implementation using SPACE/TAB/LF/CR/HASH constants and a _commentLine regex for full comment lines.
• Adds a regression test that exercises all possible stream split points around mixed separators and comment lines, verifying stable token output with comments both enabled and disabled.

File summaries

File	Description
src/N3Lexer.js	Reworks separator consumption to use character dispatch (with a comment-line regex) to reduce repeated regex work in the main tokenization loop.
test/N3Lexer-test.js	Adds coverage ensuring separators/comments behave consistently across arbitrary stream chunk splits.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

🎉 This PR is included in version 2.7.10 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀