feat(parser): add token callbacks with a provenance reference

[jeswr]

This proposal adds public onToken(token) and onTokenEnd(token) callbacks to Parser.parse, then uses them in a complete term-location/provenance reference implementation. It continues the work in jeswr/N3.js#20 on top of the merged lexer ranges from #721.

The draft keeps the callback API separate so it can be extracted and landed independently:

1. 17301c3 — public callbacks, contract tests, and documentation. This changes only N3Parser.js, the README, and a standalone callback test file. No observer calls or checks are added to the ordinary per-token path when callbacks are absent.

2. Reference implementation and tests. examples/provenance contains the location parser, compact provenance index, an EntityIndex adapter, documentation, and a runnable demo. Nothing is added to N3's exports or published package, and this commit needs no further core parser, lexer, or Store changes.

3. 2efdb07 — simplify public occurrence packing. Write the four component ranges directly in one loop, removing the separate offset-calculation helper. This retains the same 16-number storage, atomic validation, and copying behavior. Seven alternating fresh-process samples per runtime on Node 24/25 found performance-neutral public insertion times (median changes −0.6% and −1.2%).

The reference is more complete than fork PR #16's term-symbol prototype: it handles frozen/interning factories, complete compound and literal-suffix ranges, current N3 emission rules, and ordered callbacks with partial-error behavior. Those guarantees require additional state. The packed representation retains its separate handling of unfinished ranges because simpler storage alternatives did not establish equivalent performance across the tested workloads.

The callbacks observe each token once around processing, including EOF and enabled comment tokens, across synchronous and streaming parsing. Return values are ignored. onTokenEnd runs for cleanup even if processing or onToken throws; consumer exceptions stop subsequent token dispatch. The README specifies ordering with quad/comment/completion callbacks, coordinate units, and error behavior.

The reference replaces the private _readToken override with the public callbacks and retains the complete provenance behavior: duplicate occurrences with value-based quad lookup, original lexical spellings, literal suffixes, multiline ranges, nested lists/property lists/formulas/triple terms, named graphs, inverse predicates, and null ranges for generated terms. Frozen/interning RDF/JS factories work. The original proposal's parser changes have been adapted to current main, including the newer N3 inverse-predicate routing.

Scope of the extension point: these callbacks make token observation public. The reference still subclasses private grammar/factory methods and uses EntityIndex internals to correlate terms with syntax. Its README explicitly records those dependencies and the need to pin/test a compatible N3 version; this is not a claim that the complete provenance implementation now uses only public APIs.

Validation:

• Callback commit alone: 6,961 tests, 100% coverage.
• Complete proposal: 7,073 tests, 100% statement/branch/function/line coverage, including 112 provenance tests.
• Added regressions cover callback lifecycle/errors/reuse, nested empty lists with interning factories, current N3 inverse semantics, and every two-chunk split of a document with BOM, CRLF, a multiline directional literal, and nested compounds.
• ESLint (including the example), Node/browser builds, the runnable provenance demo, and independent code review pass.
• Local throughput check on macOS arm64, Node 24.12.0 and 25.1.0: five alternating fresh-process samples per case, 300 ms warmup and at least 500 ms measurement. With observers disabled, median changes across dense N-Triples, literal N-Triples, mixed Turtle, mixed N3, and streaming N-Triples ranged from −2.8% to +1.8%, with no material regression established. Registering both empty observers added roughly 0–5.6% relative to the new disabled path, before any provenance work.

Try the reference after installing dependencies:

node -r @babel/register examples/provenance/demo.js
npm test -- --runInBand test/N3ProvenanceParser-test.js

To land only the requested extension points, select the first commit; the full reference can remain a separate external experiment.