Opt-in term source spans and a provenance-tracking parser wrapper - #672
Open
ericprud wants to merge 7 commits into
Open
Opt-in term source spans and a provenance-tracking parser wrapper#672ericprud wants to merge 7 commits into
ericprud wants to merge 7 commits into
Conversation
Lexer: an absolute-offset counter and, under the new trackOffsets
option, offsetStart/offsetEnd on each token (default token shape and
hot path unchanged).
Parser: under the new onQuadSpans option, remember each term's source
token span (WeakMap, populated in _readEntity, literal completion and
synthetic-blank-node creation) and report per-position spans for every
emitted quad. Zero cost when the option is absent.
N3ProvenanceParser: wraps Parser to maintain a multiset of quad
*utterances* on the side - Map from a canonical quad key (value-based,
never object identity, so store-reconstructed quads still resolve) to
{quad, subject/predicate/object/graph: Range[]} with absolute character
offsets. Stores stay plain sets of quads at full speed; the multiset
lives in the wrapper.
All existing tests pass unchanged and coverage stays at 100%; new tests
cover utterance multiset semantics, value-keyed lookup, TriG graph
labels, RDF 1.2 annotations and span-less synthetic terms.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
apply-provenance.mjs applies the term-span transforms (a payload of context-anchored replacements generated by diffing this branch against main, plus the N3ProvenanceParser source) to a pristine N3.js source tree. Anchors must match exactly once, so upstream drift fails loudly. By construction, term-provenance == main + this script, byte for byte - so the instrumentation can also be maintained entirely out-of-tree against upstream releases if it isn't wanted in-tree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TallTed
reviewed
Aug 5, 2026
Co-authored-by: Ted Thibodeau Jr <tthibodeau@openlinksw.com>
rdfjs#613 fixed literal subjects/predicates in N3 mode. Two consequences for the provenance branch: - The subject-literal test document `"s" <p> <o> <g> .` only parsed before because the old `_completeSubjectLiteral` swallowed the token after the literal, shifting `<o>`/`<g>` into predicate/object. With the token no longer dropped, the document is an N3-invalid quad. Use `"s" <p> <o> .`, which is what the test meant to exercise. - `_readPredicate`'s new `case 'literal'` is a fourth site that stashes `_literalValue` for later completion, so it needs the same `_literalSpan` stash as the subject, object, and list-item sites, or predicate literals come back span-less. Instrument it and add the corresponding payload transform. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Numbers and booleans reach the parser as a single `literal` token whose `prefix` already carries the datatype, so they skip the `_literalValue`/`_literalSpan` handshake that the quoted-literal paths use and were constructed with no span at all: the object of `<s> <p> 42 .` came back span-less. Note the span directly from the token at all four such sites (subject, predicate, object, list item); the token covers exactly the numeric or boolean lexeme, so no offset arithmetic is needed. apply-provenance.mjs itself is unchanged -- this is four more context-anchored replacements in the payload, and the derivation still reproduces src/ byte-identically from pristine upstream sources. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The anchor spanned the whole function body, including the _emit line that carries the reifies quad's graph. That line is unrelated to the instrumentation -- the transform only wraps the blankNode() and quad() calls -- but including it made the anchor drift the moment upstream touched the graph argument (as the fix for rdfjs#676 does). Trim it to the three lines actually rewritten. Output on main is byte-identical, and the payload now applies cleanly on top of the reifies-graph, lone-reifier and blank-node-annotation fixes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jeswr
requested changes
Aug 19, 2026
Comment on lines
+1102
to
+1104
| const reifier = this._reifier || this._noteSpan(this._factory.blankNode(), null); | ||
| this._reifier = null; | ||
| this._tripleTerm = this._tripleTerm || this._factory.quad(this._subject, this._predicate, this._object); | ||
| this._tripleTerm = this._tripleTerm || this._noteSpan(this._factory.quad(this._subject, this._predicate, this._object), null); |
Member
There was a problem hiding this comment.
Why call notespan here when the term is just returned if the token is null?
Member
There was a problem hiding this comment.
I'm not sure it is appropriate for this to be maintained here - as opposed to a consuming package.
@RubenVerborgh this is a call for you to make
Comment on lines
+18
to
+21
| // Opt-in source-span tracking: after each emitted quad, invoke | ||
| // onQuadSpans(quad, {subject, predicate, object, graph}) with the | ||
| // lexer's {line, start, end} for each position's source token (null for | ||
| // synthetic terms without one). Zero cost when the option is absent. |
This was referenced Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Opt-in term source spans and a provenance-tracking parser wrapper
Editor and validation tooling needs to answer "where in the source did this quad come from?" — for red squiggles, for highlighting a validation result's triple in the document, and for the RDF/JS editor-API conversation some of us have been circling. Stores rightly stay sets of value-equal quads; provenance is a multiset of utterances, and it belongs beside the store, not in it. This PR adds the minimal parser support that makes that possible, plus a reference wrapper.
Why would I want that?
So you can match query or validation results to data and make garrish interfaces (click schema in the left column and "data" in the right, then validate and hover over constraints in the left (schema), triples in the right (data) or TestedTriples in the resulting valiation proof).
What's here
Three layers, each opt-in:
trackOffsetsoption —offsetStart/offsetEndon each token. The default token shape is byte-for-byte unchanged.onQuadSpans(quad, {subject, predicate, object, graph})option, the parser remembers each term's source token span (aWeakMappopulated in_readEntity, literal completion, and synthetic-blank-node creation) and reports per-position spans for every emitted quad. Synthetic terms without a source token (e.g.rdf:first,a) reportnull.Parserto maintain the multiset — aMapfrom a canonical quad key (value-based, never object identity, so quads reconstructed byStore.getQuads()still resolve) to utterances{quad, subject/predicate/object/graph: Range[]}in absolute character offsets.Performance
Measured with interleaved runs (5×, medians) over a 500k-triple / 19 MB document, string input:
onQuadSpanswith a no-op consumerProvenanceParserThe off-path cost is one integer add per input advance and one predictable branch per token/term; the
trackOffsetsgate keeps the token hidden class unchanged, so nothing leaks into unrelated code. The opt-in multipliers are allocation/GC (span objects, WeakMap traffic, canonical keys) and land on editor-sized documents — a 100 KB document pays ~20 ms. There's headroom (packed integer spans, pooled records, lazy keys) if the opt-in path's cost matters to anyone; I kept the first cut obvious rather than clever.Tests
All existing tests pass unchanged and coverage stays at 100 %. New tests cover the utterance multiset semantics (a quad uttered twice has two utterances), value-keyed lookup through a
Storeround-trip, TriG graph labels, RDF 1.2 annotations, and span-less synthetic terms.Who carries it — in preference order
The instrumentation is mechanical.
scripts/apply-provenance.mjsexpresses all of it as a payload of context-anchored replacements (generated by diffing this branch against main) that reproduces the instrumented files byte-identically from pristine sources, failing loudly the moment an anchor drifts. That separates where the code lives from who maintains it, so there are three arrangements rather than the usual two. Best first:1. Merge it (this PR). The ecosystem shares one parser instead of a derived one, and the instrumentation is covered by the same tests and the same 100 % gate as everything else. The script stays in-tree as documentation of what the transformation actually is.
2. Don't merge the instrumentation, but carry the script.
scripts/keepsapply-provenance.mjsplus its payload, and a CI job derives the instrumented parser and runs the provenance suite against it. I maintain the payload; rdfjs just runs it. Make itcontinue-on-errorand it never gates your work — it only tells you, in the same run, that a core-parser change broke a downstream derivation.src/is untouched by the feature, so the +0.5 % conversation goes away entirely.3. Fully out-of-tree. I keep script, payload and canary in my own repo, tracking your releases. Costs rdfjs nothing, and you find out about a break when I file an issue — or when a user does.
Merging main into this branch this week is a live argument for (2) over (3). #613 drifted none of the anchors, so the byte-identical guarantee held exactly as advertised — but it still broke the derived parser twice in ways no anchor check can catch. It added a new literal code path (predicate literals in N3 mode) that needed instrumenting, so those terms came back span-less; and by fixing a token-swallowing bug it invalidated one of my tests that had been quietly asserting the buggy behaviour. Under (2) your CI would have reported both in the same run that merged #613. Under (3) I found out days later, on my next rebase.
To be clear, (2) and (3) are both fine outcomes and I'll do the work either way; I'd just rather the breakage be visible to the people making the change than discovered downstream.
Downstream consumers (already live)
--provenanceflag that decorates each matched triple in validation output with its source ranges.Reification bugs found while testing against main
That observation about
<G> { <a> <b> <c> {| <b> <c> |}. }turned into four separate defects, none of them caused by this branch. Filed and, where the fix was small enough to be reviewable on its own, patched:rdf:reifiesemitted into the default graph instead of the enclosing one<a> <b> <c> ~ <r> .asserts the triple twice and never emitsrdf:reifies{| |}silently droppedEffect on this PR: none, once #679 is accounted for. The three PRs merge cleanly with each other in any order, and
scripts/apply-provenance.mjsapplies to every one of them individually and combined. #679 did initially drift one anchor — it spanned the whole of_readTripleTerm, including the_emitline that carries the graph, while the transform only wraps the two factory calls — so the anchor is now trimmed to the lines it actually rewrites. Output on main is byte-identical.The instrumented parser plus this branch's provenance tests pass against each of the three fixes separately and all three together, at 100 % coverage. So whichever of them land, in whatever order, this branch needs no further change.
This is also the concrete argument for arrangement (2) above: #676's fix drifts an anchor that no amount of care in this branch could have anticipated, and the canary reports it in the same run rather than at my next rebase.
Closes #377