feat(provenance): prototype RDF4J-style event streams - #19
Closed
jeswr wants to merge 11 commits into
Closed
Conversation
Lexer: an absolute-offset counter and, under the new trackOffsets
option, offsetStart/offsetEnd on each token (default token shape and
hot path unchanged).
Parser: under the new onQuadSpans option, remember each term's source
token span (WeakMap, populated in _readEntity, literal completion and
synthetic-blank-node creation) and report per-position spans for every
emitted quad. Zero cost when the option is absent.
N3ProvenanceParser: wraps Parser to maintain a multiset of quad
*utterances* on the side - Map from a canonical quad key (value-based,
never object identity, so store-reconstructed quads still resolve) to
{quad, subject/predicate/object/graph: Range[]} with absolute character
offsets. Stores stay plain sets of quads at full speed; the multiset
lives in the wrapper.
All existing tests pass unchanged and coverage stays at 100%; new tests
cover utterance multiset semantics, value-keyed lookup, TriG graph
labels, RDF 1.2 annotations and span-less synthetic terms.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
apply-provenance.mjs applies the term-span transforms (a payload of context-anchored replacements generated by diffing this branch against main, plus the N3ProvenanceParser source) to a pristine N3.js source tree. Anchors must match exactly once, so upstream drift fails loudly. By construction, term-provenance == main + this script, byte for byte - so the instrumentation can also be maintained entirely out-of-tree against upstream releases if it isn't wanted in-tree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Ted Thibodeau Jr <tthibodeau@openlinksw.com>
rdfjs#613 fixed literal subjects/predicates in N3 mode. Two consequences for the provenance branch: - The subject-literal test document `"s" <p> <o> <g> .` only parsed before because the old `_completeSubjectLiteral` swallowed the token after the literal, shifting `<o>`/`<g>` into predicate/object. With the token no longer dropped, the document is an N3-invalid quad. Use `"s" <p> <o> .`, which is what the test meant to exercise. - `_readPredicate`'s new `case 'literal'` is a fourth site that stashes `_literalValue` for later completion, so it needs the same `_literalSpan` stash as the subject, object, and list-item sites, or predicate literals come back span-less. Instrument it and add the corresponding payload transform. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Numbers and booleans reach the parser as a single `literal` token whose `prefix` already carries the datatype, so they skip the `_literalValue`/`_literalSpan` handshake that the quoted-literal paths use and were constructed with no span at all: the object of `<s> <p> 42 .` came back span-less. Note the span directly from the token at all four such sites (subject, predicate, object, list item); the token covers exactly the numeric or boolean lexeme, so no offset arithmetic is needed. apply-provenance.mjs itself is unchanged -- this is four more context-anchored replacements in the payload, and the derivation still reproduces src/ byte-identically from pristine upstream sources. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The anchor spanned the whole function body, including the _emit line that carries the reifies quad's graph. That line is unrelated to the instrumentation -- the transform only wraps the blankNode() and quad() calls -- but including it made the anchor drift the moment upstream touched the graph argument (as the fix for rdfjs#676 does). Trim it to the three lines actually rewritten. Output on main is byte-identical, and the payload now applies cleanly on top of the reifies-graph, lone-reifier and blank-node-annotation fixes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ce-events # Conflicts: # src/N3Lexer.js # src/N3Parser.js
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Prototype an architecture closer to RDF4J for lexical provenance, following the discussion around rdfjs#672.
Architecture
The parser exposes two independent event streams:
onToken(token, start, end, sourceId)reports lexical occurrencesonQuadOrigin(quad, subjectId, predicateId, objectId, graphId)reports semantic statements in terms of those occurrence IDsOccurrence IDs travel beside subject/predicate/object/graph state and through parser contexts. They are never inferred from RDF/JS term identity.
ProvenanceParseris only one consumer: it compacts token positions and quad-origin rows into typed arrays, then materializes the rdfjs#672-compatibleRange[]view lazily.This keeps the lexer/parser event surface reusable for consumers that want a CST, syntax highlighting, diagnostics, or a different provenance representation.
Performance
100,000 N-Triples (6,677,780 bytes):
main: full provenance 1.73x (335.1 ms vs 193.3 ms)The detailed controlled results and methodology are in
perf/rdf4j-provenance-results.md. The fresh host was noisy, so its ratio is included as a smoke measurement.Validation
Trade-off
This has the widest parser diff because occurrence IDs must follow every parser-state transition, including lists, paths, formulas, annotations, and triple terms. In exchange, terms and quads remain ordinary RDF/JS values and the event streams do not prescribe a retained provenance model.