(SPARQL) Full W3C Suite in CI w/ Registers for Skip/Ignore#1437
Open
aaj3f wants to merge 8 commits into
Open
Conversation
Audit found CI compiled the testsuite but never ran it, eval categories were all #[ignore]d, and the harness had fallen behind the engine (UPDATE execution, TriG/named graphs). This makes the whole rdf-tests submodule enforceable: - implement mf:UpdateEvaluationTest end-to-end over the public sparql_update transact surface (load ut:data/ut:graphData, apply, compare resulting default + named graph state isomorphically; 24 -> 67 passing) - load qt:graphData/ut:graphData as real named graphs (TriG GRAPH blocks via the transact builder) instead of dumping them into the default graph; fix labeled graphData blank nodes being silently dropped (never read ut:graph) - add CSV/TSV expected-result parsing with CSV-space projection (0 -> 5 of 6) - register the SPARQL 1.2 suites (254 tests) and the un-versioned Positive/NegativeUpdateSyntaxTest types used by them - move every suite's skip list to tests/registers/mod.rs, grouped by root cause; drop 14 stale property-path entries (13 features since landed + pp06 caught by the new check) - police registers in both directions in check_testsuite: unexpected failures AND stale entries (registered tests that now pass) fail the suite - drop the duplicate combined-manifest test fns; de-#[ignore] everything; CI now runs cargo test (36 suites, ~1420 tests, all green, ~15s wall) - document the audit, failure taxonomy, and perf-safe burn-down plan in docs/audit/2026-07-sparql-testsuite-audit.md; refresh docs/contributing/sparql-compliance.md and the Makefile for the new model
… plan
Corrects the empirical placeholder: RDF 1.2 annotation syntax ({| |}, ~,
<<( )>> restricted to rdf:reifies) is parsed, lowered to
Pattern::EdgeAnnotation, executed, and durably stored via the reified-edge
model. Actual 1.2 gaps: triple-term functions, bare triple-terms-as-values
(design decision needed), Turtle-star ingest (map onto the existing
edge_annotations reifier pipeline), CONSTRUCT annotation projection. Also
records SERVICE as Fluree-only by design and confirms UPDATE Modify parses
WITH/USING.
SPARQL, JSON-LD query, and Cypher share the IR and execution engine. Every W3C burn-down fix must classify as IR/engine-level (implicit parity, still add a JSON-LD regression test) or surface-syntax addition (SPARQL-possible => JSON-LD-possible in the same effort; Cypher assessed against the openCypher support matrix). The W3C submodule only guards the SPARQL surface — JSON-LD/Cypher regression tests are authored with each fix.
We own the JSON-LD query syntax, so SPARQL-possible => JSON-LD-possible stands with regression tests per fix. Cypher is openCypher — not our grammar to extend — so SPARQL compliance work carries no Cypher syntax obligations; it benefits from shared IR/engine fixes automatically.
Address the review's findings — all verified real; all make the harness's green trustworthy rather than changing what it covers: - (review #1) the 'no unexpected named graph' guard was dead code: the engine binds GRAPH ?g as a plain literal, so the Iri-only filter in list_named_graphs always produced []. Accept literal and IRI bindings (excluding the alias-named default graph) so the guard survives the eventual engine fix. Verified live: an update leaving a stray graph with no expected graphs now fails with 'Unexpected non-empty named graph'. - (review #2) bail when a manifest yields zero tests — a submodule restructure or manifest-parser regression must not report green. - (review #3) a registered test that dies by timeout/subprocess crash now hard-fails: the register excuses a known wrong answer, not an infra death masking a new hang or panic. - (review #4) UpdateEvaluationTest now rejects an mf:result blank node exposing none of ut:data/ut:graphData/ut:result, instead of degrading to a trivially satisfiable 'expected empty store'. - (review #5) register entries matching no discovered test now fail the suite, so dead entries (typos, upstream renames) cannot accumulate. - notes: tightened the TSV bare-integer heuristic to a single optional leading sign; documented the line-based directive-hoisting assumption; fixed the stale 'CI runs make ci' Makefile comment. Full suite remains green: 36 suites, ~1420 tests, 0 failed, 0 ignored.
…rables) Pre-implementation root-cause audits for every register cluster, produced by parallel deep-dive passes with live reproduction (run-w3c-test probes, CLI repros) rather than register-comment inference. Highlights: - update: ~90 failures collapse to 6 root causes; multi-operation ';' requests silently execute only the first op (production-facing; register comments for insert-05a / same-bnode / delete-insert-01c corrected) - parser-syntax: 62 tests -> 8 accept-more + 6 reject-more rules; four validation passes missing entirely; reject-more = user-visible changes - named-graph/dataset: GRAPH ?g literal binding is a one-line fix; default graph enumeration is the intentional #1279 extension (recommend W3C-by-default); SS5.3 answered: within-ledger datasets (Option A) - expression-semantics: 13 defects; two shallow bugs (DATATYPE/LANG reject expression args; variable-free FILTER drops all rows) explain ~40 tests; ~28 register entries reassigned to other clusters - lexer/formatter: bnode-label dot lexing + canonical xsd:double form - sparql12 wave 1: VERSION already supported (register mislabel); lang-basedir is a cross-stack representation gap; Turtle-star ingest via shared EdgeKey::to_reifies_facts greens only 2/41 eval tests alone - sparql12 wave 2: both triple-term registers are pure syntax suites; D3 decision narrowed to accept-then-defer vs documented divergence - residual-eval: property-path multiplicity framing corrected (sequences already correct + regression-guarded); pp34/35 are mis-filed graph bugs; entailment candidate wins via existing owl2rl materializer + PRAGMA Stage-2 adversarial verification runs before any claim alters registers or published statements.
Stage-2 verification of the eight cluster audits: 55 load-bearing claims attacked by independent skeptics — 40 confirmed, 13 refuted, 2 uncertain — plus a cross-cluster completeness check over every register entry. Material corrections the verification caught before implementation: - the proposed bnode-dot lexer loop breaks spec-valid _:a..b (and doesn't compile); replaced with greedy-scan + trailing-dot-rewind - the harness .rdf DAWG parser bug is real but backwards: Event::Empty state-stack skew keeps unbound solutions and DROPS bound ones - double canonicalization scope was missing export.rs N-Triples/N-Quads sites and R2RML lexical() consumers - named-graph BUG-4 (existence-row drop) is empirically false — graph-exist actually fails on base-relative IRI resolution; phantom PR deleted and replaced with PR-BASE, which fills a genuine ownership vacuum - fluree-db-api swallows parse-error diagnostics whenever an AST survives recovery, so grammar tightening must prevent AST production - EdgeKey::to_reifies_facts full variant would emit orphan flakes; ingest must use the _jsonld_compatible variant, guarded by equivalence tests Slate: 17 firm PRs + 5 decision-gated items + 2 unowned work-streams (algebra/OPTIONAL-scope and output-serialization need a ninth cluster audit). Firm slate removes ~418 of 538 registered entries; end state with all decisions favorable is ~124 remaining (63 not-applicable, 47 entailment, 14 orphans). Decisions D-1..D-12 enumerated for team sign-off; three production bugs queued for immediate issue filing.
…on transparency in PR descriptions
This was referenced Jul 6, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR makes the entire W3C SPARQL test suite (the
rdf-testssubmodule) an enforced, durable CI gate. Before this change, thetestsuite-sparqlCI job rancargo test --no-run— it compiled the harness but never executed a single test — and every evaluation category was#[ignore]d. The harness had also fallen behind the engine: SPARQL UPDATE execution (#509, #1288) and TriG/named-graph support (#1278, #1279) landed in the API, but the harness still had "not implemented" stubs and loaded named-graph data into the default graph.Now
cargo testintestsuite-sparql/runs all 36 suites — ~1,420 tests — green in ~15 s, and CI runs exactly that. Every test either passes or appears in an explicit, root-cause-grouped skip register that is policed in both directions.Standing after this PR: 815 passing / 538 registered engine or feature gaps / 67 registered not-applicable (protocol, graph-store protocol, service-description, SERVICE-needs-endpoints, entailment regimes).
The enforcement model
tests/w3c_sparql.rs— including the previously unregistered SPARQL 1.2 tree (254 tests). Nothing is#[ignore]d; nothing is silently un-run.tests/registers/mod.rs, grouped by root cause with rationale comments and pointers into the audit doc.check_testsuiteenforces the register both ways: an unregistered failure fails the suite (regression), and a registered test that now passes also fails the suite (stale entry — it must be removed in the same change that fixes the feature). This mechanism immediately proved itself: the old property-path skip list had rotted to 14 stale entries out of 18 (13 features had since landed, pluspp06freed by a harness fix in this PR).docs/audit/2026-07-sparql-testsuite-audit.md.Harness changes
mf:UpdateEvaluationTestimplemented end-to-end through the publicsparql_updatetransact surface: load initial graph-store state (ut:data/ut:graphData), apply the update, compare resulting default-graph and named-graph states isomorphically, and check for unexpected non-empty named graphs. Update-eval went from 24 → 67 passing.qt:graphData/ut:graphDatafiles are wrapped as TriGGRAPHblocks (directives hoisted) and loaded through the transact builder. Also fixes a latent bug where labeledgraphDatablank nodes were silently dropped (get_graph_datanever readut:graph).mf:PositiveUpdateSyntaxTest/mf:NegativeUpdateSyntaxTesttypes they use are now handled.docs/contributing/sparql-compliance.mdrewritten to match the registers model.New engine findings (registered, not fixed here)
The suite immediately surfaced precise engine gaps, now sitting in the registers with grouped rationale:
GRAPH ?gexposes the default graph as a graph named by the ledger alias, and binds?gas a plain literal rather than an IRI (breaks mostsparql10/graphtests even though the data now loads correctly).GRAPHblocks insideDELETE WHEREare rejected at lowering;INSERTinto a not-yet-existing named graph silently loses triples;USINGsemantics are incomplete; combinedDELETE/INSERTWHEREapplies inserts without the deletes.LOAD/CLEAR/CREATE/DROP/COPY/MOVE/ADD+SILENT, ~70 tests).Modifydoes parseWITH/USING..(_:o6.) — blocks all 4 json-res tests and affects real user data of that shape.Perf safety
This PR contains zero engine changes — it touches only the out-of-workspace
testsuite-sparqlcrate, CI config, and docs. Hot-path risk is nil by construction. The audit doc (§6) prescribes the perf-safety discipline for the follow-up engine fixes: parse/prepare-time fixes preferred, common-type fast paths preserved byte-identical,query_hot_bsbm*/insert_formatsbench guardrails withinregression-budget.jsonbudgets per PR.Test plan
cd testsuite-sparql && cargo test— 36 suites, 0 failed, 0 ignored (~15 s).cargo fmt --all -- --checkandcargo clippy --all-targets -- -D warningsclean.