Skip to content

(SPARQL) Full W3C Suite in CI w/ Registers for Skip/Ignore#1437

Open
aaj3f wants to merge 8 commits into
mainfrom
test/sparql-testsuite-full-coverage
Open

(SPARQL) Full W3C Suite in CI w/ Registers for Skip/Ignore#1437
aaj3f wants to merge 8 commits into
mainfrom
test/sparql-testsuite-full-coverage

Conversation

@aaj3f

@aaj3f aaj3f commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR makes the entire W3C SPARQL test suite (the rdf-tests submodule) an enforced, durable CI gate. Before this change, the testsuite-sparql CI job ran cargo test --no-run — it compiled the harness but never executed a single test — and every evaluation category was #[ignore]d. The harness had also fallen behind the engine: SPARQL UPDATE execution (#509, #1288) and TriG/named-graph support (#1278, #1279) landed in the API, but the harness still had "not implemented" stubs and loaded named-graph data into the default graph.

Now cargo test in testsuite-sparql/ runs all 36 suites — ~1,420 tests — green in ~15 s, and CI runs exactly that. Every test either passes or appears in an explicit, root-cause-grouped skip register that is policed in both directions.

Standing after this PR: 815 passing / 538 registered engine or feature gaps / 67 registered not-applicable (protocol, graph-store protocol, service-description, SERVICE-needs-endpoints, entailment regimes).

The enforcement model

  • Every manifest in the submodule is registered as a test fn in tests/w3c_sparql.rs — including the previously unregistered SPARQL 1.2 tree (254 tests). Nothing is #[ignore]d; nothing is silently un-run.
  • Per-suite skip registers live in tests/registers/mod.rs, grouped by root cause with rationale comments and pointers into the audit doc.
  • check_testsuite enforces the register both ways: an unregistered failure fails the suite (regression), and a registered test that now passes also fails the suite (stale entry — it must be removed in the same change that fixes the feature). This mechanism immediately proved itself: the old property-path skip list had rotted to 14 stale entries out of 18 (13 features had since landed, plus pp06 freed by a harness fix in this PR).
  • Registers can therefore only shrink. The burn-down plan (Phases B–D, with per-cluster perf-safety prescriptions) is documented in docs/audit/2026-07-sparql-testsuite-audit.md.

Harness changes

  • mf:UpdateEvaluationTest implemented end-to-end through the public sparql_update transact surface: load initial graph-store state (ut:data/ut:graphData), apply the update, compare resulting default-graph and named-graph states isomorphically, and check for unexpected non-empty named graphs. Update-eval went from 24 → 67 passing.
  • Named graphs load as real named graphs: qt:graphData/ut:graphData files are wrapped as TriG GRAPH blocks (directives hoisted) and loaded through the transact builder. Also fixes a latent bug where labeled graphData blank nodes were silently dropped (get_graph_data never read ut:graph).
  • CSV/TSV expected-result support: RFC 4180 CSV and SPARQL-TSV parsers plus lossy CSV-space projection for comparison. csv-tsv suite went 0 → 5 of 6.
  • SPARQL 1.2 suites registered per category, and the un-versioned mf:PositiveUpdateSyntaxTest/mf:NegativeUpdateSyntaxTest types they use are now handled.
  • Duplicate combined-manifest test fns removed; Makefile and docs/contributing/sparql-compliance.md rewritten to match the registers model.

New engine findings (registered, not fixed here)

The suite immediately surfaced precise engine gaps, now sitting in the registers with grouped rationale:

  • GRAPH ?g exposes the default graph as a graph named by the ledger alias, and binds ?g as a plain literal rather than an IRI (breaks most sparql10/graph tests even though the data now loads correctly).
  • GRAPH blocks inside DELETE WHERE are rejected at lowering; INSERT into a not-yet-existing named graph silently loses triples; USING semantics are incomplete; combined DELETE/INSERT WHERE applies inserts without the deletes.
  • The UPDATE grammar lacks all graph-management verbs (LOAD/CLEAR/CREATE/DROP/COPY/MOVE/ADD + SILENT, ~70 tests). Modify does parse WITH/USING.
  • Turtle lexer rejects a blank-node label immediately followed by . (_:o6.) — blocks all 4 json-res tests and affects real user data of that shape.
  • The ~76-test SPARQL 1.0 expression cluster: XSD type promotion in comparisons, open-world equality, builtin edge semantics.
  • RDF-star: a code-level sub-audit (audit doc §4.3) confirmed Fluree's edge-annotation/reified-edge model covers parse → IR → match → storage; the actual 1.2 gaps are triple-term functions, bare triple-terms-as-values (pending design decision), Turtle-star ingest, and CONSTRUCT annotation projection.

Perf safety

This PR contains zero engine changes — it touches only the out-of-workspace testsuite-sparql crate, CI config, and docs. Hot-path risk is nil by construction. The audit doc (§6) prescribes the perf-safety discipline for the follow-up engine fixes: parse/prepare-time fixes preferred, common-type fast paths preserved byte-identical, query_hot_bsbm*/insert_formats bench guardrails within regression-budget.json budgets per PR.

Test plan

  • cd testsuite-sparql && cargo test — 36 suites, 0 failed, 0 ignored (~15 s).
  • cargo fmt --all -- --check and cargo clippy --all-targets -- -D warnings clean.
  • CI job runs the full suite with a 45-minute timeout backstop (actual runtime is minutes, dominated by build).
  • No workspace crates touched; workspace CI unaffected.

aaj3f added 2 commits July 6, 2026 13:26
Audit found CI compiled the testsuite but never ran it, eval categories were
all #[ignore]d, and the harness had fallen behind the engine (UPDATE
execution, TriG/named graphs). This makes the whole rdf-tests submodule
enforceable:

- implement mf:UpdateEvaluationTest end-to-end over the public sparql_update
  transact surface (load ut:data/ut:graphData, apply, compare resulting
  default + named graph state isomorphically; 24 -> 67 passing)
- load qt:graphData/ut:graphData as real named graphs (TriG GRAPH blocks via
  the transact builder) instead of dumping them into the default graph; fix
  labeled graphData blank nodes being silently dropped (never read ut:graph)
- add CSV/TSV expected-result parsing with CSV-space projection (0 -> 5 of 6)
- register the SPARQL 1.2 suites (254 tests) and the un-versioned
  Positive/NegativeUpdateSyntaxTest types used by them
- move every suite's skip list to tests/registers/mod.rs, grouped by root
  cause; drop 14 stale property-path entries (13 features since landed +
  pp06 caught by the new check)
- police registers in both directions in check_testsuite: unexpected
  failures AND stale entries (registered tests that now pass) fail the suite
- drop the duplicate combined-manifest test fns; de-#[ignore] everything;
  CI now runs cargo test (36 suites, ~1420 tests, all green, ~15s wall)
- document the audit, failure taxonomy, and perf-safe burn-down plan in
  docs/audit/2026-07-sparql-testsuite-audit.md; refresh
  docs/contributing/sparql-compliance.md and the Makefile for the new model
… plan

Corrects the empirical placeholder: RDF 1.2 annotation syntax ({| |}, ~,
<<( )>> restricted to rdf:reifies) is parsed, lowered to
Pattern::EdgeAnnotation, executed, and durably stored via the reified-edge
model. Actual 1.2 gaps: triple-term functions, bare triple-terms-as-values
(design decision needed), Turtle-star ingest (map onto the existing
edge_annotations reifier pipeline), CONSTRUCT annotation projection. Also
records SERVICE as Fluree-only by design and confirms UPDATE Modify parses
WITH/USING.
@aaj3f aaj3f requested review from bplatz and zonotope July 6, 2026 17:41
aaj3f added 6 commits July 6, 2026 13:58
SPARQL, JSON-LD query, and Cypher share the IR and execution engine. Every
W3C burn-down fix must classify as IR/engine-level (implicit parity, still
add a JSON-LD regression test) or surface-syntax addition (SPARQL-possible
=> JSON-LD-possible in the same effort; Cypher assessed against the
openCypher support matrix). The W3C submodule only guards the SPARQL
surface — JSON-LD/Cypher regression tests are authored with each fix.
We own the JSON-LD query syntax, so SPARQL-possible => JSON-LD-possible
stands with regression tests per fix. Cypher is openCypher — not our
grammar to extend — so SPARQL compliance work carries no Cypher syntax
obligations; it benefits from shared IR/engine fixes automatically.
Address the review's findings — all verified real; all make the harness's
green trustworthy rather than changing what it covers:

- (review #1) the 'no unexpected named graph' guard was dead code: the
  engine binds GRAPH ?g as a plain literal, so the Iri-only filter in
  list_named_graphs always produced []. Accept literal and IRI bindings
  (excluding the alias-named default graph) so the guard survives the
  eventual engine fix. Verified live: an update leaving a stray graph with
  no expected graphs now fails with 'Unexpected non-empty named graph'.
- (review #2) bail when a manifest yields zero tests — a submodule
  restructure or manifest-parser regression must not report green.
- (review #3) a registered test that dies by timeout/subprocess crash now
  hard-fails: the register excuses a known wrong answer, not an infra
  death masking a new hang or panic.
- (review #4) UpdateEvaluationTest now rejects an mf:result blank node
  exposing none of ut:data/ut:graphData/ut:result, instead of degrading to
  a trivially satisfiable 'expected empty store'.
- (review #5) register entries matching no discovered test now fail the
  suite, so dead entries (typos, upstream renames) cannot accumulate.
- notes: tightened the TSV bare-integer heuristic to a single optional
  leading sign; documented the line-based directive-hoisting assumption;
  fixed the stale 'CI runs make ci' Makefile comment.

Full suite remains green: 36 suites, ~1420 tests, 0 failed, 0 ignored.
…rables)

Pre-implementation root-cause audits for every register cluster, produced by
parallel deep-dive passes with live reproduction (run-w3c-test probes, CLI
repros) rather than register-comment inference. Highlights:

- update: ~90 failures collapse to 6 root causes; multi-operation ';'
  requests silently execute only the first op (production-facing; register
  comments for insert-05a / same-bnode / delete-insert-01c corrected)
- parser-syntax: 62 tests -> 8 accept-more + 6 reject-more rules; four
  validation passes missing entirely; reject-more = user-visible changes
- named-graph/dataset: GRAPH ?g literal binding is a one-line fix; default
  graph enumeration is the intentional #1279 extension (recommend
  W3C-by-default); SS5.3 answered: within-ledger datasets (Option A)
- expression-semantics: 13 defects; two shallow bugs (DATATYPE/LANG reject
  expression args; variable-free FILTER drops all rows) explain ~40 tests;
  ~28 register entries reassigned to other clusters
- lexer/formatter: bnode-label dot lexing + canonical xsd:double form
- sparql12 wave 1: VERSION already supported (register mislabel);
  lang-basedir is a cross-stack representation gap; Turtle-star ingest via
  shared EdgeKey::to_reifies_facts greens only 2/41 eval tests alone
- sparql12 wave 2: both triple-term registers are pure syntax suites; D3
  decision narrowed to accept-then-defer vs documented divergence
- residual-eval: property-path multiplicity framing corrected (sequences
  already correct + regression-guarded); pp34/35 are mis-filed graph bugs;
  entailment candidate wins via existing owl2rl materializer + PRAGMA

Stage-2 adversarial verification runs before any claim alters registers or
published statements.
Stage-2 verification of the eight cluster audits: 55 load-bearing claims
attacked by independent skeptics — 40 confirmed, 13 refuted, 2 uncertain —
plus a cross-cluster completeness check over every register entry.

Material corrections the verification caught before implementation:
- the proposed bnode-dot lexer loop breaks spec-valid _:a..b (and doesn't
  compile); replaced with greedy-scan + trailing-dot-rewind
- the harness .rdf DAWG parser bug is real but backwards: Event::Empty
  state-stack skew keeps unbound solutions and DROPS bound ones
- double canonicalization scope was missing export.rs N-Triples/N-Quads
  sites and R2RML lexical() consumers
- named-graph BUG-4 (existence-row drop) is empirically false — graph-exist
  actually fails on base-relative IRI resolution; phantom PR deleted and
  replaced with PR-BASE, which fills a genuine ownership vacuum
- fluree-db-api swallows parse-error diagnostics whenever an AST survives
  recovery, so grammar tightening must prevent AST production
- EdgeKey::to_reifies_facts full variant would emit orphan flakes; ingest
  must use the _jsonld_compatible variant, guarded by equivalence tests

Slate: 17 firm PRs + 5 decision-gated items + 2 unowned work-streams
(algebra/OPTIONAL-scope and output-serialization need a ninth cluster
audit). Firm slate removes ~418 of 538 registered entries; end state with
all decisions favorable is ~124 remaining (63 not-applicable, 47
entailment, 14 orphans). Decisions D-1..D-12 enumerated for team sign-off;
three production bugs queued for immediate issue filing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant