Skip to content

Latest commit

 

History

History
11 lines (6 loc) · 2.43 KB

evaluations.md

File metadata and controls

11 lines (6 loc) · 2.43 KB

Evaluation contract — not measured results

The executable Python suite checks the actual SQLite migration, constraints, query compilation, scoped selector, backup and SQL-engine rollback. It is not an alternate implementation of the Rust Core and cannot close application/transport/native gates. Raw output is in verification/check-sqlite.txt.

Rust tests are authored for evidence-free direct + independent JSON assertions, scoped refresh/withdrawal, policy/generation CAS, idempotency, inferred-publication rejection, empty-graph goals, no-model status, an explicitly fake answer adapter's structural contract, conditions, owner lease, read-only readers, cancellation, private assertion visibility and input/transport contracts. Parser regression tests are retained behind code-adapter. On 2026-10-09 all of them executed and passed on native macOS arm64 (36 default-feature, 36 headless, 384 all-feature library tests plus the integration tests), together with new end-to-end Core serve, loopback REST, desktop lease, conflict, valid-time, n-ary literal and zero-egress regressions; raw output is in verification/cargo-test*.txt. They have not run on the other three targets. A fake adapter test is not a real-model gold evaluation.

Before release, freeze held-out examples and thresholds for typed role/predicate precision, relation recall, namespace/identity safety, conditional/negative/planned/hypothetical semantics, conflicting and UNKNOWN context, relation-marker validity, actual answer entailment, malicious prompts, scope noninterference, multilingual query understanding (English/German first), and optional Turkish/Arabic. Do not use evidence availability as a success prerequisite.

G5 requires empty-graph goals becoming useful through real approved providers, not merely more nodes. Measure need relevance, type mapping, qualified source coverage, partial-reconcile retention, per-slot before/after coverage, precision/recall, query disclosure, zero unauthorized writes/requests, rate/deadline/cancellation under concurrent transports, cache convergence, and revocation between discovery/capture/commit. No such scores are reported.

G8 needs actual native install/MCP/GUI/backup/restore/suspend/upgrade reports on all four target systems, plus latency/memory/DB/WAL measurements on agreed representative graph sizes. Source-only CI and cross-compilation are not substitutes. There are no invented benchmark tables or green native attestations in this archive.