Skip to content

Harden selective verification and complete competitor matrix - #35

Merged
samarth1412 merged 16 commits into
mainfrom
agent/v1.2-safety
Aug 8, 2026
Merged

Harden selective verification and complete competitor matrix#35
samarth1412 merged 16 commits into
mainfrom
agent/v1.2-safety

Conversation

@samarth1412

Copy link
Copy Markdown
Owner

What changed

  • Adds selective semantic_core_v2 and semantic_core_v2_1 verification, schemas, limits, concurrency isolation, and source-condition diagnosis.
  • Adds controlled development corpora and engineering benchmarks for hard negatives, source conditions, and evidence attribution.
  • Pins and audits RAGAS, MiniCheck, and RefChecker competitor runtimes.
  • Completes the nine-row visible-development comparison matrix across RAGTruth, ARES, and the CRAG proxy track.
  • Documents migration, research boundaries, and the SOTA engineering roadmap.

Why

The verifier needed safer selective behavior, stronger source-condition attribution, reproducible external comparisons, and explicit separation between visible development measurements and untouched evaluation evidence. The interrupted RefChecker run was caused by an Ollama auto-update from the pinned 0.32.1 runtime to 0.32.5; it was resumed and completed using the exact pinned runtime.

Impact

This gives ContextTrace a reproducible development baseline and stronger implementation safeguards without publishing a release, touching sealed labels, or claiming SOTA. CRAG remains an agreement/proxy track rather than an accuracy evaluation.

Validation

  • 609 passed, 1 skipped.
  • Coverage 86.85%; the 80% gate passed.
  • ResourceWarning checks passed.
  • All nine comparison rows reproduced from raw artifacts with exact ID, input, and hash binding.
  • Targeted Ruff and mypy checks passed.
  • git diff --check passed.

@samarth1412
samarth1412 marked this pull request as ready for review August 8, 2026 15:17
@samarth1412
samarth1412 merged commit 564210e into main Aug 8, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant