OpenEvalGate

Turn evaluation evidence into a bounded launch decision for production AI assistants and agents.

OpenEvalGate is an open-source, local evidence-admission layer for enterprise AI product and platform teams. It deterministically assesses declared review evidence and turns it into a bounded recommendation for the requested review stage.

Use it after your existing eval runner, harness, or manual review has produced evidence. OpenEvalGate validates the evidence package, fails closed on contradictions and missing critical controls, identifies launch blockers, and writes a reproducible Markdown report.

It helps teams assemble and review evidence for three questions:

1. Expected behavior: Does observed behavior match a business-owned contract?
2. Safe stopping boundaries: Does autonomy end at the right point, with the right human destination and context?
3. Release evidence: Do eval results, critical controls, mitigations, owners, and rollback plans support the requested review stage?

OpenEvalGate does not run the candidate system. Teams run evaluations with their existing tools, record the results locally, and use the CLI to validate the evidence package, identify blockers, and generate a release-assurance report.

Stable core: 0.1.0 provides Core Compatibility v1 for its defined CLI, JSON, assessment, and V1 evidence surfaces. Templates, playbooks, vendor adapters, and full product-scope stability remain experimental. See the Core Compatibility v1 and governance.

The Production Problem

A demo or model benchmark can show that a system succeeded once. A release review must also examine whether it behaves consistently within policy, stops safely, transfers work correctly, and leaves operators able to observe and roll back the system.

Intent + context + policy + risk + user-friction signals
                          |
                          v
              Escalation Control Surface
                          |
                          v
      Resolve | Clarify | Act | Approval | Escalate | Refuse
                               |
                               v
        Live handoff | async case | specialist review
              approval queue | safe fallback

OpenEvalGate assesses both under-escalation, which can create unsafe autonomy or false containment, and over-escalation, which can waste human capacity and increase user effort.

Reference Scenarios

These repository-authored scenarios are synthetic and illustrative. They are not external production deployments, independent adoption evidence, or external validation of OpenEvalGate.

Domain	Safe stopping boundary	Human path evidence	Failure demonstrated
Subscription support	Billing explanation, cancellation approval, account security, authentication refusal	Billing approval queue, account security review, deterministic block	Passing bounded controlled-launch package
Customer support	Refund approval, repeated failure, fraud signal, dependency outage	Live handoff, approval queue, async case, fraud operations	Under-escalation, wrong destination, false containment
Presales	Discount authority, roadmap claims, legal/security commitments	Account-owner follow-up, pricing approval, specialist review	Unsupported commitment, late escalation
Education	Graded work, learner safety, accommodation exception	Instructor review, safety route, accessibility approval	Missing handoff context, unnecessary escalation, failed resume

See the examples index for scenario purposes, inputs, and reproduction commands. Start with the passing subscription-support report, then compare it with the blocked refund-support report to see why evidence completeness and launch readiness are separate decisions.

Quickstart

New to the repository? Start with Getting Started for Practitioners.

python -m pip install -e .
openevalgate --version
openevalgate validate examples/subscription_support_assistant/eval_cases.yaml
openevalgate check examples/subscription_support_assistant/
openevalgate report examples/subscription_support_assistant/ \
  --output /tmp/openevalgate-subscription-report.md

The quickest path is to copy the subscription-support scenario into a new project, replace its synthetic evidence with your own artifacts, and rerun the same three commands. The report is written to /tmp so the canonical example remains unchanged.

For a clean project scaffold, use the deterministic minimal profile:

openevalgate init my-assistant --profile minimal
cd my-assistant
openevalgate validate eval_cases.yaml
openevalgate check .
openevalgate report . --format card

The generated files are synthetic placeholders and the initial result is expected to be incomplete. Replace them with reviewed evidence before treating the report as a release input. Use --format json for automation and --fail-on-blocked when a CI job should fail on a blocked recommendation. See the CLI output contract.

The generated report separates evidence completeness from observed behavior and critical controls:

Structured escalation contract: valid
Destination SLA coverage: 100%
Required-escalation recall: 67%
Over-escalation rate: 33%
Destination accuracy: 33%
Context-preservation rate: 67%
Hard blocker: critical_escalation_regression
Recommendation: Not ready

High evidence completeness cannot override a high-risk case that failed to stop or reached the wrong human destination.

How It Works

1. Define assistant scope, unsupported uses, and business-owned behavior.
2. Build golden eval cases from expected, boundary, adversarial, tail-risk, regression, and drift scenarios.
3. Define action risks, automation boundaries, human escalation, observability, rollback, and ownership.
4. Run the candidate assistant externally using an eval runner, internal harness, or manual review.
5. Save observed results in eval_results.csv and optional outputs in eval_runs/.
6. Validate the project and generate a local report.
7. Review the recommendation, blockers, mitigations, and required next actions.

See the external-runner handoff for a V1 evidence handoff, including a LangChain-shaped producer example that does not add an SDK dependency.

The framework distinguishes:

• evidence completeness;
• behavioral-evidence status;
• critical-control status;
• maximum permitted review stage;
• final bounded recommendation.

The complete gate list, scoring weights, evidence bands, and hard-gate semantics are documented in Launch Gates and Evidence Completeness Scoring. Review modes, selected-run coverage, and controlled-launch behavioral sufficiency are documented in Review Modes and Behavioral Sufficiency.

Runtime Eval-Run Identity

The Eval-Run Provenance Contract v1 defines how a small run_manifest.yaml can wrap an existing OpenEvalGate-compatible eval_results.csv. Existing compatible CSVs do not need new provenance columns.

OpenEvalGate classifies selected eval-run identity as complete, missing, or invalid. complete means the run, candidate, evaluator, result CSV, output paths, recognized output metadata, and declared artifact-index identities are coherent. It does not mean the run passed, behavioral evidence is sufficient, or controlled launch is authorized.

Projects may omit empirical results while documenting controls. Once a conventional eval_results.csv is present, an authoritative run_manifest.yaml is required.

A manifestless result file is surfaced as a provenance validation failure, while its rows are excluded from row validation, summaries, metrics, coverage, behavioral invariants, behavior-derived blockers, and launch authorization. No new CSV columns are required.

Artifact indexes remain optional. Runtime validation currently includes local SHA-256 digest verification, verified assurance, freshness and recency comparison, and review_context.yaml enforcement for local historical and current evidence. Complete authorization classification, enriched workflow and handoff claims, and broader artifact-version policy remain deferred.

Minimal manifest:

schema_version: "1"

run:
  id: run_001
  status: complete

candidate:
  id: candidate_name
  version: candidate_version

evaluation:
  kind: human
  evaluator:
    id: human_review

outputs:
  results:
    path: eval_results.csv

Where OpenEvalGate Fits

OpenEvalGate complements eval runners, observability platforms, runtime controls, security tooling, and internal review systems. Its specific job is evidence admission: checking that the package is internally coherent, provenance-aware, and sufficient for the requested review stage before a cross-functional decision.

Business intent / policy / trust risk
        |
        v
OpenEvalGate review artifacts and declared controls
        |
        v
External eval runners, candidate systems, traces, and runtime tools
        |
        v
eval_results.csv, observed outputs, incidents, and control evidence
        |
        v
OpenEvalGate validation, blockers, and bounded recommendation

Eval and LLMOps tools execute tests, collect traces, monitor behavior, or enforce runtime policy. OpenEvalGate consumes their outputs through local artifacts and organizes that evidence for a defined release review. It does not require an official vendor integration.

For CI summaries and review meetings, generate a compact decision card:

openevalgate report . --format card

For automation, consume the versioned JSON envelope instead of parsing Markdown:

openevalgate report . --format json --output /tmp/openevalgate-decision.json

See Research Context and Adjacent Tools for conceptual relationships with external guidance and tooling.

Core Primitives

• Business behavior contract: expected and unacceptable behavior owned by product and domain stakeholders.
• Golden eval set: product-specific behavior expressed as reusable test cases.
• Tail-risk review: explicit treatment of rare, high-impact failures.
• Action-risk matrix: classification of tool and action risks, including required human review.
• Automation boundary: decisions about resolution, clarification, approval, escalation, refusal, and blocking.
• Human escalation contract: destinations, service levels, payloads, fallbacks, and durable resume behavior.
• Evidence completeness: documentation and review coverage, kept separate from behavioral quality.
• Hard blockers: critical controls that can prevent advancement regardless of aggregate score.
• Review modes: documentation, shadow-launch, and controlled-launch assessments with different evidence requirements.

For multi-agent systems, optional routing_policy.yaml evidence records fixed, adaptive, deterministic, and human paths per bounded workflow. A stronger model is not treated as an authorization or safety control.

Who It Is For

• AI product managers and business or operations owners;
• ML, AI, and platform engineers;
• trust, safety, legal, compliance, and security reviewers;
• teams preparing a cross-functional review of an assistant or agent.

The same decision packet is intended to give builders reproducible CI evidence, operators and product owners clear mitigation work, reviewers stable blocker identifiers, and decision makers a bounded recommendation rather than a generic quality score.

Limitations and Non-Claims

OpenEvalGate has a stable V1 core and practitioner-defined defaults with repository-authored synthetic examples. Full product-scope stability and adoption evidence remain future 1.0 work.

• It does not execute candidate models, call LLM APIs, or evaluate live behavior by itself.
• It does not replace eval runners, observability systems, runtime guardrails, security controls, or organizational approval.
• It validates submitted artifacts and declared evidence; it does not independently verify that every claim in those artifacts is true, current, or complete.
• The current implementation validates core result identities, selected eval-run identity, expected-route consistency, duplicate identities, review timestamps, supplied output references, output identity metadata, artifact-index identity, local provenance digests, evidence freshness and recency, review-context validity, assurance, and basic route-match derivation. It does not yet provide complete authorization classification, enriched workflow-route claims, handoff claims, or every routing-policy and model-policy field.
• A recommendation is only as reliable as the quality, completeness, provenance, and freshness of the supplied evidence.
• A passing check, high evidence score, or bounded recommendation does not guarantee safe, reliable, compliant, or successful deployment.
• It does not certify compliance or provide legal, regulatory, security, or risk-management certification.
• References to NIST, ISO, OECD, OWASP, the EU AI Act, or other guidance describe conceptual relationships, not formal alignment, endorsement, partnership, or approval.
• References to external tools describe complementary workflows, not official integrations or vendor partnerships unless explicitly documented by both parties.
• Repository-authored scenarios are not external validation, independent adoption evidence, or proof of production performance.
• The scoring weights and thresholds are initial practitioner defaults, not scientific constants or a universal industry standard.
• Teams remain responsible for system design, evidence quality, runtime controls, monitoring, authorization, and release decisions.

Templates and Playbooks

The repository includes local Markdown, YAML, and CSV templates for assistant scope, behavior contracts, eval cases and results, action risk, automation boundaries, escalation, output review, observability, metrics, launch gates, and reports.

Practitioner playbooks include:

• Golden Eval Set Playbook
• Synthetic Boundary Case Guide
• Routing Playbook
• Human Escalation Playbook

Repository Structure

docs/          Guides, methodology, and playbooks.
templates/     Copyable review artifacts.
examples/      Synthetic, illustrative assistant scenarios.
openevalgate/  Deterministic local Python CLI.
tests/         Schema, policy, assessment, and report tests.

Maintainer, Citation, and Contribution

OpenEvalGate was created and is maintained by Chenye Zhu and is released under the MIT License.

For references, use CITATION.cff. Release history is in CHANGELOG.md. Contributions should follow CONTRIBUTING.md, GOVERNANCE.md, and the Code of Conduct. See SUPPORT.md for support boundaries and report suspected vulnerabilities according to SECURITY.md.

Keep the CLI small, deterministic, local-first, and dependency-light.