Skip to content

Block 1.1.0 stable promotion until authenticated repeated live-Codex evaluation #4

Description

@VKK-00

Purpose

Track the only remaining external acceptance gate for promoting 1.1.0-rc.1 to stable. Static validation, fixtures, policy checks, property tests, mutation testing, local security review, reproducible builds, and GitHub Actions are necessary but do not prove real model behavior.

Required evaluation

Run the committed live behavioral-evaluation harness against a pinned authenticated Codex model using the committed held-out pressure cases.

Minimum execution contract:

  • baseline and skill-enabled variants;
  • at least three independent runs per case and variant;
  • clean isolated Git workspace per run;
  • pinned model identifier and recorded Codex version;
  • immutable run manifests and trace digests;
  • no manual editing of generated outcomes before grading.

Required metrics

Record and publish at least:

  • routing and outcome compliance;
  • gate-compliance rate;
  • scope-escape rate;
  • fabricated-evidence rate;
  • policy-violation rate;
  • unnecessary-action count;
  • rollback success rate;
  • human-correction rate;
  • pass@1 and pass@3;
  • tokens, tool calls, duration, and cost where available.

Acceptance criteria

  • all configured promotion thresholds in the repository pass;
  • no critical policy or evidence-fabrication violation occurs in any run;
  • generated report validates against the committed schema and is bound to the evaluated commit, model, policy digest, and harness version;
  • limitations and failed cases remain visible;
  • an independent reviewer verifies the report and promotion decision.

Non-acceptance

Do not substitute deterministic routing proxies, fixture execution, instruction-coverage scores, a single model run, or an agent-authored narrative for this evidence.

Promotion rule

1.1.0 must remain a release candidate until this issue is closed with the validated evaluation artifacts linked in the closing comment.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions