Skip to content

[FR-009] Orchestrator stores failed evals with error_class (no drop from denominator) #9

Description

@explosivebit

Trace: PRD prd-v0-1-smoke-evaluation-run · FR-009 · RFC error handling taxonomy · Invariant #1 (NEVER violated)

Capability: asyncio.gather(*..., return_exceptions=True) preserves failures. Each failed eval gets error_class from enum (network/rate_limit/5xx_server/4xx_client/timeout/contract_violation/sandbox_failure).

Acceptance:

  • AC-5 from RFC: 4xx_client → 0 retries (fail-fast)
  • AC-5 from SPEC: failed eval present in evals[] (NOT dropped) + counts_by_status.failed += 1
  • SC-3 PRD: 0 dropped evals from denominator

Implementation locus: apps/eval-core-py/src/orchestrator/grid_runner.py

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/rfcRequest for Comments — implementation planphase/2-smokePhase 2 — Smoke run executionpriority/p0Critical — blocker for current Phasetype/featNew feature

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions