Skip to content

[FR-005] Orchestrator runs automatic evaluator per task and stores JSON result #5

Description

@explosivebit

Trace: PRD prd-v0-1-smoke-evaluation-run · FR-005 · RFC implementation task 4

Capability: 3 evaluators (be_01_jwt_auth, fe_01_multistep_form, doc_01_cli_readme) produce evaluator_json per task-specific schema. be_01 uses Docker sandbox pinned by digest (architect finding #5 from EVID-005 SWE-bench).

Acceptance:

  • Each evaluator deterministic on same input (NFR-003)
  • Docker image for be_01 pinned by SHA digest, not tag
  • Gold solution → evaluator → 100% pass

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/rfcRequest for Comments — implementation planphase/2-smokePhase 2 — Smoke run executionpriority/p1High — current Phase scopetype/featNew feature

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions