Export eval/org_dynamics_*.py output as an EvalPort TestCase/ResultSet suite

[adhabnr-ux]

Hi — opening this per the "open an Issue before writing code" note in CONTRIBUTING.md and the README's Roadmap item on the eval harness, rather than showing up with a PR.

I maintain EvalPort, an open interchange format (TestCase/Grader/Result/ResultSet/GraderResult) for portable LLM eval datasets, with Python (openeval) and TypeScript SDKs. I think eval/org_dynamics_question_builder.py + eval/org_dynamics_scorer.py map onto it almost 1:1, which is unusual — most eval harnesses need real restructuring, this one mostly needs a serializer.

Why I think it fits well:

Your OrgDynamicsQuestion dataclass (eval/org_dynamics_question_builder.py):

question_id: str
category: str            # ATTENTION_COST | RESOURCE_PRESSURE | CAUSAL_PRESSURE | ...
difficulty: str
day_range: Tuple[int, int]
question_text: str
ground_truth: Dict[str, Any]
evidence_search_space: List[str]
evidence_plan_ids: List[str]
requires_reasoning: bool

maps onto EvalPort's TestCase (sdk/python/openeval/types.py) almost field-for-field:

from openeval.types import TestCase, Grader

def question_to_testcase(q: dict) -> TestCase:
    return TestCase(
        id=q["question_id"],
        input=q["question_text"],
        graders=[f"gr_{q['category'].lower()}_answer",
                 f"gr_{q['category'].lower()}_trajectory"],
        retrieval_context=q["evidence_search_space"],  # exactly what this field is for
        tags=[q["category"], q["difficulty"]],
        metadata={
            "ground_truth": q["ground_truth"],
            "day_range": q["day_range"],
            "evidence_plan_ids": q["evidence_plan_ids"],
            "track_weights": q["track_weights"],
        },
    )

The scorer side maps just as cleanly. org_dynamics_scorer.py's score_answer() -> (float, bool) and score_trajectory() -> OrgDynamicsTrajectoryScore(search_coverage, correct_tools_used, no_hallucination, composite) are basically un-typed GraderResults already:

from openeval.types import GraderResult, Result

def to_grader_results(answer_score, answer_correct, traj: OrgDynamicsTrajectoryScore) -> list[GraderResult]:
    return [
        GraderResult(grader_id="gr_answer", type="custom",
                     score=answer_score, passed=answer_correct),
        GraderResult(grader_id="gr_trajectory", type="custom",
                     score=traj.composite, passed=traj.composite >= 0.55,
                     metadata={"search_coverage": traj.search_coverage,
                               "correct_tools_used": traj.correct_tools_used,
                               "no_hallucination": traj.no_hallucination}),
    ]

Result(test_case_id=q["question_id"], passed=..., grader_results=..., actual_output=json.dumps(final_answer)) closes the loop, and a run across eval_questions.json becomes one ResultSet.

Precedent: since OrgForge already orchestrates through CrewAI, adapters/crewai-openeval-adapter is the closest existing template — it normalizes CrewAI's Task/TaskOutput shapes (dict-or-object, via a _get() accessor) into TestCases from the outside, as a standalone package, without needing anything merged upstream. A orgforge-openeval-adapter could follow the identical shape: to_openeval(questions: list[dict]) -> EvalSuite-dict and results_to_openeval(scored_run) -> ResultSet-dict, living entirely in its own package so it's zero-risk to OrgForge's core.

What this would get you: the ATTENTION_COST/RESOURCE_PRESSURE/CAUSAL_PRESSURE/ASSIGNMENT_QUALITY/ORG_FRICTION question set becomes runnable by any EvalPort-compatible runner, not just agentic_eval_harness.py — useful if export_to_hf.py's HuggingFace path is meant for broader reuse beyond your own harness.

Happy to draft the adapter package myself and open it as a PR here (or as a standalone orgforge-openeval-adapter under the EvalPort adapters dir) if that's a useful direction — wanted to check alignment first per the Golden Rule in CONTRIBUTING.md.

— Sahi, independent contributor (not affiliated with this project)