feat(sdk): 'rocketride eval' — golden-dataset evaluation runner for .pipe files (assertions, LLM-as-judge, JUnit CI reports)

[nihalnihalani]

Problem Statement

There is no way to answer the question every production AI team asks: "did my last change make this pipeline worse?"

.pipe files are git-versioned JSON living next to application code — RocketRide's defining strength — but nothing can run a pipeline against a fixed set of inputs and check the outputs. A prompt tweak, a model swap, or a chunking change ships on vibes. The building blocks all exist (client.use() / client.chat(), PIPELINE_RESULT, per-run traces via pipelineTraceLevel), but there is no dataset format, no assertion layer, no scorer, and no CI-friendly report.

Industry context: built-in evaluations are now cited as a purchase criterion for AI workflow tools — n8n ships dataset-driven Evaluations in the editor, Flowise has datasets + evaluators, and LangSmith/Langfuse/Braintrust monetize exactly this loop. A July 2026 competitive review of this repo ranked evals the #1 missing capability. An IDE-first, git-native tool is uniquely positioned to own it: eval specs live next to .pipe files and run on every PR.

Related but distinct: #683 adds in-canvas evaluator nodes (eval_cobalt, dataset_cobalt) — evaluation as pipeline components. This issue is the outer harness: a regression runner that treats any .pipe file as the unit under test, from the terminal or CI, complementing (and able to exercise) pipelines that use those nodes.

Proposed Solution

rocketride eval — a test runner for pipelines (Python CLI first; TS parity as follow-up):

1. Eval spec format (<name>.eval.json, strict JSON, lives in git next to the pipeline):

{
  "pipeline": "rag-pipeline.pipe",
  "source": "chat_1",
  "cases": [
    {
      "name": "grounded-answer",
      "input": "What does the onboarding doc say about VPN access?",
      "expect": [
        { "type": "contains", "value": "VPN", "ignore_case": true },
        { "type": "not_contains", "value": "I don't know" },
        { "type": "regex", "pattern": "request.*IT" },
        { "type": "llm_judge", "criteria": "Answer is grounded in the provided documents and actionable.", "min_score": 0.7 }
      ]
    }
  ]
}

2. Deterministic assertions: contains / not_contains (with ignore_case), regex, equals, min_length/max_length, json_path (dot-path + equals/gte/lte for structured outputs), latency_max_ms.

3. LLM-as-judge — the judge is itself a pipeline. llm_judge runs a judge .pipe (a bundled default template is included; any judge pipeline can be substituted per-spec or per-case) that receives the case input, the pipeline's output, and the criteria, and returns a JSON verdict {score, reasoning}. No new LLM-provider dependencies in the SDK; judging runs on the same engine, with whatever provider node the team already uses — and judge pipelines are themselves git-versioned and eval-able.

4. Runner + reporters: rocketride eval <specs...> [--case <filter>] [--fail-fast] [--json] [--junit <path>] — connects like every other subcommand, runs cases sequentially against the engine, reports per-case/per-assertion results. Exit codes match validate (#1573): 0 all pass · 1 failures · 2 usage/spec/connection error. --junit emits standard JUnit XML so any CI renders the results natively.

5. Docs + example: eval guide, examples/rag-pipeline.eval.json, and the default judge template.

Alternatives Considered

• Evaluation nodes on the canvas (feat(test,nodes): integrated Cobalt pipeline evaluation (testing framework + dataset + evaluator) #683's direction) — right for online/in-flow scoring, but regression testing needs to live outside the pipeline under test, versioned with it, runnable headlessly in CI.
• Point users at LangSmith/Langfuse/Braintrust — real ops platforms, but they evaluate LLM calls, not .pipe files; the graduation path off RocketRide starts exactly there.
• pytest plugin instead of a CLI — considered; the CLI is language-neutral for TS/Java users and CI, and a pytest wrapper can trivially shell out to it later.

Affected Modules

• client-python (eval module + CLI subcommand)
• server (C++ engine) — no changes
• docs
• client-typescript — follow-up issue once the format stabilizes

Acceptance Criteria

• rocketride eval examples/rag-pipeline.eval.json runs every case against a live engine and reports per-assertion results; failures exit 1 with actionable output.
• All deterministic assertion types covered by unit tests (no server needed); spec-format errors reported with file/case context, exit 2.
• llm_judge executes a judge pipeline and parses its verdict robustly (malformed judge output = case error, not a crash); default judge template included.
• --json and --junit outputs; JUnit XML validates against the standard schema.
• Docs + runnable example spec against an in-repo example pipeline.

Happy to implement — PR to follow shortly.

[github-actions]

🤖 Internal: Discord sync marker

Auto-managed by the Discord notification workflow. Stores the linked Discord message ID and forum thread ID. Do not edit or delete.

[nihalnihalani]

Implementation is up in #1581 — all acceptance criteria covered: the spec format with every assertion type, pipeline-as-judge with the in-wheel default template and per-spec/per-case overrides, robust verdict parsing (malformed judge output fails the assertion, never crashes), --json and xmllint-valid --junit output, and docs + a runnable examples/rag-pipeline.eval.json.

Verification highlights (full detail in the PR): 189 unit tests with no server; a live end-to-end where the pip-installed CLI ran a two-case spec against the real dockerized engine (mock LLM SDKs) — correct exit codes with one passing and one deliberately failing case, valid JUnit written, and the packaged judge template executing live with graceful degradation on non-JSON verdicts.

As scoped in this issue, TypeScript CLI parity is a follow-up once the spec format settles through review.