DeepAgent EVAL

English (default) | Chinese

DeepAgent EVAL is a mechanism-oriented evaluation harness for the DeepAgentCode family. It runs long, real-world coding tasks from DeepSWE against a real DeepAgentCode checkout, then collects process, Session, provider, Git, persistence, and task-lifecycle evidence that can be reviewed, replayed, and attached to a bug report.

EVAL is for product reliability, not model ranking. It asks whether the agent runtime behaves correctly under realistic engineering pressure:

• Are subagents created, supervised, budgeted, and delivered back to the parent?
• Are child identity, partial transcripts, failures, retries, and explicit recovery preserved?
• Are worktrees and concurrent writers isolated correctly?
• Do code intelligence, context federation, compaction, memory, Goal, Panel, and Wiki mechanisms actually execute?
• Can an incident be reproduced from immutable event evidence rather than model self-report?

Harbor containers are not required. The bundled DeepSWE verifier is an optional secondary signal and is disabled by default.

Why this helps DeepAgentCode development

EVAL turns a real task run into a repeatable engineering feedback loop:

1. Run a real DeepSWE task through the production DeepAgentCode entry point.
2. Capture the causal evidence for the mechanism under test.
3. Classify the result as a runtime, harness, provider, environment, budget, or model-activation outcome.
4. Preserve the raw run and produce a deterministic bug report.
5. Replay evaluator changes without starting the agent or calling the provider.
6. Fix DeepAgentCode and rerun the same task, profile, provider, and repetition set.

This exposes long-range failures that isolated unit tests rarely cover: child lifecycle across provider turns, parent/child delivery, budget truncation, process restart boundaries, concurrent worktrees, post-compaction continuity, and cross-session persistence. EVAL does not decide product design and does not treat the final patch as the only success criterion; it provides reproducible evidence for the engineering team.

What is covered

A real task remains the workload. EVAL adds only a versioned workflow policy when a mechanism needs deterministic activation; the original task remains authoritative and the policy never supplies a solution.

Profile	What it exercises
natural-task	Run the original task unchanged and observe naturally activated mechanisms.
subagent-collaboration	Foreground research, structured results, parent consumption, planning, and code intelligence.
workspace-isolation	Parallel writers, worktrees, dirty-parent retention, and workspace hygiene.
context-pressure	Long turns, compaction, bounded context, Ledger, and query-log recall.
cross-session-memory	Persisted handoff, provenance, fresh-session retrieval, and promotion boundaries.
goal-loop	Plans, Goal ticks, Panel reviewer fan-out, structured verdicts, and Wiki governance.
fault-recovery	Child interruption, partial evidence retention, explicit recovery, and replay safety.
packaged-product	Package/state-isolation probes. Task-to-renderer/UI attribution remains not observable without a supported UI seam.

Inspect the current profile and contract catalog without running a task:

bun run src/index.ts --list-contracts

A product capability can be implemented but still lack a trustworthy EVAL oracle. EVAL reports that as an observation limitation instead of silently declaring success.

Evidence pipeline

local DeepSWE task and exact baseline
  -> real DeepAgentCode CLI and server sidecar
  -> SSE, Session tree/messages, provider boundaries, Git and state evidence
  -> deterministic mechanism analyzers and Tier-1 monitors
  -> JSON/Markdown assessments and bug reports
  -> optional evidence-grounded LLM diagnosis

The deterministic layer is authoritative. LLM diagnosis is optional source-location and explanation support; it cannot create or remove a finding. Each run records target/harness provenance, effective prompt hashes, relevant feature flags, event ranges, task/session identifiers, and observation completeness.

Included offline assets

The repository includes the task data and repository assets needed to run without downloading benchmark repositories at runtime:

data/
├── deepswe/
│   ├── LICENSE
│   ├── PROVENANCE.md
│   ├── DATASET_NOTICE.md
│   ├── tasks/
│   └── repos/
│       ├── manifest.json
│       └── archives/
└── models/
    ├── models.json
    └── manifest.json

The bundled snapshot contains 113 original DeepSWE tasks, five programming languages, 91 logical upstream repositories, and 92 exact repository asset keys. Archives are ordinary compressed data files, not nested Git repositories or submodules. Each task baseline is checked out from its recorded commit before the agent starts.

Verify the assets before the first campaign:

bun install --frozen-lockfile
bun run assets:verify --concurrency 3

This recomputes SHA-256 checksums, unpacks archives, runs git fsck, and checks out every baseline without network access. Asset preparation is only needed when updating the vendored dataset.

Requirements

• Bun compatible with the lockfile (the reference environment uses Bun 1.3.14) and Git.
• A local DeepAgentCode source checkout with its Bun dependencies installed for the cli runner.
• A provider credential and model supported by that DeepAgentCode checkout. The model API may use the network; benchmark data and repository assets do not.
• Enough disk space for isolated workspaces and reports.

For release evidence, keep both EVAL and target worktrees clean and committed. EVAL records commit IDs, porcelain status, and a digest covering untracked bytes. Development overrides are available, but dirty runs are marked ineligible for release evidence.

Install and check both programs before the first run:

cd /path/to/deepagent-eval
bun install --frozen-lockfile
bun run assets:verify --concurrency 3

cd /path/to/deepagent-code
bun install --frozen-lockfile
bun run packages/deepagent-code/src/index.ts --help

The cli runner uses the production DeepAgentCode credential store. Authenticate DeepAgentCode once so that ~/.deepagent/code/auth.json exists, or point EVAL at a different credential file when starting the campaign:

export DEEPAGENT_EVAL_AUTH_FILE=/absolute/path/to/auth.json

For each product process, EVAL copies that file with mode 0600 into an isolated campaign HOME and removes the copy when the process finishes. LLM_API_KEY configures only simple-agent and the optional --llm-monitor; it does not authenticate the real cli runner.

When credentials have not been configured yet, start the production provider login from the DeepAgentCode checkout:

cd /path/to/deepagent-code
bun run packages/deepagent-code/src/index.ts providers login

Quick start

Run preflight before spending provider time. It validates the strict CLI/config contract, selected task IDs, offline asset declarations, pinned model, DeepAgentCode entry and dependencies, credential availability, Git provenance, and the campaign output directory. It does not start DeepAgentCode or call the model:

bun run src/index.ts \
  --runner cli \
  --source-dir /path/to/deepagent-code \
  --deepswe-dir ./data/deepswe \
  --reports-dir ./reports/smoke \
  --task-id superjson-error-stack-serialization \
  --model deepseek/deepseek-chat \
  --preflight

Preflight prints a secret-free resolved campaign as JSON. --dry-run is an alias, and --list-tasks prints the same result with the selected task metadata. An explicit missing --config, unknown option/config field, repeated scalar option, invalid task ID, missing required credential, or existing EVAL evidence in --reports-dir exits 1 before any campaign artifact is created.

For the common source-checkout workflow, the repository wrapper defaults to the sibling ../deepagent-code checkout and creates a unique campaign directory under reports/:

VERIFY_ASSETS=1 ./run.sh \
  --runner cli \
  --model deepseek/deepseek-chat \
  --task-id superjson-error-stack-serialization

Explicit command-line values take precedence over the wrapper defaults. Set DEEPAGENT_SOURCE_DIR only when the target checkout is elsewhere.

One real task

Replace /path/to/deepagent-code with the checkout to test. The model below is an example from the bundled catalog; replace it with a model for which DeepAgentCode has a valid credential.

cd /path/to/deepagent-eval
MODEL=deepseek/deepseek-chat
bun run src/index.ts \
  --runner cli \
  --source-dir /path/to/deepagent-code \
  --deepswe-dir ./data/deepswe \
  --reports-dir ./reports/smoke \
  --profile natural-task \
  --max-tasks 1 \
  --timeout-seconds 900 \
  --model "$MODEL"

The cli runner starts the production DeepAgentCode entry point and a server sidecar, subscribes to the SSE stream, and captures recursive Session snapshots before shutdown.

Subagent campaign

The task ID in this example exists in the bundled snapshot and the selected profile requires at least three repetitions for a gate-eligible campaign.

cd /path/to/deepagent-eval
MODEL=deepseek/deepseek-chat
bun run src/index.ts \
  --runner cli \
  --source-dir /path/to/deepagent-code \
  --deepswe-dir ./data/deepswe \
  --reports-dir ./reports/subagent-campaign \
  --profile subagent-collaboration \
  --task-id python-statemachine-state-data-scoping \
  --repetitions 3 \
  --timeout-seconds 1800 \
  --model "$MODEL"

Three repetitions are a practical minimum for a stability comparison. One successful run is a smoke result, not a release claim.

Offline model catalog

EVAL uses data/models/models.json by default. To test a different pinned catalog, pass the same file to the isolated product runtime:

cd /path/to/deepagent-eval
MODEL=deepseek/deepseek-chat
bun run src/index.ts \
  --runner cli \
  --source-dir /path/to/deepagent-code \
  --deepswe-dir ./data/deepswe \
  --reports-dir ./reports/offline-catalog \
  --profile subagent-collaboration \
  --models-path /path/to/models.json \
  --model "$MODEL"

The model catalog prevents runtime catalog downloads; it does not make the model API local. Configure a local provider in DeepAgentCode if inference must also be offline.

Runner modes

Runner	Use	DeepAgentCode evidence
cli	Production DeepAgentCode CLI plus server/SSE collection.	Yes
simple-agent	Harness-only OpenAI-compatible tool-calling smoke test.	No
solution-patch	Harness/verifier pipeline self-test with a fixed patch.	No

The two harness-only runners validate EVAL wiring and local dependencies. They must never be presented as evidence about DeepAgentCode behavior.

Running and observing

Every campaign writes an atomic status.json, structured progress.jsonl, and a concise human progress.log. status.json exposes the current run, phase, Session, child, provider turn, tool, budget and last lifecycle event without copying reasoning or tool output:

jq . reports/subagent-campaign/status.json
tail -f reports/subagent-campaign/progress.jsonl

For a background run:

cd /path/to/deepagent-eval
MODEL=deepseek/deepseek-chat
mkdir -p reports/campaign-01
nohup bun run src/index.ts \
  --runner cli \
  --source-dir /path/to/deepagent-code \
  --deepswe-dir ./data/deepswe \
  --reports-dir ./reports/campaign-01 \
  --profile subagent-collaboration \
  --task-id python-statemachine-state-data-scoping \
  --repetitions 3 \
  --timeout-seconds 1800 \
  --model "$MODEL" \
  > reports/campaign-01/console.log 2>&1 &

The first SIGINT or SIGTERM asks the active run to stop and finalize evidence. A second signal force-exits. An interrupted run remains incomplete in the report.

Common options:

Option	Meaning
--source-dir	DeepAgentCode source tree; required for cli.
--deepswe-dir	Bundled cleaned DeepSWE data.
--repo-assets-dir	Override the vendored repository asset directory.
--models-catalog	Override the evaluator's pinned model catalog.
--reports-dir	One campaign evidence directory. Existing EVAL evidence makes preflight fail.
--campaign-id	Stable campaign ID; generated when omitted.
--config	Optional strict JSON config. Without it, EVAL uses built-in defaults plus CLI values; config.example.json is never loaded automatically. A missing file, unknown field, or unresolved environment placeholder is an error.
--profile	Mechanism scenario profile.
--task-id	Comma-separated task IDs.
--max-tasks	Maximum selected tasks.
--shuffle / --no-shuffle	Enable or disable seeded task shuffling from config.
--repetitions	Repetitions per task.
--timeout-seconds	Whole-task wall-clock budget.
--preserve-workspaces / --no-preserve-workspaces	Keep or remove temporary workspaces outside the evidence package.
--model / --variant / --agent	Forward model selection to DeepAgentCode.
--models-path	Use a specific local models.json in both EVAL and DeepAgentCode.
--packaged-app	Application path used by the packaged-product profile.
--required-contract	Comma-separated additional required contract IDs.
--verifier / --no-verifier	Enable or disable the optional local verifier.
--gate / --no-gate	Enable or disable the quality gate.
--llm-monitor / --no-llm-monitor	Enable or disable optional LLM diagnosis; enabling it requires LLM_API_KEY.
--allow-dirty-target / --allow-dirty-harness	Development-only provenance overrides.
--preflight / --dry-run	Print the resolved campaign and validate it without starting the Agent.
--list-tasks	Validate and print the selected DeepSWE tasks without running them.
--list-contracts	Print the current scenario and contract catalog without running a task.

Reports and handoff

A typical campaign directory looks like this:

reports/<campaign>/
├── .deepagent-eval-campaign.json
├── campaign-manifest.json
├── provenance.json
├── status.json
├── progress.jsonl
├── progress.log
├── mechanism-summary.json
├── bugs/
│   ├── <bug-id>.json
│   └── <bug-id>.md
├── diagnoses/                     # only with --llm-monitor and a finding
│   └── <diagnosis-id>/
├── verifier-logs/                 # only with --verifier
│   └── <run-id>/
├── sessions/
│   └── <task-id>-<phase-id>-<timestamp>.jsonl
└── runs/<run-id>/
    ├── events.jsonl               # multi-phase or harness-owned log
    ├── mechanism-assessment.json
    ├── diagnostic-context.json
    ├── scenario-phases.json        # multi-phase profiles
    └── campaign-state/             # persisted multi-phase product state

The exact files vary by profile and failure point. A natural-task cli run normally records its authoritative event stream under sessions/; multi-phase and harness-owned runs also write runs/<run-id>/events.jsonl. New assessments store evidence.eventLogPath relative to the campaign directory, plus campaignId and runnerMode. Durable database rows are captured as harness.durable.snapshot events in that JSONL, not as a separate file. Derived assessments and Markdown reports can be regenerated.

Treat the entire campaign directory as an immutable evidence package. Do not copy only the final Markdown. A minimal handoff contains the manifest, provenance, mechanism summary, affected run directories, bug JSON/Markdown, raw events, and Session JSONL. For transfer between machines:

tar -czf campaign-01.tgz -C reports campaign-01
shasum -a 256 campaign-01.tgz

Use sha256sum campaign-01.tgz instead of shasum on Linux.

Before sharing, review prompts, tool arguments, model output, file paths, and timing information. Authentication files are excluded from state manifests, and bug reports include only an allowlist of non-secret mechanism flags rather than arbitrary configured environment variables. Reports should still be treated as potentially sensitive.

Reading a result

Start with the campaign summary:

CAMPAIGN=reports/campaign-01
jq '.quality, .stoppedReason, .l1.total, .l1.run' \
  "$CAMPAIGN/mechanism-summary.json"

List the captured run IDs, select one, and inspect its assessment:

ls -1 "$CAMPAIGN/runs"
RUN_ID=REPLACE_WITH_RUN_ID
RUN_DIR="$CAMPAIGN/runs/$RUN_ID"
jq '{campaignId,runId,taskId,profile,runnerMode,gateEligible,observationComplete,checks,violations}' \
  "$RUN_DIR/mechanism-assessment.json"

Resolve the raw event log before following a finding:

jq -r '.evidence.eventLogPath' \
  "$RUN_DIR/mechanism-assessment.json"

Resolve a new relative path against the campaign directory. Older assessments may contain an absolute path; replay also checks the campaign sessions/ directory by basename for those reports.

The statuses are deliberately separate:

Status	Meaning
passed	Direct evidence satisfies the configured invariant.
failed	Direct evidence violates a runtime or harness invariant.
degraded	Activation miss or explicit bounded fallback without proof of a crash.
blocked	Upstream provider, workspace, budget, or harness issue prevented observation.
not_exercised	The selected task/workflow did not activate the capability.
not_observable	The collector has no trustworthy oracle for the capability.
not_claimed	The target commit does not claim the future capability.

Read status together with domain: runtime_contract, harness_contract, provider_contract, environment_contract, budget_contract, or model_behavior. A model not choosing an optional tool is not automatically a product defect; deterministic identity replacement or unsafe replay is.

Bug triage and replay

1. Open bugs/<bug-id>.md; it contains the campaign/run ID, attempt, profile, runner, product/EVAL commits, reproduction argv, assessment path, log path, code, severity, domain, and evidence references.
2. Open the run's mechanism-assessment.json and check gateEligible, observationComplete, the failed check, and its violation domain.
3. Resolve evidence.eventLogPath, then search the referenced seq values in that JSONL.
4. Follow rootSessionId, child Session IDs, task IDs, provider message IDs, and scenarioPhaseId into the related sessions/*.jsonl stream.
5. Compare those events with snapshot.workspace.git, the top-level provenance file, and scenario-phases.json when present.
6. For durable-control-plane findings, locate the snapshot event and compare taskRun, admission, runEvent, and outbox rows with Session delivery events:

EVENT_LOG="$CAMPAIGN/$(jq -r '.evidence.eventLogPath' \
  "$RUN_DIR/mechanism-assessment.json")"
rg '"type":"harness.durable.snapshot"' "$EVENT_LOG"

7. Replay after changing only the evaluator:

bun run replay \
  --run-dir "$RUN_DIR" \
  --deepswe-dir ./data/deepswe \
  --gate

Replay reads runnerMode from schema 1.1 assessments, refuses a conflicting --runner, and never starts DeepAgentCode or calls the provider. Legacy assessments without runnerMode require the original mode explicitly. Output defaults to ./replays/<run-id>/...json, outside the immutable campaign; use --output to select another non-campaign location. Replay applies the same required-contract and reportable-violation gate semantics as a run assessment.

Useful finding codes include:

• SUBAGENT_NON_RETRYABLE_FAILURE_REPLAYED
• SUBAGENT_CHILD_IDENTITY_REPLACED
• SUBAGENT_FAILURE_CONTENT_LOST
• CODE_INTEL_LOOKUP_MISSED
• SESSION_TASK_TIMEOUT
• REQUIRED_CONTRACT_HAS_NO_ORACLE

Keep the captured JSONL unchanged while comparing the original and replayed assessments. A useful bug handoff includes the bug ID, product commit, EVAL commit, task ID, provider fingerprint, reproduction command, and the smallest relevant evidence excerpt.

Quality gate and exit codes

The default gate is a product-quality gate, not a benchmark-score gate. It fails on required-contract non-pass results, reportable product/harness violations at the configured severity, or too few repetitions for a gate-eligible profile. An empty or invalid task selection is an input error and always exits 1, even with --no-gate:

• exit 0: the process completed and the enabled gate did not fail, or the gate was disabled;
• exit 1: CLI, config, preflight, task selection, or a fatal execution error prevented campaign completion;
• exit 2: the enabled quality gate failed;
• exit 130/143: the campaign finalized after SIGINT/SIGTERM.

The optional verifier is disabled by default. Add --verifier to a campaign command when that secondary signal is needed.

A verifier dependency or environment failure is auxiliary evidence and does not prove an agent mechanism failure.

Security and reproducibility

For comparable runs, keep product/EVAL commits, task ID and baseline, provider/model, profile, repetitions, timeout, feature flags, operating system, and Bun version fixed. Reports can contain prompts, model output, tool arguments, source excerpts, and local paths; review and redact the evidence package according to the receiving environment's data policy before sharing it.

Data provenance and license

DeepAgent EVAL source code is licensed under the GNU Affero General Public License v3.0 or later, matching DeepAgentCode. The bundled datasets and upstream source archives retain the separate licenses described below.

This repository redistributes task data and evaluation materials from Datacurve AI’s DeepSWE snapshot e016041a6ccf8da29906afc9a3f5a8df940a1f78.

DeepSWE-authored task specifications, evaluation materials, verifier metadata, and curation work are provided under Apache License 2.0. See PROVENANCE.md and DATASET_NOTICE.md for attribution, upstream licenses, and local cleaning/redistribution changes.

Apache-2.0 for DeepSWE does not relicense the upstream repositories referenced by individual tasks. Those source assets remain subject to their own licenses and attribution notices. DeepAgent EVAL is an independent project and is not an official Datacurve AI, DeepSWE, Pier, or Harbor implementation.

Development checks

bun test
bun run typecheck
bun run assets:verify --concurrency 3