agent-audit

中文版

agent-audit is an agent-first smart contract audit system. It splits an EVM contract audit into four layers:

1. The Rust harness CLI under src/ creates runs, fetches source code, prepares tool workspaces, aggregates audit materials, and archives run data.
2. The containerized audit environment under docker/ packages the CLI, Codex SDK, Codex CLI, and common contract security tools into a single-audit runner.
3. The container-level parallel deployment system under k3s/ uses a Redis queue and Kubernetes Jobs to run multiple runners in parallel.
4. The evaluation system under eval_docker/ and EVMbench/ reuses the audit environment, but adapts it for EVMbench Detect tasks against local benchmark repositories.

This root README explains the overall architecture and workflows. Detailed build, deployment, and runtime notes live in each subdirectory README.

What the system finds

Scanning 8,255 Ethereum mainnet contracts end to end, the system confirmed 6,466 vulnerabilities and reproduced working exploits on 432 contracts — measured balance effects on a forked mainnet, not severity estimates. 96% of those targets had no verified source code.

flowchart LR
    A["8,255 mainnet<br/>contracts scanned"] --> B["3,577 with<br/>confirmed findings"]
    B --> C["3,245 with a<br/>high-severity finding"]
    B --> D["432 with an exploit<br/>reproduced on a fork"]
    D --> E["Measured impact<br/>on balances"]

    style A fill:#e8f0fe,stroke:#4285f4
    style D fill:#e6f4ea,stroke:#34a853
    style E fill:#e6f4ea,stroke:#34a853

Loading

Result	Count
Contracts scanned	8,255
Confirmed vulnerabilities	6,466
Contracts with at least one confirmed vulnerability	3,577 (43.7%)
Contracts with a high-severity vulnerability	3,245 (39.6%)
Contracts with an exploit reproduced on a mainnet fork	432 (5.3%)
Priority vulnerability classes reached	10 / 10

What that looks like in practice: an unauthenticated flashloan() that drains a contract's entire ETH balance to a hardcoded address; an inverted ownership check that locks out the owner and admits everyone else; a permissionless oracle refresh that moves a price 6.77× in a single swap; 6,000,000 USDT permanently locked by an unhandled ERC-20 return value.

→ Full report: Scanning 8,255 On-Chain Contracts

System Structure

Directory	Role	Main outputs
src/	Local harness CLI	agent-audit binary, runs/<run_id>/ workspaces, JSON step output
docker/	Production audit runner image	agent-audit:<tag>, one container run per complete prompt
k3s/	Batch parallel execution layer	Redis Stream, dispatcher, one Kubernetes Job per task
eval_docker/	EVMbench-specific evaluation image	agent-audit-eval:<tag>, writes submission/audit.md
EVMbench/	EVMbench overlay	agent-audit-codex adapter and Dockerfile patch script
dispatcher/	k3s dispatcher program	agent-audit-dispatcher image, converts Redis messages into Jobs

Overall data flow:

Production single audit:
  prompt -> docker runner -> Codex agent -> agent-audit CLI -> runs/<run_id>/ -> sync-run -> Mongo

Production batch audit:
  address list -> Redis Stream -> dispatcher -> Kubernetes Jobs -> docker runner -> Mongo

EVMbench Detect:
  EVMbench audit repo -> eval runner -> Codex agent + local tools -> submission/audit.md

1. src/: Harness CLI

src/ is the data preparation and archival core of the audit system. It does not make audit decisions for the agent. Instead, it prepares repeatable materials and records each step as structured JSON.

CLI entrypoint:

agent-audit <subcommand>

For local development:

cargo run --bin agent-audit -- <subcommand>

Current commands:

agent-audit init-run --chain <chain> --address <address>
agent-audit fetch-source --run-id <run_id>
agent-audit run-dependency --run-id <run_id>
agent-audit prepare-slither --run-id <run_id>
agent-audit prepare-tooling --run-id <run_id>
agent-audit aggregate-materials --run-id <run_id>
agent-audit sync-run --run-id <run_id>

Core responsibilities:

• Create the runs/<run_id>/ workspace plus input/request.json and input/run_meta.json, including run creation time and the current block number when RPC is configured.
• Fetch target contract source code through settings such as AGENT_AUDIT_SOURCE_API_BASE.
• Run dependency discovery and dependency analysis, writing artifacts/dependency_*.json.
• Prepare Slither, Foundry, and Echidna tool workspaces and build manifests.
• Aggregate audit materials into reports/materials_manifest.json.
• Sync run metadata and files to MongoDB through sync-run.

Common configuration comes from the repository-local .env:

AGENT_AUDIT_DEFAULT_CHAIN
AGENT_AUDIT_SOURCE_API_BASE
AGENT_AUDIT_SOURCE_API_KEY
AGENT_AUDIT_SOURCE_HEADERS_JSON
AGENT_AUDIT_RPC_URL
AGENT_AUDIT_MONGO_URI
AGENT_AUDIT_MONGO_DB
AGENT_AUDIT_RUNS_DIR

init-run executes the full preparation flow: source fetch, dependency analysis, tool workspace preparation, and material aggregation. The other subcommands are used to rerun or supplement individual steps for an existing run.

2. docker/: Containerized Audit Environment

docker/ packages the production audit environment into a runner image. The image contains:

• agent-audit CLI
• Node.js, @openai/codex-sdk, codex
• Slither, solc-select
• Foundry: forge, cast, anvil
• Echidna
• Heimdall for bytecode-oriented review of unverified contracts
• Container-specific .codex configuration and audit skills

Build:

./docker/build.sh

Run a single audit:

./docker/run.sh --prompt "Check AGENTS.md and audit 0x0000000000000000000000000000000000000000 on eth."

Direct Docker invocation:

docker run --rm \
  -v "$(pwd)/.env:/opt/agent-audit/.env:ro" \
  -e FULL_PROMPT="Check AGENTS.md and audit 0x0000000000000000000000000000000000000000 on eth." \
  agent-audit:0.1

The production runner accepts exactly one complete prompt:

• FULL_PROMPT environment variable
• or --prompt "..." argument

The container does not assemble business fields such as address, chain, or instructions. The caller must include the target, chain, audit requirements, and output requirements in the prompt. After startup, the Codex agent reads AGENTS.md, calls the agent-audit CLI and security tools as needed, then completes the audit and archival flow.

When verified source is unavailable, the CLI records review targets in runs/<run_id>/artifacts/bytecode_targets.json and runtime bytecode under runs/<run_id>/artifacts/bytecode/. prepare-tooling still prepares bytecode-only Foundry and Echidna workspaces for fork-backed harnesses, and creates managed Heimdall command workspaces under runs/<run_id>/artifacts/heimdall/ so command text, stdout, stderr, exit code, failure text, and output directories are saved deterministically; material aggregation includes those paths when present.

See docker/README.md for details.

3. k3s/: Container-Level Parallel Deployment

k3s/ is the single-server Kubernetes deployment for parallelizing multiple single-audit runners. It does not change the audit logic. It only turns queued tasks into one-shot Jobs.

Topology:

Redis Stream
  -> agent-audit-dispatcher Deployment
  -> one Kubernetes Job per task
  -> agent-audit runner container
  -> sync-run / Mongo archive

Key components:

• k3s/redis.yaml: input queue.
• dispatcher/: Redis Stream to Kubernetes Job bridge.
• k3s/dispatcher-*.yaml: dispatcher Deployment, RBAC, configuration, and Secret.
• k3s/runner-configmap.yaml: runner Job template, including image, resources, TTL, and runs/ volume.
• k3s/runner-secret.yaml: runner API key, source, RPC, and Mongo configuration.

Deployment entrypoint:

k3s kubectl apply -f k3s/namespace.yaml
k3s kubectl apply -f k3s/runner-secret.yaml
k3s kubectl apply -f k3s/dispatcher-secret.yaml
k3s kubectl apply -k k3s/

Submit a single task:

k3s kubectl -n agent-audit exec deploy/agent-audit-redis -- \
  redis-cli XADD agent-audit:tasks '*' \
    task_id audit-20260505-001 \
    full_prompt 'Check AGENTS.md and audit 0x0000000000000000000000000000000000000000 on eth.' \
    image ghcr.io/lunaticabs/agent-audit:v1.0.0

Submit a batch from an address list:

k3s kubectl -n agent-audit port-forward svc/agent-audit-redis 6380:6379
python3 scripts/enqueue_redis.py \
  --chain eth \
  --address-file scripts/addresses/addrs.txt \
  --host 127.0.0.1 \
  --port 6380 \
  --image ghcr.io/lunaticabs/agent-audit:v1.0.0

Use Kubernetes Job and Pod state as the runtime source of truth:

k3s kubectl -n agent-audit get jobs,pods -w

Redis is the input queue, not final state storage. Audit evidence and run files are written to MongoDB through the runner's internal sync-run flow.

See k3s/README.md for details.

4. eval_docker/ and EVMbench/: Evaluation System

The evaluation system targets EVMbench Detect. It reuses the Codex and security tooling from the production image, but switches the data source from "chain address plus source service" to "local audit repository mounted by EVMbench".

eval_docker/ provides the evaluation runner:

./eval_docker/build.sh
APIAPI_API_KEY=... ./eval_docker/run.sh --audit-dir /path/to/audit

Evaluation runner rules:

• The Codex working directory is the benchmark-provided audit repo.
• Final output must be written to submission/audit.md.
• The production address/chain data plane is intentionally disabled.
• Production pipeline commands such as agent-audit init-run, fetch-source, and sync-run are blocked by the evaluation guard.
• Temporary notes, logs, and tool outputs may be created under the benchmark-local repository, but the grader reads only submission/audit.md.

EVMbench/ provides an overlay script that injects the agent-audit-codex adapter into an EVMbench checkout:

./EVMbench/overlay.sh --evmbench-dir /path/to/evmbench

Then build and run Detect from the EVMbench checkout:

cd /path/to/evmbench
uv sync
docker build -f ploit/Dockerfile -t ploit-builder:latest --target ploit-builder .
uv run docker_build.py --split debug

APIAPI_API_KEY=... \
uv run python -m evmbench.nano.entrypoint \
  evmbench.audit_split=debug \
  evmbench.mode=detect \
  evmbench.apply_gold_solution=False \
  evmbench.log_to_run_dir=True \
  evmbench.solver=evmbench.nano.solver.EVMbenchSolver \
  evmbench.solver.agent_id=agent-audit-codex \
  runner.concurrency=1

See eval_docker/README.md and EVMbench/README.md for details.

Workflows

A. Local Development and Material Preparation

cargo build
  -> agent-audit init-run
  -> runs/<run_id>/input
  -> runs/<run_id>/artifacts
  -> runs/<run_id>/slither_project / foundry_project / echidna_project
  -> runs/<run_id>/reports/materials_manifest.json

This flow validates the CLI, source provider, dependency analysis, and material aggregation. The agent can then continue from the prepared run workspace and call Slither, Foundry, Echidna, cast/anvil, and other tools as needed.

B. Single Production Audit

docker runner
  -> Codex reads FULL_PROMPT and AGENTS.md
  -> Codex calls agent-audit CLI to prepare evidence
  -> Codex runs security tools as needed
  -> Codex writes report / findings
  -> agent-audit sync-run archives run files to Mongo

This flow is suitable for manual audits, prompt debugging, image validation, and reproducing a single target.

C. Batch Parallel Audit

address file or external scheduler
  -> Redis Stream message(task_id, full_prompt, image)
  -> dispatcher
  -> Kubernetes Job
  -> same docker runner as single audit
  -> Mongo archive

This flow is suitable for large address batches. Effective parallelism is determined mainly by k3s scheduling capacity, runner Job CPU and memory requests/limits, and node resources.

D. EVMbench Detect Evaluation

EVMbench task
  -> mounted local audit repository
  -> eval runner
  -> Codex uses local tools only
  -> submission/audit.md

This flow is for benchmark evaluation. It does not access the production source/RPC/Mongo data plane and does not use the address/chain pipeline to generate audit materials.

Development Checks

Recommended command after code changes:

cargo xtask check

Equivalent command:

cargo run -p xtask -- check

Fixed check order:

cargo fmt --all --check
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo test --workspace --all-features

For documentation or deployment changes, at least confirm that the README and script commands still match the subdirectory documentation.