Racket-orchestrated ProgramBench harness. An OCaml toolkit turns a coding agent into a programming-language engineer: the scaffold provides ~5K lines of lexer, parser, value, environment, and runtime infrastructure so the agent only writes the core evaluator.
10 PLT tasks from ProgramBench, scored by programbench eval (% of tests passed).
| Task | Sonnet 4.6 (MSWEA) | veldt (Sonnet 4) | zagent (Sonnet 4) |
|---|---|---|---|
| jqlang/jq | 1.0% | 55.6% | 9.6% |
| lua/lua | 34.7% | 46.7% | 12.1% |
| luajit/luajit | 71.5% | 44.0% | 14.8% |
| tree-sitter/tree-sitter | 37.2% | 35.9% | 26.5% |
| paradigmxyz/solar | 42.9% | 33.7% | 24.9% |
| parcel-bundler/lightningcss | 49.9% | 27.1% | 9.6% |
| tinycc/tinycc | 9.3% | 4.9% | 3.4% |
| typst/typst | 0.0% | 8.0% | 4.6% |
| bellard/quickjs | 0.0% | 0.8% | 0.7% |
| php/php-src | 0.0% | 0.6% | 2.3% |
| Average | 24.5% | 25.7% | 10.8% |
- Sonnet 4.6 — ProgramBench leaderboard baseline, mini-SWE-agent, no scaffold
- veldt — older/cheaper Sonnet 4, mini-SWE-agent + OCaml scaffold (~$3/task)
- zagent — Sonnet 4, zagent captain, no scaffold (~$5/task)
veldt matches Sonnet 4.6 at 9× lower cost. The scaffold is worth a full model generation on jq (+54.6pp) and typst (+8.0pp).
Each ProgramBench task provides a compiled binary. The agent must reverse-engineer it and produce an equivalent executable. veldt splits this into two phases:
- Recon — probe the reference binary (help text, error messages, edge cases) and write a report
- Impl — read the report, fill in the OCaml scaffold, build and test against the reference binary
Both agents run on Vers VMs restored from a golden snapshot that has OCaml 5.2, Docker, and the scaffold pre-installed.
HOST (Racket orchestrator)
│
├── for each task:
│ ├── restore VM from golden commit
│ ├── pull ProgramBench Docker image
│ ├── lock down network (LLM API + DNS only)
│ ├── inject OCaml scaffold into container
│ ├── run recon agent → recon.md
│ ├── run impl agent → modified scaffold
│ ├── dune build → executable
│ ├── collect submission.tar.gz
│ └── delete VM
│
└── output: ~/.veldt/runs/<timestamp>/
The scaffold lives in scaffold/ocaml/ and provides the boilerplate
that every language implementation needs. The agent only writes the
evaluator in bin/main.ml.
scaffold/ocaml/
├── bin/
│ ├── main.ml # ← agent writes this (interpreter/compiler core)
│ └── parser.ml # generic parser entry point
├── lib/
│ ├── lexer.ml # configurable lexer (360 lines)
│ ├── token.ml # token type definitions
│ ├── pratt.ml # Pratt parser combinator (231 lines)
│ ├── value.ml # runtime values: int, float, string, array, object, …
│ ├── env.ml # scoped environment (let bindings, closures)
│ ├── interp.ml # interpreter driver / eval loop skeleton
│ ├── pos.ml # source positions for error messages
│ ├── cli.ml # CLI argument parsing (297 lines)
│ ├── json_print.ml # JSON output (matching jq/node formatting)
│ ├── json_seq.ml # JSON sequence parsing
│ ├── number_format.ml # numeric formatting (449 lines — IEEE 754 edge cases)
│ ├── format_strings.ml # printf-style format strings (299 lines)
│ ├── pretty.ml # pretty-printer for ASTs and values
│ ├── regex_match.ml # POSIX regex matching (372 lines)
│ ├── unicode.ml # unicode utilities (359 lines)
│ ├── datetime.ml # date/time formatting (425 lines)
│ ├── exit_protocol.ml # exit code conventions (292 lines)
│ ├── stream_decompose.ml # stream processing utilities (326 lines)
│ └── lang.ml # re-exports everything as `open Lang`
├── templates/
│ └── main_interp.ml # starter template for interpreters
├── compile.sh # dune build → ./executable
└── dune-project
4,839 lines of infrastructure the agent doesn't have to write.
# Prerequisites
brew install minimal-racket
raco pkg install net-lib http-easy
# Run all 10 PLT tasks
racket veldt.rkt run-ocaml --golden a12a47c8-6983-4f46-8ce5-b745100953b5
# Run a single task
racket veldt.rkt run-ocaml --golden a12a47c8 --tasks "jqlang__jq.b33a763"
# List PLT tasks
racket veldt.rkt list-plt
# Check running VMs / clean up
racket veldt.rkt status
racket veldt.rkt cleanupCommit: a12a47c8-6983-4f46-8ce5-b745100953b5 (tag: veldt-ocaml-golden)
| Component | Version | Purpose |
|---|---|---|
| OCaml | 5.2.1 | Solution language (pattern matching, algebraic types) |
| opam | 2.3.0 | Package manager (pre-installed, no internet needed) |
| dune | 3.18.2 | Build system |
| Docker | 29.4.2 | ProgramBench task containers |
| mini-swe-agent | 2.2.8 | Coding agent (LLM tool-calling loop) |
| zagent | latest | Alternative agent harness (Zig-based) |
| pb-lockdown.sh | — | iptables: blocks internet, allows LLM API |
After pulling the Docker image, pb-lockdown.sh blocks all outbound except:
- LLM APIs (api.anthropic.com, api.openai.com)
- DNS (port 53)
- Docker bridge (container management)
- Established connections (keeps SSH alive)
No PyPI, crates.io, npm, GitHub, or apt-get.
~/.veldt/runs/<timestamp>/
├── results.txt # PASS/FAIL per task
├── <instance-id>/
│ ├── recon.md # reconnaissance report
│ ├── submission/
│ │ ├── bin/main.ml # agent's implementation
│ │ ├── lib/ # scaffold libraries
│ │ ├── compile.sh
│ │ └── prebuilt_binary # (if VM-built binary included)
│ ├── submission.tar.gz # ready for `programbench eval`
│ └── agents/
│ ├── recon/agent.log
│ └── impl/agent.log
# Install programbench
pip install programbench
# Evaluate a run
programbench eval ~/.veldt/runs/<timestamp>/ -w 3 --docker-cpus 4Note: ProgramBench eval Docker images are linux/amd64 only.
On arm64 Macs, some tasks may fail with results_read_failed due to
QEMU emulation issues.
veldt/
├── veldt.rkt # CLI entry point
├── lib/
│ ├── orchestrator-ocaml.rkt # OCaml run loop (VM lifecycle, agent coordination)
│ ├── prompts-ocaml.rkt # recon + impl prompts, system prompt, file-writing rules
│ ├── plt-classify.rkt # classify tasks: interpreter, compiler, parser-gen, DSL
│ ├── tasks.rkt # ProgramBench task manifest
│ ├── vers.rkt # Vers CLI client (VM create/delete/exec/copy)
│ ├── run-log.rkt # timestamped logging
│ ├── orchestrator.rkt # original Go pipeline (deprecated)
│ ├── prompts.rkt # original Go prompts (deprecated)
│ └── gitea.rkt # Gitea management (deprecated)
├── scaffold/ocaml/ # OCaml toolkit (injected into every task container)
├── config/
│ ├── plt-tasks.txt # 10 PLT task instance IDs
│ ├── tasks.json # full 200-task manifest
│ └── golden.txt # golden VM commit ID
└── scripts/
├── build-golden-vm-ocaml.sh # golden VM build script
├── build-golden-vm.sh # original Go golden (deprecated)
└── fetch-tasks.py # fetch task metadata from HuggingFace