Scaffolding Agent

A self-scaffolding AI agent that plans and executes tasks using LLM function calling — and can synthesise its own tools at runtime when an existing tool can't satisfy a task. Built on a small, composable runtime with provider-agnostic LLM access (OpenAI, Anthropic, and any OpenAI-compatible endpoint), sandboxed execution, shell allowlisting, retry-with-backoff, structured JSON logging, and Pydantic-validated configuration.

The design goal is a minimal, trustworthy agent core: plans are data (JSON steps), not executable code, so every run is observable, replayable, and bounded by explicit limits. New tools are generated as Python functions and registered with the tool registry only after passing validation.

---

Features

• Self-scaffolding — the agent generates new tools on demand when an unknown tool is referenced in a plan.
• Provider-agnostic — OpenAI, Anthropic, and OpenAI-compatible providers behind one adapter interface.
• Plans as data — execution is JSON step-by-step, interpreted by the runtime; no eval of LLM-generated code.
• Sandboxed Python — python_eval runs under RestrictedPython with no imports, filesystem, or network access and a 5s timeout.
• Shell allowlist — only explicitly allowlisted commands execute; everything else is rejected at the policy layer.
• Resilient LLM calls — exponential backoff (1s, 2s, 4s…) on transient failures, capped at 3 retries.
• Structured logging — JSON to stdout with task_id, step, duration, outcome; API keys auto-redacted.
• Validated configuration — scaffolder.yml and environment variables are checked by Pydantic schemas.

Quick Start

pip install -e .

export OPENAI_API_KEY=sk-...      # or ANTHROPIC_API_KEY=sk-ant-...

python -m scaffolder run "your task description here"
python -m scaffolder run "task" --llm openai:gpt-4o
python -m scaffolder run "task" --config scaffolder.yml

Architecture

A three-layer core keeps planning, execution, and tool synthesis separate. Each layer talks to the next through a narrow interface, so each is independently testable.

flowchart LR
    Task([Task]) --> Planner
    Planner[Planner<br/>LLM → validated JSON plan] --> Runtime
    Runtime[Runtime<br/>interprets steps<br/>resolves vars<br/>handles errors]
    Runtime --> Registry[Tool Registry<br/>shell · files · web · llm · eval]
    Runtime --> Scaffolder[Scaffolder<br/>synthesises new tools<br/>on demand]
    Scaffolder -- "registers new tool" --> Registry
    Registry --> Output([Result])

Loading

• Planner — calls the LLM with the current tool registry and the task. The LLM returns a JSON plan; the planner validates structure (required fields, tool names exist) before handing it to the runtime.
• Runtime — walks each step of the plan, resolves {{variables}} between steps, dispatches tool calls, and surfaces errors. A step budget (max_steps) bounds every run.
• Scaffolder — when a plan references a tool the registry has never seen, the scaffolder asks the LLM to generate a Python implementation, validates it, and registers it. Capped at max_tool_generations per run.

Plans are data, not executable code. The runtime interprets each step — this keeps every run replayable from the JSON and stops LLM output from running arbitrary code.

Project Structure

scaffolder/
├── __main__.py          # python -m scaffolder entry point
├── cli.py               # argparse + runtime wiring
├── config.py            # Pydantic config schema + loader
├── planner.py           # LLM → JSON plan + validation
├── runtime.py           # step interpreter + variable resolution
├── scaffolder.py        # on-demand tool synthesis
├── tool_registry.py     # tool registration + lookup
├── security.py          # RestrictedPython + shell policy
├── logging.py           # structlog JSON output + secret redaction
├── llm_adapter.py       # OpenAI / Anthropic / OpenAI-compatible
└── tools/
    ├── shell.py
    ├── file_ops.py
    ├── web.py
    ├── llm_call.py
    └── python_eval.py
tests/                   # 51 tests across 19 files

Built-in Tools

Tool	Purpose
shell	Run allowlisted shell commands (echo, ls, cat, pwd, date)
read_file / write_file	Filesystem operations
web_fetch / web_search	HTTP fetch + web search
llm_call	Invoke the LLM mid-execution for sub-tasks
python_eval	Evaluate Python in a RestrictedPython sandbox
plan	Trigger replanning with the current tool registry

Security & Reliability

The agent includes hardening intended for internal, trusted use. It is not production-grade isolation — there is no container, and plan execution is single-process — but every trust boundary has an explicit guard.

• RestrictedPython sandbox for python_eval — no imports, no filesystem, no network; 5s timeout (configurable via python_timeout).
• Shell allowlist — only listed commands execute; configurable in scaffolder.yml:

  shell_allowlist: ["echo", "ls", "cat", "pwd", "date", "grep"]

• LLM retry — exponential backoff (1s, 2s, 4s…) up to max_retries (default 3); base_delay configurable.
• Structured logging — JSON to stdout with task_id, step, duration, outcome; API keys auto-redacted.
• Config validation — Pydantic schemas validate scaffolder.yml and environment variables before any run.

Testing

The project ships with 51 tests across 19 files covering the planner, runtime, scaffolder, tool registry, security policy, config, logging, LLM adapter, and integration runs against sandboxed and allowlisted paths.

pytest -q

Tests are dependency-light and use monkeypatch to stub LLM calls, so they run without an API key.

Configuration

llm:
  provider: openai
  model: gpt-4o
  max_tokens: 4096
max_steps: 25
max_tool_generations: 5
shell_allowlist:
  - echo
  - ls
  - cat
  - pwd
  - date
python_timeout: 5

Limits

• 25 max steps per plan
• 5 max tool generations per run
• 5s python_eval execution timeout
• Single-process; no container isolation

Author

David Escoto — built as a study in minimal, observable agent architecture.

License

MIT