Continual learning infra for self-improving agents

English | 中文

Reef is the first open-source infrastructure for continual self-improving agents. It connects agent inference, feedback, learning, and versioned delivery. Use it to train model weights with Slime and SGLang, or improve an agent's harness, including its prompts, rules, and skills.

Get started | Roadmap | Launch post | Join Discord

When to use Reef

Use Reef when you want your agent to keep improving simply by learning from how you interact with your agent.

Your goal	Learning path	What you need
Keep getting stronger model designed for you	Model weight training	A trainable model, a supported GPU stack, and feedback your recipe can use
Get your harness to self-improve	Harness optimization	A model endpoint, representative tasks, and an evaluator; no local training GPUs
Scientific discoveries	Test-time training	An execution environment, a correctness checker, and a measurable objective

How Reef fits your stack

Ability	Inference engine (vLLM, SGLang, …)	RL training framework (Slime, veRL, AReaL, …)	Reef
Serves live traffic	✅	❌	✅
Trains weights	❌	✅	✅
Version management	❌	❌	✅
Stays live through updates	❌	❌	✅
Evolves beyond weights (skills, harness)	❌	❌	✅

How it works

Reef processes each learning cycle in four steps. The table also shows which modules implement each step.

Step	What happens	Where it lives
1 · Serve	Serve agent requests and record interactions.	service/ — agent requests and interaction records runtime/ — inference and artifact updates
2 · Observe	Match feedback to recorded interactions.	records.py — stored interactions and feedback train/processors/ — feedback matching and eligibility
3 · Grow	Produce an update from eligible records.	recipe/ — recipe integration train/ — batches and update jobs
4 · Commit	Apply the configured selection policy and publish accepted updates.	train/evaluation/ — candidate evaluation artifact/ — version history surface/ — artifact delivery

Installation

💡 Note

Reef's artifact and checkpoint functionality requires the git-lfs system package. Reef initializes Git LFS locally for its artifact repositories.

We recommend uv for managing packages, and the commands below use it.

From PyPI

uv venv && source .venv/bin/activate
uv pip install reef-infra
python3 -c "import reef; print(reef.__version__)"

From source

git lfs install
git clone https://github.com/Human-Agent-Society/reef.git
cd reef
uv venv && source .venv/bin/activate
uv pip install -e .
python3 -c "import reef; print(reef.__version__)"

Use the source checkout for development and for the training examples below.

Using Reef

Reef supports two learning surfaces: model weights and agent harnesses. The deployment's recipe determines which surface its scenarios update.

Weight-training deployment

Start the deployment

The following example starts the SAO (arXiv:2607.07508) example deployment. Run it from a Reef checkout in an environment that satisfies the GPU requirements in Evolve your model.

uv pip install -e ".[slime]" && uv pip install --no-deps --group runtime

export MODEL_PATH="Qwen/Qwen2.5-1.5B-Instruct"
export REEF_TOKEN="reef-local"

reef serve -c recipes/sao/examples/sao/serve.yaml \
  --reef.model_path "$MODEL_PATH" \
  --reef.port "8900"

curl -f http://127.0.0.1:8900/healthz          # ready to serve

Send an inference request and report feedback

Send inference requests through Reef and report a score for each response. The SAO recipe uses each eligible scored rollout to run a training step.

Reef's inference endpoint is OpenAI- and Anthropic-compatible: /v1/chat/completions and /v1/messages take the provider's own request body. A request includes the x-reef-scenario header; a new name creates a scenario using the deployment's configured recipe. Requests do not select recipes.

The response body uses the provider's OpenAI-compatible format. Reef adds the x-reef-agent-record-id response header. Its value is the receipt that a later report uses to identify this interaction. A report can contain a numeric score, textual or structured feedback, and the receipts it evaluates. This example reports both a score and a short explanation.

import os
import httpx

reef = httpx.Client(
    base_url="http://127.0.0.1:8900",
    headers={"Authorization": f"Bearer {os.environ['REEF_TOKEN']}", "x-reef-scenario": "hello-reef"},
    timeout=300,
)

# Send a provider-compatible inference request
response = reef.post(
    "/v1/chat/completions",
    json={
        "model": os.environ["MODEL_PATH"],
        "messages": [{"role": "user", "content": "Return exactly: reef is ready"}],
    },
)

response.raise_for_status()
receipt = response.headers["x-reef-agent-record-id"]
answer = response.json()["choices"][0]["message"]["content"]

# Sending report about the inference
matched = answer.strip() == "reef is ready"

reef.post(
    "/reef/report",
    json={"score": float(matched), "feedback": "matched" if matched else "wrong answer", "references": [receipt]},
).raise_for_status()

feedback carries the richer signal, plain text or a structured object, for recipes that read more than a scalar. The endpoint will validate the report schema (reef/core/reports/).

Watch it learn and grow

Once the recipe has enough feedback, it runs a training step and synchronizes the updated weights to the serving runtime. Later inference requests use the current version without restarting Reef.

Harness-evolving deployment

Improve harness skills using a model API instead of GPUs. Set model.path in deployment.yaml to your model name, then run from your Reef checkout and activated Python environment:

export REEF_UPSTREAM_URL="https://api.openai.com"  # No /v1 suffix
export REEF_UPSTREAM_API_KEY="your-openai-api-key"
reef serve -c tutorials/evolve-your-harness/configs/deployment.yaml

For another provider, use its base URL, model name, and API key. This config deploys Reef on 8901 with reef-local as its access token.

In another terminal, install the harness and run a task:

export REEF_TOKEN="reef-local"   # the script writes it into the installed harness's
                                 # model binding; keep it exported for `report`
curl -fsS -H "Authorization: Bearer $REEF_TOKEN" \
  'http://localhost:8901/reef/harness/install?adapter=pi' | bash
reef-pi -p "fix the failing test in auth.py"

# After running your tests, report the actual result:
reef-pi report --score 0 --feedback "missed the empty-token case"

Failed reports trigger a candidate skill update. Reef evaluates it against the current harness on the tutorial's three coding tasks and publishes it only if it wins. See the tutorial to customize the tasks and evaluation.

Recipes and examples

Choose a recipe based on your workload's feedback and the artifact you want to update. The implementations live in this repository's recipes/ cookbook, are selected by dotted class reference, and do not ship in the Reef wheel.

Workload	Recipe guide	Updated artifact	Examples and results
A stream of tasks scored by tests or a verifier	SAO	Model weights	Example · Results
Agent traffic with useful next-state signals and no explicit reports	OpenClaw-RL	Model weights	Example
Repeated, scored attempts at one problem	TTT-Discover	Model weights	Example · Results
Scored code search with a trainable guidance model and a frozen executor	Guidance-TTT / TTTD	Guidance-model weights	Example · Results
Agent feedback used to evolve its skill pool	SkillClaw	Harness skills; no training GPUs	Example
Scores and transcripts used to improve prompts and instructions	GEPA	Harness; fixed model weights	Example and results

For a small walkthrough of feedback, candidate edits, and publication, start with the coding harness tutorial. Each result page documents its task, evaluation setup, measurements, and limitations.

Architecture

sequenceDiagram
    accTitle: How Reef serves, records, trains, evaluates, and publishes
    autonumber
    participant H as Harness
    participant S as Scenario
    participant I as Inference
    participant T as Trainer
    participant G as Training*
    participant E as Artifact evaluation

    opt Harness recipe: pull the served tree
      H->>S: GET /reef/harness for scenario
      S-->>H: Harness tree and release
      Note over H: Agent runs on that tree
    end
    Note over H,I: Serve and record each request
    H->>S: Inference request for scenario
    S->>S: Freeze current release
    S->>I: Provider-native request
    I-->>S: Provider response
    S->>S: Validate frozen release and store record
    S-->>H: Response and receipt
    H->>S: Feedback quotes the receipt
    S->>T: Eligible record
    opt Processor has a batch
      Note over S,E: Produce, evaluate, and select a candidate
      T->>G: Prepared step
      G-->>T: Candidate artifact ready
      T->>E: evaluate(candidate)
      E-->>T: Evaluation result
      T->>E: decide(candidate, result)
      E-->>T: Select or reject
      alt Candidate selected
        T->>S: Commit new release
      else Candidate rejected
        Note over S,I: Previous release keeps serving
      end
    end

Loading

See the architecture guide for the request path, scenarios, and release lifecycle.

Learn more

The documentation is organized in the following order:

• Quickstart: install Reef, connect a client, and inspect the version history
• HTTP API: use the HTTP API and report feedback
• Write a recipe: configure how Reef processes data and produces updates
• Evolve your harness: evolve a harness instead of model weights
• Evolve your model: configure and operate a training deployment
• Recipes: additional references on the cookbook implementations in this repository
• Architecture: Overall architecture of Reef
• Glossary: Explanation of the terminologies used

Community & Contributing

Working on continual self-improving agent?

• Join Discord to share your recipes, ask implementation questions, and discuss new features.
• Join the GitHub Discussions to ask questions, share ideas, and connect with the community.
• Start contributing with the contribution guide.
• Propose designs through an RFC issue.
• Report suspected vulnerabilities privately by following the security policy.

If Reef looks useful to you, please give it a ⭐ — it helps the community to discover and contribute to the project.

The Team

Reef brings together people exploring how agents can learn from experience and improve over time. The people below help turn that idea into working infrastructure.

This list is non-exhaustive, with team members listed alphabetically by last name:

Wenhao Chai, Shuangrui Ding, Hao He, Haoze He, Chonghe Jiang, Nan Jiang, Xuan Jiang, Xiaochen Li, Paul Liang, Bo Liu, Boyuan Long, Qiuyang Mang, Zhenting Qi, Ao Qu, Mingruo Qu, Zhaokai Wang, Xuezhi Yan, Hanfei Yu, Haofei Yu, Simon Yu, Han Zheng, Kaichen Zhou, Zijian Zhou, Jiacheng Zhu, Dingyi Zhuang.

Star History

Acknowledgements

We are particularly grateful to these projects which power important parts of Reef:

• SGLang — high-performance inference
• slime — model weight training
• cordis — harness evolution