Skip to content

PR4: LLM roles + multi-round loop + history injection + cost guard - #6

Merged
Protocol-zero-0 merged 3 commits into
mainfrom
feat/pr4-llm-loop-history
May 10, 2026
Merged

PR4: LLM roles + multi-round loop + history injection + cost guard#6
Protocol-zero-0 merged 3 commits into
mainfrom
feat/pr4-llm-loop-history

Conversation

@Protocol-zero-0

Copy link
Copy Markdown
Owner

Closes #4. See overall roadmap at #5.

What this PR adds

Built on top of PR #2 (merged main). Four deliverables as specified in Issue #4:

Block 1 — Three roles wired to real LLM / coding agent

  • roles/planner.py: calls Anthropic or OpenAI API; produces plan with summary, steps, abort signal
  • roles/executor.sh: calls aider or claude-code (configurable); writes executor_output.json
  • roles/evaluator.py: calls LLM to judge accept/reject; reports cost_usd + tokens_used

All choices (provider, model, api_key env var, coding tool) are config-driven — nothing hardcoded. Roles read config.json from their run directory (written by governor).

Block 2 — --loop flag (multi-round evolution)

cli.py gains --loop: runs run_once() in a loop until any hard stop triggers. Each iteration saves state atomically, so a crash mid-run leaves the ledger consistent.

Block 3 — History injection

Governor._build_history() scans ledger/runs/*/reflection.json before each run and injects the last N entries into planner_input.json. N is configurable via history.max_entries. Without this, multi-round planner has no memory.

Block 4 — Cost guard

HardStops gains max_total_usd and max_total_tokens (both default 0 = unlimited). HardStopState accumulates cost across rounds. precheck and record_outcome enforce limits alongside existing iteration limits. Evaluator reports per-round cost so kernel can track spend.

New config shape (evolution.yml)

llm:
  provider: anthropic        # anthropic | openai
  model: claude-sonnet-4-6
  api_key_env: ANTHROPIC_API_KEY

coding_agent:
  tool: aider                # aider | claude-code

history:
  max_entries: 10

hard_stops:
  max_iterations: 10
  max_consecutive_failures: 3
  max_total_usd: 1.00
  max_total_tokens: 500000

Verification

# All 39 tests pass (19 pre-existing + 20 new)
python3 -m pytest tests/ -v

# Multi-round loop with stub roles
evolution-kernel --config examples/evolution.yml --repo <repo> --ledger /tmp/led --loop

Not in this PR (see roadmap #5)

  • Goal evaluator (knows when mission is accomplished) → PR5
  • Strategist role → PR5
  • k-branch parallel exploration → PR6
  • Process sandbox → PR7

Four deliverables:
- config: add llm/coding_agent/history sections; add max_total_usd and
  max_total_tokens to hard_stops for configurable cost guard
- hard_stops: accumulate total_usd/total_tokens in state; precheck and
  record_outcome enforce cost limits alongside iteration limits
- governor: inject history[] into planner_input each round (last N
  reflections from ledger); history_max_entries is configurable
- cli: add --loop flag; _run_loop drives multi-round evolution until any
  hard stop triggers; _make_governor helper deduplicates setup
- roles/planner.py: LLM planner (anthropic/openai, configurable via
  config.json in run dir)
- roles/executor.sh: coding-agent wrapper (aider/claude-code, configurable)
- roles/evaluator.py: LLM evaluator; reports cost_usd + tokens_used so
  kernel can enforce cost guard

All choices (LLM provider/model/key, coding tool) are config-driven, nothing
hardcoded. 39/39 tests pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR advances the kernel from a single-run MVP toward a multi-round “evolution runtime” by adding (1) configurable real role implementations (planner/executor/evaluator), (2) a --loop CLI mode to run multiple rounds until hard stops trigger, (3) planner history injection from prior runs, and (4) spend/token hard stops with persisted accumulation across rounds.

Changes:

  • Add default roles/ implementations for planner (LLM), executor (coding agent), and evaluator (LLM with cost reporting).
  • Add --loop to run repeated Governor.run_once() iterations while persisting hard-stop state across rounds.
  • Add history injection into planner_input.json, plus config schema extensions (llm, coding_agent, history, and cost guard fields).

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
tests/test_pr4.py Adds unit/integration coverage for new config fields, cost guard behavior, history injection, and --loop.
roles/planner.py Implements an LLM-backed planner role reading planner_input.json + config.json and writing plan.json.
roles/executor.sh Implements executor role that calls Aider/Claude Code based on coding_agent.tool and writes executor_output.json.
roles/evaluator.py Implements LLM evaluator role that emits accept/reject recommendation plus cost_usd/tokens_used.
examples/evolution.yml Updates example config with new llm, coding_agent, history, and cost hard-stop fields; points roles to roles/.
evolution_kernel/hard_stops.py Extends persisted hard-stop state to track cumulative USD and token usage; enforces new limits.
evolution_kernel/governor.py Injects recent run history into planner_input.json via _build_history().
evolution_kernel/config.py Adds config dataclasses/parsers for llm, coding_agent, history, and new cost hard-stops.
evolution_kernel/cli.py Adds --loop mode and wires cost/tokens into persisted hard-stop state updates.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread roles/executor.sh Outdated
Comment on lines +63 to +66
python3 -c "
import json
print(json.dumps({'changed_files': $CHANGED, 'tool': '$TOOL', 'summary': '''$SUMMARY'''}, indent=2))
" > "$OUTPUT"
Comment thread roles/planner.py Outdated

m = re.search(r"\{.*\}", text, re.DOTALL)
if m:
plan = json.loads(m.group())
Comment thread evolution_kernel/governor.py Outdated
except Exception:
pass
n = self.history_max_entries
return entries[-n:] if n > 0 else entries
Comment thread evolution_kernel/cli.py
)
hard_stops.save_state(args.ledger, new_state)
_print_result(result, halted=new_state.halted, halt_reason=new_state.halt_reason)
if new_state.halted:
Comment thread evolution_kernel/cli.py Outdated
if not allowed:
_record_halted(args.ledger, state, why)
print(json.dumps({"halted": True, "reason": why}, indent=2, sort_keys=True))
return 0
Comment thread evolution_kernel/cli.py Outdated
Comment on lines +124 to +126
result = governor.run_once(goal, run_id=args.run_id)
cost_usd = float(result.evaluation.get("cost_usd", 0.0))
tokens_used = int(result.evaluation.get("tokens_used", 0))
Comment thread evolution_kernel/config.py Outdated
Comment on lines +226 to +231
max_total_usd = float(value.get("max_total_usd", 0.0))
max_total_tokens = int(value.get("max_total_tokens", 0))
if max_total_usd < 0:
raise ConfigError("`hard_stops.max_total_usd` must be >= 0")
if max_total_tokens < 0:
raise ConfigError("`hard_stops.max_total_tokens` must be >= 0")
Comment on lines +220 to +226
data = self._read_json(reflection)
entries.append({
"run_id": data.get("run_id", run_dir.name),
"accepted": data.get("accepted", False),
"summary": data.get("reason", ""),
"metrics": data.get("metrics", {}),
})
Protocol-zero-0 and others added 2 commits May 10, 2026 18:49
- executor.sh: use jq to encode output JSON (was unsafe shell interpolation)
- planner.py: wrap json.loads in try/except JSONDecodeError; fall back to
  plain-text plan instead of crashing on malformed LLM output
- config.py: explicit type check before float/int conversion in
  _parse_hard_stops; raises ConfigError with clear message instead of raw
  ValueError
- cli.py: add _safe_cost() helper with defensive float/int parsing for
  cost_usd/tokens_used fields; fix exit codes to return 3 consistently on
  halt in both single-run and --loop modes; call _record_halted() when
  record_outcome triggers a halt in loop mode (was missing audit entry)
- governor.py: store plan_summary in reflection.json; use it in
  _build_history() so planner sees actual attempt descriptions, not generic
  decision reason strings; simplify unreachable else branch in slice

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Lead with value ("Give an LLM a goal") not architecture internals
- Add concrete coverage-improvement example with realistic output
  including a rejection + recovery round showing history injection
- Add ledger directory tree with all artifacts annotated
- Add full configuration reference with all PR4 fields
- Add capabilities table: working vs coming-next (PR5/6/7)
- Fix CI badge URL: hitome0123 → Protocol-zero-0/evolution-kernel
- Remove Token-Ignition mentions from main README
- Remove legacy --goal CLI docs (code unchanged, just not advertised)
- Move architecture diagram after value sections, not before

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Protocol-zero-0
Protocol-zero-0 merged commit cf06eee into main May 10, 2026
4 checks passed
@Protocol-zero-0
Protocol-zero-0 deleted the feat/pr4-llm-loop-history branch May 10, 2026 19:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PR4: 接入真 LLM + 多轮进化循环 + History 注入 + Cost Guard

2 participants