PR4: LLM roles + multi-round loop + history injection + cost guard - #6
Merged
Conversation
Four deliverables: - config: add llm/coding_agent/history sections; add max_total_usd and max_total_tokens to hard_stops for configurable cost guard - hard_stops: accumulate total_usd/total_tokens in state; precheck and record_outcome enforce cost limits alongside iteration limits - governor: inject history[] into planner_input each round (last N reflections from ledger); history_max_entries is configurable - cli: add --loop flag; _run_loop drives multi-round evolution until any hard stop triggers; _make_governor helper deduplicates setup - roles/planner.py: LLM planner (anthropic/openai, configurable via config.json in run dir) - roles/executor.sh: coding-agent wrapper (aider/claude-code, configurable) - roles/evaluator.py: LLM evaluator; reports cost_usd + tokens_used so kernel can enforce cost guard All choices (LLM provider/model/key, coding tool) are config-driven, nothing hardcoded. 39/39 tests pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This was referenced May 10, 2026
There was a problem hiding this comment.
Pull request overview
This PR advances the kernel from a single-run MVP toward a multi-round “evolution runtime” by adding (1) configurable real role implementations (planner/executor/evaluator), (2) a --loop CLI mode to run multiple rounds until hard stops trigger, (3) planner history injection from prior runs, and (4) spend/token hard stops with persisted accumulation across rounds.
Changes:
- Add default
roles/implementations for planner (LLM), executor (coding agent), and evaluator (LLM with cost reporting). - Add
--loopto run repeatedGovernor.run_once()iterations while persisting hard-stop state across rounds. - Add history injection into
planner_input.json, plus config schema extensions (llm,coding_agent,history, and cost guard fields).
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
tests/test_pr4.py |
Adds unit/integration coverage for new config fields, cost guard behavior, history injection, and --loop. |
roles/planner.py |
Implements an LLM-backed planner role reading planner_input.json + config.json and writing plan.json. |
roles/executor.sh |
Implements executor role that calls Aider/Claude Code based on coding_agent.tool and writes executor_output.json. |
roles/evaluator.py |
Implements LLM evaluator role that emits accept/reject recommendation plus cost_usd/tokens_used. |
examples/evolution.yml |
Updates example config with new llm, coding_agent, history, and cost hard-stop fields; points roles to roles/. |
evolution_kernel/hard_stops.py |
Extends persisted hard-stop state to track cumulative USD and token usage; enforces new limits. |
evolution_kernel/governor.py |
Injects recent run history into planner_input.json via _build_history(). |
evolution_kernel/config.py |
Adds config dataclasses/parsers for llm, coding_agent, history, and new cost hard-stops. |
evolution_kernel/cli.py |
Adds --loop mode and wires cost/tokens into persisted hard-stop state updates. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+63
to
+66
| python3 -c " | ||
| import json | ||
| print(json.dumps({'changed_files': $CHANGED, 'tool': '$TOOL', 'summary': '''$SUMMARY'''}, indent=2)) | ||
| " > "$OUTPUT" |
|
|
||
| m = re.search(r"\{.*\}", text, re.DOTALL) | ||
| if m: | ||
| plan = json.loads(m.group()) |
| except Exception: | ||
| pass | ||
| n = self.history_max_entries | ||
| return entries[-n:] if n > 0 else entries |
| ) | ||
| hard_stops.save_state(args.ledger, new_state) | ||
| _print_result(result, halted=new_state.halted, halt_reason=new_state.halt_reason) | ||
| if new_state.halted: |
| if not allowed: | ||
| _record_halted(args.ledger, state, why) | ||
| print(json.dumps({"halted": True, "reason": why}, indent=2, sort_keys=True)) | ||
| return 0 |
Comment on lines
+124
to
+126
| result = governor.run_once(goal, run_id=args.run_id) | ||
| cost_usd = float(result.evaluation.get("cost_usd", 0.0)) | ||
| tokens_used = int(result.evaluation.get("tokens_used", 0)) |
Comment on lines
+226
to
+231
| max_total_usd = float(value.get("max_total_usd", 0.0)) | ||
| max_total_tokens = int(value.get("max_total_tokens", 0)) | ||
| if max_total_usd < 0: | ||
| raise ConfigError("`hard_stops.max_total_usd` must be >= 0") | ||
| if max_total_tokens < 0: | ||
| raise ConfigError("`hard_stops.max_total_tokens` must be >= 0") |
Comment on lines
+220
to
+226
| data = self._read_json(reflection) | ||
| entries.append({ | ||
| "run_id": data.get("run_id", run_dir.name), | ||
| "accepted": data.get("accepted", False), | ||
| "summary": data.get("reason", ""), | ||
| "metrics": data.get("metrics", {}), | ||
| }) |
- executor.sh: use jq to encode output JSON (was unsafe shell interpolation) - planner.py: wrap json.loads in try/except JSONDecodeError; fall back to plain-text plan instead of crashing on malformed LLM output - config.py: explicit type check before float/int conversion in _parse_hard_stops; raises ConfigError with clear message instead of raw ValueError - cli.py: add _safe_cost() helper with defensive float/int parsing for cost_usd/tokens_used fields; fix exit codes to return 3 consistently on halt in both single-run and --loop modes; call _record_halted() when record_outcome triggers a halt in loop mode (was missing audit entry) - governor.py: store plan_summary in reflection.json; use it in _build_history() so planner sees actual attempt descriptions, not generic decision reason strings; simplify unreachable else branch in slice Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Lead with value ("Give an LLM a goal") not architecture internals
- Add concrete coverage-improvement example with realistic output
including a rejection + recovery round showing history injection
- Add ledger directory tree with all artifacts annotated
- Add full configuration reference with all PR4 fields
- Add capabilities table: working vs coming-next (PR5/6/7)
- Fix CI badge URL: hitome0123 → Protocol-zero-0/evolution-kernel
- Remove Token-Ignition mentions from main README
- Remove legacy --goal CLI docs (code unchanged, just not advertised)
- Move architecture diagram after value sections, not before
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4. See overall roadmap at #5.
What this PR adds
Built on top of PR #2 (merged main). Four deliverables as specified in Issue #4:
Block 1 — Three roles wired to real LLM / coding agent
roles/planner.py: calls Anthropic or OpenAI API; produces plan withsummary,steps,abortsignalroles/executor.sh: callsaiderorclaude-code(configurable); writesexecutor_output.jsonroles/evaluator.py: calls LLM to judge accept/reject; reportscost_usd+tokens_usedAll choices (provider, model, api_key env var, coding tool) are config-driven — nothing hardcoded. Roles read
config.jsonfrom their run directory (written by governor).Block 2 —
--loopflag (multi-round evolution)cli.pygains--loop: runsrun_once()in a loop until any hard stop triggers. Each iteration saves state atomically, so a crash mid-run leaves the ledger consistent.Block 3 — History injection
Governor._build_history()scansledger/runs/*/reflection.jsonbefore each run and injects the last N entries intoplanner_input.json. N is configurable viahistory.max_entries. Without this, multi-round planner has no memory.Block 4 — Cost guard
HardStopsgainsmax_total_usdandmax_total_tokens(both default 0 = unlimited).HardStopStateaccumulates cost across rounds.precheckandrecord_outcomeenforce limits alongside existing iteration limits. Evaluator reports per-round cost so kernel can track spend.New config shape (
evolution.yml)Verification
Not in this PR (see roadmap #5)