Design stage. This concretizes the direction from the RFC in #252 (now locked). #252 stays the RFC/alignment thread and the related-work survey; this issue is the buildable prototype spec and the parent for the W1–W6 workstreams (filed as sub-issues below).
Summary
A persistent, non-modifying repository monitor. Each cycle it runs sync → refresh → observe → reflect → investigate → report → persist on one target repo and writes back a cross-cycle repository memory. It perceives the repo through two facets: its current state (CodeMiner's index + code graph) and its evolution over time (the memory). Its defining action is to actively investigate — form a hypothesis about an emerging risk and write a test that exposes it, run that test in an isolated sandbox, and report the result with evidence. It never touches the production repo.
What we prove (H1). Replaying a repo through its own history, a Guardian arm that carries memory across cycles surfaces higher-precision / earlier maintenance findings than an identical memoryless arm, at a bounded per-cycle cost. The headline is that Δ, measured — a null/negative result is still a valid answer to the research question.
First target: CodeMiner itself. Runs on: 1× H100 80 GB, local model (vLLM + Qwen3-Coder-Next) behind the repo's litellm layer, one ephemeral container per cycle.
| Milestone |
Meaning |
~Day |
| M1 |
One cycle runs end-to-end in a per-cycle container on the host |
7 |
| M2 |
Memory + graph-diff persist and are used across cycles |
16 |
| M3 |
Replay harness yields precision / recall / lead-time / cost |
21 |
| M4 |
Memory-vs-memoryless headline number + threats |
27 |
In scope: graph-diff drift signal, repository memory, LLM reflect/investigate with test synthesis, future-history replay + memory ablation — one Python repo, ~5–10 cycles. Deferred: applied/candidate patches (Phase 4), multi-repo, daemon/scheduling, web UI.
1 · Architecture & repository memory
The cycle (7 steps). Steps 1–3 and 7 are deterministic (no model); only reflect (4) and investigate (5) call the LLM — which is what lets the memoryless ablation be a clean toggle on what step 4 may read.
- Sync — resolve/checkout commit
t.
- Refresh — incremental index + code-graph update for
t.
- Observe — signals: churn + graph-diff drift vs. the last snapshot + test deltas.
- Reflect — LLM reads memory + signals → ranked hypotheses.
- Investigate — for top hypotheses, retrieve evidence and synthesize a risk-revealing test (§ below).
- Report — findings + evidence + reasoning trace to
.md/.json; no patches.
- Persist — write this cycle's graph snapshot, findings, and test state to memory.
Repository memory (repo_memory/<repo_id>/, append-only, keyed by commit): per-cycle graph snapshots (CodeGraph.save_graph), per-cycle findings JSON, and a small index.sqlite:
| Table |
Purpose |
cycles |
one row/cycle; token + wall-clock cost for H1 |
symbols |
symbol lifetime → churn / removal |
edges |
dependency evolution → drift |
findings |
prior findings (+ evidence_test, evidence_diff) for recall / de-dup / escalate |
test_deltas |
per-test outcome trajectory over time |
Reads are paged/summarized so a long history never blows the model context. Backend is deliberately boring (SQLite + JSON + graph snapshots); a vector/graph DB is a month-2 decision.
Graph-diff drift is the new deterministic signal the memory facet unlocks: diff the current CodeGraph against the prior snapshot for new/removed edges on high-fan-in symbols, fan-in spikes, and public-API arity/signature changes with lagging dependents. The graph is a detector, not an LLM input — it runs deterministically and emits a plain-English shortlist (parse_config changed arity; 47 dependents; 3 not updated). Only that prose + retrieved code reaches the model. Its job is selection at repo scale, and it's the sharpest test of H1: cross-file structural drift is exactly what a single-snapshot memoryless arm cannot see.
Active investigation = test synthesis. A risk is confirmed not by argument but by a test the agent writes to expose it:
- Gather evidence —
HybridRetrievePipeline.query → evidence rows + source spans.
- Synthesize a risk-revealing test (primary action) — the agent writes a new targeted test whose failure demonstrates the risk (e.g. calls
parse_config the way the 47 dependents still do, asserts the old behaviour). It fails on the current commit — that failure is the evidence. Written to the sandbox overlay, run with the repo's own pytest, never committed.
- Corroborate — a red test only counts if it's red for the hypothesized reason: differential run (same test passes on the prior-cycle checkout, fails now → an agent-generated
PASS→FAIL) and/or fix-probe (a minimal overlay edit reverting the suspected cause makes it go green → FAIL→PASS).
- Record verdict + reasoning trace + synthesized test + any fix-diff into the
Finding.
Cheaper first-pass probes (no test written): existing-test run, call-site grep, import/typecheck. Everything is gated by a per-cycle budget.
Non-modifying invariant (hard constraint). The agent works only inside a per-cycle container on a disposable overlay checkout. Synthesized tests and fix-edits live on the throwaway layer and die with the container — surfaced as evidence a risk is real, never as "apply this." report.py asserts a no-patch invariant; that test stays green.
2 · Environment — how to run & dispatch sandboxes
Host: one remote Linux box, 1× H100 PCIe 80 GB, CUDA 13, driver 580. Model served locally and reached through the repo's litellm — no cloud keys.
Model server (vLLM). vLLM's paged KV-cache + continuous batching use the 80 GB card properly and keep headroom for Guardian's long prompts (memory + retrieved code + findings). OpenAI-compatible, so nothing in Guardian changes — only the
api_base litellm points at.
Model: Qwen3-Coder-Next (80B-total / 3B-active MoE, 256K context, ~40 GB in 4-bit → ~35–40 GB KV headroom). Coding-agent specialist that fits one H100 and runs at ~3B speed. Non-thinking only — fine, since reasoning traces come from the
investigate loop, not model internals. Fallbacks: Qwen3.6-27B fp16 (dense, adds <think>) or Qwen3.6-35B-A3B 4-bit (leanest).
# Shell A — model server (GPU)
python -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen3-Coder-Next \
--tool-call-parser qwen3_coder --max-model-len 131072 \
--gpu-memory-utilization 0.90 --port 8000
curl -s http://localhost:8000/v1/models | python -m json.tool # verify
# Shell B — Guardian (conda env `codeminer`), litellm → http://localhost:8000/v1
# 1. smoke: one cycle at HEAD, BM25-only, no LLM
python scripts/guardian_cycle.py --repo /path/to/target --index-types bm25 \
--no-investigate --out reports/
# 2. one cycle with retrieval + local-model investigation
python scripts/guardian_cycle.py --repo /path/to/target --investigate --out reports/
# 3. future-history replay, memory arm, 8 commits, per-cycle budget
python scripts/guardian_replay.py --repo /path/to/target \
--commits commits.txt --arm memory --budget-tokens 300000 \
--sandbox container --out runs/target_memory/
Entry points: existing scripts/guardian_cycle.py (one cycle) + new thin scripts/guardian_replay.py (the month-1 driver: replays a commit list, one sandboxed cycle each with knowledge ≤ t, appends to memory). No daemon/cron this month — the harness drives cycles so runs are deterministic and comparable.
Sandbox dispatch — one ephemeral container per cycle (primary, M1-required). Each cycle runs against a throwaway checkout at commit t, never the production tree — this is what makes "non-modifying" structural rather than a promise.
scratch/<repo_id>/
mirror.git/ # one bare mirror, fetched once
wt/<n>_<commit>/ # git worktree at t — the checkout to mount (ephemeral)
cache/ memory/ # index cache + repository memory — persist across cycles
Per cycle: (1) git worktree add --detach materializes t cheaply (shares the mirror's objects); (2) launch a container from a pinned image (guardian-runtime), mounting /repo as an overlay — worktree = read-only lower, tmpfs upper absorbs edits and is the evidence-diff; cache/+memory/ read-only, the memory write happens host-side after exit; (3) run sync/refresh/observe/investigate inside, --network none + one localhost allow-rule to the host model server, CPU/RAM cap + timeout; (4) --rm the container, remove the worktree. Runtime: Podman (rootless, daemonless) with Docker fallback; container needs no GPU. --sandbox worktree is retained as a debug mode — not what the evaluation runs in.
Cost budget per cycle. Deterministic signals use no model; reflect triages, only high-confidence hypotheses reach the pricier investigate. Enforced by --budget-tokens (via llm/usage.py), paged memory reads, and prompt caching. Tokens + wall-clock are logged to the cycles table so the ablation reports quality at a cost.
3 · Evaluation
Answers one question — H1 — via future-history replay: run Guardian as if living through the repo's past, knowledge bounded at each cycle's commit, and check whether its unprompted findings anticipated what actually happened later.
Protocol. Pick replay commits t0 < … < tk; at each ti the sandbox is at ti and memory/index contain only info ≤ ti. Build post-ti ground truth (later issues / bug-fix PRs / reverts) and map each future fix to the symbols it touched, reusing dataset/gt_locate.py + the span/symbol normalization in eval/retrieval_eval.py. Score: did Guardian flag that symbol at ti?
Headline ablation — memory vs. memoryless. Two arms, identical except what reflect may read; memoryless is a flag (--arm memoryless) that refuses the memory reads, so any difference is attributable to memory, not implementation drift.
|
Memory |
Memoryless |
Current-state facet (index + graph @ t) |
✅ |
✅ |
| Graph-diff vs. prior snapshot |
✅ |
❌ |
| Prior findings / symbol & test history |
✅ |
❌ |
| Model, budget, retrieval, signals |
identical |
identical |
Metrics: finding precision, recall @ horizon, lead-time (how early), memory-unique findings (confirmed findings memory produced that memoryless didn't — direct H1 evidence), cost (tokens + wall-clock), and the non-modifying invariant (100% of reports carry zero applied diffs). Headline = memory-vs-memoryless on {precision, recall, lead-time} at matched cost; reported as paired per-cycle points, directional (not a powered significance test) at n = 1 repo.
Baselines beyond the ablation. The ablation isolates memory; to position the number we also run a short ladder on the same commits / ground truth / horizon / budget, all scored by the same score_agent_localization (a baseline just emits Findings): churn/static ranker (rank symbols by git churn + complexity — "do you need an agent over a heuristic?"), graph-diff-only (emit the drift shortlist directly — "does LLM investigation add precision over the raw detector?"), and random + hindsight-oracle (floor & ceiling). Target shape is a monotone ladder: memory > memoryless > graph-diff-only > churn-rank > random. Optional: linter-suite-as-findings, and a reactive agent fed the real later issue text (stretch). External systems — RepoAudit (agentic auditing, precision-at-cost; benchmarks vs. Infer/CodeGuru), LocAgent/CoSIL (graph localizers), SWT-Bench (validates the differential-run test-synthesis mechanism) — are comparators on sub-tasks, since none is persistent / memory-carrying (that gap is the contribution).
Threats: hindsight leakage (nothing after ti reaches the sandbox/index — audited); ground-truth incompleteness (recall is a lower bound); n = 1; LLM nondeterminism (fixed decoding, model-free signals); judging bias (rubric blind to arm); invalid synthesized tests (a test failing for the wrong reason — import errors discarded, and the differential run separates "risk is real" from "test is broken"; we report the validity-gate pass rate as harness health).
4 · Schedule
| Week |
Focus |
Exit |
| 1 |
Runtime incl. container (W4) + graph-diff (W1) + memory schema (W2.1) + replay skeleton (W5.1) |
M1 — a cycle runs end-to-end inside a per-cycle container; non-modifying audit passes. |
| 2 |
Memory read/write + memoryless toggle (W2) + reflect over memory (W3.1–3.2) + ground truth (W5.2) |
M2 — cycle k provably uses k−1's memory; drift findings appear. |
| 3 |
Test-synthesis investigation + trace (W3.3–3.5) + metrics (W5.3–5.4 start) |
M3 — replay yields precision/recall/lead-time/cost for one arm. |
| 4 |
Run the ablation (W5.4–5.5) + write-up & demo (W6) |
M4 — memory-vs-memoryless headline number, threats stated, demo in hand. |
If time slips, cut in this order: richer investigation probes (keep one) → number of replay cycles (keep ≥ 4) → container network hardening (fall back to localhost-only allow-rule; never drop the container) → never cut the memoryless arm — it is the result.
Workstreams (tracked as sub-issues)
The six workstreams below are filed as sub-issues of this issue so the progress bar tracks them. Spine: W4 (runtime) + W1 (graph-diff) unblock W2 (memory) → W3 (investigation); W5 (eval) runs partly in parallel and consumes all; W6 writes up. Critical path: W4.1 → W4.3 (container) → W1.1 → W2.1 → W2.2/2.5 → W3.1 → W5.1 → W5.4.
Full task breakdown and reuse map live in Repository_Guardian/prototype_design.md.
Summary
A persistent, non-modifying repository monitor. Each cycle it runs
sync → refresh → observe → reflect → investigate → report → persiston one target repo and writes back a cross-cycle repository memory. It perceives the repo through two facets: its current state (CodeMiner's index + code graph) and its evolution over time (the memory). Its defining action is to actively investigate — form a hypothesis about an emerging risk and write a test that exposes it, run that test in an isolated sandbox, and report the result with evidence. It never touches the production repo.What we prove (H1). Replaying a repo through its own history, a Guardian arm that carries memory across cycles surfaces higher-precision / earlier maintenance findings than an identical memoryless arm, at a bounded per-cycle cost. The headline is that Δ, measured — a null/negative result is still a valid answer to the research question.
First target: CodeMiner itself. Runs on: 1× H100 80 GB, local model (vLLM + Qwen3-Coder-Next) behind the repo's
litellmlayer, one ephemeral container per cycle.In scope: graph-diff drift signal, repository memory, LLM reflect/investigate with test synthesis, future-history replay + memory ablation — one Python repo, ~5–10 cycles. Deferred: applied/candidate patches (Phase 4), multi-repo, daemon/scheduling, web UI.
1 · Architecture & repository memory
The cycle (7 steps). Steps 1–3 and 7 are deterministic (no model); only
reflect(4) andinvestigate(5) call the LLM — which is what lets the memoryless ablation be a clean toggle on what step 4 may read.t.t..md/.json; no patches.Repository memory (
repo_memory/<repo_id>/, append-only, keyed by commit): per-cycle graph snapshots (CodeGraph.save_graph), per-cycle findings JSON, and a smallindex.sqlite:cyclessymbolsedgesfindingsevidence_test,evidence_diff) for recall / de-dup / escalatetest_deltasReads are paged/summarized so a long history never blows the model context. Backend is deliberately boring (SQLite + JSON + graph snapshots); a vector/graph DB is a month-2 decision.
Graph-diff drift is the new deterministic signal the memory facet unlocks: diff the current
CodeGraphagainst the prior snapshot for new/removed edges on high-fan-in symbols, fan-in spikes, and public-API arity/signature changes with lagging dependents. The graph is a detector, not an LLM input — it runs deterministically and emits a plain-English shortlist (parse_configchanged arity; 47 dependents; 3 not updated). Only that prose + retrieved code reaches the model. Its job is selection at repo scale, and it's the sharpest test of H1: cross-file structural drift is exactly what a single-snapshot memoryless arm cannot see.Active investigation = test synthesis. A risk is confirmed not by argument but by a test the agent writes to expose it:
HybridRetrievePipeline.query→ evidence rows + source spans.parse_configthe way the 47 dependents still do, asserts the old behaviour). It fails on the current commit — that failure is the evidence. Written to the sandbox overlay, run with the repo's ownpytest, never committed.PASS→FAIL) and/or fix-probe (a minimal overlay edit reverting the suspected cause makes it go green →FAIL→PASS).Finding.Cheaper first-pass probes (no test written): existing-test run, call-site grep, import/typecheck. Everything is gated by a per-cycle budget.
2 · Environment — how to run & dispatch sandboxes
Host: one remote Linux box, 1× H100 PCIe 80 GB, CUDA 13, driver 580. Model served locally and reached through the repo's
litellm— no cloud keys.Model server (vLLM). vLLM's paged KV-cache + continuous batching use the 80 GB card properly and keep headroom for Guardian's long prompts (memory + retrieved code + findings). OpenAI-compatible, so nothing in Guardian changes — only the
api_baselitellmpoints at.Model: Qwen3-Coder-Next (80B-total / 3B-active MoE, 256K context, ~40 GB in 4-bit → ~35–40 GB KV headroom). Coding-agent specialist that fits one H100 and runs at ~3B speed. Non-thinking only — fine, since reasoning traces come from the
investigate loop, not model internals. Fallbacks: Qwen3.6-27B fp16 (dense, adds
<think>) or Qwen3.6-35B-A3B 4-bit (leanest).Entry points: existing
scripts/guardian_cycle.py(one cycle) + new thinscripts/guardian_replay.py(the month-1 driver: replays a commit list, one sandboxed cycle each with knowledge ≤t, appends to memory). No daemon/cron this month — the harness drives cycles so runs are deterministic and comparable.Sandbox dispatch — one ephemeral container per cycle (primary, M1-required). Each cycle runs against a throwaway checkout at commit
t, never the production tree — this is what makes "non-modifying" structural rather than a promise.Per cycle: (1)
git worktree add --detachmaterializestcheaply (shares the mirror's objects); (2) launch a container from a pinned image (guardian-runtime), mounting/repoas an overlay — worktree = read-only lower, tmpfs upper absorbs edits and is the evidence-diff;cache/+memory/read-only, the memory write happens host-side after exit; (3) runsync/refresh/observe/investigateinside,--network none+ one localhost allow-rule to the host model server, CPU/RAM cap + timeout; (4)--rmthe container, remove the worktree. Runtime: Podman (rootless, daemonless) with Docker fallback; container needs no GPU.--sandbox worktreeis retained as a debug mode — not what the evaluation runs in.Cost budget per cycle. Deterministic signals use no model;
reflecttriages, only high-confidence hypotheses reach the pricierinvestigate. Enforced by--budget-tokens(viallm/usage.py), paged memory reads, and prompt caching. Tokens + wall-clock are logged to thecyclestable so the ablation reports quality at a cost.3 · Evaluation
Answers one question — H1 — via future-history replay: run Guardian as if living through the repo's past, knowledge bounded at each cycle's commit, and check whether its unprompted findings anticipated what actually happened later.
Protocol. Pick replay commits
t0 < … < tk; at eachtithe sandbox is attiand memory/index contain only info ≤ti. Build post-tiground truth (later issues / bug-fix PRs / reverts) and map each future fix to the symbols it touched, reusingdataset/gt_locate.py+ the span/symbol normalization ineval/retrieval_eval.py. Score: did Guardian flag that symbol atti?Headline ablation — memory vs. memoryless. Two arms, identical except what
reflectmay read; memoryless is a flag (--arm memoryless) that refuses the memory reads, so any difference is attributable to memory, not implementation drift.t)Metrics: finding precision, recall @ horizon, lead-time (how early), memory-unique findings (confirmed findings memory produced that memoryless didn't — direct H1 evidence), cost (tokens + wall-clock), and the non-modifying invariant (100% of reports carry zero applied diffs). Headline = memory-vs-memoryless on {precision, recall, lead-time} at matched cost; reported as paired per-cycle points, directional (not a powered significance test) at n = 1 repo.
Baselines beyond the ablation. The ablation isolates memory; to position the number we also run a short ladder on the same commits / ground truth / horizon / budget, all scored by the same
score_agent_localization(a baseline just emitsFindings): churn/static ranker (rank symbols by git churn + complexity — "do you need an agent over a heuristic?"), graph-diff-only (emit the drift shortlist directly — "does LLM investigation add precision over the raw detector?"), and random + hindsight-oracle (floor & ceiling). Target shape is a monotone ladder: memory > memoryless > graph-diff-only > churn-rank > random. Optional: linter-suite-as-findings, and a reactive agent fed the real later issue text (stretch). External systems — RepoAudit (agentic auditing, precision-at-cost; benchmarks vs. Infer/CodeGuru), LocAgent/CoSIL (graph localizers), SWT-Bench (validates the differential-run test-synthesis mechanism) — are comparators on sub-tasks, since none is persistent / memory-carrying (that gap is the contribution).Threats: hindsight leakage (nothing after
tireaches the sandbox/index — audited); ground-truth incompleteness (recall is a lower bound); n = 1; LLM nondeterminism (fixed decoding, model-free signals); judging bias (rubric blind to arm); invalid synthesized tests (a test failing for the wrong reason — import errors discarded, and the differential run separates "risk is real" from "test is broken"; we report the validity-gate pass rate as harness health).4 · Schedule
kprovably usesk−1's memory; drift findings appear.If time slips, cut in this order: richer investigation probes (keep one) → number of replay cycles (keep ≥ 4) → container network hardening (fall back to localhost-only allow-rule; never drop the container) → never cut the memoryless arm — it is the result.
Workstreams (tracked as sub-issues)
The six workstreams below are filed as sub-issues of this issue so the progress bar tracks them. Spine: W4 (runtime) + W1 (graph-diff) unblock W2 (memory) → W3 (investigation); W5 (eval) runs partly in parallel and consumes all; W6 writes up. Critical path: W4.1 → W4.3 (container) → W1.1 → W2.1 → W2.2/2.5 → W3.1 → W5.1 → W5.4.
Full task breakdown and reuse map live in
Repository_Guardian/prototype_design.md.