Skip to content

Commit 3eea761

Browse files
yjwyjw
yjw
authored and
yjw
committed
docs: clarify AET positioning and workflows
1 parent d78b4e6 commit 3eea761

2 files changed

Lines changed: 220 additions & 129 deletions

File tree

‎README.md‎

Lines changed: 126 additions & 68 deletions
Original file line numberDiff line numberDiff line change
@@ -10,60 +10,56 @@
1010

1111
> AET is the local evidence layer between an agent’s work and a claim that the work is ready.
1212
13-
Modern coding agents are becoming increasingly capable of writing code, but improving how they work is still largely a manual and heuristic process. Skills are rewritten after failures, prompts are refined through trial and error, and successful sessions are often forgotten while unsuccessful ones are difficult to explain.
13+
**Agent Engineering Toolkit (AET)** makes coding-agent work inspectable before
14+
it is trusted, and improvable only when the improvement is itself evidenced.
15+
It records the instructions available to an agent, the human-approved change
16+
boundary, explicit command execution, produced artifacts, and any verification
17+
gap. Those records can travel with a delivery—or, when failures repeat, become
18+
the input to a bounded Skill-improvement experiment.
1419

15-
**Agent Engineering Toolkit (AET) treats evidence—not prompts or model weights—as the optimization target for coding agents.**
20+
This answers two different engineering questions with one evidence model:
1621

17-
Instead of asking an agent to reflect on what it *thinks* happened, AET records what actually happened:
18-
19-
- what instructions and Skills were available;
20-
- what changes were approved by humans;
21-
- what commands were explicitly executed;
22-
- what evidence those commands produced;
23-
- what remains unverified;
24-
- and whether the collected evidence still matches the current repository.
25-
26-
These records become reusable engineering artifacts rather than disposable execution logs.
27-
28-
```text
29-
Coding Session
30-
│
31-
▼
32-
Collect structured evidence
33-
│
34-
▼
35-
Understand recurring failures
36-
│
37-
▼
38-
Generate bounded Skill improvements
39-
│
40-
▼
41-
Replay + Validation Gate
42-
│
43-
▼
44-
Human Review & Adoption
45-
│
46-
▼
47-
Better Skill
48-
│
49-
▼
50-
Next Coding Session
51-
```
52-
53-
Unlike conventional agent reflection, AET never allows unrestricted self-modification.
54-
55-
Every candidate improvement must remain inside explicitly editable regions, preserve immutable contracts, replay successfully in isolation, pass independent validation and held-out gates, and finally be adopted explicitly by a human.
56-
57-
Evidence therefore serves two purposes simultaneously:
58-
59-
- it explains **why** an agent can be trusted today;
60-
- it determines **how** the agent is allowed to improve tomorrow.
61-
62-
As more coding sessions accumulate, AET transforms isolated execution records into an evidence-driven engineering feedback loop, enabling coding agents to become more reliable without changing the underlying model.
63-
64-
AET is **not** an agent runtime, an autonomous coding framework, or a prompt optimizer.
65-
66-
It is an evidence-driven self-evolution framework that makes coding agents continuously improvable through verifiable engineering evidence.
22+
| Question | AET answer |
23+
| --- | --- |
24+
| “Can we honestly say this agent delivery is ready?” | Audit instructions, review an approved diff, trace an explicit proof, and hand off the resulting evidence. |
25+
| “Can this recurring agent failure improve the Skill safely?” | Mine structured failures, patch only marked Skill regions, replay baseline and candidate, gate the result, then require a human adoption decision. |
26+
27+
The important distinction is that AET is not a self-reporting layer. A natural
28+
language answer never substitutes for a recorded command, artifact, snapshot,
29+
or explicit `UNKNOWN` state.
30+
31+
## Why AET exists
32+
33+
Coding agents make it inexpensive to change a repository, but not necessarily
34+
easy to answer the questions that matter at handoff: *Which instructions were
35+
in scope? Was this command really run? Does the output still describe this
36+
workspace? What is verified, and what remains unknown?*
37+
38+
Most teams solve fragments of this with chat transcripts, CI logs, prompt
39+
edits, or manual checklists. AET gives those fragments a local, structured,
40+
hash-bound form and keeps their meanings deliberately narrow. That makes the
41+
system useful for ordinary delivery work today, without pretending that every
42+
agent session should become training data tomorrow.
43+
44+
## Why AET
45+
46+
- **Evidence-first, not confidence-first.** `UNKNOWN` stays a verification gap;
47+
it is never discounted into a pass.
48+
- **Smallest safe surface.** `audit` and `review` inspect; only `trace` executes
49+
the explicit argv after `--`.
50+
- **Local by default.** Evidence collection, review, Experience Store, and
51+
federation do not require a hosted telemetry service or transcript archive.
52+
- **Proof remains fresh-or-stale.** A successful command and a workspace that
53+
later changed are represented as separate facts.
54+
- **Learning is constrained.** Candidate Skills are hash-bound, limited to
55+
marked editable blocks, validated independently, staged, and adopted only by
56+
an explicit human action.
57+
- **Behavior can be observed.** Static Skill-document checks remain Gate 0;
58+
opt-in Scripted, Codex, and Claude Code runners can evaluate actual isolated
59+
task executions with deterministic scoring.
60+
61+
AET is **not** an agent runtime, a general autonomous coding framework, a
62+
hosted monitoring product, or a system that auto-edits production Skills.
6763

6864
## Start here
6965

@@ -95,28 +91,85 @@ found a problem,” not “no audit JSON was produced.”
9591
| Which existing findings should be handled first? | `aet triage` | Explainable ordering; never a changed finding status. |
9692
| Can repeated evidence failures improve a Skill safely? | `aet learn` | Evidence-only experience set, bounded candidate, Gate, and staged review copy. |
9793

94+
## Where AET fits
95+
96+
These tools are complementary. The comparison is about the job each one owns,
97+
not a claim that one should replace the others.
98+
99+
| Tool category | Best used for | What AET adds or deliberately does not do |
100+
| --- | --- | --- |
101+
| Coding-agent runtime (Codex, Claude Code, Copilot) | Planning and executing the work in a repository. | AET does not replace the runtime; it records the local evidence needed to make its delivery claims reviewable. |
102+
| CI, tests, linters, and security scanners | Checking code or a deployment against their own rules. | AET can trace an explicit check and bind its artifact to intent, workspace freshness, and a handoff; it does not replace the checker. |
103+
| Skill authoring / governance system ([Yao Meta Skill](https://github.com/yaojingang/yao-meta-skill)) | Creating, packaging, compiling, evaluating, and governing reusable cross-platform Skill assets. | AET focuses on the evidence around a coding-agent delivery and on bounded improvement of an in-use Skill. Use Yao to engineer the Skill product; use AET to evidence and constrain work performed with it. |
104+
| Skill optimizer ([SkillOpt](https://github.com/microsoft/SkillOpt)) | Training a Skill document from scored rollouts and held-out validation. | AET provides local engineering evidence semantics—intent boundaries, explicit command proof, artifact handling, freshness, and human adoption—rather than a general benchmark optimizer. |
105+
| Transcript analytics / agent observability | Searching broad session history, dashboards, or fleet telemetry. | AET defaults to structured Evidence Only records and local storage; it intentionally does not ingest an unbounded transcript archive. |
106+
107+
### Choose AET when
108+
109+
- You need a credible handoff after an agent changed code: not just “tests passed,”
110+
but the command, exit status, declared artifact, approved scope, and freshness.
111+
- You need to preserve the difference between **PASS**, **FAIL**, and
112+
**UNKNOWN** instead of collapsing uncertainty into a score.
113+
- You want to improve a Skill from repeated engineering failures without letting
114+
an optimizer silently weaken safety semantics or overwrite production guidance.
115+
- You need a local, portable evidence format that works alongside—not inside—an
116+
existing agent runtime and CI system.
117+
118+
### Do not choose AET as
119+
120+
- a replacement for writing tests, running CI, reviewing code, or securing a
121+
deployment;
122+
- a substitute for an agent runtime or task planner;
123+
- a promise that an Agent understood a file merely because it was discovered or
124+
attested as read; or
125+
- an automatic self-modification daemon. `propose`, `gate`, `stage`, and
126+
`adopt` are intentionally separate actions.
127+
98128
## Architecture
99129

100130
```mermaid
101131
flowchart TB
102-
A["Instructions and Skills"] --> B["audit\nstatic facts"]
103-
C["Approved intent + Git diff"] --> D["review\nscope facts"]
104-
E["Explicit argv after --"] --> F["trace\nexecuted proof"]
105-
G["Local Git + docs\noptional explicit remote export"] --> H["evolve\ncited history"]
106-
B --> I["Versioned Evidence IR\nstatus + hashes + snapshots"]
107-
D --> I
108-
F --> I
109-
I --> J["Evidence Pack / Viewer\nRun / Context / Decision"]
110-
H --> K["Evolution Pack"]
111-
I -. "repeated, structured\nEvidence Only records" .-> L["learn\npattern → bounded patch → replay → Gate"]
112-
L --> M["stage for human review"]
113-
M -. "explicit --yes" .-> N["adopt + Decision Ledger"]
132+
subgraph delivery["Delivery evidence plane"]
133+
A["Instructions + Skills"] --> B["audit\nstatic instruction facts"]
134+
C["Human intent + Git diff"] --> D["review\nscope and proof contract"]
135+
E["Explicit argv after --"] --> F["trace\nexecuted proof + declared artifacts"]
136+
G["Repository context + decisions"] --> H["context / decision / evolve\nhash-bound local history"]
137+
B --> I["Evidence IR\nstatus + hashes + workspace snapshots"]
138+
D --> I
139+
F --> I
140+
H --> I
141+
I --> J["Handoff\nEvidence Pack / Viewer / Run"]
142+
end
143+
144+
subgraph learning["Optional evidence-gated evolution"]
145+
I -. "structured Evidence Only\nrepeated failures" .-> K["harvest + mine\npattern support"]
146+
K --> L["propose\nbounded Patch IR"]
147+
L --> M["replay\nstatic Gate 0 or isolated host rollout"]
148+
M --> N["gate\ncore + validation + held-out + cost"]
149+
N --> O["stage\nhuman review copy"]
150+
O -. "explicit --yes" .-> P["adopt\nSkill + Decision Ledger"]
151+
N -. "fail / inconclusive" .-> Q["reject record\nnegative constraint"]
152+
end
153+
154+
P --> A
114155
```
115156

116-
The dashed path is deliberately optional. AET does not treat every artifact as
117-
training data, and a passed Gate does not modify the production Skill.
157+
The learning path is deliberately optional. AET does not treat every artifact
158+
as training data; static text checks are not presented as observed behavior;
159+
and a passing Gate does not modify a production Skill.
118160

119-
## A normal delivery
161+
## How to use AET
162+
163+
Start with the job, not the biggest workflow:
164+
165+
| If you need to… | Start with | Add only if needed |
166+
| --- | --- | --- |
167+
| Check whether an Agent’s local guidance is usable | `aet audit` | `context` when you need a hash-bound record of discovered/read assets. |
168+
| Deliver an Agent-authored change | `audit` + `review` + `trace` | `evidence pack` for a portable handoff; `run` when the delivery has multiple lifecycle steps. |
169+
| Explain why a repository looks this way | `aet evolve plan` | `collect/build/report` after reviewing the collection plan. |
170+
| Improve a recurring Skill failure | `learn harvest` + `mine` | `propose/replay/gate/stage`; use a real host runner only when explicitly configured. |
171+
172+
### Recipe: evidence-backed delivery
120173

121174
```bash
122175
# 1. Audit instructions and review the approved diff. Neither runs tests.
@@ -155,7 +208,7 @@ Evidence Only JSON → inspect → mine → bounded Patch IR → isolated replay
155208
→ core + validation + held-out Gate → stage → human adopt or reject
156209
```
157210

158-
### Phase 0–6, available now
211+
### What is implemented today
159212

160213
| Phase | What is implemented |
161214
| --- | --- |
@@ -168,6 +221,11 @@ Evidence Only JSON → inspect → mine → bounded Patch IR → isolated replay
168221
| 6. Sleep | Local bounded loop, append-only `SKILL_EVOLUTION` event history, budgets, target-change check, and stage-only terminal action. |
169222
| 7. Real-host evaluation | Isolated fixture workspaces, normalized command/final-answer events, deterministic behavioral scoring, repeated paired rollouts, and `PASS`/`FAIL`/`INCONCLUSIVE`/`INFRASTRUCTURE_ERROR` statistics. |
170223

224+
The real-host fixture is a proof-handoff smoke test, not a broad claim that
225+
every task distribution is solved. For an adoption-grade observed Gate, bring
226+
your own separated core, validation, and held-out tasks; configure sufficient
227+
paired rollouts; and treat `INCONCLUSIVE` as a non-passing result.
228+
171229
```bash
172230
# Phase 1: build and inspect a local Evidence Only experience set.
173231
aet learn harvest --evidence .aet/evidence --output .aet/learn/experiences.json

0 commit comments

Comments
 (0)