Fitness gate: judge generated objects by evidence they cannot edit - #1
Closed
andreBurnt wants to merge 16 commits into
Closed
Fitness gate: judge generated objects by evidence they cannot edit#1andreBurnt wants to merge 16 commits into
andreBurnt wants to merge 16 commits into
Conversation
A manifest could describe a method's shape but not its meaning: nothing said whether calling it changes the world, what its output must look like, or which properties every answer has to satisfy. The fitness gate needs all three, so methods can now declare effects (read|act), a JSON Schema for their output, metamorphic relations, and a known entity that must appear in a healthy answer. All fields are optional; existing manifests are untouched.
A cassette records one exchange: the request an object made, the answer the world gave, and what the object made of it. Capped per method, LRU evicted, credentials stripped at the door. This is the memory the fitness gate replays against every candidate.
evaluate() is the gate a generated source must pass before it deploys: every cassette must be reproduced in meaning, every declared output schema satisfied, with all I/O served from the recording. No LLM, no network, nothing the candidate can edit. Relations and the mutation gate land next.
Track validation success per method; if a method with outputSchema has zero successful validations and at least one probe threw, fail schema with a detail noting the method threw on all probes. This prevents first-create candidates that always throw from spuriously passing schema validation.
Metamorphic relations judge a method by how its answers relate: same call twice agrees, no element repeats, order holds, a tighter filter returns a subset, and the entity every healthy answer contains is there. Declared in the manifest, checked by the gate, invisible to the LLM.
Secondary invocations (idempotent re-call, subset-on-tighter-filter loose/tight) must be wrapped in try/catch to fail the relation instead of rejecting the promise. Numeric filter arguments now sort numerically (a - b) instead of lexicographically; string sorts via localeCompare as before. Covers both cases in two new integration tests: stateful invoker failure on idempotent, and numeric argument ordering for tighter/looser comparisons.
A fixed set of deterministic mutations -- flipped comparisons, dropped filters, emptied returns, renamed keys -- is applied to every candidate. If the cassettes, schemas, and relations cannot tell most mutants from the original, the evidence is too weak to certify a deploy, and the gate says so with a kill ratio.
Verify that the mutation-gate kill-counting logic is exercised with a source that has real mutation points (comparisons, filter calls), confirming mutants are correctly identified and counted as killed when replay fails.
ObjectCreator gains a fitness action: the draft runs in the sandbox with every call shimmed, HTTP served from the object's own cassettes, and the verdict recorded against a digest of the judged source. deploy_spawn and deploy_update now consult that gate -- no verdict, failed verdict, or edited-since-judged draft all refuse. The semantic reviewer stays advice; this is the part that is not.
HttpClient consults a per-object recorder before and after each request: record mode keeps every successful exchange as a cassette, replay mode serves recorded answers and refuses to improvise. Objects nobody registered pass through untouched. This is how the gate's memory grows without anyone authoring fixtures.
HttpClient's ask guide promises every caller { status, statusText, headers,
body, ok } with body ALWAYS a raw string — that is the contract an LLM writes
its handler against. The fitness invoker was returning { status, body:parsed },
so a candidate that did the documented JSON.parse(res.body) failed judgment
while one written against a shape the runtime does not produce passed.
Cassettes now keep rawBody (the response text verbatim) alongside parsedOutput,
so replay can return the same characters the world sent — JSON.stringify of a
parsed JSON string primitive is not the same text. Entries recorded before
rawBody existed derive it on load.
WebFetch leaves HTTP_TARGETS: its live FetchResult is nothing like an
HttpResponse, so shimming it taught candidates a second fictional contract.
A WebFetch-using candidate now fails with a plain unstubbed-I/O message.
matchRequest is exact method+url only. The host+path fallback served ?q=1's
recording to ?q=other, which defeats argument-dependent replay outright; the
loose behaviour survives as matchRequestLoose for callers that want a sample
of an endpoint rather than an answer to a question.
With zero cassettes — the only state this branch actually produces at runtime,
since nothing calls setRecorder yet — the gate failed honest objects for
reasons that were about the evidence, not the candidate: a declared
outputSchema failed on a {} probe, and a real handler map scored 0/N killed.
Only contract-free sources passed, which is exactly backwards. An empty store
now returns an unverified PASS with each check saying so, and probes nothing.
The mutation gate also failed open on the one input it could not read.
generateMutants wrapped every source as a function body, so the canonical
handler map did not parse and a syntax error and a clean source were both
"no mutation points". It now returns null for "parsed under no dialect", and
that fails the mutation check outright; zero sites from a real parse still
passes.
'_http' cassettes are raw traffic captured on the object's behalf, not method
calls. Replaying them asked the invoker for a handler no object can have, so
any object that had ever recorded could never be healed again. They now
participate only as HTTP stubs.
Handlers were invoked unbound, so `this` was undefined inside them. The house style is a thin handler over a private helper — `return this.shape(rows)` — and every such object failed judgment on a `this` the gate itself withheld. Handlers are now bound to a minimal stand-in for ScriptableAbject's handler proxy: sibling handlers, the call/dep/find shims, an inert data/saveData/emit/ changed/observe, assert-like ensure/invariant, and an id. Nothing could interrupt a candidate that never returned, either: the vm's timeout is synchronous-only, so a mutant that flips a loop guard and awaits inside it hung evaluate forever. Compile and invocation now race a deadline, default 5s and injectable for tests. A timing-out mutant is killed by replay's catch; a timing-out candidate fails it.
A verdict was keyed on the source alone, so redrafting the manifest kept it valid — even though the schema and relation checks are judgments of the source AGAINST those declarations. sourceDigest becomes verdictDigest(source, methods). It was also keyed on nothing at all where the object was concerned. opFitness read cassettes for state.targetObjectId, while deploy_update can resolve a different explicit objectId/targetName — one it stamps AFTER the old gate ran. A verdict built from one object's recorded traffic could therefore wave a deploy onto another. The judged target is recorded and the gate refuses a mismatch; deploy_update now resolves its target before consulting the gate. On a heal the agent edits source without redrafting a manifest, so methods came back empty and schema/relations were vacuous. opFitness now reads the live target's manifest via describe, cached so the deploy gate recomputes the same digest, with a note in the op summary when the read comes back empty or fails.
…e is Two verdicts nothing exercised: a source whose recording kills only one of its four mutants (0.25, below the 0.8 threshold) must fail, and the whole dialect an LLM actually writes — parenthesized handler map, a helper reached through `this`, an HttpClient call checked with res.ok and parsed out of res.body, against a cassette with a raw body — must pass every check with the mutation gate armed. The comments still described a world with two mechanical checks and no fitness gate, which is now the only non-mechanical reason a deploy is refused.
andreBurnt
force-pushed
the
ratchet/c2-fitness-gate
branch
from
August 24, 2026 14:20
f7cd197 to
f4392b7
Compare
An object whose only recorded traffic is `_http` has a store that is not
empty but is unattributed: `replayable` filters those cassettes out, so
every method-level check probes with a single `{}` call. A method that
needs arguments throws on it, `checkRelations` swallowed the throw, and
the function fell through to "all declared relations hold" having
evaluated nothing — on data whose own recording violated the relations
it declared.
checkSchema already had the guard (`validatedCount === 0` fails closed);
relations did not. It now counts probes that produced an output and,
when none did, reports the methods it could not judge. The pass is kept
rather than inverted, matching `evaluate`'s no-evidence path: an
unverified pass, honestly labelled, and the mutation gate still refuses
to certify a candidate nothing can kill.
The verdict line the loop reads listed check NAMES, so three vacuous
passes rendered identically to three earned ones. `summarizeVerdict`
now names the checks that verified nothing.
Nothing on this branch can reach that state — `setRecorder` has no
non-test callers and no writer for `cassettes:<objectId>` exists yet —
but the recorder wiring in the next PR makes `_http`-only the normal
shape of a populated store.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Owner
Author
|
Superseded by the upstream PR: mempko#11 Same branch and same head commit. This one was staged on the fork while the design question was open, so it has no reason to stay open in parallel. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
C2 of the Ratchet Proposal (PR 1 of 5, staged on the fork pending design buy-in on the manifest shape — do not merge upstream before that).
What this adds
effects: 'read'|'act',outputSchema(JSON Schema), metamorphicrelations, and aknownEntityprobe — all optional, legacy manifests untouched.deploy_spawn/deploy_updaterefuse without a passing verdict digest-bound to the exact draft source, its manifest methods, and the target object.runSandboxedwith the documented HttpClient response shape ({status, statusText, headers, ok, body:string}), a handler-proxythis, and a hard invocation deadline. With zero cassettes the verdict is an honest unverified pass — the gate says what it does not know instead of inventing probes.Known seams (deliberate, follow-up PRs)
compose_organism/extract_organelledeploy already-existing source without the fitness gate.getBase64/fetchBase64bypasses the recorder (separate code path); binary fetches are invisible to the gate.HttpClientis stubbed in the sandbox invoker; WebFetch/WebParser/Storage-based objects fail closed with 'unstubbed I/O' until PR 2 stubs them.Honest scope: the gate is wired and enforced; cassette recording starts accumulating evidence only once PR 2's harness drives traffic, so today's verdicts on first-create rest on compile + (where evidence exists) schema/relations/mutation.
Tests: 51 passing across 8 files (
pnpm tsx --test src/core/manifest-contract.test.ts src/protocol/{cassette,fitness,mutants,cassette-recorder}.test.ts src/objects/object-creator-fitness.test.ts src/objects/capabilities/http-client-recorder.test.ts src/protocol/health-monitor.test.ts);tsc --noEmitclean; no new dependencies (ajv + acorn were already present).