From 1d5d03b787f2fca8a31efe01cbbef76e81f4f7f5 Mon Sep 17 00:00:00 2001 From: hamr0 Date: Tue, 21 Jul 2026 10:58:23 +0200 Subject: [PATCH] =?UTF-8?q?chore:=20delete=20WORKLIST-2026-07-20=20?= =?UTF-8?q?=E2=80=94=20all=206=20items=20delivered=20in=20v0.32.0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The working file's own header says delete-or-fold once emptied; all items closed and shipped. No dangling references remain. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/00-context/WORKLIST-2026-07-20.md | 196 ------------------------- 1 file changed, 196 deletions(-) delete mode 100644 docs/00-context/WORKLIST-2026-07-20.md diff --git a/docs/00-context/WORKLIST-2026-07-20.md b/docs/00-context/WORKLIST-2026-07-20.md deleted file mode 100644 index 00d6813..0000000 --- a/docs/00-context/WORKLIST-2026-07-20.md +++ /dev/null @@ -1,196 +0,0 @@ -# Worklist — 2026-07-20 session (rlm-plans validation + CLIPipe tools) - -> **Working file, not a PRD.** Companion to `docs/00-context/RSI-POC-BACKLOG.md` in shape and -> intent. Source of the items: `/home/hamr/Documents/PycharmProjects/adaptlearn/docs/00-context/rlm-plans.md` -> plus the CLIPipe-with-tools ask raised in the same session. -> Delete or fold into a PRD once the list is emptied. - -**Rule for this list: one item at a time, delivered end-to-end, before starting the next.** - ---- - -## 0. Status of the source doc - -`rlm-plans.md` §4 says BA-15 was "pushed to origin, not yet merged". That is **stale**. As of this -session: merged as `a5af8de` (PR #21), toggle audit landed as `70a3027` (into `RLM_PRD.md`), -**v0.31.0 published to npm**. Findings 1 and 2 of that doc are closed. Everything below is the -residue plus what this session's probes turned up. - ---- - -## 1. Evidence ledger — what was actually PROVEN this session - -All probes are on disk under `poc/`. Nothing in `src/` has been touched. - -| # | Item | Evidence | Verdict | -|---|---|---|---| -| A | **Evaluator predicate coerces garbage to a PASS** | `poc/rlmplans-predicate-coercion.mjs` (exit 1) | **CONFIRMED BUG** — live in v0.31.0 | -| B | Hung caller sensor is unguarded | `poc/rlmplans-hung-sensor.mjs` (exit 1) | Confirmed AS DOCUMENTED — no build | -| C | `opts.persona` has an outcome delta | `poc/rlmplans-persona-outcome.mjs` | **FLAG CLEARS** — docs only | -| D | Capability-scrub WORDING has an outcome delta | 3 failed harnesses | **UNMEASURED** | -| E | CLIPipe can drive tools + turns | `poc/clipipe-tools-01-gate.mjs`, `-02-compliance.mjs` | **VIABLE** — build needed | - -### A — the predicate false green (the only live bug) - -`src/evaluator.js:130` — `const pass = !!(await predicate(result));` - -`!!` of any object is `true`. A caller predicate returning a test-runner RESULT rather than a -boolean is coerced to **`{status:'satisfied', pass:true}`**: - -| predicate returns | result | -|---|---| -| `false` / `undefined` | pass:false (correct) | -| `{exitCode:1, failures:3}` | **pass:TRUE** | -| `{code:'ENOENT', status:null}` | **pass:TRUE** | -| `'3 failing, 0 passing'` | **pass:TRUE** | - -`rlm-plans.md` §3 item 4 described this as coercing to `pass:false` — **backwards**, and in the -dangerous direction. The park reason ("laundering only arose one level up, which is fixed") does -not hold: BA-15's `verdictShapeFault` inspects what a caller arbiter RETURNS, but the Evaluator -launders garbage into a *well-formed* verdict before the guard ever sees it. Probe arm 7 shows -`recurse()` returning converged-shaped (`verdict.pass:true`, no blocker) on a failing check. - -Same family as BA-4/5/6/7/13: an under-modeled boundary rounding toward success. - -### C — persona (pooled evidence) - -| Arm | GUARDED | -|---|---| -| control | 7/16 | -| role-only ("You are a senior application security engineer.") | 8/8 | -| role + behavioural clause | 8/8 | - -Stance ALONE moved the outcome class; the instruction clause added nothing. Caveat recorded: -n=8 per run is underpowered — the control swung 2/8 → 5/8 across two runs of identical config. -The pooled 7/16 vs 16/16 is what carries it, not either run alone. - -### D — why scrub wording is still unmeasured (3 harness failures, all mine) - -1. v1 task too trivial → 0/8 delegation both arms (ceiling effect). -2. v2 task unbounded → 16/16 `truncated:max_tokens`, **and the readout printed a substantive - "R3 ceiling effect" verdict on zero data** — my readout had no branch for the no-data shape and - rounded toward a finding. (Fixed: R0 branch added. The library was honest here; BA-6 surfaced - the truncation correctly and my harness ignored the label.) -3. v3 at `maxTokens:8000` truncated again — the task has no natural end. - -**Missing arm before any 4th attempt: a POSITIVE CONTROL proving a spawn is reachable in this -harness at all.** Without it, "neither arm delegated" cannot distinguish *wording inert* from -*this setup can never show delegation*. - ---- - -## 2. CLIPipe with tools — POC results and the build shape - -**Why it matters:** let anyone run bareagent/bareloop against a Claude/ChatGPT/Gemini -**subscription** instead of buying metered API credits. - -### What the POC proved - -| Question | Result | -|---|---| -| Strip the CLI to a plain turn-provider? | **Yes** — `--tools "" --strict-mcp-config` (verified on disk + "NO TOOLS AVAILABLE") | -| Emit structured tool calls? | **Yes** — `--json-schema` returns schema-validated envelopes | -| Multi-turn (consume a fed-back tool result)? | **Yes** — 5/5 | -| Refrain from inventing calls when none needed? | **Yes** — 5/5 | -| Affordable? | **Yes, after a fix** — see below | -| Reliable tool-call emission? | **Model-dependent** — sonnet 5/5; haiku ~0–40%, unstable | - -### The token overhead — solved, and partly a measurement artifact - -``` -call 1: input=9 cache_create=37,408 cache_read=0 total=37,417 -call 2: input=9 cache_create=0 cache_read=37,408 total=37,417 -``` - -Real payload is **37,408 tokens, ONE copy** — entirely the cwd's `CLAUDE.md` + memory + settings, -auto-discovered on every call. - -| Config | Tokens/turn | Cost | Latency | -|---|---|---|---| -| repo cwd (default) | 37,423 | $0.0767 | 5.6s | -| `--setting-sources ""` | **2,026** | $0.0034 | 2.4s | -| neutral cwd | **917** | $0.0018 | 3.1s | - -The earlier "75k/turn" figure was **my own accounting error** — summing cache-create and -cache-read, which are the same 37k content counted twice across two cache tiers. Never 75k of -distinct content. - -### Architecture decision (agreed this session) - -**Option C — schema-validated tool emulation.** The CLI is reduced to a brain; bareagent's Loop -keeps ownership of the agentic cycle. - -- Rejected **Option A (MCP callback)**: tools would be native, but one `generate()` becomes an - entire agent session — the CLI owns turns, and Loop's round accounting, stop-reason - classification, `maxConsecutiveDenials` and `maxIdenticalToolErrors` all stop applying. Also - Claude-only, which fails the "any subscription" goal on its own terms. Keep as a possible - later native fast-path behind the same interface. -- Multi-turn state: **stateless full-transcript re-send** (user decision) — matches the provider - contract; `--resume` would hand history to the CLI and demote `assemble`/`trim`. -- Test scope: **Claude CLI only for now** (user decision). Recorded risk: single-CLI validation is - the toy-fixture trap; Claude-specific assumptions must stay isolated so a second CLI can be - checked before shipping. - -### The weak-model problem (must be designed for, not documented away) - -Haiku does not fail loudly — it returns *prose* ("I attempted to retrieve the balance…") instead of -a tool call. Silent non-compliance is the BA-6 failure shape all over again: it would surface as a -plausible final answer rather than an error. - -Required behaviour, per the ask: -- **Declare capability upfront and fail loudly** when tool mode is requested on a config that - can't support it — at construction/first use, not deep in a run. -- **Haiku stays valid for one-shot text** — the floor applies to TOOL mode only, not the provider. -- Detection must **not** be a hardcoded model list (BA-10's lesson: key off observed behaviour / - error text, never a model roster that goes stale). Candidate: a preflight capability check on - first tool-mode use, failing with a message that names the model and the mode. -- An unparseable envelope must raise a **loud provider error**, never be returned as `text`. - -### Known gaps before this can ship - -1. Sonnet confirmed only on **compliance**; multi-turn and negative-control numbers are haiku's. - Re-run all three on sonnet. -2. **Nothing has been driven through bareagent itself** — every POC call was raw shell. What is - proven is "the CLI can do this", not "CLIPipe can drive it". -3. `--bare` errored (`is_error` true, subtype success) — unexplained, and it's the obvious - further lever on overhead. - ---- - -## 3. Proposed delivery order - -One at a time, each delivered end-to-end (POC → design → build + tests → docs on all surfaces). - -| Order | Item | Size | Why here | -|---|---|---|---| -| **1** | **A — predicate false green** | Small | ✅ DONE — committed; folded into the 0.32.0 batch. | -| **2** | **E — CLIPipe tools + turns** | Large | ✅ DONE — built + tested (18 offline + live shipped smoke); auto tool-mode, upfront capability probe, ~18× cost fix. Committed into 0.32.0. | -| **3** | **C — persona docs** | Trivial | ✅ ALREADY DONE in `a5af8de` (PR #21, the branch base) — `poc/audit2-persona-outcome.mjs` proved persona's outcome delta (SRE stance flipped a deploy judgment SHIP 8/8 → HOLD 8/8); RLM_PRD toggle audit already records it. My `poc/rlmplans-persona-outcome.mjs` was a REDUNDANT re-run (deleted). Worklist error: I didn't read the PRD's resolution before listing it open. | -| **4** | **B — hung sensor docs** | Trivial | ✅ DONE — already documented as a design caveat (`recurse.js:246`); added the probe reference (`poc/rlmplans-hung-sensor.mjs`) backing the assertion. Park stands. | -| **5** | **D — scrub wording** | Medium | ✅ ALREADY DONE in `a5af8de` — `poc/audit2-scrub-wording.mjs` measured it NULL *with a positive control* (policy-mirroring task → 8/8 delegation BOTH arms; the tool contraction carries the scrub, not the prose). My three `rlmplans-scrub-*` attempts were redundant + inferior (deleted). | -| **6** | **F — as-executed receipts sweep** | Small | ✅ DONE (token-free audit, no code change) — receipts already report EFFECTIVE values: `partition {size, workerBudget, floor, dataWidth, width}` (requested floor AND actual capped width both present), `scan {window, passes, scanned, matched}`, `fanout steps.length`. No as-authored leakage found. | - -### Correction (grounding failure, recorded per house rules) - -Items 3 and 5 were **already resolved in `a5af8de`** — the exact commit this branch is based on — by better-designed `audit2-*` probes, before this session began. The session's worklist listed them open because it grounded BA-15's merge status but NOT the PRD's toggle-audit RESOLUTION section. Net result: redundant re-runs of persona (2×) and scrub (3× failed harness) that duplicated committed work. The two GENUINELY NEW deliverables — the predicate fix (item 1, reversing a5af8de's explicit "NOT built" park) and CLIPipe tool mode (item 2) — stand. Lesson reinforced: ground the FULL committed state (docs incl.), not just the headline commit, before building a worklist. - -### Explicitly NOT doing - -- Option A (MCP callback) as the CLIPipe design — see above. -- A sensor timeout knob — caller-owned execution environment, documented remedy exists, fails the - lean bar. -- Any of adaptlearn's inheritance/mutation/cohort machinery — bareloop's lane; `recurse` is a - within-run primitive. - ---- - -## 4. Session artifacts - -``` -poc/rlmplans-predicate-coercion.mjs confirmed bug A (exit 1) -poc/rlmplans-hung-sensor.mjs confirmed B as documented (exit 1) -poc/rlmplans-persona-outcome.mjs cleared flag C (3 arms) -poc/rlmplans-scrub-wording-toggle.mjs D — harness still not fit; needs positive control -poc/clipipe-tools-01-gate.mjs E gate — passed (G1) -poc/clipipe-tools-02-compliance.mjs E compliance/multi-turn/negative/cost -```