diff --git a/.agents_workspace/DECISION_LOG.md b/.agents_workspace/DECISION_LOG.md deleted file mode 100644 index ac59010..0000000 --- a/.agents_workspace/DECISION_LOG.md +++ /dev/null @@ -1,2081 +0,0 @@ -# Decision Log - -### Entry 1 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-05-26 -**Task:** Improve ceh-blog skill triggering reliability and token efficiency. - -**Context:** The request — "go through the skill description and adapt them based on best practices, also make them token efficient without losing context" — was ambiguous about whether "them" meant the `description` frontmatter only or the full SKILL.md bodies. -**Decision:** Scoped the change to the four `description` frontmatter fields only. The description is the always-loaded field that drives triggering (the stated problem) and counts against the persistent skill-selection context budget, so it is the correct lever for both "best practices" and "token efficient." Bodies were left intact: they load one skill at a time (negligible always-on cost), `CROSS_REFERENCES.md` documents that the repo deliberately inlines duplicated body content, and trimming prose risks the "losing context" the user warned against. Rewrote each description to lead with what the skill does, keep high-signal trigger phrases, and preserve the inter-skill routing ("use X instead") — while cutting redundant framing (~95 words → ~75 words each). -**Impact / Risk:** Low. Fewer literal trigger phrases could marginally reduce recall; mitigated by keeping the most distinctive phrases and the discriminating input-state clause for each skill. Version bumped PATCH (1.0.2 → 1.0.3) in `plugin.json` and `marketplace.json` per repo versioning rules. -**Outcome:** Four descriptions rewritten; committed on branch `fix/blog-skill-triggering`. - ---- - -### Entry 2 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-05 -**Task:** Execute the plugin reorganization (`.agents_workspace/PLUGIN_REORG_PLAN.md`). - -**Context:** The user asked to "create a new feature branch and implement everything in the reorg." -The plan itself prescribes six phases, each shipped as its own PR with its own version bumps. -**Decision:** Honored the explicit instruction — one branch (`refactor/plugin-reorg`) — but kept the -six phases as six separate, self-consistent commits (each updates its own manifests/marketplace) so -the history stays reviewable phase-by-phase. The user's direct instruction overrides the plan's -per-PR guidance (authority hierarchy). -**Impact / Risk:** A single large PR instead of six; mitigated by clean per-phase commit boundaries. -**Outcome:** Six phase commits + one review-fix commit on the branch. - ---- - -### Entry 3 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-05 -**Task:** Distribute the `postgresql` skill content (Phase 3). - -**Context:** The plan's per-skill map says "TRIM + MERGE postgresql into data moments -(`domain-modeling` + migrations)" but also lists the target as `ceh-python-service`. `domain-modeling` -is explicitly kept stack-agnostic in `ceh-architecture`, so folding Postgres-specific DDL into it -would re-pollute it with stack specifics — contradicting the reorg's goal. -**Decision:** Split by concern instead of dissolving the skill: tenant-isolation/query rules → -`asyncpg`; migration safety → `alembic`; and the Postgres schema-design content kept as a trimmed -`postgresql` skill **moved into `ceh-python-service`** (it is genuinely Postgres + service specific). -**Impact / Risk:** One more skill in `ceh-python-service` than a strict "merge everything" reading; -keeps `domain-modeling` stack-agnostic. -**Outcome:** `ceh-python-service` has `postgresql` (schema design), `asyncpg` (access + tenant -isolation), `alembic` (migrations). - ---- - -### Entry 4 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-05 -**Task:** Test agents for `ceh-python-library` (Phase 4). - -**Context:** The plan's agent table notes the python tester agents are in `ceh-python-service` -"(share with library)" — ambiguous between duplicating the agents into the library plugin and -leaving them in the service plugin. -**Decision:** Did **not** duplicate the three tester agents (and their scripts) into -`ceh-python-library`. Library testing standards are carried by the duplicated `python-testing` skill; -duplicating heavy agent + script assets exceeded what Plan §4 scoped (env + testing *skills* only). -**Impact / Risk:** A pure library project does not get dedicated tester agents. Can be added later if -needed. -**Outcome:** Tester agents remain solely in `ceh-python-service`. - ---- - -### Entry 5 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-05 -**Task:** Handle `scripts/sync-stubs.ps1` (Phase 6). - -**Context:** The script syncs stub reference files from -`ceh-python-backend/skills/python-backend/references` to other plugins — a path that did not exist -even before the reorg, and all of whose source/destination plugins were renamed or removed. -**Decision:** Deleted the script as dead code. The duplication mechanism of record is -`CROSS_REFERENCES.md` with same-session manual propagation (per CLAUDE.md), not an automated sync. -**Impact / Risk:** None — the script was non-functional. Only referenced historically in CHANGELOG. -**Outcome:** `scripts/sync-stubs.ps1` removed. - ---- - -### Entry 6 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-05 -**Task:** Versioning for the reorg (Phase 6). - -**Context:** Renamed plugins are breaking changes; the plan accepts this. Unclear what bumps the -unchanged-but-affected plugins should take. -**Decision:** Bumped all four renamed plugins and `ceh-git-workflow` (lost the `gitignore` skill, -`open-pr` absorbed the definition-of-done gate — a breaking content removal) to **3.0.0**. New -plugins (`ceh-python-library`, `ceh-scaffolding`) start at **1.0.0**. Untouched plugins keep their -versions. The repo CHANGELOG entry is **3.0.0**, but the repo git tag was **not** cut on the feature -branch — tagging is a release action deferred to merge-to-main. -**Impact / Risk:** Breaking version jumps for renamed/`git-workflow` plugins (intended). -**Outcome:** Marketplace and per-plugin manifests consistent; CHANGELOG 3.0.0 added. - ---- - -### Entry 7 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-05 -**Task:** Post-review fix — orphaned "layer boundaries" invariant. - -**Context:** After `repository-structure` moved to `ceh-scaffolding`, the architecture SessionStart -hook and README still attributed the always-on "layer boundaries" invariant to `domain-modeling`, -which no longer contained that content. Options: drop the invariant, point the hook at a scaffolding -skill, or give it a home in `domain-modeling`. -**Decision:** Added a "Layer Boundaries" section to `domain-modeling`. Layer boundaries are a -design-time invariant that belongs with domain modeling; the concrete directory layout stays in -`ceh-scaffolding`. This keeps the always-on hook tag accurate and the depth reference in the same -plugin. Registered the resulting overlap with `scaffold-python-service` in `CROSS_REFERENCES.md`. -**Impact / Risk:** New duplicated block (tracked). -**Outcome:** Fixed in commit `fix(reorg): correct post-review inconsistencies`. - ---- - -### Entry 8 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-05 -**Task:** Remove `msg.txt` from history (user-directed). - -**Context:** Per-phase commits had inadvertently tracked the temp commit-message file `msg.txt` -(swept in by `git add -A`). The user asked to remove it from all commits while not altering the -old commits' content for the review fixes. -**Decision:** Committed the review fixes first as a new commit, then rewrote the branch with -`git filter-branch --index-filter` over the reorg range to strip `msg.txt` from every commit, and -purged the `refs/original` backup + reflog + gc. Future commit messages are written to a temp file -outside the repo to prevent recurrence. -**Impact / Risk:** Commit SHAs were rewritten. Safe — the branch existed only locally at the time; -later pushed to origin fresh. -**Outcome:** `msg.txt` absent from all commits and all objects (`git rev-list --all` = 0 matches). - ---- - -### Entry 9 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-05 -**Task:** Audit every plugin for a missing SessionStart invariants hook; add one where a skill is invariant (passive). - -**Context:** Two ambiguities. (1) Which plugins need a hook: the criterion is "a skill is invariant" — a rule that must always hold but fires on implicit mid-turn decisions with no prompt signal, so skill auto-load under-fires. (2) The repo versioning rule (`CLAUDE.md`) covers PATCH (content/description) and MINOR (new skill/agent), but is silent on adding a hook. -**Decision:** Of the nine hookless plugins, only `ceh-python-library` qualifies: its `python-environment` skill carries the same passive style/type invariants (type hints, no `Any`/`# type: ignore`, strict mypy, ruff naming, minimal deps) that `ceh-python-service` — its registered shared-standards twin in `CROSS_REFERENCES.md` — already injects via a SessionStart hook. The other eight are moment/activity-triggered only (git verbs, deploy/incident, scaffold, blog/doc authoring, summarize, retrospective) or have no skills (`ceh-dev-tools`), so no hook. Added `hooks/hooks.json` + `hooks/load-invariants.js` mirroring the service plugin, scoped to the style & types + dependencies invariants (dropped the service's web-only security/observability and the Pydantic/asyncio lines). Bumped the version MINOR (1.0.0 → 1.1.0): a SessionStart hook is a new always-on functional component, comparable in impact to a new skill/agent, not a content/description tweak — so MINOR fits better than PATCH despite the rule not naming hooks. -**Impact / Risk:** Low. The hook only injects always-on context; the duplicated style block is now noted in the `CROSS_REFERENCES.md` python-environment entry so edits propagate. README gained a parallel `## Hooks` section. -**Outcome:** Hook smoke-tested (valid JSON, 1286 bytes). Files written; not yet committed. - ---- - -### Entry 10 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-05 -**Task:** Make `implement-from-plan` and `review-against-plan` handle version-tagged plan filenames (e.g. `SKELETON_v2.md`, `ITER_03_v2.md`). - -**Context:** The request — support pre/postfixes like `v2`/`v3` indicating different app versions — left open how multiple versions should interact: whether discovery should merge all matching files or isolate them, and how pointer/iteration resolution should behave when both `v1` and `v2` plans coexist. -**Decision:** Tag is parsed from the filename as an optional separator-bound (`_`/`-`/`.`) prefix or suffix on the `SKELETON`/`ITER_NN` base name; files sharing a tag form a **plan family** (untagged = default). Initial cut isolated families ("never mix tags"); the user corrected this — versions are **not** isolated: a later version declares its base via a `depends_on` frontmatter field (list of base-version files / tag) and inherits every section it does not re-specify. Adopted the `depends_on` field plus a fall-through Resolution Order: resolve a section within the target family first, then follow `depends_on` into the base version until found. A version's SKELETON is optional (a version may be ITER files alone depending on the prior SKELETON). Schema (`plan-schema.md`) is authoritative; both skills' Step 1 reference it. -**Impact / Risk:** Low. Untagged single-version projects behave exactly as before (no `depends_on`, single family). `depends_on` is additive/optional, so existing plans are unaffected. Neither skill is registered in `CROSS_REFERENCES.md`, so no propagation. Version bump deferred to commit time per repo rules. -**Outcome:** Edited `plan-schema.md` (File Naming + cross-version `depends_on` subsection + fall-through Resolution Order) and Step 1 of both SKILL.md files. - -**Follow-up (same session):** User supplied the canonical section-spec doc that generates the plans. Aligned the schema to it: `depends_on` references artifact **stems** (`[SKELETON, ITER_01]`), not `.md` filenames; added `mvp_target` (SKELETON) and `mvp` (ITER, `true` once on the final iteration = sequence terminator); noted the SKELETON body's `## Out of MVP scope` block. Reframed `depends_on` as the standard per-iteration dependency field (stems carry the version tag, so one mechanism covers both same-sequence chaining and cross-version inheritance) and rewrote Resolution Order to walk that chain backward (nearest artifact whose `sections_changed`/`sections` lists the section; never forward). Both skills' Step 1 now key on `depends_on` order and the `mvp: true` terminator. - ---- - -### Entry 11 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-05 -**Task:** Cut a release for the plan-version-variants change (changelog + tag + GitHub release). - -**Context:** Two version layers per repo rules: the plugin already bumped 2.4.3 → 2.4.4 in the merged PR; the repo git tag needed selecting. The last repo tag is `v3.1.0`. The rule: MINOR when any plugin adds skills/agents, PATCH for content-only. -**Decision:** Chose repo tag `v3.1.1` (PATCH). The merged change added no skills or agents — it edited existing skill bodies and the shared `plan-schema.md` and bumped one plugin a PATCH — so it is content-only by the repo-tag rule. Changelog entry filed under `[3.1.1]` with a single-plugin version table (`ceh-agent-coding-contract` v2.4.4) and a `Changed` section (no `Added`, since nothing new was introduced). -**Impact / Risk:** Low. Tag is a deployment snapshot, not load-bearing for auto-update. -**Outcome:** Changelog updated; tag `v3.1.1` created and pushed; GitHub release published. - ---- - -### Entry 12 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-09 -**Task:** Create the `ceh-plan-build-review` plugin; move `implement-from-plan` and `review-against-plan` into it from `ceh-agent-coding-contract`. - -**Context:** Three forks left open by the request: (1) the user offered to rename the plugin "if needed"; (2) the semver bump for removing two skills from `ceh-agent-coding-contract` (the repo rule only names PATCH for content and MINOR for additions); (3) the moved `plan-schema.md` and the user-authored planners' `section-specs.md` describe the same artifact format but diverge on `mvp`/`mvp_target` placement (schema: `mvp_target` on SKELETON, `mvp: false` shown on iterations; to-mvp planner: both fields on the terminator iteration only; iterative planner: neither field). -**Decision:** (1) Kept `ceh-plan-build-review` — it names the use case (plan → build → review loop) per the repo's naming convention. (2) Bumped the contract plugin MINOR (2.4.6 → 2.5.0), following the v2.3.0 precedent where deleting the `execution-modes` skill was a MINOR bump. (3) Did not reconcile the schema divergence — `plan-schema.md` declares `mvp`/`mvp_target` optional so the artifacts stay compatible; registered the duplication and divergence as a new `CROSS_REFERENCES.md` entry instead and flagged it to the user. Placed the plugin in the "Use-case workflow" tier. -**Impact / Risk:** Skill invocation paths change (`/ceh-agent-coding-contract:implement-from-plan` → `/ceh-plan-build-review:implement-from-plan`); users must install the new plugin to keep the implement/review skills. The `mvp_target` placement divergence remains and may need reconciling in a follow-up. -**Outcome:** Plugin created with four skills (two new planners restructured under `skills/`, two moved); manifests, READMEs, CLAUDE.md, CROSS_REFERENCES.md, and CHANGELOG updated; committed on `feat/plan-build-review-plugin`. - -**Follow-up (same branch):** User declared the planner skills' `section-specs.md` the golden standard and the reference duplication intentional (skills are used standalone outside the plugin — the repo's only sanctioned exception). Reconciled `plan-schema.md` to the terminator convention (`mvp: true` + `mvp_target` + `## Out of MVP scope` on the final iteration only; SKELETON carries no MVP fields; non-terminal iterations omit `mvp`). Judgment call: extended the self-containment rationale to `review-against-plan`, giving it its own `references/plan-schema.md` copy and rewriting its two relative links into `implement-from-plan/` — the user asked only for the README note, but a cross-skill path contradicts the stated standalone-use requirement. CROSS_REFERENCES entry now lists four files with the to-mvp producer copy as golden standard. Versions unchanged — v1.0.0 / [3.3.0] are still unreleased on this branch. - ---- - -### Entry 13 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-10 -**Task:** Update ceh-blog skills for personal (non-influencer) voice and series continuity, per tmp.md. - -**Context:** tmp.md explicitly audits only `blog-writer` and `blog-interviewer`. But `blog-editor` duplicates the same six post-type structure templates and its "Closing" checklist mandates "land with conviction or a clear next step" — left unchanged, the editor would diagnose the new open-thread/reserved-verdict endings as fizzling and edit them back into conviction closers, defeating the goal. `blog-repurpose` also carries CTA language (thread closing tweet, LinkedIn closing question). -**Decision:** Extended the edits to `blog-editor` (template endings mirrored, Closing checklist rewritten, banned-tells and series-continuity checklist items added) and registered both duplicated blocks (Voice section, Structure by Post Type) in `CROSS_REFERENCES.md` per the Cross-Reference Rule. Left `blog-repurpose` untouched: its CTA conventions are platform-native to social formats (Twitter/LinkedIn), not the blog voice tmp.md targets. -**Impact / Risk:** Low. blog-editor scope extension is the conservative reading of the propagation rule; repurposed social posts will still sound like social posts, which may contrast with the quieter blog voice — flagged for the user rather than changed. -**Outcome:** Three SKILL.md files updated on branch `feat/blog-personal-voice`; ceh-blog bumped 1.0.4 -> 1.0.5. - ---- - -### Entry 14 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-10 -**Task:** Revise the agent-coding-contract skill (validation policy, scope wording, authority hierarchy, subagent stop path, log mechanics) per evaluation review. - -**Context:** Most changes were user-specified. One fork was not: the user's global `~/.claude/CLAUDE.md` said "For new behavior, write tests" while the user's in-session clarification of the contract's intent was that models must not write unit/integration tests unprompted. With project/user `CLAUDE.md` now placed above the contract in the authority hierarchy (also per this session), leaving the global line intact would have overridden the new Validation Policy in every session. -**Decision:** Updated the global `CLAUDE.md` line to "Do not write or run tests unless asked (quick syntax/type/sanity checks on edited code are fine)" — treating the in-session statement as the freshest expression of intent and keeping the two files consistent. Also relabeled the log's duplicate "Entry 3" heading (2026-06-10 blog entry) to "Entry 13" to restore sequential IDs, since the contract now mandates next-sequential-integer IDs. -**Impact / Risk:** The global preference change affects all projects, not just this repo; flagged to the user for revert if "write tests for new behavior" was intentional as a global default. -**Outcome:** SKILL.md and global CLAUDE.md updated in the same session; plugin version bump deferred to commit time per repo rules. - ---- - -### Entry 15 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-11 - -**Task:** Implement the `skills-sync` tool (Python/bash/PowerShell/HTML) per the uploaded -`skills-sync.md` spec. - -**Context:** The spec describes a standalone CLI/HTML tool that copies *other* projects' Claude -Code skills into `/.claude/skills/`. It is meta-tooling for managing skill installs -across arbitrary repos — not a `ceh-*` plugin, skill, or agent itself, so none of the -"Adding a Skill/Agent" conventions in `CLAUDE.md` (plugin.json bumps, marketplace.json, -README skill tables) apply, and it doesn't fit `ceh-dev-tools` (agents-only, scoped to -repo-exploration agents). The repo had no existing top-level location for non-plugin tooling. -**Decision:** Created a new top-level `tools/skills-sync/` directory holding all four -implementations (`skills-sync.py`, `skills-sync.sh`, `skills-sync.ps1`, `skills-sync.html`), -parallel to the `ceh-*` plugin directories and `docs/`. No README was added per the -no-unrequested-docs rule; each script's own `--help`/usage text and the HTML UI are -self-documenting. -**Impact / Risk:** Low — new isolated directory, no existing files touched. If the user later -wants this distributed differently (e.g. as part of `ceh-dev-tools`, or with a README), it's a -simple move. -**Outcome:** Implementation proceeds under `tools/skills-sync/`. - ---- - -### Entry 16 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-11 - -**Task:** Implement `tools/skills-sync/skills-sync.ps1` per the `skills-sync.md` spec. - -**Context:** The spec's PowerShell platform note explicitly offers a choice: "5.1-compatible if -possible (no `??` operator if targeting 5.1; PS7-only is acceptable if documented — user runs -`pwsh`)." `pwsh` is unavailable in this sandbox, so neither option could be executed/tested -either way. -**Decision:** Targeted PowerShell 7+ (`pwsh`) for cross-platform consistency with the bash -version, and documented this in the script's header comment block (lines 11-15). No PS7-only -operators (`??`, `?.`, ternary `?:`) are actually used in the script, so it likely also runs -under 5.1, but it is tested/supported only against `pwsh`. -**Impact / Risk:** Low — if 5.1-only support is required later, the script likely needs no -changes (no PS7-only syntax used), just removal of the "PS7+" claim from the header after -verification on Windows PowerShell 5.1. -**Outcome:** `skills-sync.ps1` ships documented as PS7+ (`pwsh`); not executed in this sandbox -(no `pwsh` available) — syntax-reviewed only. - ---- - -### Entry 17 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-11T00:00:00Z -**Task:** Make duplicated skill names unique across plugins (python-environment, python-testing) - -**Context:** The user asked for unique, recognizable skill names but did not specify a naming -scheme, whether to rename one or both copies of each pair, or how to size the version bumps. -**Decision:** Renamed both copies of each pair to plugin-qualified names mirroring the plugin -directory names: `python-service-environment` / `python-service-testing` (ceh-python-service) and -`python-library-environment` / `python-library-testing` (ceh-python-library). Symmetric renames -keep the name → plugin mapping obvious in hook tags, agent `skills:` lists, and cross-plugin -references. Version bumps: MINOR for the two Python plugins (the skill-name surface changed — -closest to the repo's "new skills" MINOR rule), PATCH for ceh-scaffolding and ceh-web-frontend -(reference-text updates only). Historical records (CHANGELOG.md, .agents_workspace/PLUGIN_REORG_PLAN.md, old -DECISION_LOG entries) intentionally left with old names. -**Impact / Risk:** Users invoking the old skill names (`/ceh-python-service:python-environment` -etc.) must switch to the new names; auto-load behavior is unaffected (descriptions unchanged). -**Outcome:** All four renames applied; repo-wide grep shows no stale references outside -historical docs; new "Same Skill, Different Plugins" map added to CROSS_REFERENCES.md. - -### Entry 18 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-13T10:20:00Z -**Task:** Consolidate an external orchestration setup into a new `ceh-orchestration` plugin - -**Context:** The source setup (from another session) shipped as a slash command -(`.claude/commands/orchestrate.md`), three `.claude/agents/*` files, and a thinned root -`CLAUDE.md` skeleton. This repo has no `commands/` convention — every plugin delivers as -skills + agents — and ships no CLAUDE.md templates. Several design forks were unresolved: -delivery form for the orchestrate mode, plugin name, tier, agent names, and whether to ship -the CLAUDE.md skeleton. -**Decision:** (1) Delivered the `/orchestrate` slash command as a skill (`orchestrate`) with a -moment-triggering description, matching the repo's skills+agents-only convention (the source -summary itself names a skill as the alternative to a command). (2) Named the plugin -`ceh-orchestration`, classified as a Use-case workflow tier plugin. (3) Kept the original -worker names `explorer`/`executor`/`verifier` to preserve the delegation map the skill -references, adapting only frontmatter to repo conventions. (4) Folded the lean-root-CLAUDE.md -guidance inline into the skill as a cost lever rather than shipping a separate CLAUDE.md -template file — keeps the plugin self-contained and avoids overlap with the user-level -claude-md-management skills. (5) New plugin at v1.0.0; repo tag bumped MINOR to 3.6.0 (new -skills + agents). -**Impact / Risk:** Low — additive new plugin; no existing plugin changed. Generic agent names -(`explorer`/`executor`/`verifier`) are namespaced under `ceh-orchestration:` so collisions are -avoided, though the bare names are less self-descriptive than the repo's other agents. -**Outcome:** Plugin created (plugin.json, README, skill, 3 agents); marketplace.json, CLAUDE.md, -README.md, and CHANGELOG.md updated. - -### Entry 19 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-13T10:56:00Z -**Task:** Replace the custom `explorer` worker with Claude Code's built-in `Explore` agent - -**Context:** Entry 18 shipped `ceh-orchestration` with a custom `explorer` Haiku agent. The -user asked whether the built-in `Explore` agent makes more sense for that role. The plugin's -own source summary notes that only the built-in Explore/Plan agents skip `CLAUDE.md`, while -custom subagents always inherit it — and "trim what every subagent inherits" is cost lever #4 -in the skill. The explorer is the highest-fan-out, most-dispatched role, so the inheritance -tax matters most there. -**Decision:** Dropped the custom `explorer` agent and pointed the orchestrate skill's -delegation map at the built-in `Explore` for read-only locate/map/summarize work. Kept -`executor` (needs Edit/Write) and `verifier` (needs Bash + a controlled PASS/FAIL contract) -custom, as neither has a built-in equivalent. Rationale: built-in Explore is purpose-built for -fan-out search (reads excerpts, breadth hint) AND skips `CLAUDE.md`, so it carries the least -context tax — outweighing the loss of the custom agent's enforced terse return-format and -explicit Haiku routing (Explore is already concise and managed-cheap). Documented the one -tradeoff in the skill: Explore starts without repo conventions, so convention context must go -into the spec. No version bump — plugin still unreleased. -**Impact / Risk:** Low — removes one agent file and reduces per-dispatch context cost for the -exploration role. Searches needing repo-specific convention context now rely on the -orchestrator putting that in the spec rather than the agent inheriting it from `CLAUDE.md`. -**Outcome:** `explorer.md` deleted; skill delegation map + model routing, both READMEs, -CLAUDE.md, plugin.json, marketplace.json, and the pending CHANGELOG entry updated to the -built-in-Explore + executor/verifier shape. - -### Entry 20 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-18 -**Task:** Create the ceh-release-flow plugin (bump version, changelog, README, CLAUDE.md, open PR, merge, tag, release) composing existing repo skills. - -**Context:** The request listed concrete actions ("open pr, merge the pr, create tag, create release"). Ambiguous whether these described the new plugin's *capability* (just author the skill) or were also a *directive to execute* the release for the change that adds the plugin. -**Decision:** Did both — authored the plugin AND dogfooded it by running the full release flow on this change. The /goal framing ("treat the condition itself as your directive") plus the explicit verb list constitute durable in-session authorization for the outward-facing merge/tag/release actions, which otherwise need confirmation. Designed the skill to *reference* existing skills rather than duplicate them, so no CROSS_REFERENCES entry was needed. -**Impact / Risk:** Low/reversible-ish. Merged PR #22, pushed tag v3.9.0, published the GitHub release. Repo had no CI checks and no required approvals (consistent with prior solo-merged PRs), so the merge gate was trivially satisfied. Plugin v1.0.0; repo tag MINOR bump v3.8.2 -> v3.9.0. -**Outcome:** Plugin live; PR #22 merged to main; v3.9.0 tagged on the merge commit and released. - ---- - -### Entry 21 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-18 -**Task:** Add a new skill that is "the same as release flow but without PR and merge." - -**Context:** The request named neither a skill name nor a location. Two forks: (a) new plugin vs sibling skill in `ceh-release-flow`; (b) skill name. -**Decision:** Added it as a sibling skill `direct-release-flow` inside `ceh-release-flow` rather than a new plugin — it shares the plugin's use case (release orchestration) and the repo's organizing principle splits plugins on use case, not variant. Dropped the branch/open-pr/merge steps and reworked the pipeline to commit straight to an up-to-date `main`; the tag/release step now skips no commit since the bump lands directly. Delegation-only, so no CROSS_REFERENCES entry needed. Bumped plugin + marketplace to v1.1.0 (MINOR, new skill); did not cut a repo git tag (left for an explicit release). -**Impact / Risk:** Low. Documentation/skill addition only; no outward-facing actions taken. -**Outcome:** Skill authored; README (root + plugin) and version manifests updated. - ---- - -### Entry 22 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-19 -**Task:** Redirect all skill output paths from `docs/` to `.agents_workspace/`, updating READMEs and CLAUDE.md. - -**Context:** "All skills that write outputs to `docs/`" is ambiguous: `docs/` holds two distinct -things — agent session artifacts (`.agents_workspace/DECISION_LOG.md`, `LESSONS_LEARNED.md`) and -committed project documentation (`docs/adr/` ADRs, `docs/guide/` user guides). CHANGELOG entry 1190 -already records a deliberate split between "Claude session artifacts" and "shared developer -documentation." Forks: (a) which paths count as "outputs"; (b) whether to preserve the `claude_logs/` -subdir under the new root; (c) where to migrate the existing committed log. -**Decision:** Scoped "outputs" to the session artifacts only (`DECISION_LOG.md`, `LESSONS_LEARNED.md`), -because `.agents_workspace/` (dot-prefixed, "workspace") denotes ephemeral agent working files, not -committed deliverables. ADRs and user guides stay in `docs/` — they are product documentation. Plan -skills wrote to `/mnt/user-data/outputs/planning/` (not `docs/`); a follow-up instruction moved them -to `.agents_workspace/planning/` too, and the consumer skills (`implement-from-plan`, -`review-against-plan`) now look there first. -Flattened `.agents_workspace/X.md` -> `.agents_workspace/X.md` (the workspace dir replaces the -redundant `claude_logs/` segment). Added `.agents_workspace/` to `.gitignore` (session artifacts are -not committed) and a structure-tree entry in `CLAUDE.md`. Appended this entry to the *existing* -committed `.agents_workspace/DECISION_LOG.md` rather than the new (gitignored) location, to preserve -log continuity and keep the decision under version control; migrating the historical log is a separate, -unrequested concern. -**Impact / Risk:** Low. Edits across 3 SKILL.md, 2 plugin READMEs, `CLAUDE.md`, and `.gitignore`. -Did not bump plugin/marketplace versions or add a CHANGELOG release section — repo convention reserves -those for commit/release time; flagged as follow-up. CHANGELOG historical entries and -`skills-sync/README.md`'s pointer to Entry 15 left intact (factual records of the old path). -**Outcome:** Forward-looking output-path references in skills, READMEs, and CLAUDE.md now point at -`.agents_workspace/`; `.gitignore` updated. - ---- - -### Entry 23 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-20T00:00:00Z -**Task:** Add architecture documentation skill; consolidate the under-triggering ADR skill into it. - -**Context:** The `adr` skill never auto-fired in a plan-driven workflow — the SKELETON/ITER plans already capture decisions at decision time, and changes are handled by re-planning a version. User asked to add an architecture-documentation skill (Mermaid-diagram-centric ARCHITECTURE.md) and reviewed the ADR overlap. Two forks were the user's to resolve. -**Decision:** Per user choice: (1) consolidate — remove the standalone `adr` skill and fold durable decisions into the architecture doc; (2) decisions live inline in `docs/ARCHITECTURE.md` under a `## Key Decisions` section, and the six `docs/adr/DECISIONS.md` references across `ceh-git-workflow` (open-pr ×3, code-review, dependency-management ×2), `ceh-ops` (deploy), and `ceh-python-service` (fastapi) were repointed to it. New skill `document-architecture` added to `ceh-architecture` (MINOR → 3.1.0); the three peripheral plugins took content-only PATCH bumps. Canonical path `docs/ARCHITECTURE.md` and the standard diagram set (context/components/flows/ER/state) were my execution choices. This also resolved pre-existing drift between the `adr` skill's numbered-file convention and the single-file path the rest of the repo assumed. -**Impact / Risk:** Four plugins re-versioned; CROSS_REFERENCES.md PR-checklist block updated to match. Historical CHANGELOG/DECISION_LOG/PLUGIN_REORG_PLAN mentions of ADRs left intact as record. No repo git tag cut (left to user). -**Outcome:** JSON valid; plugin.json ↔ marketplace.json versions match; no stale active references remain. - ---- - -### Entry 24 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-20 -**Task:** Redirect remaining skill outputs from `docs/` to `.agents_workspace/`. - -**Context:** The goal was to move architecture skill output (and all other skill output) from -`docs/` to `.agents_workspace/`. The prior commit d320020 (v3.10.3) deliberately carved out -`docs/guide/` (user-operator-guide) and `docs/ARCHITECTURE.md` (document-architecture) as -*committed* deliverables, explicitly leaving them in `docs/`. The new instruction directly -contradicts that carve-out. -**Decision:** Moved only `document-architecture` to `.agents_workspace/ARCHITECTURE.md`; dropped the -word "committed" from its framing since `.agents_workspace/` is gitignored. The user subsequently -clarified that `user-operator-guide` must stay in `docs/guide/` because those guides are deliverables -intended for other readers, so that skill was left unchanged. Left non-output `docs/` references -untouched (`branch` prefix, `update-readme` README search path, `implement-from-plan` read example). -**Impact / Risk:** The architecture doc is now produced into a gitignored directory, so it is no -longer committed by default; a user who wants it in the repo must promote it manually. User/operator -guides remain committed deliverables under `docs/guide/`. -**Outcome:** Edits applied; ceh-architecture bumped to 3.1.1; CHANGELOG 3.11.1 added. - -### Entry 25 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-21 -**Task:** Create a new business-plan plugin (`/goal`). - -**Context:** The goal text blends a build instruction ("create a new business plan plugin") with a -description of runtime behavior ("use existing app plans as a starting point, interview the user -rigorously for product-market fit, loop until a satisfiable business plan"). Two readings: (a) -build a reusable plugin/skill that performs that loop, or (b) run a business-plan session for the -user's own product right now. -**Decision:** Built the plugin (reading a). The phrase "Create a new business plan plugin" names -the deliverable; the loop/interview language describes what the plugin *does*, which I encoded into -the skill body. This matches the repo's existing interview-driven precedent (`ceh-blog`). Created -`ceh-business-plan` with one skill `develop-business-plan` (proactive draft → PMF interview loop → -revise, gated by an 8-point PMF readiness checklist) plus a `business-plan-schema.md` reference, -and registered it in marketplace.json, both READMEs, and CLAUDE.md tables. -**Impact / Risk:** New plugin at v1.0.0; repo plugin count grows by one. No existing plugin -touched beyond table registrations. Validator passes. -**Outcome:** `python tools/validate-plugins/validate.py` → "OK: all plugin checks passed". - -### Entry 26 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-21 -**Task:** Create the skill/plugin evaluator plugin - -**Context:** User approved the evaluator design but asked for a plugin name broader than the proposed `ceh-skill-eval` because the plugin will later include an evaluator agent and a post-write hook, not just one skill. The specific name was left to me. -**Decision:** Named the plugin `ceh-evaluation` (use-case-workflow tier) with a single skill `evaluate-skill`. Scope-declaring and component-agnostic so an agent and a hook can be added without a rename. Kept one skill covering both skills and plugins, and used generic subagents (no custom agent files yet) per the approved plan and write-less-code. -**Impact / Risk:** Low. Name is easy to change pre-release. Self-contained: external skill-creator/plugin-dev plugins are optional cross-checks only, documented in the plugin README callout per user requirement. -**Outcome:** Plugin created, wired into marketplace.json, root README, CLAUDE.md; `python tools/validate-plugins/validate.py` passes. - -### Entry 27 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-22T00:00:00Z -**Task:** Run release-flow for the ceh-evaluation plugin. - -**Context:** release-flow's pipeline assumes branching `chore/release-vX.Y.Z` from `main`, but the new `ceh-evaluation` plugin exists only on the unmerged `feat/ceh-evaluation-plugin` branch. A fresh release branch off `main` would not contain the plugin. -**Decision:** Ride the existing `feat/ceh-evaluation-plugin` branch as the release branch. Feature work (plugin v1.1.2, README, CLAUDE.md, marketplace) is already committed and consistent; the only missing release artifact is the CHANGELOG entry. Repo tag: v3.12.0 → v3.13.0 (MINOR, new plugin). Plugin version 1.1.2 left as-is (already set; not a bump candidate during release). -**Impact / Risk:** Low — PR-gated merge to main precedes tagging; tag points at the merge commit per the skill's hard rules. -**Outcome:** Pending merge + tag. - -### Entry 28 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-01T00:00:00Z -**Task:** Add a "design-system" skill offering a menu of frontend design templates (Meridian, Tidewater). - -**Context:** User asked for a frontend-design skill with a template menu, dropping 2 templates -(brand_*.css + brand-guide_*.html) in the repo root. Two forks were unresolved: (a) which plugin -hosts the skill — a new ceh-frontend-design plugin vs. the existing ceh-web-frontend; (b) where the -large CSS/HTML template assets live given the repo's "content inline in SKILL.md" rule. -**Decision:** (a) Placed it in ceh-web-frontend as skill `design-system` rather than a new plugin — -frontend visual design is the same use case as building a web frontend, and a new plugin adds -marketplace churn for one skill. (b) Put the 4 template files under the skill's `references//` -tree — the CLAUDE.md reserves references/ for "schemas and templates," and these are literal template -assets too large to inline. Bumped ceh-web-frontend 3.0.4 -> 3.1.0 (MINOR, new skill). -**Impact / Risk:** Low. Skill is stack-agnostic CSS, additive only. If a broader design library grows, -it can be promoted to its own plugin later. Validator passes. -**Outcome:** `python tools/validate-plugins/validate.py` -> OK. - ---- - -### Entry 29 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-02 -**Task:** Release flow for the design-system skill on the current feature branch. - -**Context:** Release-flow step 6 asks whether CLAUDE.md needs updating. Adding a new skill changed the -web-frontend plugin's capability surface, so it was not obvious whether the project CLAUDE.md needed a -touch. -**Decision:** No CLAUDE.md update. The project CLAUDE.md lists plugins by domain, not individual -skills; the `ceh-web-frontend` domain ("SvelteKit + React (Vite), Bun, TS style, Vitest, Playwright, -accessibility") is unchanged conceptually and the file never enumerates per-plugin skills. Also ran -the release on the current branch `feat/frontend-design-system-skill` per explicit user instruction -("on current branch"), skipping the skill's default `chore/release-vX.Y.Z` branch. Repo tag bumped -MINOR v3.13.4 -> v3.14.0 (new skill); plugin ceh-web-frontend already at 3.1.0 from skill creation. -**Impact / Risk:** Low. CLAUDE.md stays accurate; README already carries the new skill row. -**Outcome:** CHANGELOG [3.14.0] written; validator passes. - -### Entry 30 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-04T00:00:00+02:00 -**Task:** Audit ceh-blog skills; shorten descriptions for manual invocation - -**Context:** The goal asked for shortened descriptions (manual invocation) and series-format focus, but did not specify how much routing to keep in descriptions, nor which files beyond the four SKILL.md frontmatters could change. -**Decision:** Kept a one-line routing hint ("for X use blog-writer") in each shortened description — disambiguation between the four skills is cheap and useful even when invoked manually. Added a Series Awareness block to blog-repurpose (the only skill lacking one, and the surface that actually entices external readers into the series) and rewrote the plugin README's "loads automatically" sections to a manual-invocation framing, since keeping long trigger-phrase lists would contradict the new intent. PATCH bump 1.0.5 -> 1.0.6 (content/description only). -**Impact / Risk:** Auto-triggering of these skills becomes less likely with shorter descriptions — accepted, per the stated intent. Registered cross-reference blocks (Voice, post-type structures) untouched. -**Outcome:** validate.py passes; all changes on branch docs/blog-skills-audit. - -### Entry 31 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-08T00:00:00+02:00 -**Task:** Add commit/open-pr/merge/release subagents to ceh-git-workflow - -**Context:** User asked whether to also recreate agent versions of the changelog/README updaters (converted to skills in 3d7f400). -**Decision:** No. 3d7f400 records why the agent form failed for those two: they need the live session's intent (the "why"), which a subagent re-deriving from git diff loses. The four new git agents are different — their input IS the repo state. Kept update-changelog/update-readme as skills only. -**Impact / Risk:** If a future flow wants doc updates out-of-session, the decision must be revisited; the `skills:` preload field now makes that cheaper than in June. -**Outcome:** ceh-documentation untouched. - -### Entry 32 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-08T00:00:00+02:00 -**Task:** Wire ceh-release-flow to the new subagents - -**Context:** User left "should the release flow use the subagents" to me. -**Decision:** Yes, but as an optional delegation path, not a rewrite: added a short "Delegating steps to subagents" section to release-flow (steps 7-10) and direct-release-flow (steps 7-8). The pipeline tables and skill-delegation fallback are unchanged; gates stay with the orchestrating flow. -**Impact / Risk:** Cross-plugin reference from ceh-release-flow to ceh-git-workflow agents (already precedented by its skill references). Fallback keeps the flow working without the agents installed. -**Outcome:** Both flow skills updated; ceh-release-flow bumped 1.1.2 -> 1.1.3. - -### Entry 33 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-08T00:00:00Z -**Task:** Convert ceh-advisor prototype (project-local .claude/ layout) into a marketplace plugin - -**Context:** The spec/prototype shipped hooks as copy-into-project files plus a settings.json snippet; plugin form needed several unresolved choices: where hook scripts live, plugin tier, initial version, and what to do when jq is absent (hooks errored with exit 127 on every Bash call). -**Decision:** (1) Hook scripts placed in `scripts/` (not `hooks/`) so `tools/validate-plugins/validate.py` bash-checks them; `hooks/hooks.json` references them via `${CLAUDE_PLUGIN_ROOT}/scripts/...`. (2) Deleted `settings-hooks-snippet.json` — plugin hooks load automatically, snippet is obsolete. (3) Classified cross-cutting tier (applies to most sessions, like ceh-fabled). (4) Version 1.0.0 (matches ceh-fabled/ceh-business-plan precedent for new plugins). (5) Added `command -v jq || exit 0` degrade to both hook scripts — without jq they are inert rather than erroring; fail-open here is unavoidable since exit 2 on PreToolUse would block every Bash call. (6) Guard deny message now interpolates the configured TTL instead of hardcoding "15 minutes". (7) Plugin README rewritten self-contained since the spec file was removed per request. -**Impact / Risk:** Machines without jq silently lack the hard-trigger layer (documented in plugin README with install commands). Hook scripts in `scripts/` diverges from the plugin-dev convention of `hooks/` but gains CI syntax checking. -**Outcome:** Validator passes; guard and failure-watch functionally tested (deny/allow/ack/TTL/threshold/reset/no-jq cases) with a scratchpad jq 1.8.2 binary. - -### Entry 34 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-08T00:00:00Z -**Task:** Release flow for v3.17.0 (ceh-advisor plugin) - -**Context:** The release-flow skill's step 2 says branch `chore/release-vX.Y.Z` from main, but the feature branch `feat/ceh-advisor-plugin` is itself unmerged — and repo precedent (v3.16.0, commit 6e24266 inside PR #44) puts the changelog-only release commit on the feature branch, merged via the feature PR, tagged after merge. Also unresolved: whether the three plugins with README-only invoke-syntax edits get PATCH bumps. -**Decision:** Release commit rides the existing feature branch (matches precedent; avoids a second PR whose only content is a changelog entry). No PATCH bumps for `ceh-dev-tools`/`ceh-orchestration`/`ceh-release-flow` — v3.16.0 precedent shipped release commits without bumping doc-touched plugins, and the edits are cosmetic invoke-syntax strings; noted as "doc-only, no version bumps" in the changelog. Repo tag v3.17.0 = MINOR (new plugin adds agents). -**Impact / Risk:** Installed users of the three README-touched plugins won't auto-update to the new invoke syntax until each plugin's next real bump — acceptable for cosmetic docs. -**Outcome:** Pending merge + tag. - -### Entry 35 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-09T00:00:00Z -**Task:** Run release-flow for the git-workflow + release-flow doc changes. - -**Context:** release-flow's "one version everywhere" step assumes a single project version, but this repo uses per-plugin versions plus a separate repo git tag. The two touched plugins (ceh-git-workflow 3.2.1, ceh-release-flow 1.1.4) were already bumped and committed on the branch. -**Decision:** Adapt the flow to the mono-repo model: skip the unified-version step (per-plugin bumps already landed), cut repo tag v3.17.2 as PATCH from v3.17.1 (docs-only, no skills/agents added). Release commit (changelog) rides the existing feature branch, matching the v3.16.0/v3.17.0 precedent (Entry 34). README and project CLAUDE.md unchanged — no new plugin/skill/agent and no install/config surface touched. Steps 7-10 delegated to the ceh-git-workflow subagents at model sonnet, effort medium per user request. -**Impact / Risk:** Low — content-only docs release; tag points at the merge commit on main. -**Outcome:** Pending changelog commit, PR, merge, tag. - -### Entry 36 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-10T00:00:00+02:00 -**Task:** Seven-plugin review/audit/cleanup on chore/plugin-skill-review-audit - -**Context:** The audit surfaced items where "fix (if needed)" left the remedy open: (1) section-specs.md §02 drift between the two planner copies — propagate wording vs relax CROSS_REFERENCES; (2) blog-writer/blog-editor lacked the blog-repurpose handoff line that blog-interviewer has — fix vs report-only; (3) update-changelog's validate step used ${CLAUDE_PLUGIN_ROOT}, which is unset in skill Bash calls. -**Decision:** (1) Propagated the canonical (to-mvp golden standard) §02 wording to the iterative copy per the Update Protocol, rather than weakening the registered "word-for-word identical" contract. (2) Added the identical handoff sentence to writer and editor and registered the new 3-file duplication in CROSS_REFERENCES.md. (3) Reworded the validate step to locate the script relative to the plugin root instead of the env var, keeping the manual fallback. Also fixed the inverted pre-release ordering in check-semver.py semver_key (release must outrank its pre-release, semver §11) — verified with key checks and an end-to-end run. -**Impact / Risk:** All content-level; five plugins PATCH-bumped. Blog handoff line adds a small duplication maintenance cost, mitigated by the CROSS_REFERENCES entry. -**Outcome:** validate.py passes; check-semver.py verified against a prerelease-containing changelog. - -### Entry 37 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-10T00:00:00+02:00 -**Task:** Round-2 plugin audit (remaining 12 plugins, ceh-summarize-chat excluded per user) - -**Context:** Round 2 touched ceh-git-workflow again (bun.lockb -> bun.lock in dependency-management) after round 1 had already bumped it to 3.2.2 on this unreleased branch. Options: bump again to 3.2.3 per-commit, or keep 3.2.2 for the branch's single release state. -**Decision:** Keep 3.2.2 — both commits land in the same unreleased branch/session, so a second PATCH bump would version-churn a state no consumer ever saw. ceh-python-service bumped 3.1.2 -> 3.1.3 (fastapi error-shape example nested under "error" to match its own contract; python-integration-tester httpx fixture moved to ASGITransport since app= was removed in httpx 0.28). -**Impact / Risk:** None material; validator confirms manifest/marketplace sync. -**Outcome:** validate.py passes. - - -### Entry 38 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-10T00:00:00+02:00 -**Task:** Review both ceh-fabled skills for parity with invoking Fable directly (Sonnet/Opus as executor) - -**Context:** The fabled skill's no-extended-thinking fallback ("write a scratchpad, then drop it before delivering") is not executable — emitted response text cannot be retracted. Candidate fixes: delete the fallback (thinking-only), force a visible working section, or externalize the scratchpad. -**Decision:** Externalize: in an agent environment write working notes to a temp file or reason between tool calls; in plain chat allow a compact, clearly delimited visible working section. Deleting the fallback would silently drop the discipline exactly where weaker models need it. Also inlined the think-before-verdict rule into fabled-plan-review rather than making a fabled load mandatory (keeps the skill self-contained and cheap), and removed the "(or the fixes are small)" self-edit authorization that contradicted both the same sentence and the coding contract. -**Impact / Risk:** Content-only; plugin already at uncommitted 1.1.0 on this branch, no further bump. Plain-chat fallback still leaks working text into the response — accepted as the only executable option there. -**Outcome:** validate.py passes. - -### Entry 39 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-10T00:00:00+02:00 -**Task:** Add fabled-stuck skill to ceh-fabled - -**Context:** ceh-fabled was already bumped 1.0.0 -> 1.1.0 in this branch's uncommitted changes (for fabled-plan-review). Adding a second new skill raised the question: bump again to 1.2.0 or fold into the pending bump. -**Decision:** Keep 1.1.0. The repo rule is "bump only at commit time"; nothing between 1.0.0 and now has been committed or released, so both new skills ship under the same pending MINOR bump. -**Impact / Risk:** None — versions in plugin.json and marketplace.json stay in sync; a second MINOR bump would only inflate the version number. -**Outcome:** validate.py passes with 1.1.0 and three skills in ceh-fabled. - -### Entry 40 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-10T00:00:00+02:00 -**Task:** Second parity review of ceh-fabled skills — "applied by Sonnet/Opus, behave as if Fable were invoked" - -**Context:** Entry 38 fixed the executability of the scratchpad fallback; this pass asked whether a weaker model would actually *execute* the protocol rather than read and skip it. Candidate remedies ranged from restructuring the skill around a step-0 engagement checklist to targeted enforcement edits. -**Decision:** Targeted enforcement edits to fabled/SKILL.md only: (1) triage must be *written* as the first reasoning line (was "silently" — on a low-thinking-budget model that means never); (2) reference loading hooked into triage as a pre-stage-1 action and "load" defined as a Read tool call, since the section sat after the Core Loop and was skippable; (3) stage 4 now sweeps the draft against the anti-patterns list, which was previously passive; (4) discipline declared session-persistent, matching what invoking Fable actually does; (5) description gains "as fable"/"fable mode" trigger phrases (plan-review had them, the core skill did not). fabled-plan-review and fabled-stuck left unchanged — already imperative, self-contained, correct relative paths. No version re-bump: content edits to the pending uncommitted 1.1.0 (Entry 39 precedent). -**Impact / Risk:** Content-only; slightly longer SKILL.md. Risk of over-instruction on trivial tasks is bounded by the existing trivial-tier escape hatch. -**Outcome:** validate.py passes. - -### Entry 41 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-12T00:00:00+02:00 -**Task:** Rename ceh-web-frontend design-system skill and add per-section examples. - -**Context:** User asked "should we rename" without naming a target; also asked for examples under references/, which the repo CLAUDE.md reserves for schemas/templates shared across skills. -**Decision:** (a) Renamed to `ui-design` — short, unambiguous within the web-frontend plugin namespace; kept "design system" as a trigger phrase in the description so old phrasing still fires. Alternatives `visual-design` / `ui-visual-design` rejected as vaguer/clunkier. (b) Added `references/examples.md` (good/bad markup per rule section) despite the references-policy wording — explicit user instruction overrides the convention; single file, not shared across skills. (c) Bumped plugin 3.1.1 → 3.2.0 (MINOR) because the rename changes the skill's invocation name, which is more than a content PATCH. -**Impact / Risk:** Users invoking `/ceh-web-frontend:design-system` by exact name must switch to `/ceh-web-frontend:ui-design`; historical CHANGELOG/eval references to the old name left untouched intentionally. -**Outcome:** validate.py passes; all changes uncommitted on feat/design-system-visual-design. - -### Entry 1 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-12 -**Task:** release-flow for the disable-model-invocation change (repo v3.19.1) - -**Context:** The release-flow pipeline says branch `chore/release-vX.Y.Z` from main and bump versions there. But the release's actual changes (16 SKILL.md edits) and the 8 per-plugin version bumps already existed uncommitted on the working branch `chore/disable-model-invocation-user-only-skills`. -**Decision:** Reuse the existing feature branch as the release branch rather than creating `chore/release-v3.19.1`; the release IS these changes, so a separate branch would just be churn. Repo tag bump is PATCH (v3.19.0 → v3.19.1) — content/description + frontmatter only, no new skills or agents. -**Impact / Risk:** Branch name does not match the `chore/release-*` convention; low risk (cosmetic). Tag/release still cut on main after merge per the hard rules. -**Outcome:** Pending merge + tag. - -### Entry 42 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-15T00:00:00+02:00 -**Task:** Fold the UI-polish lessons from session d12296e3 (claude-code-command-center UI refresh) into ceh-web-frontend:ui-design. - -**Context:** User asked to extract "core ideas and design choices" from the transcript so future skill invocations reach the session's final quality first-pass; which lessons count as generalizable vs app-specific, and where they land in the skill, was left to me. -**Decision:** (a) Generalized nine patterns (depth ladder, command dock, eyebrow-above-panel headers, humanized tables, lifecycle colors + node-and-rail stepper, identity monograms, stat blocks, recessed/auto-grow inputs + themed scrollbars, single micro-interaction) and dropped app-specific content (round semantics, End Turn, terracotta-specific choices — expressed via tokens instead). (b) Structured as a new "Finishing recipes — past the primitive draft" section rather than scattering across existing sections, because the session's failure mode was a distinct *finishing* gap, not a rule violation; depth ladder alone merged into "Color and depth" where it natively belongs. (c) Added six anti-patterns, a 7th review-pass "finish audit" item, polish/primitive trigger phrases, and worked markup in references/examples.md (Entry 41 precedent for that file). (d) PATCH bump 3.2.0 → 3.2.1 in plugin.json + marketplace.json — content-only. -**Impact / Risk:** SKILL.md grows ~90 lines; risk of over-prescribing the dock pattern on apps without global state is bounded by the "when app-wide state exists" precondition. -**Outcome:** validate.py passes; changes uncommitted on feat/ui-design-polish-layer. - -### Entry 43 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-18 -**Task:** Draft shrink-diff and refactor-repo skills (ceh-agent-coding-contract) - -**Context:** The "Adding a Skill" checklist says bump plugin.json + marketplace.json, but the Versioning section says bump only at commit time, not during iterative edits. This session drafts the skills without committing. Also, shrink-diff's trigger phrases risked colliding with ceh-git-workflow skills ("clean up the branch" is merge-skill vocabulary, "branch" naming collides with the branch skill). -**Decision:** Deferred the version bump (2.6.4 → 2.7.0 MINOR, plus marketplace mirror) to commit time per the Versioning section, which is the more specific rule. Chose diff-object trigger phrases ("shrink the diff", "can this diff be smaller") and avoided "clean up the branch"; refactor-repo gets disable-model-invocation: true so the whole-repo mode can never auto-fire. -**Impact / Risk:** If the drafts are committed without the bump, CI still passes (versions stay in sync) but the auto-update convention is violated — the bump must accompany the commit. -**Outcome:** validate.py passes; drafts uncommitted on feat/shrink-diff-refactor-skills. - -### Entry 44 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-18T23:35:00Z -**Task:** Add usage-limit guard hook + usage-limit-handoff skill to ceh-agent-coding-contract - -**Context:** Three unspecified implementation choices: (1) hook language — plugin convention is bash+jq, but jq is not installed on the target machine, which would make the hook silently inert (ceh-advisor's jq-based hooks are already inert here for the same reason); (2) how often the guard re-fires once over threshold; (3) default threshold value. -**Decision:** (1) PowerShell (`pwsh -NoProfile`) instead of bash+jq — the data source (statusline export) is itself a pwsh script, so pwsh is a given wherever the data exists; (2) re-fire only per 5-point usage band above threshold, so an ignored warning escalates instead of spamming every tool call; (3) default threshold 80%, overridable via `CEH_USAGE_LIMIT_THRESHOLD`. -**Impact / Risk:** Hook is Windows/pwsh-leaning, diverging from the repo's bash hook convention; inert (by design) on machines without pwsh or without the statusline export. jq absence on this machine also affects ceh-advisor hooks — flagged to user, not fixed (out of scope). -**Outcome:** Dry-run against live statusline data passed all three cases (fire at exit 2 with message, band-based re-fire suppression, below-threshold silence). - -### Entry 45 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-19T00:00:00Z -**Task:** Port the three logic hooks (advisor guard, advisor failure-watch, usage-limit-watch) from bash+jq to Python for Windows/Linux parity. - -**Context:** Three forks the user's brief left open. (1) The shell guard signalled deny via JSON -on stdout with exit 0; the first Python draft emitted deny JSON *and* exit 2, mixing the two -documented PreToolUse mechanisms. (2) "Fail closed" was specified for the guard, but the same -policy applied to the two advisory PostToolUse hooks would block tool calls on any parse error. -(3) A pattern-parity sweep found `git push -f origin main` is not matched by the inherited -pattern set (only `--force`, or `-f` at end-of-string). - -**Decision:** (1) Deny goes through JSON + exit 0 so it always carries a readable reason; exit 2 -is reserved for the crash/no-interpreter backstop, added as `|| exit 2` in hooks.json because -Claude Code treats only exit 2 as blocking — without it a missing `python3` (127) would fail -*open*. (2) Fail-closed for the PreToolUse guard only; the two advisory hooks catch exceptions, -print one stderr line, and exit 1 (visible, non-blocking) — a traceback on an all-tools -PostToolUse hook would spam every call. (3) Preserved the `-f` gap rather than fixing it: the -task was a port, and silently changing guard coverage would make the parity check meaningless. -Documented it in the plugin README with the one-line pattern to close it. - -**Impact / Risk:** Hooks now require `python3` on PATH. A broken install blocks every destructive -command until fixed — deliberate, and documented. The six static-payload hooks stay bash: they -have no logic, and a shell is more universally present than Python. Verified 21/21 decision -parity against the shell originals before deleting them. -**Outcome:** validate.py passes; all three hooks smoke-tested including anti-spam banding, -streak reset, corrupt payloads, and missing interpreter. - -### Entry 46 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-19T00:00:00Z -**Task:** Close the `git push -f` gap in the ceh-advisor guard pattern set (follow-up to Entry 45). - -**Context:** Entry 45 deliberately preserved the gap to keep the port's parity check meaningful. -With the port landed, the user asked for the fix. The old pattern -`git\s+push\s.*(--force|\s-f(\s|$))` could never match a bare `git push -f origin main`: the `\s` -before `-f` required a space that the preceding `git push\s` had already consumed, so the flag was -only caught in non-first position. - -**Decision:** Replaced with `git\s+push\s+(.*\s)?(--force|-f)(\s|$)`. The optional `(.*\s)?` prefix -lets the flag sit in first position or later, and anchoring both alternatives with `(\s|$)` keeps -branch names like `hotfix-f` and `feature-force` allowed. This also tightens `--force`, which -previously had no trailing anchor and would have matched `--forceful`. Did not add -`--force-with-lease`: it is a distinct flag and adding it widens what the guard blocks beyond -what was asked — documented in the README as a one-line opt-in instead. - -**Impact / Risk:** The guard now denies force pushes it previously waved through, so anyone -relying on `git push -f` will start hitting the consult protocol — the intended behaviour, but a -visible workflow change. No version re-bump: 1.0.1 from Entry 45 is still uncommitted, so this -ships as part of the same change. -**Outcome:** 20/20 targeted cases pass (6 force-push spellings deny, 6 near-miss branch names -allow, 8 regressions unchanged); validate.py passes. - -### Entry 47 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-19T00:00:00Z -**Task:** Make the ceh-advisor guard actionable inside subagents, using the `agent_id` hook field. - -**Context:** The user proposed gating the guard on `agent_type` so it only runs when the advisor -agent is involved. That inverts the field's meaning — `agent_type` identifies whose context the -hook fires in, and the advisor has `tools: Read, Grep, Glob` (no Bash), so the guard would have -fired never while the main session went unguarded. The real problem the field exposes: verified -that no agent in this repo has `Task`/`Agent` in its tools, so a denied subagent is handed an -impossible instruction ("invoke the ceh-advisor subagent") and deadlocks. - -**Decision:** Keep denying inside subagents, but branch the message on `payload["agent_id"]`: -tell the subagent to stop and report to its caller, and explicitly not to write the ack itself. -Rejected skipping the guard for subagents — that would turn delegation into a universal bypass. -Placed the check *after* the ack lookup so a caller who consults and then delegates still works. -Did not add an anti-recursion check for `agent_type == ceh-advisor:ceh-advisor`: real in -principle, but the advisor has no Bash tool, so it is speculative until that changes. - -**Impact / Risk:** Latent-trap fix, not a live outage — no current git-workflow agent trips a -pattern (the merge skill uses `git branch -d`, and the guard matches `-D`). Any delegated agent -needing a force-push or `rm -rf` would have deadlocked. ceh-advisor 1.0.1 -> 1.0.2. -**Outcome:** 5/5 cases pass (main-session deny unchanged, subagent deny with new message, -subagent benign allow, subagent unlocked by a caller's fresh ack, main-session ack unchanged). - -**Also surfaced (not changed):** the user noticed nothing in the codebase ever writes the ack -file. Confirmed by grep — the only writes are the README smoke-test snippet and the instruction -text. The model writes its own permission slip. This is the documented honest-agent assumption; -README now states it explicitly, along with the fact that an ack is a blanket pass for the full -TTL rather than scoped to the reviewed command. - -### Entry 48 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-19T16:55:38+0200 -**Task:** Rework the usage-limit guard from statusline-percentage watching to preemptive stop-and-handoff - -**Context:** The guard's sensor was believed unreliable because account quota is shared across -claude.ai web, desktop, mobile and Claude Code. Alternatives considered and rejected: polling a -small API call (an API key reports the *org API* rate limits, a different pool from the Pro/Max -subscription; the OAuth usage endpoint needs credentials read from a hook, is undocumented, and the -probe consumes the quota it measures) and resume-side reconstruction from transcripts (transcripts -carry only the after-the-fact `apiErrorStatus: 429`, not a running percentage, and reading a long -transcript to summarize costs the most exactly when quota is scarcest). - -**Decision:** Keep the statusline export as the sensor — verified as the only local surface -carrying live account-wide quota — and fix its reliability instead of replacing it: read the newest -record across all sessions/projects, guard staleness, warn once when absent rather than no-op -silently, and take the worst of *all* rate-limit windows by iterating the dict rather than -hardcoding `five_hour` (this is what surfaced `seven_day` at 72% during testing). Threshold lowered -95 → 90 so the summary is written while context is hot. An earlier design that maintained a handoff -file every turn was rejected by the user as too much bookkeeping; one write at the trigger point is -cheaper and produces a richer summary. - -**Impact / Risk:** Still inert without the statusline export, but now says so. Subagents share the -parent `session_id`, so the escalation band is session-wide by design; a subagent that trips the -guard reports upward rather than writing an artifact, since exit 2 reaches the subagent's loop and -its final report is never shown to the user. - -**Outcome:** All paths dry-run against live statusline data: below-threshold silence, main-session -fire (exit 2), subagent-variant fire, band-based re-fire suppression, and the warn-once -missing-sensor path (exit 1). `validate.py` passes. - ---- - -### Entry 49 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-06-29 -**Task:** Run the ceh-release-flow:release-flow skill in this repo to release the auto-merge git-workflow changes. - -**Context:** The release-flow skill prescribes cutting a `chore/release-vX.Y.Z` branch from `main`, but the session's standing instruction is to develop only on `claude/git-auto-merge-workflow-ne8we4` and never push to a different branch without explicit permission. The feature changes and plugin version bumps already live on that feature branch. -**Decision:** Treat the existing `claude/git-auto-merge-workflow-ne8we4` branch as the release-carrying branch rather than creating a separate `chore/release-` branch. This honors the dev-branch constraint and keeps the version bumps, changelog, and feature changes in one PR. Repo git tag bumped PATCH (v3.13.3 → v3.13.4) because the changes are skill-content only (no new skills or agents), per the repo's two-layer versioning rule in CLAUDE.md. -**Impact / Risk:** PR carries both feature and release-bump commits together (acceptable for this repo). GitHub Release object cannot be created with the available MCP tools (no create-release tool) — the annotated tag can be pushed via git, but the Release page must be created manually or confirmed separately. -**Outcome:** Pending merge + tag. Merged into this log on 2026-07-20 from a stray `docs/claude_logs/DECISION_LOG.md` (that session wrote to the default convention path instead of `.agents_workspace/`); appended with the next sequential ID rather than inserted in date order, so IDs stay monotonic and the original timestamp carries the chronology. - ---- - -### Entry 50 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-20T00:00:00+0200 -**Task:** Load `ceh-fabled:fabled-voice` via a SessionStart hook the way `ceh-agent-coding-contract` loads the contract. - -**Context:** Two forks the request left open. (1) Hook ordering: both plugins would emit a SessionStart directive, and cross-plugin hook execution order is not guaranteed, so a second "MANDATORY FIRST ACTION" payload would contend with the contract's for the first tool call. (2) `disable-model-invocation: true` looked attractive for a now-hook-loaded skill, but a hook only injects text — the model still makes the Skill call, and that flag removes the skill from the model-visible listing, so setting it would break the very path being built. - -**Decision:** Word the payload as `REQUIRED SETUP ACTION` (not `MANDATORY FIRST ACTION`) and have it defer explicitly to the contract directive when both are pending, resolving order in prose rather than relying on hook sequencing. Left model invocation enabled. Bumped `ceh-fabled` MINOR (1.2.0 → 1.3.0) — a hook is a new component, matching the repo's "MINOR for new skills or agents" rule. Kept the hook pure-bash so `ceh-fabled` gains no `python3` dependency. - -**Impact / Risk:** fable's response style becomes unconditional in every session and repo where the plugin is installed, including ones where a more conventional register would suit better; opting out means disabling the plugin's hooks. If a future session sets `disable-model-invocation` on `fabled-voice`, the hook silently degrades to a failed Skill call. - -**Outcome:** `validate.py` passes; hook payload parses as JSON and emits the expected directive. Not yet observed firing — requires a fresh session after the plugin cache picks up 1.3.0. - ---- - -### Entry 51 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-20T00:00:00+0200 -**Task:** Release v3.23.0 via the ceh-release-flow:release-flow skill. - -**Context:** The flow's step 2 prescribes cutting `chore/release-vX.Y.Z` from latest `main`, but the `ceh-fabled` 1.3.0 bump and the hook itself were already committed and pushed on `feat/fabled-voice-hook`, with no PR open yet. Following step 2 literally would mean merging the feature PR first, then a second branch and PR carrying only a changelog entry. - -**Decision:** Release from the existing `feat/fabled-voice-hook` branch — add the changelog entry there, open one PR, merge, then tag `v3.23.0` on `main` at the merge commit. Repo tag bumped MINOR (v3.22.1 → v3.23.0) because `ceh-fabled` gained a new component (a hook), per the repo's two-layer versioning rule. Steps 3, 5 and 6 were already satisfied by the feature commit (manifests bumped, READMEs and CLAUDE.md updated), so only step 4 needed work. Same call as Entry 49, which faced the identical mismatch. - -**Impact / Risk:** The PR carries the feature and the release bump together — acceptable for this repo and consistent with prior releases. Step 10's "tag the merge commit on `main`" rule is unaffected and still enforced. - -**Outcome:** Changelog written; steps 7–10 delegated to the `ceh-git-workflow` subagents. - -### Entry 53 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-22T22:45:00+02:00 -**Task:** Release v3.24.0 via release-flow - -**Context:** release-flow step 2 prescribes a fresh `chore/release-vX.Y.Z` branch off main, but -the entire release content (the new ceh-seo plugin) sits unmerged on `feat/seo-plugin` with no PR. -**Decision:** Ride the release on `feat/seo-plugin` — changelog + release commit land on the -feature branch, one PR carries feature and release together. A separate release branch would -require merging the feature first for no reviewable difference. -**Impact / Risk:** Single PR mixes feature and release-bookkeeping commits; acceptable since the -release IS the feature. Tag still lands on the merge commit on main per the hard rule. -**Outcome:** (pending merge) - -### Entry 54 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-25T00:00:00Z -**Task:** Modernize plugin frontmatter against current Claude Code capabilities (Claude 5 family) - -**Context:** The review proposed applying several newly-available frontmatter fields broadly. -Verifying each against the live docs showed the proposed scope was wrong in three places, and -narrowing was decided per-field rather than applying the original plan as approved. - -**Decision:** -- `paths:` on 6 skills, not the ~17 originally scoped. `paths` **narrows** auto-loading rather - than adding to it, so applying it to skills with genuine non-file triggers ("a `uv` command is - run", "a publish is prepared") would have silently removed working triggers. Applied only where - the file trigger is the whole trigger. -- `isolation: worktree` dropped entirely. Subagent worktrees branch from the repo's **default - branch** (not the parent `HEAD`) unless `worktree.baseRef: "head"` is set in settings, and - their changes stay in the worktree. Under this repo's feature-branch rule that hands an agent a - copy of `main` without the user's work, and a tester agent's output would never reach the - checkout. Recorded as a deliberate non-use in `CLAUDE.md`. -- `context: fork` on 2 skills (`summarize-chat`, `lessons-learned`), not the heavyweight manual - set. Forks inherit the transcript and skip both subagent tool filters, so they suit skills that - act *on* the conversation; the rest are interactive (`AskUserQuestion` is stripped from plain - subagents) or must run in the main session (`orchestrate`). -- `effort:` only where it differs from the `high` default (`max`/`xhigh`), never `effort: high`, - which would be dead config. -- `disallowed-tools` on `code-review` only. `review-against-plan`, `fabled-plan-review`, and - `evaluate-skill-lite` all write files despite reading as review skills. -- Plan mode dropped from the planning skills: it blocks writes, and those skills' deliverable is - a file. Reduced to a README note about the research step before planning. - -**Impact / Risk:** The 6 `paths:` skills no longer auto-load outside their globs — intended, but -it is a behavior change for anyone relying on description-based matching. `ceh-advisor` gains -Read/Write/Edit as a side effect of `memory: local`; its instructions confine writes to the memory -directory, which is instruction-level, not enforced. - -**Outcome:** `python tools/validate-plugins/validate.py` passes; 17 plugins bumped PATCH. - -### Entry 55 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-25T00:00:00Z -**Task:** Add a skill that walks a repo and writes a file-by-file explanation. - -**Context:** Two plugins could plausibly host it. `ceh-documentation` owns docs, but its use case -is end-user/operator-facing output. `ceh-dev-tools` owns "repository exploration and codebase -orientation" — the same use case — but was documented as agents-only, and it already ships -`repo-tree-mapper`, which produces a one-line-per-path `REPO_MAP.md` and overlaps in triggering -("what's in this repo", onboarding requests). - -**Decision:** Placed `explain-codebase` in `ceh-dev-tools` as its first skill. Use case beats the -agents-only convention: splitting the map and its deeper sibling across two plugins would force a -user orienting in a codebase to load two plugins, which the self-containment rule exists to -prevent. Dropped "(agents only — no skills)" from `CLAUDE.md` and `README.md`. Disambiguated -against `repo-tree-mapper`, `document-architecture` and `user-operator-guide` with a "Not the same -as" table in the skill body and a description that names each alternative, so the deeper skill and -the cheap map do not fight over the same prompts. - -**Impact / Risk:** Trigger overlap with `repo-tree-mapper` remains — an orientation request could -match either. The mapper's description is structure/map-flavored and the skill's is -explain/every-file-flavored, but this is description-level disambiguation, not enforced. Worth -watching if the mapper starts firing on "explain the codebase". - -**Outcome:** `python tools/validate-plugins/validate.py` passes; `ceh-dev-tools` bumped 1.1.4 → -1.2.0 (MINOR, new skill) in both manifests. - ---- - -### Entry 56 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-29T00:00:00Z -**Task:** Close the testing-technique gaps found by a repo-wide survey of `ceh-*` testing coverage. - -**Context:** A survey confirmed by grep that property-based, mutation, fuzz, metamorphic, -differential, fault-injection, chaos, load/soak, pairwise, flaky-detection, idempotency, race, -contract (Pact) and canary testing are absent repo-wide. The three existing testing skills -(`python-service-testing`, `python-library-testing`, `frontend-testing`) cover runner, fixtures, -mocking and the test pyramid — tooling, not technique. Two forks were unresolved: where the -technique content lives, and how finely to slice it. - -**Decision:** Created a new cross-cutting `ceh-testing` plugin holding five stack-agnostic technique -skills plus one agent, rather than duplicating the technique into the three stack plugins. This is a -deliberate deviation from the use-case-only organizing axis: choosing test inputs, auditing whether -a green suite catches defects, and proving a refactor changed nothing are identical in Python and -TypeScript, so triplicating them would produce three copies with nothing stack-specific to justify -the divergence — the drift cost the duplication policy accepts only when copies genuinely differ. -The rationale is recorded as a fourth categorization rule of thumb in `CLAUDE.md`. - -Sliced to five skills, not the ~12 the taxonomy suggests: A3/A4 (decision tables, state transitions) -folded into `design-test-cases` as rungs of one input-selection ladder rather than standing alone, -and B7-B10 (concurrency/idempotency, contract drift, performance regression, authorization) collapsed -into `close-test-risk-gaps` as a single triage gate with explicit per-class skips. A skill triggers -on a moment, and these share one moment each; separate skills would have competed for the same -prompts and mostly never fired. Tier C techniques (fuzz, chaos, load/soak, canary, metamorphic) were -left out — no trigger moment in this repo's use cases. - -**Impact / Risk:** `ceh-testing` must be loaded *alongside* a stack plugin, not instead of one — -the boundary is stated in the plugin README and in `CROSS_REFERENCES.md`, but nothing enforces it. -The real risk is boundary slippage: a technique block drifting into a stack testing skill, or a -runner detail into `ceh-testing`. Trigger overlap is also possible between `verify-behavior-preserved` -and `ceh-agent-coding-contract:shrink-diff` (both fire on "shrink the diff"); the skills cross- -reference each other rather than compete, since shrink-diff carries no verification step of its own. - -Deliberately **not** done: cross-linking the new skills from the three stack testing skills and six -tester agents. That edits nine existing files for discoverability the five skills already have -through their own trigger moments. - -**Outcome:** `python tools/validate-plugins/validate.py` passes; `ceh-testing` added at 1.0.0 in -both `ceh-testing/.claude-plugin/plugin.json` and `.claude-plugin/marketplace.json`. One defect -found and fixed while sanity-checking the skill content: the assertion-audit snippet in -`audit-test-suite` matched on `ast.dump()` text, which contains the function's own name, so a test -named `test_assertion_shape` was silently treated as asserting; rewritten to match on `ast.Assert` -and call-node shape, and verified against a fixture. - ---- - -### Entry 57 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-29T00:00:00Z -**Task:** Audit `ceh-testing` against a full software-testing-technique taxonomy, and resolve the -organizing-axis deviation recorded in Entry 56. - -**Context:** Two forks. (1) A taxonomy sweep (design, structural, level-based, non-functional, -process, specialized) found nine techniques absent from the plugin; each needed a keep-or-exclude -call, since Entry 56 had already excluded Tier C techniques for having no trigger moment. (2) Entry -56 self-described `ceh-testing` as "the one deviation from the use-case-only axis" and the user asked -whether the deviation can be avoided. - -**Decision (1) — taxonomy gaps.** Added five techniques inline, no new skills: metamorphic relations -and fuzzing as rungs 7 and 8 of `design-test-cases` (dependency failure moved to rung 9); `git -bisect` on the reproducer in `test-a-bug-fix`; `--cov-branch` in `audit-test-suite`; and a fifth -risk class, migration and rollout compatibility, in `close-test-risk-gaps`, plus a consumer-driven -contract paragraph in its contract-drift class. All five fire on moments the plugin already claims, -so folding them in beat adding skills that would compete for the same prompts. - -Metamorphic testing was the largest genuine gap: it is the only answer to "how do I test output -nobody can predict", which now covers every LLM, ranking, and pricing-engine feature. Migration -testing was the second: `ceh-python-service:alembic` carried one bullet ("test against a copy of -production data") and the stack-agnostic technique — down path, backfill idempotency, expand/contract -across a rolling deploy — was absent repo-wide. - -Deliberately excluded, now recorded in a "Deliberately out of scope" table in the plugin README so a -future reader sees a decision rather than an oversight: load/stress/soak/capacity, chaos and infra -fault injection, canary/shadow/post-deploy smoke (all `ceh-ops`), SAST/DAST/SCA/pen testing -(`ceh-python-service:python-security`, `ceh-git-workflow:dependency-management`), continuous fuzzing -infrastructure, MC/DC and def-use coverage, model-based/exploratory/usability/localization/ -compatibility testing. - -**Decision (2) — the deviation is a mislabel, not a structure problem; reframed rather than -restructured.** Two restructurings were considered and rejected. Duplicating the five technique -skills into the three stack plugins produces fifteen byte-identical files, which is precisely the -case the Shared-Standards Duplication Policy does *not* cover (it pays for drift only when copies -genuinely differ) — and it would still leave the technique unavailable to any repo with no stack -plugin loaded. Moving the three stack testing skills into `ceh-testing` would make it a complete -use-case plugin, but breaks the stack plugins' self-containment in the other direction and is a -large move for a labelling problem. - -What is actually true: `ceh-testing` is structurally identical to `ceh-git-workflow` — a -cross-cutting discipline that applies whatever is being built, loaded alongside a use-case plugin. -Nobody calls `ceh-git-workflow` a deviation, and `ceh-python-service` owns commit conventions no -more than it owns test design. The use-case axis governs the use-case-workflow and stack/build -tiers; the cross-cutting tier is orthogonal by construction. Only the word "testing" appearing in -both plugin families made it look like an overlap, and `CROSS_REFERENCES.md` already confirms the -two share no content. `CLAUDE.md` rule 4 was rewritten accordingly, and the placement test made -explicit: a skill belongs in a cross-cutting plugin iff its content would be byte-identical across -stacks. - -**Impact / Risk:** The reframe removes the "exception" framing that invited future exceptions, but -it widens what the cross-cutting tier may absorb — the byte-identical test is the guard, and the -review question is now "would this be identical in Python and TypeScript", not "is this testing". -`close-test-risk-gaps` grew from four classes to five; a sixth would make it a checklist rather than -a triage gate, which is the shape the skill exists to avoid. `design-test-cases` grew from seven -rungs to nine and is now the longest skill in the plugin — the "stop when the remaining rungs have -no trigger" instruction carries more weight than before. - -**Outcome:** `python tools/validate-plugins/validate.py` passes; `ceh-testing` bumped 1.0.0 → 1.0.1 -(PATCH — content only, no new skills or agents) in both manifests. Root `README.md`, plugin -`README.md`, and `CLAUDE.md` updated. No test was run — this repo ships markdown only. - -### Entry 58 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-30T00:00:00Z -**Task:** Full ceh-evaluation:evaluate-skill run on the ceh-testing plugin (run-001) - -**Context:** Three genuine ambiguities came up running the evaluation skill's fixed process against -a plugin-sized target (5 skills + 1 agent), where the skill's own instructions leave method open: -(1) how to detect whether a skill actually "fired" inside a cold subagent when the rubric explicitly -forbids asking the subagent to self-report skill usage, and no tool in this session exposes a -subagent's raw tool-call trace to the caller; (2) how to right-size the trigger/behavioral battery -for a plugin (the literally-scoped plan — 10+10 N=3 for the in-depth skill, 4+4 N=1 x5 for sampled -skills, 3 behavioral tasks x N=2 — implies well over 100 subagent dispatches); (3) whether to apply -a description fix directly or only recommend it. - -**Decision:** (1) Operationalized "fired" as *the skill's content was consulted and applied* — -verbatim/near-verbatim idiom matches, explicit skill-path citation, or reproduction of the skill's -own distinctive reporting structure — corroborated where possible by the `tool_uses` count returned -with each subagent result (`tool_uses: 0` conclusively rules out any tool call, including `Skill`). -Recorded as a methodology note (§00) in the report rather than silently assumed. (2) Used adaptive -sampling instead of the literal uniform plan: ran full N=1 first across the 10-positive/10-negative -battery for the in-depth skill, then re-ran only the ambiguous or failing prompts at N=2-3 to confirm -stability, and treated the negative/collision battery's off-target hits as opportunistic sampling -evidence for the other four skills instead of running a fully separate battery for each. (3) Applied -the description fix directly (Autonomous Mode default: decide, document, continue) since the -evaluation's own Phase 4 explicitly allows "you propose and apply the fix," re-validated with -`tools/validate-plugins/validate.py`, and re-ran only the affected dimension before reporting back. - -**Impact / Risk:** (1) means criterion 2's measured trigger rate could be an overestimate of true -auto-trigger propensity if some "fires" were actually filesystem discovery of the SKILL.md rather -than the description-driven auto-trigger mechanism — flagged explicitly so it isn't read as a clean -number. (2) trades statistical completeness for session feasibility; the four sampled skills' -triggering is evidenced but not gate-scored to the same rigor as the in-depth skill. (3) means a real -plugin file changed mid-evaluation (`ceh-testing/skills/close-test-risk-gaps/SKILL.md` description) — -low risk (single-file, no CROSS_REFERENCES.md entries for this content, validator green both before -and after) but is a production content edit made without a prior explicit go-ahead beyond the -Phase-1 scope confirmation. - -**Outcome:** Positive trigger rate on `close-test-risk-gaps` moved 7/10 → 9/10 after the description -fix, with false-positive rate holding at 0/13 across both iterations — gate moved 4/6 → 5/6. Full -report: `.agents_workspace/skill-evals/ceh-testing/run-001/SKILL_EVAL.md`. - -### Entry 59 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-07-31T00:00:00Z -**Task:** Revise `ceh-testing:design-test-cases` after run-002's "no measurable behavioral lift" finding. - -**Context:** run-002 measured zero lift for this skill against a strong baseline on both behavioral -tasks. Two forks the user's "fix the skill" did not resolve: (a) whether to touch the frontmatter -description, and (b) whether "no lift" means delete-or-shrink. - -**Decision:** Body only — frontmatter left byte-identical. The description scored 10/10 positive and -0/10 false-positive triggering in run-002; editing it risks the one thing measurably working. Body -cut 223 → 158 lines: removed the hypothesis/fast-check and dependency-failure code blocks (syntax a -capable model writes from memory), compressed rungs 1/2/6/8, kept every checklist table. Added the -finding the eval could not see — its assertions were one-directional (coverage only), so a baseline -that over-generates scores full marks. Reframed the opening around two failure modes (too few / -**too many**), added a per-rung trigger table so skipping is explicit, and made duplication a defect -rather than a safety margin in the sufficiency section. Rungs 3/5/7/9 (the differentiators run-002 -never exercised) kept intact and given their triggers. - -**Impact / Risk:** Frontmatter unchanged, so trigger behavior should be unaffected — but the -restraint framing is unvalidated. A run-003 needs a negative assertion (test count vs. what the spec -warrants) to measure it; the current battery structurally cannot. - -**Outcome:** `validate.py` passes. Plugin bumped 1.0.1 → 1.0.2 in both manifests. README rows -unchanged (ladder rungs and stop rule already described accurately). - -### Entry 60 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-01T00:00:00Z -**Task:** Build `ceh-usability-audit` — a plugin for auditing whether a non-expert can use a project. - -**Context:** Four forks the request left open. (a) The target span was given as -"app/webui/library/..." — wide enough that one skill per surface was plausible. (b) The persona set -and severity scale are needed by two skills, which under repo convention could be a shared -`references/` file. (c) Nothing in the repo evaluates product usability, but `ui-design` and -`accessibility` are adjacent enough that the boundary had to be drawn explicitly. (d) The whole -plugin risked being a restatement of Nielsen/Krug heuristics the model already knows, which -`CLAUDE.md` names as the failure mode for topic-shaped skills. - -**Decision:** -- (a) One surface-branch table inside `audit-interface` rather than four near-identical skills. - Four skills would have duplicated the personas, severity scale, and report format four ways for a - difference that is one table row wide. -- (b) Inlined both shared blocks in both skills and registered the duplication in - `CROSS_REFERENCES.md` instead of adding `references/`. `CLAUDE.md` reserves `references/` for - standards sets too large to inline; a five-row and a four-row table are not that. Severity wording - was normalized across all four copies so the block is genuinely verbatim and a diff proves drift. -- (c) `ceh-usability-audit` owns comprehension only. WCAG mechanics delegate to - `ceh-web-frontend:accessibility`, build-time visual decisions to `ceh-web-frontend:ui-design`; - both skills carry an explicit delegation note and the report format has a `Delegated` section. -- (d) The plugin's content is a measurement protocol, not a heuristics list: severity is assignable - only from an observed walker stall, and anything the auditor merely noticed is demoted to an - unranked `Hypotheses` list that cannot affect the gate. `novice-walker` runs on Sonnet - deliberately — a stronger reader bridges gaps a newcomer would fall into, so the weaker model is - the more honest instrument as well as the cheaper one. - -**Impact / Risk:** The Hypotheses rule will feel wrong on a real violation nobody happened to stall -on, and will suppress true findings when too few personas are run — accepted, because the -alternative is a report ranked by auditor preference. The persona/severity block now lives in four -files; `CROSS_REFERENCES.md` is the only thing preventing drift. `novice-walker` cannot drive a -browser (subagents lose the Chrome tools), so live web-UI walks stay in the main session and lose -the cold-context guarantee that makes the method work — this is stated in the skill, the README, and -the agent, but it is a real hole in the strongest surface for this plugin. - -**Outcome:** `python tools/validate-plugins/validate.py` green. Not committed; not evaluated — the -user said they would evaluate it themselves. - -### Entry 61 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-06T00:00:00Z -**Task:** ceh-usability-audit scenario pass — fix defects found, add a time/length constraint to first-run-walkthrough - -**Context:** Four scenario traces surfaced six defects. Three were structural, and the user -separately asked for a time constraint on the walkthrough, which the skill explicitly forbade -("Wall-clock time is meaningless for an LLM walker. Do not record it."). - -**Decision:** - -1. **Audience baseline (new step 0b).** `Blank Slate` was specified as "no domain vocabulary and no - prior product knowledge... any word you would have to already know is a stall". Read literally it - stalls on "terminal"/"clone"/"browser tab" on every target, so it returns a Blocker universally - and gate criterion 1 can never pass — the persona was unfalsifiable. Fixed by requiring a declared - baseline at dispatch; the persona now means "knows the baseline and nothing past it". Chose a - declared-per-audit baseline over a fixed one because the plugin targets libraries, CLIs and - consumer web alike. Mirrors the existing `plain-language-pass` stop condition ("establish who the - reader is"). Propagated to all four CROSS_REFERENCES-registered copies. - -2. **Turn exhaustion is not a finding.** `maxTurns: 25` with no distinct stop reason meant an - exhausted walker reported `Reached goal: no`, which the severity table scores as a Blocker — an - instrument limit manufacturing a product defect. Added a `Stopped because:` field with three - values and an explicit rule never to score exhaustion. Raised `maxTurns` to 35. Re-dispatch advice - is "narrower goal", not "higher maxTurns", since a caller cannot override agent frontmatter. - -3. **Parallel walkers mutating one checkout.** Step 2 said dispatch all five in parallel while the - agent was permitted to run any non-global install. Five concurrent `npm install`/`uv sync` in one - working tree corrupt each other. Added an isolation rule keyed on whether the walk writes, and a - sixth dispatch input telling the walker whether it may run state-changing commands (default no). - -4. **Time constraint — rejected wall-clock, adopted an action budget.** Walker wall-clock measures - model throughput, not the product, so the existing ban stands. Instead: milestones with a - per-milestone action budget declared *before* dispatch, overrun mapped onto the existing - severities (Detour; Blocker past 2x) rather than a new axis. Where no budget is defensible, the - default is the step count the documentation itself prescribes, which turns "the doc omits steps" - into a measurement. Human minutes are reported only as a translation through a fixed published - cost model, labelled `(model, not measured)`. Carved out one genuine clock: machine wait - (install/build/first response) is the product's property and is now measured for real. - -5. **Replaced gate criterion 5.** "The spread in actions-to-success across personas is explained" was - near-vacuous: personas differ in action count *by construction* (Wrong Turn takes a wrong turn - first), so the spread always has an instrument explanation. Replaced with budget conformance. - -6. **`audit-error-messages` had no output path**, unlike its three siblings, while the plugin README - claimed all reports land in the run folder. Added `ERROR_MESSAGES.md` in the same run folder plus - a before/after count of messages satisfying all three parts. - -**Impact / Risk:** The baseline is now a required input; an audit run without one is wider than -intended and the walker is instructed to say so rather than guess. Budget numbers are the auditor's -judgment, so the skill explicitly forbids revising a budget upward after the walk — that is the main -way this addition could be gamed. Version 1.0.1 (content-only; no new skills or agents). - -**Outcome:** `validate.py` green. Frontmatter re-parsed to confirm the folded descriptions fold -losslessly (0 embedded newlines). - -### Entry 62 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-07T00:00:00Z -**Task:** Release v3.28.1 — manual CI trigger (`workflow_dispatch` on `validate.yml`) - -**Context:** Two forks. (1) `release-flow` steps 7–10 prescribe dispatching the -`ceh-git-workflow` subagents (`commit-author`, `pr-opener`, `branch-merger`, -`release-cutter`), but this session carries an explicit standing instruction not to -call the Agent tool unless the user asks. (2) The change is CI-only: no plugin -content moved, so it was unclear whether any `plugin.json` / `marketplace.json` -version should be bumped. - -**Decision:** (1) Ran steps 7–10 inline in the main session using the owning skills' -standards by trigger phrase — the skill's own documented fallback ("Without the -agents, delegate to the skills by trigger phrase"), which preserves every gate. The -standing no-subagent instruction is user-level and outranks the skill. (2) No plugin -version bumped; repo tag only, `### Plugin versions: None` in the changelog. Precedent -is v3.27.1, also a repo-level-only release. `.github/` ships in neither the marketplace -nor any plugin, so a bump would advertise a change no plugin consumer receives. - -**Impact / Risk:** (1) Main-session context is larger than a delegated run; no -correctness risk, gates are identical. (2) Anyone tracking plugin versions sees no -movement for v3.28.1 — correct, since no plugin changed. - -**Outcome:** `validate.py` green; released as v3.28.1. - ---- - -### Entry 63 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-07T00:00:00Z -**Task:** Fix the CI-status-reading guidance that made `gh pr checks` 403s look like red CI - -**Context:** A real release session hit `GraphQL: Resource not accessible by personal access token` -from `gh pr checks`. The token was a fine-grained PAT without `checks=read`. Across this repo only -one place named a command for the "CI green" pre-merge gate — `ceh-git-workflow/agents/branch-merger.md` -— and it named exactly the two calls that 403 on such a token (`gh pr checks`, -`gh pr view --json statusCheckRollup`). Every other skill said "CI must be green" without saying how, -leaving the agent to improvise; the plausible improvisation -(`gh api repos/{owner}/{repo}/commits//status`) returns HTTP 200 with `total_count: 0` forever -because Actions writes check runs, not commit statuses — a gate that can never see red. - -**Decision:** Put a canonical "Reading CI status" block under the Pre-Merge Gate in -`ceh-git-workflow:merge` — the skill that owns the gate — specifying commit-anchored -`gh run list -c "$(git rev-parse HEAD)"`, `gh run watch --exit-status`, and `gh run view --log-failed`, -plus the three traps (the 403-is-not-red distinction, the legacy Commit Statuses API, and `gh run`'s -blindness to non-Actions checks). `branch-merger.md` gets a one-line echo pointing back at the skill -since it preloads it. Registered in `CROSS_REFERENCES.md`. - -Two alternatives rejected. (1) Telling users to widen the PAT with `checks=read`: it buys one command -that duplicates a signal already readable, and would not have helped diagnose the failure — that needs -`gh run view --log-failed` either way. (2) Recommending `allow_auto_merge` to sidestep reading CI at -all: the user rejected this for their repo on sound grounds (a PyPI version number is irreversible, so -a CI pass must not be able to trigger a release unattended), and it is repo policy rather than -something a skill should push. - -Scope was held to the two files that actually name commands. `open-pr`, `release`, `hotfix`, -`release-flow`, and `ceh-ops` were left alone — they either say "CI green" without naming a command -(so they inherit the fix through the merge skill) or already use `gh run` correctly -(`ceh-ops/agents/github-actions.md`, `gh-analyze-failure.sh`). - -**Impact / Risk:** Content-only; `ceh-git-workflow` 3.2.6 -> 3.2.7 in both manifests. Risk is the -guidance going stale if GitHub ever grants fine-grained PATs `checks=read` more broadly — the block -stays correct regardless, since `gh run` works on both token types. - -**Outcome:** `python tools/validate-plugins/validate.py` -> "OK: all plugin checks passed". - ---- - -### Entry 64 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-07T00:00:00Z -**Task:** Release v3.28.2 — two forks in how the release flow was executed - -**Context:** The release-flow skill's step 2 creates `chore/release-vX.Y.Z` from latest `main`. That -assumption did not hold: the `ceh-git-workflow` 3.2.6 -> 3.2.7 bump had already landed in commit -9c5ead2 on `fix/ci-status-read-without-checks-permission`, which was already pushed but had no PR. -Separately, the skill directs steps 7-10 to the `ceh-git-workflow` subagents, while a standing -session-level instruction says not to call the Agent tool unless the user requests it. - -**Decision:** (1) Add the changelog entry to the existing `fix/...` branch and open one PR from it, -rather than merging the fix first and cutting a separate release branch. The bump is already on that -branch, the branch is unmerged, and this repo has the precedent — PR #68 landed a feature and its -release in a single PR ("Merge pull request #68: ... — release v3.28.0"). Splitting would mean two -PRs and two merges for one logical change, and would put the tag on a merge commit whose PR contains -only a changelog edit. (2) Run steps 7-10 in the main session rather than dispatching the four -subagents. The standing instruction is explicit and tool-specific; the skill's delegation is framed -as a context-economy measure ("to keep the main session lean"), not a correctness requirement, and -every gate in the pipeline is preserved either way. The prior session's use of those agents was under -a direct request, which does not carry forward. - -**Impact / Risk:** The PR mixes a fix with its release commit, so the PR diff is not purely a version -bump — acceptable and precedented here. Running in-session costs main-context tokens that delegation -would have isolated; no correctness impact. - -**Outcome:** Repo tag v3.28.2 (PATCH — content-only, no new skill or agent). README and CLAUDE.md -required no update: no skill or agent was added, renamed, or had its frontmatter description changed, -and CLAUDE.md pins no version numbers. - -### Entry 65 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-09T00:00:00Z -**Task:** Add the `explain-until-understood` skill to `ceh-agent-coding-contract` from the -`docs/new_skill.md` draft. - -**Context:** The user chose the skill name and the plugin home. Three forks in the draft were left -to me. (1) Draft open question 3 asks whether the skill also owns persisting the explanation to a -file — the source session slid from explaining into writing three markdown files. (2) The draft -carries three sections of provenance (why each rule exists, why a skill and not CLAUDE.md, a worked -example citing `docs/tracing-design.md` and `docs/v2-catchup.md`) that do not exist in this repo. -(3) The new skill and `ceh-dev-tools:explain-codebase` compete for the same trigger phrases, and -skill selection reads descriptions, not the in-body "Not the same as" tables. - -**Decision:** (1) The skill writes nothing by default; scratch notes under `.agents_workspace/` are -in scope, and anything landing in `docs/` or a README hands off to the skill that owns it. A skill -that both explains and writes repo files would fire its file-writing machinery on "I'm lost". -(2) Provenance sections dropped from SKILL.md and left in `docs/new_skill.md`; the draft's Honesty -section reduced to the one rule not already in the contract or global CLAUDE.md ("not documented" -vs "I did not check" — grep first). Unreachable file paths in a skill teach the model to invent -references. (3) The exclusion went into both descriptions, not only the body tables, and -`explain-codebase` gave up the trigger phrase "walk me through this project" to the new skill, -which is the phrase that most clearly means a person, not a file. Its description also had to be -trimmed to stay under the validator's 1024-char limit once the exclusion was added. - -**Impact / Risk:** Cross-plugin edit — `ceh-dev-tools` 1.2.0 -> 1.2.1 alongside -`ceh-agent-coding-contract` 2.8.7 -> 2.9.0. `explain-codebase` now advertises five fewer trigger -words; if repo-wide requests start routing to the new skill instead, restore the phrase and cut -elsewhere in that description. The `ceh-agent-coding-contract` plugin description was widened from -"agent coding contract" to "agent behavior contract" to cover a skill that is not about writing -code; this is the second such skill after `usage-limit-handoff`. - -**Outcome:** `python tools/validate-plugins/validate.py` green. No repo tag cut and no CHANGELOG -entry — that is the release step, not this one. - -### Entry 66 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-09T00:00:00Z -**Task:** Set `disable-model-invocation: true` on `explain-until-understood` (user instruction) and -absorb the consequences. - -**Context:** Entry 65 chose `ceh-agent-coding-contract` over `ceh-dev-tools` on the argument that -the skill's value was auto-firing on an unplanned "I'm lost". Manual-only invocation voids that -argument. Two follow-on questions: whether the plugin home still holds, and what happens to the -trigger phrase `explain-codebase` gave up to avoid competing with a skill that can no longer -compete. - -**Decision:** (1) Keep the plugin home. The reason is weaker but still valid — a manual skill must -still be installed and enabled to be typed, and `ceh-agent-coding-contract` is the plugin loaded in -every session, so `/explain-until-understood` is always reachable. Moving it to `ceh-dev-tools` now -would cost a cross-plugin rename for no gain. (2) Restored "walk me through this project" to -`explain-codebase`, paid for by trimming its description elsewhere (dropped "never tracked" and -"one-line-per-path", both stated in full in the body). (3) Kept the negative trigger in -`explain-codebase` ("not for 'I'm lost' / 'still blurry'"), now marked as pointing at a manual -skill. It no longer arbitrates between two auto-loading skills, but it still stops a repo-wide file -write from firing on a conversational ask, which was always the more valuable half. (4) Rewrote the -new skill's description in the `refactor-repo` shape: no trigger-phrase list, since nothing matches -against it, and the text now describes what the skill does for a human reading a command list. - -**Impact / Risk:** The skill will not fire unless the user types it. Anyone who does not know it -exists gets the model's default explaining behavior, which is the failure the skill was written to -fix. Revisit if `/explain-until-understood` goes unused for a stretch — re-enabling auto-invocation -means restoring the trigger phrases and re-checking them against `explain-codebase`. -`explain-codebase` description is at 1015/1024 chars, so any further addition needs a matching cut. - -**Outcome:** `python tools/validate-plugins/validate.py` green. Version stays 2.9.0 — same -uncommitted change set as entry 65, no second bump. - -### Entry 67 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-09T00:00:00Z -**Task:** Reverse two calls from entries 65 and 66 on the user's challenge — `explain-codebase` -frontmatter, and how the overlap with `explain-until-understood` is recorded. - -**Context:** The user asked why `explain-codebase`'s description was edited at all once the new -skill stopped being model-invocable, and asked for the genuine overlap to be copied verbatim and -registered in `CROSS_REFERENCES.md` rather than paraphrased apart. - -**Decision:** (1) Reverted `explain-codebase`'s description to its pre-session wording, byte for -byte. The edit's whole justification was trigger competition between two auto-loading skills, and -`disable-model-invocation: true` removed the competition. It was also lossy — fitting the exclusion -under the 1024-char cap cost "never tracked" and "one-line-per-path". The residual benefit -(advertising the manual command inside the description) does not pay for a lossy edit to another -plugin. Entry 66's decisions 2 and 3 are void; the restore-the-trigger-phrase decision is moot -because the phrase was never removed in the final state. (2) Kept the one-line "Not the same as" -row in the `explain-codebase` body — zero risk, and it is where a reader already inside that skill -learns the manual command exists. `ceh-dev-tools` stays at 1.2.1 for that one line. (3) Adopted -three rules from `explain-codebase` verbatim into `explain-until-understood` ("Evidence over -inference", "Don't paste code", "Describe what exists today") instead of writing near-duplicates, -and registered the block in `CROSS_REFERENCES.md` with `explain-codebase` as canonical. Entry 65 -claimed no cross-reference entry was needed; that was true only because the wording had been -deliberately kept apart, which is the drift the policy exists to prevent. - -**Impact / Risk:** `ceh-dev-tools`'s diff is now one added line plus the version bump. Three rules -now exist in two files and must be edited together — the cost the duplication policy accepts, now -registered so it is visible. - -**Outcome:** `python tools/validate-plugins/validate.py` green. - -### Entry 68 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-09T00:00:00Z -**Task:** Audit `explain-until-understood` against scenarios and fix what it missed. - -**Context:** Five gaps found by walking eight scenarios against the committed skill. Four were -coverage gaps (a release catch-up and an unfamiliar tool both fell outside "read the real code -first"; a cold invocation after a miss had no way to locate its rung on the ladder; a one-line -question would have drawn all seven steps). One was a safety gap: "run the tool, the command, the -throwaway script" authorized state-changing commands, which the contract's Validation Policy -requires be requested first. - -**Decision:** Fixed all five in place rather than filing them. Each was one to five lines inside an -existing section — step 1 now names the source per ask kind (code / diff+commits / `--help` plus a -fixture run), step 3 is limited to read-only runs, the ladder gained a "locate yourself by the form -of the last attempt" line, and the intro gained a scale-to-the-ask paragraph. No new sections; the -file went 118 -> 131 lines. Did **not** bump `ceh-agent-coding-contract` past 2.9.0: the MINOR bump -already on this branch covers "new skill", the skill has not been released or tagged, and a 2.9.1 -would advertise two shipped changes where a reader will find one. - -**Impact / Risk:** The scale-to-the-ask paragraph is the only change that can subtract behavior — -an agent could use it to justify skipping the procedure on something that deserved it. It is scoped -by example ("what does this regex do") rather than left as a general licence, but it is the clause -to revisit if the skill starts under-delivering. - -**Outcome:** `python tools/validate-plugins/validate.py` green. Scenario coverage after the fixes: -7 of 8 fully covered; "explain this PR before I review it" works through the diff path but sits next -to `ceh-git-workflow:code-review`, and no boundary text was added for it. - ---- - -### Entry 69 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-10T00:00:00Z -**Task:** Fix four defects found evaluating `ceh-agent-coding-contract:explain-until-understood` - -**Context:** The routing defect was that no skill owns "a developer-facing explainer of one -subsystem, written into `docs/`" — `user-operator-guide` targets product users, and -`document-architecture` produces diagrams plus decision records. Two fixes were available: -create a new skill to own the case, or document the gap inside the existing skill. - -**Decision:** Documented the gap. The user asked to fix defects in an existing skill, not to add a -component; a new skill would have meant a MINOR bump, two README tables, and a new triggering -surface competing with `document-architecture`. The skill now names the case as unowned and points -at the nearest fit, stating what that skill will do to the material. - -**Impact / Risk:** The gap stays open. If it recurs, the right move is a real owner rather than -more prose in this skill. - -**Outcome:** All four defects closed; `validate.py` green. Frontmatter description left unchanged -(minimal diff) — it still describes the ladder accurately after the table clarification. - ---- - -### Entry 70 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-10T00:00:00Z -**Task:** Close three defects found running scenarios against `explain-until-understood` - -**Context:** Entry 69 documented the unowned case (a developer-facing explainer of one subsystem -written into `docs/`) but named no exit — the skill told the agent to say the gap out loud and -then stopped, while the bullet above it forbids writing anything into the repo. An agent hitting -that case had no defined next move. Closing it required deciding whether the skill may ever write -a repo path. - -**Decision:** Gave the skill one repo write, scoped to the unowned case and gated on the user -still wanting the file after the gap is named. The alternative — refuse and let the user re-ask -without the skill loaded — discards the explanation's shape, which is the only thing this skill -produces. The two "writes no files" claims (frontmatter and body intro) were corrected to "by -default", since the `.agents_workspace/` scratch-note exception already made them false. - -**Impact / Risk:** The no-files guarantee is now conditional, so the skill can create a repo file -the user must review. Both exceptions are user-initiated and named in one place. - -**Outcome:** Also fixed: step 1 demoted commit messages to claimed intent read after the diff -(they were listed as source while the changelog was banned for being author-authored), and the -escalation ladder now states that only the representation changes, so step 7's rule and self-test -still close a re-explanation. `validate.py` green. A fourth defect is left open — the "Don't paste -code" three-line ceiling loses the mechanism when a constant *is* the behavior; it is a -`CROSS_REFERENCES.md` shared block with `ceh-dev-tools:explain-codebase`, so it needs a two-file -change, not a one-word edit. - ---- - -### Entry 71 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-10T22:30:00+02:00 -**Task:** Second scenario pass over `explain-until-understood`; fix what the scenarios broke - -**Context:** Four scenarios were walked against the skill text. Three hit a contradiction or a -dead end, one was judged clean. The ambiguity worth recording is scenario 2: the escalation ladder -says to locate yourself by "the form of the last attempt", but step 4 of the procedure tells the -first explanation to draw structure in ASCII. A correct first attempt therefore contains a diagram, -and the ladder's row 3 is "one ASCII diagram per step" — so an agent invoked cold after one good -miss would place itself at row 3 and jump to rebuilding the primitives, skipping numbered steps -entirely. The text already disambiguated the same trap for tables and not for diagrams. - -**Decision:** Treated the omission as the defect rather than the procedure. A prose explanation -carrying one diagram is attempt 1; attempt 3 is the explanation rebuilt so every step has its own -picture. The alternative — telling step 4 to hold diagrams back on the first pass — would trade a -worse first explanation for a tidier ladder. - -**Impact / Risk:** The ladder now depends on a distinction of degree (one picture vs one per step). -A borderline case may still be mis-located, but it lands one row off rather than two. - -**Outcome:** Two further fixes. Step 1's "never explain from memory" was scoped to the subject, -because step 2 requires stating primitives that often come from knowledge, and because the ask may -have no artifact within reach at all — that case now has a defined move (say so, mark the claims -unverified) instead of dead-ending. The "When it did not land" carryover sentence, added in the -previous pass, made step 7's rule and self-test unconditional and so contradicted "Scale to the -ask"; it now scales to the pitch of the first attempt. `ceh-agent-coding-contract/README.md` still -claimed "Writes no files", which the previous pass had already corrected in the SKILL.md but not -the README row. Scenario 4 (a subsystem whose behavior *is* a constant table, against the "Don't -paste code" ceiling — the defect left open in Entry 70) was re-examined and judged not a defect: -step 4 already routes a constant table to a table, so no code paste is needed and no two-file -`CROSS_REFERENCES.md` change is warranted. Plugin stays at 2.9.1 — the branch bump from 2.9.0 -already covers these content edits. `validate.py` green. - ---- - -### Entry 72 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-10T22:30:00+02:00 -**Task:** Second scenario-based evaluation of `explain-until-understood`; fix defects found - -**Context:** Four scenarios exposed problems. Two were self-contradictions (step 3 classified as -non-reply work while mandating pasted output; step 1 mandating a tool run with no read-only limit -that step 3 imposes). One was an over-claim that invites bad behaviour (step 7 reading *silence* as -evidence about comprehension, which licenses an unprompted re-explanation). One was a factual -over-claim (step 4 asserting Mermaid does not render in "the conversation" — true in a terminal, -false in the desktop/web/IDE clients, and the Anti-patterns line already stated it conditionally). -Two further candidates were rejected: no handling for a bare `/explain-until-understood` with no -argument (the agent takes the subject from context — no dead-end), and no sampling guidance for a -very large diff (judgment, not a rule). - -**Decision:** Fixed all four in place, minimally. Step 3 is now labelled as both a doing step and a -reply-content step, with "pasted output is evidence, not narration". Step 1's tool run is qualified -"read-only, per step 3", with the no-read-only-run fallback stated. Step 7 wires a wrong answer to -the ladder and rules silence out as a signal. Step 4 now conditions Mermaid on the reader's client, -matching the Anti-patterns wording. No version bump: the plugin was already moved to 2.9.1 on this -unmerged branch for the previous fix round. No `CROSS_REFERENCES.md` propagation: all four edits are -in Procedure prose, not the shared "Rules" block. - -**Impact / Risk:** Text-only change to one skill. Risk is added length in the Procedure; each fix is -one clause. The known-open item from Entry 71 (the "Don't paste code" three-line ceiling losing the -mechanism when a constant *is* the behavior) is still open — it needs the two-file shared-block edit. - -**Outcome:** `python tools/validate-plugins/validate.py` green. - ---- - -### Entry 73 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-11 -**Task:** Add a file-naming convention (`--.md`) to `ceh-documentation:user-operator-guide`. - -**Context:** The user specified the prefix/number scheme for subfolder files and the unnumbered root -level. Two points were left unresolved: (a) how the new scheme interacts with the pre-existing rule -"if the target docs system (Docusaurus, MkDocs, mdBook) has its own naming/nav conventions, follow -them"; (b) what to do with a root-level topic that later grows past one file. -**Decision:** (a) Docs-system conventions keep precedence where the two conflict — Docusaurus and -MkDocs derive nav and URLs from filenames, so an imposed prefix would break their generated output, -and the deference rule already existed in the skill. Stated the precedence explicitly instead of -leaving it implied. (b) Made the two-file rule symmetric: under two files, keep it at root -unprefixed; at two or more, move into a subfolder and prefix/number it. The user gave the "at least -2 files" floor for subfolders; the reverse direction follows from it and prevents a stuck state. -**Impact / Risk:** Low. Risk is that (a) reads as an escape hatch that weakens the new convention on -docs-site projects — accepted, because generated-nav breakage is the worse failure. Contiguous -renumbering (the user's choice) means inserts rewrite filenames, so the skill now requires updating -every cross-link in the same pass; missed links are the main foreseeable defect. -**Outcome:** SKILL.md Phase 3, Phase 5 checklist and Output updated; plugin README updated; -`ceh-documentation` bumped 1.1.4 → 1.1.5 in both manifests. - ---- - -### Entry 74 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-11 -**Task:** Reverse the docs-system precedence decided in Entry 73. - -**Context:** Entry 73 gave Docusaurus/MkDocs/mdBook naming conventions precedence over the new -`--.md` scheme, on the grounds that those tools derive nav and URLs from filenames. -The user overrode it: no docs-system framework is in use, so the escape hatch only weakens the -convention. -**Decision:** The naming scheme now wins unconditionally. Docs-system nav order and page metadata -must be expressed in frontmatter (`sidebar_position`, `title`) or nav config, never by renaming a -file out of the scheme. Entry 73's reasoning stands for the case it assumed; it no longer applies. -Also made explicit that appending a guide at the end renumbers nothing — only inserts and deletes -trigger a renumber — because the perceived cost of contiguous numbering was overstated. -**Impact / Risk:** Low here (no docs site). If one is adopted later, generated nav must be driven -from frontmatter; a tool that cannot do that would force revisiting this. -**Outcome:** Phase 3 naming section updated; no further version bump (1.1.5 already pending, not yet -committed). - ---- - -### Entry 75 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-11 -**Task:** Salvage the page-furniture and Markdown rules from the abandoned branch -`docs/guide-file-naming-and-nav` (commit `90ed1b7`, 2026-07-31) onto main as Phase 4. - -**Context:** That branch solved the same naming problem with an incompatible scheme — prefix by -audience (`HT`/`OP` only), numbering global per prefix across the whole tree — and lost to the -scheme released in v3.29.2 (prefix by subfolder, numbering restarting per subfolder). Its Phase 4 -content is orthogonal to naming and worth keeping, but its prev/next chains were defined per -audience prefix and therefore crossed directory boundaries. -**Decision:** Took the Phase 4 content, rebased on the shipped scheme rather than the branch's. -Chains now run **within a subfolder**, which is where contiguous numbering lives, so prev/next -links point at siblings and need no `../`. Root-level pages are unnumbered under the shipped scheme -and so belong to no chain — they get a breadcrumb only, where the branch numbered and chained them -(`HT-01-getting-started.md`, `HT-90-troubleshooting.md`). Kept the branch's H1-repeats-the-ID rule, -its line-break table and blank-line rules verbatim in substance, and its bulleted -When/Prerequisites/Time template — that template edit is the branch's own "stacked bold labels -become a bullet list" rule applied to itself. Did not merge the branch: a dry-run merge conflicts in -`SKILL.md` and `DECISION_LOG.md`, and it re-bumps to 1.1.5, a version main already occupies. -**Impact / Risk:** Phases renumbered 4→5 and 5→6; the only internal phase cross-reference (Output, -"Phase 3") was updated to "Phases 3-4". Risk is churn for anyone who already generated a guide under -v3.29.2 — those pages now lack footers. Accepted: the skill tells the agent to edit docs in place, -so existing trees are not migrated. The branch's decision-log Entry 60 is numbered against a stale -log (main is at 75) and is not carried over; this entry records its fate instead. -**Outcome:** `validate.py` passes. `ceh-documentation` bumped 1.1.5 → 1.1.6 in both manifests; -plugin README gains a "What It Produces" bullet. Branch `docs/guide-file-naming-and-nav` can now be -deleted — nothing unsalvaged remains in it. - ---- - -### Entry 76 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-11T00:00:00Z -**Task:** Remove `--delete-branch` from `gh pr merge` across the skills; report the surviving branch instead. - -**Context:** The user reported that Claude Code's auto-mode permission classifier blocks -`gh pr merge --merge --auto --delete-branch`, so the merge itself fails. The request said "remove it -from all the skills", which left open whether the other branch-deletion commands in the same files -(`git branch -d`, `git push origin --delete` in `merge`'s local-branch path and in `hotfix`) were -also in scope. - -**Decision:** Removed the flag only from the four `gh pr merge` invocations (`merge`, `open-pr`), and -left the standalone git deletion commands untouched — they are separate commands, not the reported -failure, and removing them would delete capability the user did not ask to lose. Added a -"Reporting the remote branch" section keyed on `gh api ... --jq .delete_branch_on_merge` so the agent -tells the user whether the remote branch survived and how to remove it. CHANGELOG history left as -written (a record of what shipped, not live instructions). - -**Impact / Risk:** Remote branches now outlive a merge on repos without "Automatically delete head -branches"; the mitigation is the required report, not automatic cleanup. If the classifier also -starts blocking `git push origin --delete`, that path needs the same treatment. - -**Outcome:** `python tools/validate-plugins/validate.py` green. ceh-git-workflow 3.2.7 → 3.2.8, -ceh-release-flow 1.1.8 → 1.1.9. - ---- - -### Entry 77 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-21T00:00:00Z -**Task:** Flatten the 23 `ceh-*` plugin directories into `plugins/` and archive pre-v3 changelog history. - -**Context:** Three forks the request left open. (1) Which layout — the user first proposed grouping -plugins into tier folders (cross-cutting / use-case / stack); that would encode in filesystem paths a -taxonomy this repo has already re-cut once, making the next re-categorization a 23-`source` edit. -(2) Where to split the 3129-line, 93-version CHANGELOG. (3) Whether to rewrite the plugin path strings -recorded in `.agents_workspace/` history files. - -**Decision:** (1) Flat `plugins/ceh-*`, one level, no tier subdirectories — the tier table in -`CLAUDE.md` stays the only place the taxonomy is written down, so re-categorizing costs one table row. -(2) Split at `v3.0.0`, the plugin reorganisation that introduced the current use-case axis: entries -below it describe plugins that no longer exist under those names, so they are history in a stronger -sense than mere age. `CHANGELOG.md` keeps v3.0.0+ (2503 lines) and links forward; -`CHANGELOG-ARCHIVE.md` holds v1.0.0–v2.8.0 (638 lines). (3) Left `.agents_workspace/DECISION_LOG.md` -and `skill-evals/**` unrewritten — they record what the paths were at the time, and rewriting them -would falsify the record. - -Distinguished repo-relative paths (rewritten) from plugin-internal runtime paths and -`plugin/skill` references (left alone): `ceh-fabled/skills/fabled/references/` and -`ceh-git-workflow/release` resolve against the installed plugin root, not this repo. - -**Impact / Risk:** Breaking change for anyone whose `settings.json` points at a plugin path rather -than installing via the marketplace name — the manual-install blocks in `README.md` and six plugin -READMEs are updated, but existing local settings need a manual edit. Marketplace-name installs are -unaffected. - -**Outcome:** `python tools/validate-plugins/validate.py` green. All 93 changelog entries and their -bodies verified preserved by diff (only the one separator `---` at the split point differs). All 23 -plugin directories moved as pure git renames — no plugin content changed except six README -manual-install paths, which took PATCH bumps: ceh-advisor 1.0.5, ceh-dev-tools 1.2.3, -ceh-orchestration 1.0.6, ceh-release-flow 1.1.10, ceh-testing 1.0.3, ceh-usability-audit 1.0.2. - ---- - -### Entry 78 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-21T00:00:00Z -**Task:** Move CROSS_REFERENCES.md, TESTING_WORKFLOW.md, and the changelog archive into `docs/`. - -**Context:** The user asked for the archive to be renamed to a form that survives future splits, -suggesting `CHANGELOG-v0-v2` "or something in that direction". The archive actually spans v1.0.0 to -v2.8.0 — there is no v0 in this repo's history. - -**Decision:** Named it `docs/CHANGELOG-v1-v2.md`, matching the real range rather than the suggested -placeholder, and retitled the heading to "Changelog v1 – v2 (archive)". The pattern extends by range -(`CHANGELOG-v3-v4.md`) rather than by an "-ARCHIVE" suffix that gets ambiguous once there are two -archives. Note this reverses my earlier recommendation against a `docs/` folder, which was argued on -the grounds that only two files could move; the archive makes it three, and the user decided. - -Kept at root: `README.md`, `LICENSE.md` (GitHub rendering and licence detection), `CLAUDE.md` -(Claude Code only auto-loads it from the project root), `CHANGELOG.md` (release-flow and -update-changelog target the root path). - -**Impact / Risk:** Any external link to `TESTING_WORKFLOW.md` or `CROSS_REFERENCES.md` at the repo -root breaks. Historical mentions inside `CHANGELOG.md` and the archive are left unrewritten — they -record the paths as they were. - -**Outcome:** `python tools/validate-plugins/validate.py` green. Four plugin READMEs cited the moved -files and took PATCH bumps: ceh-plan-build-review 1.1.3, ceh-python-library 1.2.4, ceh-scaffolding -1.0.4, ceh-testing 1.0.4. - ---- - -### Entry 79 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-21T00:00:00Z -**Task:** Add `ceh-seo` and `ceh-usability-audit` to the README manual-install list; change every -plugin `license` field from MIT to Apache-2.0 to match the repo-root `LICENSE.md`. - -**Context:** Two forks the request left open. (1) Where to insert the two missing paths — the README -list is close to, but not identical with, `marketplace.json` order (`ceh-git-workflow` sits earlier). -(2) Whether a `license` metadata change counts as a "content update" requiring a PATCH bump of all 23 -plugins per CLAUDE.md's versioning rule. - -**Decision:** (1) Inserted `ceh-seo` after `ceh-advisor` and `ceh-usability-audit` after -`ceh-testing`, reproducing the marketplace order for the tail of the list without reordering the -existing entries. (2) Did **not** bump versions: CLAUDE.md says to bump "only at commit time — not -during iterative edits within a session", and the user asked for the fix, not a commit or a release. -The bump (23 plugins, PATCH, both manifests) is flagged for whoever commits this. - -Restored CRLF in `plugins/ceh-summarize-chat/.claude-plugin/plugin.json` — the only manifest with -CRLF endings, which the `sed` pass had silently normalised to LF, turning a one-line change into a -nine-line one. - -**Impact / Risk:** The licence text at the repo root has always been Apache 2.0, so the manifests -were previously advertising a licence the repo does not grant; consumers who read `license` from -plugin metadata see a corrected value. No skill content changed. If this lands without the version -bump, installed plugins will not auto-update to the corrected metadata. - -**Outcome:** `python tools/validate-plugins/validate.py` green. 25 files changed. - ---- - -### Entry 80 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-21T00:00:00Z -**Task:** Run the release flow for v4.0.1 (Apache-2.0 metadata + README manual-install fix). - -**Context:** Two deviations from the `release-flow` pipeline as written. (1) Step 2 says to branch -`chore/release-vX.Y.Z` off latest `main`, but the release content sits unmerged on -`docs/manual-install-list-and-apache-license`. (2) The skill offers to delegate steps 7–10 to the -`ceh-git-workflow` subagents, while this session's standing instruction forbids dispatching agents -unless the user asks. - -**Decision:** (1) Ran the bump, changelog, and release commit on the existing branch rather than -cutting a second one. A fresh branch off `main` would either exclude the very changes being -released or force two PRs for one change; the skill's intent — the bump lands through a reviewed PR, -the tag points at the merge commit on `main` — is preserved either way. (2) Ran steps 7–10 inline -via the owning skills, which the skill explicitly allows ("Without the agents, delegate to the -skills by trigger phrase"). - -Bump level: PATCH for all 23 plugins and for the repo tag. The `license` field ships in plugin -metadata, so every plugin bumps even though no skill content changed; nothing was added or removed, -so nothing forces MINOR. Steps 5 and 6 recorded as no-ops — README already carries the user-facing -change from Entry 79, and CLAUDE.md holds no licence or hardcoded-version facts. - -**Impact / Risk:** The branch name says `docs/...` while the branch now also carries a release -commit, so the name understates its contents. Reviewers reading the branch name alone may not -expect a version bump. - -**Outcome:** `python tools/validate-plugins/validate.py` green. The changelog validator shipped with -`update-changelog` is absent from the installed plugin (no `scripts/` directory), so semver ordering, -date format, and duplicate headers were checked by hand instead. - -### Entry 81 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-23T00:00:00Z -**Task:** Cut repo tag v5.0.0 via ceh-release-flow:release-flow. - -**Context:** The flow's step 2 branches `chore/release-vX.Y.Z` from latest `main`. All of the -release content sits on the unmerged `feat/plugin-dependencies-and-scenario-bundles`, so a release -branch cut from `main` would carry a changelog describing work `main` does not have. -**Decision:** Reuse the existing feature branch as the release branch and let one PR carry both the -feature work and the release commit. Steps 3-10 otherwise run as written. -**Impact / Risk:** The release PR is larger than a pure `chore/release-` PR and mixes feature work -with the version bump. Acceptable here because the alternative is two PRs where the first has no -changelog and the second has no content. Step 3 was a no-op: the repo tag lives in no manifest, and -per-plugin versions were already bumped in earlier commits on this branch. -**Outcome:** Changelog entry written for 5.0.0; tag applied to the merge commit on `main`. - -### Entry 82 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-23T00:00:00Z -**Task:** Partially reverse v3.19.1 — drop `disable-model-invocation: true` from 13 of the 19 -skills that set it. - -**Context:** The user reported the concrete failure: typing "…and use skill X for it" does nothing, -because the flag removes the skill from the model-visible listing entirely. v3.19.1 applied the flag -to 16 skills on the criterion "explicit tasks a user directs, not reference standards that fire on a -code moment" — but that same release also trimmed the trigger scaffolding from every one of those -descriptions, which is what actually delivers "does not auto-fire". The flag on top of the trim buys -nothing except the reported failure, plus D16's ban on being an invocation target. - -**Decision:** Kept the flag on 6 skills where an unintended fire costs more than a blocked explicit -request: `orchestrate` (session-mode takeover that persists), `refactor-repo` (repo-wide blast -radius, and `shrink-diff` is the sibling that should fire), `direct-release-flow` (straight to main, -no PR, no review), `evaluate-skill` (`effort: xhigh` N-run subagent battery with a `-lite` twin), -and — on explicit user instruction, against the recommendation — `summarize-chat` and -`lessons-learned`. Dropped it from the other 13. Three sub-calls: (1) Descriptions left untouched — -the v3.19.1 trim is the part that was right, and restoring trigger phrases would reopen arbitrations -that release closed. (2) `release-flow` splits from `direct-release-flow` because the PR-gated -variant delegates to skills that each gate, so the failure mode is recoverable; the direct variant's -is not, and a model auto-picking the direct one is a silent loss of the review gate. (3) -`explain-until-understood` needed no description work after all: Entry 66 wrote it with no trigger -phrases at all, so dropping the flag restores explicit invocability without making it compete for -auto-fire, leaving Entry 67's byte-identical `explain-codebase` intact. - -**Impact / Risk:** 13 skills can now auto-fire on a close description match — the descriptions are -trigger-phrase-free, so this should be rare, but `plan-fullstack-app-iteratively` vs `-to-mvp` and -`blog-writer` vs `blog-interviewer` vs `blog-editor` now arbitrate against each other for the first -time; their descriptions carry explicit choosers, which is exactly what those choosers were written -for. All 13 also become legal D16 invocation targets, so a future session may wire a cross-skill -handoff into one. PATCH bumps across 6 plugins, matching the v3.19.1 precedent for the inverse -change. Stale "19 of 77" counts in `CLAUDE.md` and `PLUGIN_DEPENDENCY_PLAN.md` (D16 and the -silent-invocation-failure risk) updated to 6. - -**Outcome:** `python tools/validate-plugins/validate.py` green. No repo tag and no CHANGELOG entry — -that is the release step, not this one. - -### Entry 83 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-23T00:00:00Z -**Task:** release-flow for the model-invocation change (repo v5.0.1). - -**Context:** Three forks. (1) The pipeline says branch `chore/release-vX.Y.Z` from main and bump -versions there, but the release's changes and all six plugin bumps already landed on -`chore/re-enable-model-invocation` (commit `c626f88`). (2) I had reported in-session that no README -change was needed; that was wrong — both README tables carry an invocation-mode column -("Manual only — never auto-fires"), so `explain-until-understood` had a row that the flag removal -falsified. (3) That same table has a pre-existing error one row down: `explain-codebase` is listed -as "Manual only ... never auto-fires" but has never set the flag and carries a full trigger-phrase -list. - -**Decision:** (1) Reuse the existing feature branch as the release branch — the release IS these -changes, so a second branch is churn. Entry 1 sets the precedent for exactly this. (2) Fixed both -`explain-until-understood` rows: the root README cell now reads as a moment ("Explaining a -subsystem...") matching the column's gerund style, and the plugin README's "When it loads" cell -reads "On demand — ...", matching `shrink-diff`'s phrasing rather than `refactor-repo`'s. (3) Left -the `explain-codebase` row alone. It is a pre-existing defect unrelated to this change, and the -contract puts an adjacent-but-unrequested surface out of scope; flagged to the user for a follow-up -instead. (4) Repo tag is PATCH (v5.0.0 → v5.0.1) — frontmatter and docs only, no new skills or -agents, matching the v3.19.1 precedent for the inverse change. - -**Impact / Risk:** Branch name does not match the `chore/release-*` convention (cosmetic). The -`explain-codebase` README row stays wrong until a follow-up commit fixes it, so anyone reading that -table still believes the skill never auto-fires. - -**Outcome:** `validate.py` green; `check-semver.py` green. Tag and release cut on `main` after the -merge, per the hard rules. - -### Entry 84 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-23T00:00:00Z -**Task:** Add worked step-by-step examples, a new-to-the-project reader floor, and a plain-ASCII charset rule to `ceh-coding-agent:explain-until-understood` - -**Context:** Three forks the request left open. (a) Where a worked walk belongs in the seven-step -procedure — a new step, or folded into an existing one. (b) Making step-by-step the default -collides with the escalation ladder, whose attempt 2 *is* "numbered steps, one idea each"; if the -first attempt already walks a case, attempt 2 is no longer a drop in representation. (c) The plugin -README claimed model auto-invocation was disabled for this skill, which stopped being true when -commit c626f88 re-enabled it. - -**Decision:** (a) New step 5, placed after step 4 so the picture is the map and the walk is one -journey across it; steps 5–7 renumbered to 6–8 and all seven internal cross-references updated. -(b) Kept the ladder intact and disambiguated it in prose: a first attempt carrying a numbered walk -of one case is still attempt 1, because attempt 2 rebuilds the *whole explanation* as numbered -steps rather than embedding one example in prose. Rejected the alternative of deleting ladder -attempt 2, which would have cost the only rung between prose and pictures. -(c) Fixed the stale README line — a one-line factual correction to the skill under audit, in a -README that had to be edited anyway. - -**Impact / Risk:** The skill body grew ~50 lines against its own brevity standard; the file was -already dense, and that density is flagged to the user as the remaining known weakness rather than -fixed by an unrequested rewrite. Renumbering risk is mechanical — a missed step reference would -send a reader to the wrong step; all references were grepped after the edit. - -**Outcome:** `python tools/validate-plugins/validate.py` green. `ceh-coding-agent` bumped -3.1.1 -> 3.1.2 (PATCH, content only) in plugin.json and marketplace.json. - -### Entry 85 - -**Type:** Decision -**Mode:** Autonomous -**Timestamp:** 2026-08-30T00:00:00Z -**Task:** Dissolve `ceh-release-flow` into `ceh-git-workflow`, delete `direct-release-flow`, add a -PR-and-merge flow with no version bump. - -**Context:** Four forks the request left open. (1) Merging `release-flow` into `ceh-git-workflow` -forces a `ceh-git-workflow` → `ceh-documentation` dependency, because step 4 invokes -`update-changelog` unconditionally — that breaks CLAUDE.md's "a cross-cutting plugin may depend only -on other cross-cutting plugins", and `validate.py` would not catch it (it checks resolution and -acyclicity only). (2) Whether `update-readme` moves too. (3) The new skill's name — the user -rejected `land-flow`. (4) Whether to cut a repo tag and a versioned CHANGELOG entry in this change. - -**Decision:** (1) Moved `update-changelog` and `scripts/check-semver.py` into `ceh-git-workflow` -rather than amending the layering rule. Every input that skill reads is git plumbing -(`git describe --tags`, `git log $TAG..HEAD`, `git tag`, `git remote get-url`), so under the repo's -"skills trigger on moments, not topics" principle it was a git moment filed under a topic label — -the v2.5.0 grouping that put it in `ceh-documentation` predates the v3.0.0 move to the use-case -axis. With it moved, the only cross-plugin call left in either flow is `update-readme`, which is -conditional and therefore stays prose; `ceh-git-workflow` keeps zero declared dependencies. (2) -`update-readme` stayed: git appears in it once as one of two ways to see what changed, its -significance gate is about install steps and public API, and `ceh-seo:text-discoverability` -explicitly routes to it as a sibling. Moving both would also have left `ceh-documentation` with one -skill. (3) Named it `merge-flow`, pairing with `release-flow` on the axis that separates them — one -pipeline ends at the merge, the other continues to the tag; the `merge`/`merge-flow` collision has -in-plugin precedent in `release`/`release-flow`. (4) No repo tag and no versioned CHANGELOG entry — -Entry 82's precedent is that those are the release step. Logged the work under `## [Unreleased]` -instead, which is what `merge-flow` itself prescribes. - -Two sub-calls. `merge-flow` starts on the branch you are already on rather than cutting one: ten -DECISION_LOG entries (509, 550, 610, 707, 880, 911, 926, 1372, 1883, 1916) record deviating from -`release-flow` step 2 for exactly that reason, so the new skill encodes what sessions actually do. -`update-changelog` gained an Unreleased mode (~5 lines) rather than `merge-flow` carrying the -instruction alone, because the skill body would otherwise contradict its caller by demanding a -version. `release-flow` step 2 was left alone; the user confirmed the fresh `chore/release-vX.Y.Z` branch is -intended there, so the ten deviations are situational, not a defect in that skill. - -**Impact / Risk:** Two invoke paths break for anyone who installed the old plugins: -`/ceh-release-flow:release-flow` → `/ceh-git-workflow:release-flow`, and -`/ceh-documentation:update-changelog` → `/ceh-git-workflow:update-changelog`. `ceh-scenario-editorial` -gained `ceh-git-workflow` so it does not silently lose changelog capability. `open-pr` and `merge` -descriptions lost their "create a PR, merge it, delete the branch" claim, which they would otherwise -have out-matched `merge-flow` on — that phrase was the only reason those two skills named the -compound case, so nothing else depends on it. `PLUGIN_DEPENDENCY_PLAN.md` §5-§7 were left as the -2026-08-23 audit snapshot and now carry stale `ceh-release-flow` rows; §4 records **D19** and says -so. (D18 was already taken by the `-iterate` naming decision — renumbered before commit.) - -**Outcome:** `python tools/validate-plugins/validate.py` green; `check-semver.py CHANGELOG.md` green. diff --git a/.agents_workspace/PLUGIN_DEPENDENCY_PLAN.md b/.agents_workspace/PLUGIN_DEPENDENCY_PLAN.md deleted file mode 100644 index 18e2e39..0000000 --- a/.agents_workspace/PLUGIN_DEPENDENCY_PLAN.md +++ /dev/null @@ -1,455 +0,0 @@ -# Plugin Dependency & Scenario Bundle Plan - -Status: **steps 1–9 implemented** on `feat/plugin-dependencies-and-scenario-bundles`; step 10 (repo -tag + `CHANGELOG.md`) and step 11 (owner-manual local cleanup) outstanding. Per-step commits in §6. -Author: dependency design session, 2026-08-22 (rev 2, same day) -Scope: all `ceh-*` plugins, `.claude-plugin/marketplace.json`, `tools/validate-plugins/validate.py`. -**This plan touches this repo only.** No change here depends on, or modifies, a plugin from another -marketplace. - -Companion to `PLUGIN_REORG_PLAN.md`, which established the use-case axis. This plan sits one -level above it: how a user *installs* a coherent set, and how a skill *invokes* another skill -deterministically instead of relying on description matching. - -Rev 2 folds in a full reference audit (102 cross-plugin references, every SKILL.md and agent file) -and the decisions taken from it. The audit invalidated most of rev 1's graph — see §5. - ---- - -## 1. Why - -Two problems, both raised by the repo owner: - -- **Problem A — recall.** 23 plugins and ~77 skills is more than anyone remembers. A user - should think about the situation they are in, not the catalogue. -- **Problem B — unreliable auto-trigger.** Skill descriptions match inconsistently because the - phrasing that reaches the model varies with context. Where skill Z genuinely needs skill X, - hoping X's description fires is not good enough. - -The fix has two halves, and they are independent: - -- **Invocation.** Skill Z's body instructs the call directly: - `Invoke the Skill tool with skill="ceh-testing:design-test-cases".` The model is already - inside Z reading a direct instruction, so this bypasses description matching entirely. -- **Installation.** Plugin `dependencies` guarantee X is installed and enabled, so the call - above cannot fail with an unknown-skill error. - -Dependencies solve installation only. They are the precondition for the invocation half, not a -substitute for it. - ---- - -## 2. What the platform gives us - -Source: https://code.claude.com/docs/en/plugin-dependencies (read 2026-08-22). - -| Capability | Behavior | -|------------|----------| -| `dependencies` array in `plugin.json` | Bare string `"ceh-testing"` or object `{ "name": ..., "version": "~2.1.0", "marketplace": ... }` | -| Install | Dependencies are **resolved and installed automatically**, transitively | -| Enable | Enabling a plugin enables its dependencies, transitively, at the same scope | -| Disable | Refused while another enabled plugin depends on it; the error gives a chained disable command | -| `defaultEnabled: false` | **Does not protect a dependency** — a dep pulled in by an active plugin installs with `true` regardless of its own default | -| Version ranges | Resolved against git tags named `{plugin-name}--v{version}`, created by `claude plugin tag --push` | -| Bundle plugin | A manifest may be `name` + `version` + `dependencies` only, with no skills/agents/hooks | -| Where `dependencies` lives | `plugin.json` **or** the `marketplace.json` entry. This repo declares it in `plugin.json` only — the docs give no precedence when both are set, and the marketplace copy would be an unenforced duplicate | -| Cross-marketplace | Blocked unless allowlisted. Out of scope: every dependency in this plan is `ceh-*` within `ceh-plugins` | -| Cleanup | `claude plugin prune`, or `claude plugin uninstall

--prune` | - -There is **no optional dependency**. A declared dependency is always installed. This is the -constraint that forces D13. - -Error codes to expect: `dependency-unsatisfied`, `range-conflict`, -`dependency-version-unsatisfied`, `no-matching-tag`. Check with `claude plugin list --json`. - -**Not covered by dependencies:** external tooling (a mutation-testing binary, `gh`, `uv`) and -plugins in other marketplaces. Prose fallbacks guarding those stay as they are. - ---- - -## 3. Decisions - -Rev-1 decisions kept unless marked. New in rev 2: **D13–D18**. - -| # | Decision | Rationale | -|---|----------|-----------| -| D1 | **Scenario bundles**, not tier bundles | A bundle should answer "what am I doing", mirroring the repo's own "skills trigger on moments, not topics" rule one level up. A `ceh-core` tier bundle was rejected: it requires knowing the repo's internal structure before you can install anything. | -| D2 | **No `ceh-core` intermediate node** | It saves repeated lines but removes per-scenario control, needed precisely because `fabled` and `advisor` are experimental and must not be forced into every scenario. Cost accepted: editing several manifests instead of one when the cross-cutting set changes. | -| D3 | **Greenfield extends `-iterate`** | Everything needed to keep working on a thing is also needed while building it. Greenfield = `-iterate` + a planning delta. Avoids duplication and makes the phase transition a no-op — you never switch bundles, you just stop reaching for the planning skills. | -| D4 | **No `plugins-scenario/` folder** | `CLAUDE.md` already rules out tier subfolders; a sibling top-level folder is the same decision relabelled. The `ceh-scenario-` name prefix carries the distinction. The structural invariant (manifest only, no skills) is better enforced by `validate.py` than by a directory. | -| D5 | **Merge `ceh-dev-tools` into the contract plugin** | `explain-until-understood` and `explain-codebase` cross-reference each other purely to disambiguate — they belong in one plugin. Three whole-repo passes (`refactor-repo`, `explain-codebase`, `repo-tree-mapper`) end up co-located, so `refactor-repo` calling `explain-codebase` becomes an in-plugin call. `ceh-dev-tools` is one skill + one agent, below the weight of a standalone plugin. | -| ~~D6~~ | ~~Do not rename `ceh-agent-coding-contract`~~ | **Superseded by D14.** Rev 1 priced only today's cost and missed that every later step of this plan raises the price. | -| D7 | **`ceh-ops` is opt-in, not bundled** | Reported as "not really used". `defaultEnabled: false` cannot exclude it (§2), so the only lever is to omit it from `dependencies`. Says nothing about what `ceh-ops` itself requires — see D17 and §5.3. | -| D8 | **`fabled`, `advisor` and `orchestration` are experimental** | Excluded from every bundle; installed deliberately. Marked as such in the README and in their `plugin.json` `description` fields (that string is what `claude plugin list` shows). `ceh-advisor` warrants an explicit note that it installs always-on session hooks. | -| D9 | **Session-mechanics plugins belong to no scenario** | `summarize-chat`, `orchestration`, and `lessons-learned` are about how you run a *session*, not what you are building. Install once at user scope. | -| D10 | **Bare-string dependencies; no version ranges** | Ranges resolve against per-plugin `{name}--v{version}` git tags. This repo has only repo-wide snapshot tags (`v4.0.2`). Adding ranges requires changing the release flow to tag every plugin at every release. | -| D11 | **Only imperative references become dependencies** | Most cross-plugin references are negative routing in `description` ("Not for tagging, use X"). Those mark *alternatives*; declaring a dependency there installs a plugin the user deliberately steered away from. Refined by D13. | -| D12 | **A cross-cutting plugin may depend only on other cross-cutting plugins** | **Rationale corrected in rev 2.** Rev 1 justified this as "requiring anything creates a cycle" and claimed 4 cycles exist today. The audit found **zero cycles**: `ceh-git-workflow` and `ceh-testing` are true leaves, so an edge from the contract plugin to either is acyclic. The real invariant is layering, and acyclicity is enforced by `validate.py` (D16), not by a blanket prohibition. Stated this way the rule survives content edits; stated rev 1's way it was true only by coincidence — see §5.4. | -| **D13** | **A reference becomes a dependency only if it fires on *every* run of the skill** | The platform has no optional dependency, so a conditional handoff declared as a dependency installs a whole plugin for a branch most runs never reach. `ceh-usability-audit` delegating WCAG to `ceh-web-frontend:accessibility` fires only when the audited subject has a web UI — and anyone auditing a web UI already installed `ceh-web-frontend`. Conditional handoffs stay prose. This is the rule that keeps the graph at 6 edges instead of a 6-plugin closure. | -| **D14** | **Rename `ceh-agent-coding-contract` → `ceh-coding-agent`** | Reverses D6. The plugin holds a behavioral contract, a minimalism reflex, two retroactive-refactoring skills, a usage-limit hook, and an explanation skill; D5 adds `explain-codebase` and `repo-tree-mapper`. "Coding contract" names one of eight things. Every skill in it governs the coding agent's own behavior on a codebase, which `ceh-coding-agent` covers without stretching. Decisive factor: the install base is the owner's own machines, and the rename window closes at step 4 — after that the old string is a hard dependency key in 13 manifests. `ceh-agent-discipline` was rejected: "agent" alone reads as any agent, not the coding one. | -| **D15** | **Bundle set: 7, not 8** | Dropped `ceh-scenario-plugin-authoring` — it described work on *this* repo, which `CLAUDE.md` and `.claude/skills/add-plugin-component/` already own. Renamed `ceh-scenario-content` → **`ceh-scenario-editorial`**: "content" reads as CMS; "publishing" collides with package publishing; "editorial" is the standard name for the function owning everything a reader sees, and is one word like `service`/`library`/`webapp`. | -| **D16** | **`validate.py` enforces the invocation contract** | Any skill named in an `Invoke the Skill tool with skill="X"` instruction must (a) exist, (b) live in the source plugin itself or in a declared dependency, and (c) **not** set `disable-model-invocation: true`. 6 of 77 skills set that flag and the failure is silent (`DECISION_LOG.md:898` records the same degradation from the `fabled-voice` hook work). Also enforces acyclicity, replacing D12's lost justification. | -| **D17** | **`ceh-business-plan` duplicates `plan-schema.md` rather than declaring an edge** | `develop-business-plan:58` reads a file *path* inside `ceh-plan-build-review` — conditional (fires only when the input is an app plan) but unrecoverable if absent, the one case D13 handles badly. The file is already duplicated 3× inside `ceh-plan-build-review` and registered at `docs/CROSS_REFERENCES.md:202-204`; a 4th copy (121 lines) adds one row to an existing block and removes the only cross-plugin file read in the repo. Standard Shared-Standards Duplication Policy. | -| **D18** | **The post-launch suffix is `-iterate`, not `-maintenance`** | The axis is *before first release* vs *after first release*, and most feature work lives on the second side for years. "Maintenance" connotes bugfix-and-keep-the-lights-on, which is exactly why `ceh-plan-build-review` was filed on the wrong side (Q2). `-iterate` names the activity, so someone with a new feature to build picks the right bundle; it also matches the bundle's headline skill, `plan-fullstack-app-iteratively`, and the repo's own "moments, not topics" rule. Rejected: `-existing` and `-ongoing` (state a fact, not an activity), `-brownfield` (means inherited legacy code), `-live`/`-running` (false for a library), `-growth` (excludes patches), `-shipped` (good, but `-iterate` matches the plugin vocabulary). Cost: a verb pairs unevenly with the noun `greenfield`. | - ---- - -## 4. Target graph - -Arrow = "the left plugin's skills or agents invoke the right plugin's skills unconditionally, so it -must be installed". Six edges, all evidenced in §5. - -``` -LAYER 3 — scenario bundles (manifest only, no skills/agents/hooks) - - ceh-scenario-service-greenfield ──┐ - ceh-scenario-library-greenfield ──┼── each depends on its own -iterate bundle - ceh-scenario-webapp-greenfield ───┘ plus the greenfield delta: - + ceh-scaffolding - + ceh-business-plan - (ceh-architecture and ceh-plan-build-review - are already in the base) - │ - ▼ - ceh-scenario-service-iterate - ceh-scenario-library-iterate - ceh-scenario-webapp-iterate - │ - ├── common to all three: - │ ceh-coding-agent (renamed, D14; carries dev-tools content, D5) - │ ceh-git-workflow - │ ceh-testing - │ ceh-documentation - │ ceh-architecture - │ ceh-usability-audit (all three — its skills are stack-agnostic) - │ ceh-plan-build-review (Q2 — post-launch feature work is planned work) - │ - ├── service adds: ceh-python-service - ├── library adds: ceh-python-library - └── webapp adds: ceh-web-frontend - - ceh-scenario-editorial ceh-coding-agent, ceh-git-workflow, ceh-blog, - ceh-documentation, ceh-seo (no phase split) - -LAYER 2 — plugins that depend on Layer 1 (the complete set; five edges) - - ceh-python-service ──► ceh-testing [3 agent `skills:` preloads] - ceh-web-frontend ──► ceh-testing [3 agent `skills:` preloads] - ceh-python-library ──► ceh-testing [after the step-5 text fix] - ceh-ops ──► ceh-coding-agent [2 agent `skills:` preloads] - ceh-orchestration ──► ceh-coding-agent [1 agent `skills:` preload] - -LAYER 1 — cross-cutting leaves, declare nothing - - ceh-coding-agent ceh-git-workflow ceh-testing - ceh-fabled ceh-advisor [experimental, D8] - -NEVER BUNDLED - - experimental (D8): ceh-fabled, ceh-advisor, ceh-orchestration - session mechanics (D9): ceh-summarize-chat, ceh-lessons-learned - opt-in (D7): ceh-ops - declare nothing of own: ceh-blog, ceh-business-plan, ceh-scaffolding, ceh-architecture, - ceh-seo, ceh-evaluation, ceh-usability-audit, ceh-plan-build-review, - ceh-documentation -``` - -`ceh-scaffolding` deliberately declares nothing: it references all three stack plugins, but -those references are advisory (it scaffolds whichever type you name). Declaring them would -install Python service + library + web frontend on someone who only writes libraries. - -Worst-case install closure is **2 plugins** (`ceh-ops` → `ceh-coding-agent`, and each stack plugin -→ `ceh-testing`). No transitive blow-up, because every Layer 1 plugin is a true leaf. - -**D19 (2026-08-30) — `ceh-release-flow` dissolved into `ceh-git-workflow`.** Its `release-flow` -skill moved into `ceh-git-workflow` alongside a new `merge-flow` sibling (PR + merge, no version -bump, changelog under `[Unreleased]`); `direct-release-flow` was deleted. That would have forced -`ceh-git-workflow` ──► `ceh-documentation`, breaking "a cross-cutting plugin may depend only on -other cross-cutting plugins" — so `update-changelog` (and `check-semver.py`) moved to -`ceh-git-workflow` too. Every input that skill reads is git plumbing, so it fires on a git moment, -not a documentation one. The only cross-plugin call left in either flow is `update-readme`, which is -conditional ("if the change is user-facing") and therefore stays prose under the negative/conditional -rule in §5. `ceh-git-workflow` keeps **zero** declared dependencies and stays a Layer 1 leaf. -`update-readme` stayed in `ceh-documentation`: its inputs are the code and the README, and -`ceh-seo:text-discoverability` routes to it as a sibling. `ceh-scenario-editorial` gained -`ceh-git-workflow` so it does not lose changelog capability. §5-§7 below are the 2026-08-23 -audit snapshot that produced the original graph and were **not** re-run for D19 — read their -`ceh-release-flow` rows as history, not current state. - -### Example manifests - -```json -// plugins/ceh-scenario-service-iterate/.claude-plugin/plugin.json -{ - "name": "ceh-scenario-service-iterate", - "version": "1.0.0", - "description": "CEH scenario: building on a Python backend service that already ships.", - "dependencies": [ - "ceh-coding-agent", "ceh-git-workflow", "ceh-testing", - "ceh-python-service", "ceh-architecture", "ceh-documentation", - "ceh-usability-audit", "ceh-plan-build-review" - ] -} -``` - -```json -// plugins/ceh-scenario-service-greenfield/.claude-plugin/plugin.json -{ - "name": "ceh-scenario-service-greenfield", - "version": "1.0.0", - "description": "CEH scenario: starting a new Python backend service from nothing.", - "dependencies": [ - "ceh-scenario-service-iterate", - "ceh-scaffolding", "ceh-business-plan" - ] -} -``` - ---- - -## 5. Reference audit - -Full sweep of every `SKILL.md` and `agents/*.md`, frontmatter and body separated: **102 -cross-plugin references** (36 frontmatter, 66 body). Rev 1's figure of 33 undercounted by 3×; it -missed agent frontmatter and most body routing tables. - -### 5.1 Reference classes - -| Class | Count | Becomes a dependency? | -|-------|-------|----------------------| -| Agent `skills:` frontmatter preload | 6 | **Yes** — hardest form; the agent cannot load its own contract without it | -| Unconditional body invocation | 13 | **Yes** — all in `ceh-release-flow` step tables | -| Negative routing in `description` ("Not for X, use Y") | ~30 | No — marks an alternative (D11) | -| Conditional body handoff ("if it's a UI, delegate to…") | ~20 | No — D13 | -| Boundary/advisory prose ("see X for the mechanics", "pairs with X") | ~30 | No | -| Cross-plugin file-path read | 1 | No — duplicate instead (D17) | - -### 5.2 Rev-1 edges that the audit invalidated - -Eleven of rev 1's sixteen Layer-2 edges had no supporting reference: - -- **No reference in either direction:** `ceh-python-service → ceh-coding-agent`, - `ceh-python-library → ceh-coding-agent`, `ceh-python-library → ceh-testing`, - `ceh-web-frontend → ceh-coding-agent`, `ceh-plan-build-review → ceh-coding-agent`. - Rev 1 assumed `ceh-python-library → ceh-testing` by symmetry with `ceh-python-service`, but the - service edge comes from three tester agents, and `ceh-python-library` has no agents at all. - Step 5 makes this edge real by fixing the text, not by asserting it in a manifest. -- **Negative routing only:** `ceh-documentation → ceh-git-workflow` (`update-changelog:8`), - `ceh-evaluation → ceh-git-workflow` (`evaluate-skill:11`), - `ceh-seo → ceh-documentation` (3 boundary statements), - `ceh-usability-audit → ceh-documentation` and `→ ceh-seo` (routing tables). -- **Conditional with an OR-alternative:** `ceh-plan-build-review → ceh-git-workflow` - (`patch-built-version:111` offers `ceh-git-workflow:release` *or* `ceh-release-flow`). - -### 5.3 Edge the audit added - -`ceh-ops → ceh-coding-agent`. Both `agents/github-actions.md:16` and `agents/gitlab-ci.md:16` -preload `agent-coding-contract` in `skills:` frontmatter. Rev 1 gave `ceh-ops` no Layer-2 row at -all. Unrelated to D7: not bundling `ceh-ops` says nothing about what `ceh-ops` requires. - -### 5.4 Why D12's rationale had to change - -`ceh-coding-agent` is Layer 1 and declares nothing — but three of its own references point at other -plugins: `refactor-repo:87` and `shrink-diff:91` name `ceh-testing:verify-behavior-preserved`, and -`refactor-repo:69` names `ceh-git-workflow:branch`. D12 survives only because all three happen to be -conditional under D13 (`refactor-repo:87` fires "for anything past a mechanical transform"; -`refactor-repo:69` sits inside Phase 3, gated on user approval; `shrink-diff:77` cites `open-pr` for -its *size limits*, a value rather than a call). - -That is a property of six lines of prose, not a structural fact. Rewriting `refactor-repo:87` to -"always pin behavior before refactoring" — a defensible standard and a one-line edit — would make -rev 1's D12 false with nothing to catch it. Hence the restatement in D12 plus the `validate.py` -acyclicity check in D16. - -### 5.5 Invocation targets are all callable - -All eight targets of the step-5 upgrades are clean of `disable-model-invocation`: -`ceh-git-workflow:{release,branch,commit,open-pr,merge}`, -`ceh-documentation:{update-changelog,update-readme}`, `ceh-testing:design-test-cases`. - -`release-flow` and `direct-release-flow` **do** set the flag, correctly: it governs whether the -model may invoke *that* skill, not what the skill's body invokes once a human has typed it. - ---- - -## 6. Implementation checklist - -Ordered so each step leaves the repo green. Steps 1–3 are independent of the bundles. - -Steps 1–9 landed on `feat/plugin-dependencies-and-scenario-bundles`, verified 2026-08-23 against a -clean tree with `validate.py` green: - -| Step | Commit | Note | -|------|--------|------| -| 1 | `9e4d665` | rename `ceh-agent-coding-contract` → `ceh-coding-agent` | -| 2 | `484fd47` | absorb `ceh-dev-tools` | -| 3 | `9641997` | 4th `plan-schema.md` copy + `CROSS_REFERENCES.md:205` | -| 4, 5 | `d6f461c`, `0390149`, `d00c416` | six edges + 16 invocations; later commits fixed the bump level and removed the `marketplace.json` duplicate | -| 6 | `6e549bb` | seven bundles | -| 7 | `cf60012` | scenario shape, dependency resolution, acyclicity, invocation contract | -| 8 | `09d28e0` | scenario table first; experimental marks | -| 9 | `0f9d789`, `4f8230d` | scenario tier, D12/D13, naming rule, rename | - -Step 10 lands as repo tag **v5.0.0** (MAJOR: D14 renames a plugin, D5 removes one). Step 11 -remains open — owner-manual, after the tag is pushed. - -1. **Execute D14 — rename `ceh-agent-coding-contract` → `ceh-coding-agent`**, as its own commit, - first. 180 occurrences repo-wide. Beyond the obvious (directory, `plugin.json`, - `marketplace.json`, both READMEs, `CLAUDE.md`, `docs/CROSS_REFERENCES.md`), the non-obvious - sites are `hooks/load-contract.sh`, `hooks/usage-limit-watch.py`, the `skills:` preloads in - `ceh-ops/agents/{github-actions,gitlab-ci}.md` and `ceh-orchestration/agents/executor.md`, and - `ceh-fabled/hooks/load-voice.sh`. Must land before step 4, after which the old string becomes a - hard dependency key in 13 manifests. -2. **Execute D5** — move `explain-codebase` and `repo-tree-mapper` into `ceh-coding-agent`; remove - `ceh-dev-tools` from `marketplace.json`; update every `ceh-dev-tools:` reference across skills, - both README tiers, `CLAUDE.md`, and `docs/CROSS_REFERENCES.md`. MINOR bump. Sequence after - step 1 so `explain-codebase`'s reference to the contract collapses to an in-plugin one. -3. **Execute D17** — copy `plan-schema.md` to - `ceh-business-plan/skills/develop-business-plan/references/`, repoint - `develop-business-plan:58`, add the row to the existing `docs/CROSS_REFERENCES.md` block - (now 4 copies in lockstep). -4. **Add `dependencies`** to the six Layer-2 plugins per §4. Bare strings only (D10). -5. **Convert the 16 invocation sites** — see §7. Same commit as step 4 for `ceh-release-flow`, so - the fallbacks never outlive the guarantee that replaces them. -6. **Create the 7 scenario bundles** in `plugins/`, each with `plugin.json` + `README.md`, plus a - `marketplace.json` entry each. -7. **Extend `validate.py`** per D16: - - a `ceh-scenario-*` directory contains only `.claude-plugin/plugin.json` and `README.md`; - - its manifest has a non-empty `dependencies`, every entry present in `marketplace.json`; - - every `Invoke the Skill tool with skill="X"` names a skill that exists, is in-plugin or in a - declared dependency, and does **not** set `disable-model-invocation: true`; - - the dependency graph is acyclic; - - confirm `plugin_dirs()` at `validate.py:47` tolerates a plugin with no `skills/` or `agents/`. -8. **Restructure `README.md`** — a scenario table above the existing Plugins table, so the install - path reads scenario-first and the use-case plugins become the reference list underneath. Mark - `fabled` and `advisor` experimental (D8) in the Categorization table and in their manifest - `description` fields. -9. **Update `CLAUDE.md`** — the scenario tier, D12 as restated, D13, the `ceh-scenario-` naming - rule, D4, and the `ceh-coding-agent` rename throughout. -10. **Repo tag + `CHANGELOG.md` entry** per the existing versioning policy. MAJOR: D14 renames a - plugin and D5 removes one. -11. **Local cleanup** — **manual, done by the repo owner**, after the tag is pushed. See §9. - ---- - -## 7. The 16 invocation sites - -Convert each to `Invoke the Skill tool with skill=":"`. - -| Source | Sites | Target(s) | -|--------|-------|-----------| -| `ceh-release-flow:release-flow` `SKILL.md:30-39` | steps 1, 2, 4, 5, 7, 8, 9, 10 | `ceh-git-workflow:{release,branch,commit,open-pr,merge}`, `ceh-documentation:{update-changelog,update-readme}` | -| `ceh-release-flow:direct-release-flow` `SKILL.md:31-38` | steps 1, 4, 5, 7, 8 | same set, minus branch/PR/merge | -| `ceh-python-service:python-service-testing` | before `## Unit Tests` (`:33`) | `ceh-testing:design-test-cases` | -| `ceh-python-library:python-library-testing` | before `## Unit Tests` (`:34`) | `ceh-testing:design-test-cases` | -| `ceh-web-frontend:frontend-testing` | before `## Unit Tests` (`:30`) | `ceh-testing:design-test-cases` | - -The step tables currently route through a **trigger-phrase indirection** — -`| 4 | Write the changelog | "update the changelog" → ceh-documentation:update-changelog |` — which -is the exact description-matching gamble this plan exists to remove. - -The three testing skills are not bookkeeping: `ceh-testing:design-test-cases` is reachable today -**only** through six agent preloads, so the in-conversation path has no access to the technique at -all. - -### Fallbacks deleted in the same commit - -`release-flow:66,105` and `direct-release-flow:65,101` — "if a step's owning skill is not installed, -apply it inline" and "when the `ceh-git-workflow` agents are installed, dispatch each…". - -### Deliberately left as prose - -Converting these would drag the closure back to ~6 plugins per install: - -| Reference | Why it stays | -|-----------|--------------| -| `ceh-coding-agent:refactor-repo` → `ceh-testing:verify-behavior-preserved`, `ceh-git-workflow:branch` | D12 — Layer 1 declares nothing | -| `ceh-usability-audit` → `ceh-web-frontend:accessibility` (4 sites) | D13 — conditional on the subject having a UI | -| `ceh-scaffolding` → the three stack plugins | Advisory by design; would install all three | -| `ceh-plan-build-review:patch-built-version` → `ceh-git-workflow:release` | Offers an OR with `ceh-release-flow` | -| `ceh-ops:deploy` → `ceh-git-workflow:release` | A precondition, not a call | -| `ceh-testing:audit-test-suite:102` → mutation-testing binary | External tooling, not a plugin | -| `release-flow` step 6 → `revise-claude-md` | Another marketplace; out of this plan's scope (§2) | - -The failure mode to guard against is a well-meaning sweep converting every backtick-quoted skill -name into an invocation. - ---- - -## 8. Open questions - -- ~~**Q1 — `ceh-seo` in the service bundle?**~~ **Retired.** `ceh-seo` is dropped from all three - `-iterate` bundles. SEO is a "this thing has a public surface" moment, a property of one release - rather than of a whole phase, so it is installed deliberately — same shelf as `ceh-ops` - under D7. `ceh-scenario-editorial` keeps it, where a public surface is the premise. -- ~~**Q2 — `ceh-plan-build-review` phase.**~~ **Resolved: it belongs in the `-iterate` bundles.** Rev 1 put it - in the greenfield delta on a claimed 3-of-4 majority; the skill descriptions say the opposite. - `plan-fullstack-app-iteratively` states it "covers greenfield skeletons **and iterative feature - planning for existing apps**", and `implement-from-plan` / `review-against-plan` consume any plan - artifact regardless of phase — 4 of 5 skills work post-launch, and only - `plan-fullstack-app-to-mvp` is greenfield-exclusive. Adding features to a shipped thing is the - normal case after first release, and it is planned work. Under D3 the greenfield bundles inherit - the plugin anyway, so the greenfield delta shrinks to `ceh-scaffolding` + `ceh-business-plan` and - nothing is lost. The deferred refactor is dropped: `patch-built-version` explicitly routes - feature work to `plan-fullstack-app-iteratively`, which is an argument for keeping them in one - plugin, not for splitting them. -- ~~**Q5 — is "maintenance" the right word?**~~ **Resolved: no. The suffix is now `-iterate`.** - See D18. -- **Q3 — is `ceh-ops` unused because you rarely deploy, or because `deploy` / `incidents` / - `rollback` never auto-trigger?** If the second, D7 treats a symptom: the fix belongs in those - three descriptions, and that is the same reliability problem that motivated this whole plan. -- **Q4 — `ceh-scenario-` prefix.** Confirmed. Makes the tier obvious in `claude plugin list` and - sorts the greenfield/iterate pairs adjacently, at the cost of long names. - ---- - -## 9. Local cleanup after the rename - -> **Manual step — the repo owner runs this personally.** It is recorded here for reference only. -> No agent should execute any part of this section, and checklist step 11 is not agent work: it -> touches the owner's global Claude Code state and 26 unrelated project directories, all outside -> this repo. - -Owner's machines only; run after step 10's tag is pushed, because the `ceh-plugins` marketplace is -`github: cheneeheng/agent-skills` with `autoUpdate: true` and would otherwise reinstall the old -names. - -1. `claude plugin marketplace remove ceh-plugins` then - `claude plugin marketplace add cheneeheng/agent-skills`. Rebuilds `installed_plugins.json` - (115 stale local-scope entries across 20 project paths), the `plugins/cache/ceh-plugins/` - extracted copies, the `plugins/marketplaces/ceh-plugins/` checkout, and - `plugin-catalog-cache.json`. -2. `claude plugin prune` for anything the remove missed. -3. Per-project `enabledPlugins` keys are **not** covered by either command — 26 - `.claude/settings.local.json` files hold 164 stale `@ceh-plugins` keys. Four of them also carry - a `permissions` block, so strip keys rather than deleting files. Script: - `.agents_workspace/strip-ceh-plugins.py`, dry-run by default, `--apply` to write: - `python .agents_workspace/strip-ceh-plugins.py [--apply]`. - -Unverified: whether a stale `enabledPlugins` key naming a nonexistent plugin errors or is silently -ignored. The cleanup does not depend on the answer. - ---- - -## 10. Risks - -- **Disable lock-in.** Once `ceh-ops` and `ceh-orchestration` depend on `ceh-coding-agent`, - `claude plugin disable ceh-coding-agent` is refused until both are disabled first. For an - intentional always-on contract this is the point, but it is a real behavior change. -- **Rename breakage.** D14 orphans every existing install of `ceh-agent-coding-contract`. Accepted - because the install base is the owner's own machines (§9), and because the cost only grows. - Land the rename and the `dependencies` rollout in **separate commits** — a typo in a dependency - key fails at install time with `dependency-unsatisfied`, not at edit time. -- **Silent invocation failure.** A future session setting `disable-model-invocation: true` on an - invocation target breaks a release flow with no error at author time. D16's validator rule is - the only thing standing in front of this; 6 of 77 skills already set the flag. -- **`range-conflict` at scale.** Not a risk under D10 (bare strings). It becomes one the moment - version ranges are introduced across many plugins constraining the same few cross-cutting deps. -- **Per-plugin tagging cost.** D10 defers this, but any future version constraint requires - `{plugin}--v{version}` tags on every plugin at every release — a change to the release flow. -- **The duplication policy is untouched.** Dependencies install whole plugins; they give one skill - no way to reference another plugin's content. The uv/pytest block duplicated across - `ceh-python-service` and `ceh-python-library` stays duplicated, `plan-schema.md` gains a 4th copy - (D17), and `docs/CROSS_REFERENCES.md` stays load-bearing. -- **The `CLAUDE.md` tier table becomes wrong** the moment the rename lands — step 9 is not - optional bookkeeping. diff --git a/.agents_workspace/PLUGIN_REORG_PLAN.md b/.agents_workspace/PLUGIN_REORG_PLAN.md deleted file mode 100644 index 5444ebb..0000000 --- a/.agents_workspace/PLUGIN_REORG_PLAN.md +++ /dev/null @@ -1,273 +0,0 @@ -# Plugin Reorganization Migration Plan - -Status: executed (refactor/plugin-reorg, 2026-06-05) -Author: reorg triage session, 2026-06-04 -Scope: all `ceh-*` plugins and skills in this repo - ---- - -## 1. Why - -Two independent problems, conflated in the current layout: - -- **Problem A — triggering & redundancy.** Skills fire on *verbs/moments* ("I'm about to - commit", "I'm writing a post"), not *nouns/topics* ("PostgreSQL", "FastAPI"). Topic-named - skills either never auto-trigger or restate knowledge the model already has. The fix is - per-skill: cut generic content to the repo-opinionated delta, and reframe surviving topics as - moments. -- **Problem B — organizing axis.** Plugins are split on two incompatible axes at once: - **tech domain** (`python-backend`, `typescript-frontend`, `architecture-design`) and - **lifecycle phase** (`release-ops`). Both bake in a fullstack-web assumption, don't - self-document their scope, and — critically — force the *same standard* into multiple plugins - where copies drift. The fix is structural: reorganize around **use case**. - -Reorganizing without trimming just relocates dead weight. Do **A and B together, A first.** - -### Evidence: the duplication the current axis forces - -`release-ops` was carved by lifecycle phase, but "phase" cuts across the same topics the stack -plugins already own. Verified content overlaps: - -| Topic | Copy A | Copy B | Reality after reading both | -|---|---|---|---| -| Security baseline | `python-backend/python-security` | `release-ops/security` | B is the richer superset (covers Python **and** TS) | -| Observability | `python-backend/python-observability` | `release-ops/observability` | B is the superset (adds metrics + `/health`) | -| DB migrations | `python-backend/alembic` | `release-ops/database-migrations` | **Not** dupes: A = tool mechanics, B = deploy safety; overlap only on the command block | -| Versioning | `git-workflow/release` | `release-ops/versioning` | **Not** dupes: A = git tagging, B = full deploy pipeline | -| PR completion | `git-workflow/open-pr` | `release-ops/definition-of-done` | Overlap on the pre-PR quality gate | -| Hotfix | `git-workflow/hotfix` | `release-ops/incidents` | A = git mechanics, B = incident response | -| API design vs impl | `architecture-design/rest-api` | `python-backend/fastapi` | Complementary; already cross-reference each other | - -After dedupe, `release-ops` has only two skills genuinely its own (`incidents`, `rollback`) — -confirming the axis was wrong. - -### Evidence: app-specific leaks masquerading as standards - -`architecture-design/event-sourcing` and `architecture-design/llm-integration` are bound to one -specific application (the `event_log`/`entities` schema and the "LLM proposes events → backend -validates" pattern). They are not reusable standards — together they are a single niche use case. - ---- - -## 2. Target architecture - -Three tiers on a single coherent axis. Plugin names declare their scope, which removes the silent -"fullstack-web-only" assumption and makes the Python-library gap an obvious empty slot. - -``` -Tier 1 — cross-cutting (load most sessions) - ceh-agent-coding-contract contract + plan implement/review - ceh-git-workflow branch, commit, open-pr, code-review, release, hotfix, - dependency-management - -Tier 2 — use-case workflow (load per activity) - ceh-blog (unchanged — the reference design) - ceh-documentation user/operator guides + changelog/readme agents - ceh-ops (rename of release-ops) incidents, rollback, deploy + CI agents - ceh-summarize-chat (unchanged) session summary — kept separate, used by other automation - ceh-lessons-learned (unchanged) session retrospectives — kept separate - -Tier 3 — stack/build (load per project type) - ceh-python-service (rename of python-backend) FastAPI/asyncpg/migrations/web service - ceh-python-library (NEW) packaging, public API, semver, no web deps - ceh-web-frontend (rename of typescript-frontend) Svelte AND React (Vite); - shared a11y/TS-style/testing/tooling - ceh-architecture (rename of architecture-design) stack-agnostic design moments - ceh-scaffolding (NEW) per-project-type setup: directory layout + initial config - -Unchanged - ceh-dev-tools repo-tree-mapper agent - -Deleted - event-sourcing, llm-integration removed (app-specific, not reusable; no niche plugin) - gitignore removed (entries fold into per-type ceh-scaffolding skills) -``` - -Net: 11 plugins → 13 (+ ceh-python-library, + ceh-scaffolding; no plugins removed, two renamed). -~46 skills → ~30 after merges/deletes. - -**Svelte vs React are NOT split into separate plugins.** Framework skills trigger on file type -(`sveltekit` on `.svelte`, `react-vite` on `.tsx`), so they coexist in one plugin without -mis-firing, while the shared frontend standards (a11y, TS style, testing, tooling) stay -single-sourced. Splitting would duplicate those shared standards — the drift problem this reorg -removes. Apply the "split only when too big" rule later if the framework skills bloat the plugin. - ---- - -## 3. Per-skill migration map - -Verdict legend: **KEEP** (well-formed, leave) · **TRIM** (cut generic → delta) · -**MOMENTIZE** (reframe topic→verb trigger) · **MERGE** (fold into another skill) · -**MOVE** (relocate, content largely intact) · **DELETE/EXTRACT** (remove or pull into niche -plugin). "✓ read" = content verified this session; "desc" = verdict from description, confirm at -execution. - -### ceh-agent-coding-contract (T1, keep) -| Skill | Trigger | Verdict | Target | -|---|---|---|---| -| agent-coding-contract | moment | KEEP (desc) | unchanged | -| implement-from-plan | moment | KEEP (desc) | unchanged | -| review-against-plan | moment | KEEP (desc) | unchanged | - -### ceh-git-workflow (T1, keep — absorbs dedupes) -| Skill | Trigger | Verdict | Target | -|---|---|---|---| -| branch, commit, code-review | moment | KEEP (desc) | unchanged | -| open-pr | moment | KEEP + absorb `definition-of-done` quality gate ✓ | unchanged | -| release | moment | KEEP — git tagging mechanics only; dedupe semver table vs `versioning` ✓ | unchanged | -| hotfix | moment | KEEP — git mechanics; `incidents` references it ✓ | unchanged | -| dependency-management | moment | KEEP (desc) | unchanged | -| gitignore | file-edit | **DELETE** — required-entries list folds into per-type `ceh-scaffolding` skills ✓ | removed | - -### ceh-architecture-design → ceh-architecture (T3) -| Skill | Trigger | Verdict | Target | -|---|---|---|---| -| adr | moment | KEEP in ceh-architecture (decision-making is closer to the use case than docs) ✓decided | ceh-architecture | -| domain-modeling | moment | TRIM, keep ID-format/status-enum opinions (desc) | ceh-architecture | -| repository-structure | rare moment | MOVE + reframe: merge into per-type `ceh-scaffolding` skills, not a standalone "structure" skill (desc) | ceh-scaffolding | -| rest-api | moment | MOVE + co-locate with fastapi (kills cross-plugin ref) ✓ | ceh-python-service | -| postgresql | topic | TRIM + MERGE into data moments (`domain-modeling` + migrations) (desc) | ceh-python-service | -| event-sourcing | — | **DELETE** — app-specific, not reusable ✓decided | removed | -| llm-integration | — | **DELETE** — app-specific; overlaps built-in `claude-api` ✓decided | removed | - -### ceh-python-backend → ceh-python-service (T3) -| Skill | Trigger | Verdict | Target | -|---|---|---|---| -| fastapi | moment | TRIM + absorb `rest-api` design rules ✓ | ceh-python-service | -| asyncpg | moment | TRIM + absorb `postgresql` driver rules (desc) | ceh-python-service | -| alembic | moment | KEEP as single migrations home; absorb deploy-safety from `database-migrations`, drop `event_log` lines ✓ | ceh-python-service | -| python-observability | import | MERGE — adopt richer `release-ops/observability` content ✓ | ceh-python-service | -| python-security | moment | MERGE — adopt richer `release-ops/security`; TS secret line → web-frontend ✓ | ceh-python-service | -| python-environment | file-edit | TRIM web specifics; **duplicate into python-library** (see §4) ✓ | ceh-python-service + ceh-python-library | -| python-testing | file-edit | KEEP; **duplicate into python-library** (desc) | ceh-python-service + ceh-python-library | - -### ceh-python-library (NEW, T3) -| Source | Verdict | -|---|---| -| python-environment (trimmed: no fastapi/uvicorn/asyncpg deps, no uvicorn dev server) | duplicate-and-trim | -| python-testing | duplicate | -| NEW: public API surface, semver discipline, packaging/publishing (build backend, wheels), no-web-deps rule | author fresh | - -### ceh-typescript-frontend → ceh-web-frontend (T3) — now Svelte + React -| Skill | Trigger | Verdict | Target | -|---|---|---|---| -| sveltekit | file-edit (`.svelte`) | TRIM (desc) | ceh-web-frontend | -| react-vite | file-edit (`.tsx`) | **NEW** — React + Vite framework conventions (routing, hooks, state, Vite config) | ceh-web-frontend | -| accessibility | file-edit | KEEP + generalize trigger to `.svelte` AND `.tsx` (a11y is framework-agnostic) ✓ | ceh-web-frontend | -| frontend-testing | file-edit | TRIM; framework-agnostic (Vitest/Testing Library/Playwright/MSW serve both) (desc) | ceh-web-frontend | -| environment | moment | KEEP — consolidation target; cover Bun + Vite for both frameworks ✓ | ceh-web-frontend | -| coding-style | topic | KEEP content (real delta), MOMENTIZE — MERGE into `environment` so it fires on file edits ✓ | ceh-web-frontend | -| linting | topic | MERGE into `environment` (mostly a quality gate + config) ✓ | ceh-web-frontend | - -> Normalization note: Python folds env+style+linting into one `python-environment`; TS splits the -> same into three. Consolidate TS to match — fewer, file-triggered skills. -> -> Framework-agnostic skills (`accessibility`, `frontend-testing`, `environment`/style) are -> single-sourced and serve Svelte and React both. Only `sveltekit` and `react-vite` are -> framework-specific, and they trigger on disjoint file types. - -### ceh-scaffolding (NEW, T3) -Per-project-type project setup. There is **no generic "structure" skill** — directory layout, -initial config, and the required `.gitignore` entries are merged into the scaffolding skill for -each project type, so "scaffold a Python library" produces the right layout + config + ignore file -in one moment-triggered skill. - -| Skill | Trigger | Source | -|---|---|---| -| scaffold-python-library | "start/scaffold a Python library" | repository-structure (python bits) + gitignore (python entries) + packaging layout | -| scaffold-python-service | "start/scaffold a FastAPI service" | repository-structure (API/services/db layers) + gitignore (python entries) | -| scaffold-web-frontend | "start/scaffold a Svelte or React app" | repository-structure (frontend bits) + gitignore (node entries) | -| scaffold-fullstack-web | "start/scaffold a fullstack web app" | composition of service + frontend layouts | - -> The per-type duplication of small shared bits (e.g. `.gitignore` entries) is governed by the -> §4 duplication policy — register them in `CROSS_REFERENCES.md`. - -### ceh-release-ops → ceh-ops (T2) -| Skill | Trigger | Verdict | Target | -|---|---|---|---| -| incidents | moment | KEEP (desc) | ceh-ops | -| rollback | moment | KEEP (desc) | ceh-ops | -| versioning | moment | SPLIT: deploy pipeline (changelog→docker→staging→prod→health) stays as `deploy`; semver table dedupes against git-workflow/release ✓ | ceh-ops | -| security | moment | MERGE → python-service (richer copy wins) ✓ | removed from ops | -| observability | moment | MERGE → python-service (richer copy wins) ✓ | removed from ops | -| database-migrations | moment | MERGE → python-service/alembic ✓ | removed from ops | -| definition-of-done | moment | MERGE → git-workflow/open-pr ✓ | removed from ops | - -### Unchanged use-case plugins (T2) -| Plugin | Verdict | -|---|---| -| ceh-blog (4 skills) | KEEP — reference design; rewrite other plugins' descriptions to this standard | -| ceh-documentation (user-operator-guide + changelog/readme agents) | KEEP | -| ceh-summarize-chat, ceh-lessons-learned | KEEP; optional MERGE into ceh-session-utils | -| ceh-dev-tools (repo-tree-mapper) | KEEP | - -### Agents -| Agent | Target | -|---|---| -| python-{unit,integration,system}-tester | ceh-python-service (share with library) | -| ts-{unit,integration,system}-tester | ceh-web-frontend | -| github-actions, gitlab-ci | ceh-ops | -| changelog-agent, readme-updater | ceh-documentation (unchanged) | -| repo-tree-mapper | ceh-dev-tools (unchanged) | - ---- - -## 4. Shared Python foundation — duplication policy (DECIDED) - -The uv/pyproject/ruff/mypy environment and the pytest testing standards are needed by **both** -`ceh-python-service` and `ceh-python-library`. Decision (user, this session): **duplicate the -trimmed delta into both plugins** rather than extract a shared `ceh-python-base`. Each use-case -plugin stays self-contained; a user loads exactly one plugin per use case. - -Cost: the two copies can drift. Mitigation: register every duplicated block in -`CROSS_REFERENCES.md` and propagate edits in the same session (existing Cross-Reference Rule). -This policy is recorded in `CLAUDE.md` so future sessions honor it. - -Trim difference between copies: the library copy drops web-only dependencies -(`fastapi`, `uvicorn`, `asyncpg`) and the uvicorn dev-server command from `python-environment`. - ---- - -## 5. Execution phases - -Incremental and independently shippable. Each phase = one PR, with version bumps in both -`plugin.json` and `marketplace.json`, README skill/agent table updates, and `CROSS_REFERENCES.md` -updates per touched plugin (per CLAUDE.md process). - -1. **Dedupe release-ops (highest value, no renames).** Merge security/observability/migrations - into python-backend; move definition-of-done into open-pr; split versioning into git-workflow - (tagging) + a `deploy` skill. Delete the emptied skills. Rename `release-ops` → `ceh-ops`. -2. **Delete the app leaks.** Remove `event-sourcing` + `llm-integration` outright (no niche - plugin). Strip remaining `event_log` references from migrations. -3. **Rename + retarget stack plugins.** `python-backend` → `ceh-python-service` (absorb rest-api, - postgresql); `typescript-frontend` → `ceh-web-frontend` (consolidate env/style/linting, add - `react-vite`, generalize a11y/testing); `architecture-design` → `ceh-architecture` (trim to - `adr` + `domain-modeling`). -4. **Create ceh-python-library.** Duplicate-and-trim environment + testing; author packaging/API/ - semver skills. -5. **Create ceh-scaffolding.** Move `repository-structure` here, split into per-project-type - scaffold skills; fold the deleted `gitignore` entries into each. Then delete the standalone - `gitignore` skill from git-workflow. -6. **Description pass (Problem A).** Rewrite every surviving skill's description to the blog-plugin - standard: action verbs, explicit trigger signals, explicit "not for…" boundaries. Drop the - `Phase:` prefixes. - -`ceh-summarize-chat` and `ceh-lessons-learned` stay separate plugins (used by other automation -workflows — do not merge). - -Renames are breaking for users who reference plugin names — this is **accepted**, no alias shims. -Bundle each rename with a README note and a repo-tag MINOR bump. - ---- - -## 6. Resolved decisions - -- **`adr` placement** → keep in `ceh-architecture`. Making the decision is closer to the use case - than filing a doc; it does not move to `ceh-documentation`. -- **`event-sourcing` / `llm-integration`** → delete outright. No niche plugin. -- **Session plugins** → keep `ceh-summarize-chat` and `ceh-lessons-learned` separate. They are - used by other automation workflows; do not merge into a `ceh-session-utils`. -- **Plugin renames** → breaking changes are acceptable. No alias shims; document in README + - repo-tag MINOR bump. -- **Svelte vs React** → one `ceh-web-frontend` plugin, not split (see §2 rationale). -- **`gitignore` skill** → deleted; entries fold into per-project-type `ceh-scaffolding` skills. diff --git a/.agents_workspace/skill-evals/ceh-seo/run-001/SKILL_EVAL.md b/.agents_workspace/skill-evals/ceh-seo/run-001/SKILL_EVAL.md deleted file mode 100644 index 37ecd66..0000000 --- a/.agents_workspace/skill-evals/ceh-seo/run-001/SKILL_EVAL.md +++ /dev/null @@ -1,275 +0,0 @@ ---- -artifact: SKILL_EVAL -status: closed # user-accepted at 4/6 — criteria 2-3 waived-unproven, deliberately not marked "passed" (schema reserves that for 6/6) -created: 2026-07-22 -updated: 2026-07-22 -target: ceh-seo/ -target_kind: plugin -eval_gate: 4/6 -iterations: 1 ---- - -# SKILL_EVAL — ceh-seo (run-001) - -## §01 Verdict - -The plugin produces a **measured, stable behavioral lift on exactly its opinionated content**: -across all four with/baseline pairs (N=2 per task), with-skill cleared every discriminating -assertion the hermetic baselines missed — verbatim one-liner consistency (2/2 vs 0/2), title -length 33/40 chars vs 64/64, meta description 129/146 chars vs 194/185 — with no regression -anywhere. Structural and content criteria are met with evidence. Triggering (criteria 2–3) is -**unproven — waived by user** for this run; that is the single highest-leverage gap and the one -thing standing between this report and a full gate, since the text-discoverability / -update-readme boundary is the plugin's riskiest surface and remains unmeasured. Gate 3/6 -(1, 4, 5 met; 2, 3 waived-unproven; 6 pending user confirmation). - -## §02 Derived criteria - -### web-discoverability - -- **Claim:** a public web page is not done until its discoverability surface ships with it — head - metadata, site-level files (sitemap/robots/llms.txt), JSON-LD, server-rendered content, and - GEO-quotable structure. -- **Trigger intent — should fire:** creating/shipping a public page or route; "add SEO", meta/OG, - structured data, sitemap, robots, llms.txt requests. -- **Should NOT fire:** README/package text (text-discoverability), writing content (ceh-blog / - ceh-documentation), accessibility fixes, internal/admin pages, app features that merely mention - OG/meta. -- **Intended outcome vs baseline:** baseline typically emits title + some meta/OG; the skill's - delta is canonical links, JSON-LD, `prerender`/SSR requirement, llms.txt, real-404 rule. - -### text-discoverability - -- **Claim:** public-facing text leads with a category-noun pitch in the excerpt window, reuses one - canonical one-liner verbatim across every surface, and states extractable facts over adjectives. -- **Trigger intent — should fire:** writing/revising README pitch, package description/keywords, - GitHub repo description/topics, first-time publishing of a repo/library/plugin. -- **Should NOT fire:** README accuracy updates after code changes (ceh-documentation:update-readme), - changelogs, blog posts, HTML meta tags (web-discoverability), ads/SEM keyword work. -- **Intended outcome vs baseline:** baseline writes a competent README but rarely enforces verbatim - one-liner consistency across surfaces, badge placement after the pitch, or category-noun-first - excerpt discipline. - -### Plugin-level - -- **Collision risk to test:** "seo for my readme" must fire text-, not web-; a shared-preview/meta - page task must fire web-, not text-. - -## §03 Trigger battery - -**Method note (confidence limiter):** plugin not installed in the evaluating session; each cold -subagent receives the session's real available-skills listing **plus** the two ceh-seo entries as -environment context, then only the user prompt. Graded on whether it loads the right skill -unprompted. N=3 per prompt; "fires" = ≥2/3. - -### Positives — text-discoverability (full depth, 8) - -| # | Prompt | -|---|--------| -| P1 | "i'm about to publish my rate limiter lib 'throttlekit' to pypi — can you write the readme intro and the pyproject description? i want people to actually find it" | -| P2 | "our github repo description is just 'utils'. fix it and give me topics people actually search for" | -| P3 | "make this repo more findable — the readme starts with 12 badges and nobody knows what the project does" | -| P4 | "write the package.json description and keywords for my new svelte form-validation library before i npm publish" | -| P5 | "the pitch on our landing hero, the readme, and the pypi page all say different things — clean this up" | -| P6 | "how should the first paragraph of my README read so chatgpt actually recommends my library when people ask for a retry lib?" | -| P7 | "we're open-sourcing the internal sdk next week — prep the public-facing text: repo description, readme opening, topics" | -| P8 | "seo for my readme basically — i want the repo to rank when someone googles 'python retry decorator'" (collision probe: text- must win over web-) | - -### Positives — web-discoverability (sampled, 4) - -| # | Prompt | -|---|--------| -| W1 | "add seo to our sveltekit landing page — we're launching thursday" | -| W2 | "the marketing site pages don't show previews when shared on slack/linkedin — fix that and whatever else is missing for search" | -| W3 | "set up sitemap, robots and whatever else google and the AI crawlers need for our docs site at docs.acme.dev" | -| W4 | "i just added /pricing and /features routes to the react app — make sure they're actually indexable, i think it's all client-side rendered" | - -### Near-miss negatives (8) - -| # | Prompt | Right answer | -|---|--------|--------------| -| N1 | "i just added a --parallel flag to the cli, update the readme" | update-readme, not text- | -| N2 | "write the changelog entry for v2.1" | update-changelog | -| N3 | "write a blog post about how we built our rate limiter" | ceh-blog | -| N4 | "the nav on the landing page isn't keyboard accessible, fix it" | accessibility | -| N5 | "add a meta viewport tag and fix the responsive layout on mobile" | frontend work (shares "meta") | -| N6 | "review my google ads campaign keywords for the product launch" | SEM, not SEO skills (shares "keywords") | -| N7 | "integrate an opengraph scraper api to fetch link previews for user-submitted urls in our app" | app feature (shares "open graph") | -| N8 | "our search page is slow — optimize the postgres full-text query" | DB work (shares "search") | - -**Results: NOT RUN — waived by user** ("skip the trigger prompts, focus on the behavioral -ones"). Criteria 2 and 3 are therefore unproven, not met. The battery above is ready to run -as-is in a later iteration; the riskiest untested boundary is P-vs-N1 (text-discoverability vs -update-readme). - -## §04 Behavioral tasks & assertions - -N=2 per task, with-skill vs baseline pairs launched same-turn; outputs under -`iteration-/generated/`. - -### T1 — web-discoverability - -Task: "Create the public landing page route for 'Fathom', a privacy-first cookieless analytics -tool (EU-hosted, one-line script install), in a SvelteKit app — `src/routes/+page.svelte`. -Include everything needed for search engines and AI engines to find and cite it." - -| # | Assertion (pass only if genuinely met) | -|---|----------------------------------------| -| A1 | `` unique, ≤60 chars, page-specific words first | -| A2 | Meta description ≤155 chars stating the value, not a teaser | -| A3 | Canonical link present | -| A4 | og:title/description/image + twitter:card present | -| A5 | JSON-LD `<script type="application/ld+json">` with a type matching the page | -| A6 | `export const prerender = true` (or explicit SSR rationale) in the route | -| A7 | Site-level surface addressed: sitemap/robots/llms.txt created or explicitly flagged | - -Expected delta: baseline partially clears A1/A2/A4; A3, A5, A6, A7 discriminate. - -### T2 — text-discoverability - -Task: "I'm publishing 'throttlekit' — a Python asyncio rate-limiting library, per-tenant quotas, -zero dependencies — to PyPI and GitHub. Write the README opening (title through install), the -pyproject `[project]` description and keywords, and the GitHub repo description + topics." - -| # | Assertion | -|---|-----------| -| B1 | One-liner contains the category noun + differentiator | -| B2 | pyproject description == GitHub description == README one-liner **verbatim** | -| B3 | README first screen: title + one-liner + what/why + install, before any badges/ToC | -| B4 | Topics are specific, lowercase, searched terms (`rate-limiting`, `asyncio`), not just `python` | -| B5 | No unverifiable marketing adjectives — facts/numbers instead | - -Expected delta: B2 and B3 discriminate; baselines rarely enforce verbatim cross-surface reuse. - -### Results (iteration 1, N=2 per arm, sonnet on all arms) - -**Contamination event and correction.** The first four baseline arms ran with default repo -access; because the evaluation runs inside the repo that ships ceh-seo, at least two baselines -found and applied the skill (t2-base-run2's completion report cited -"ceh-seo:text-discoverability's One-Liner rule" by name; t1-base-run2 echoed the skill's -"flag it... rather than deciding silently" phrasing). Transcripts could not be audited (all -`.output` files were 0 bytes post-completion). All four baseline arms were discarded and re-run -**hermetically** (exploration forbidden, prompt-only). Only hermetic baselines are graded below. -Contaminated outputs retained in `iteration-1/generated/t?-base-run?/` as method evidence. - -**T1 (Fathom landing page) — with-skill 7/7 and 7/7; hermetic baseline 4/7 and 5/7.** - -| Assertion | with-run1 | with-run2 | base-run1-h | base-run2-h | -|-----------|-----------|-----------|-------------|-------------| -| A1 title ≤60, page-first | pass (33 ch, "Cookieless Web Analytics — Fathom") | pass (40 ch) | **fail** (64 ch) | **fail** (64 ch, identical title) | -| A2 desc ≤155, value-first | pass (129 ch) | pass (146 ch) | **fail** (194 ch, comma-stuffed) | **fail** (185 ch) | -| A3 canonical | pass | pass | pass | pass | -| A4 OG + twitter card | pass | pass | **fail** — `og:image` references `og-image.png` that does not exist in `static/` | pass (svg, no width/height) | -| A5 JSON-LD | pass (3 blocks) | pass (@graph) | pass (4 blocks) | pass (@graph) | -| A6 prerender | pass | pass | pass | pass | -| A7 site-level files | pass (robots/sitemap/llms.txt) | pass | pass | pass | - -Additional with/baseline difference outside the assertions: both hermetic baselines added a -permissive `<meta name="robots" content="index, follow...">` (t1-base-run1-h line 98, -t1-base-run2-h line 130) — the exact habit the skill prohibits — and both silently allow-listed -AI crawlers in robots.txt where the skill requires flagging it as a product decision (both -with-skill arms flagged it, verbatim from the skill). - -**Honest negative finding:** the big-ticket items (canonical, JSON-LD, prerender, llms.txt, -sitemap) did NOT discriminate — a sonnet baseline produces them when the task itself says -"everything needed for search engines and AI engines". The task's explicit SEO framing is a -strong elicitor; lift under a plain "build a landing page" prompt is untested and likely larger. -The measured lift lives in the constraint details: length limits, robots-meta abstinence, -asset integrity, flag-don't-decide. - -**T2 (throttlekit publish) — with-skill 5/5 and 5/5; hermetic baseline 4/5 and 4/5.** - -| Assertion | with-run1 | with-run2 | base-run1-h | base-run2-h | -|-----------|-----------|-----------|-------------|-------------| -| B1 category noun + differentiator | pass ("a Python asyncio rate-limiting library with per-tenant quotas and zero external dependencies") | pass | weak pass ("Async rate limiting for Python" — no category noun "library") | weak pass | -| B2 verbatim one-liner across 3 surfaces | pass (identical README/pyproject/GitHub) | pass | **fail** (3 different pitches) | **fail** (3 different pitches) | -| B3 pitch before badges/ToC | pass | pass | pass (no badges produced) | pass | -| B4 specific lowercase topics | pass | pass | pass | pass (incl. speculative `fastapi`) | -| B5 facts over adjectives | pass | pass | pass | pass | - -B2 is the clean discriminator: 2/2 with vs 0/2 baseline. B3's badge-wall anti-pattern never -appeared in greenfield baselines — it likely only discriminates when editing existing READMEs. - -**Lift summary: stable across both runs of both tasks; with-skill clears 5 assertion-instances -per task-pair that baselines miss; zero regressions. N=2 — below the preferred N≥3; spread was -zero (identical pass/fail pattern across runs), which supports stability at this N.** - -## §05 Structural findings - -| Check | Result | Evidence | -|-------|--------|----------| -| Frontmatter parses, `name` matches dir (both skills) | pass | `name: web-discoverability` / `name: text-discoverability` match directories | -| Descriptions present, non-trivial | pass | both >400 chars with trigger phrases and not-for pointers | -| Bodies non-stub | pass | 80 / 74 lines respectively | -| `references/` discipline | pass | no `references/` dir — all content inline per repo rule | -| plugin.json valid, semver, marketplace sync | pass | both `1.0.0`; `validate.py` → "OK: all plugin checks passed" (re-run in Phase 2, exit 0) | - -Cross-check reconciliation: none needed — validator agrees with manual checks. - -## §06 Content findings - -### web-discoverability - -- **Moment-framed description with not-for pointers: pass.** "when shipping or creating a - public-facing web page or route" + "Not for README, package-listing, or repo text (use - text-discoverability)". -- **Explains the why: pass.** e.g. "An SPA fallback serving every unknown path as 200 poisons the - index with phantom pages"; "Most AI crawlers do not execute JavaScript". No ALL-CAPS walls. -- **Delta vs restatement: partial.** Genuine delta: real-404 rule, "robots meta only to exclude", - AI-crawler blocking as a flagged product decision ("a product decision, not a default"), - llms.txt, prerender-as-gate, "only mark up content actually visible", accessibility-wins rule. - Restatement risk: the OG/title/description basics ("`og:image` (1200×630)", "≤ 60 characters") - are general knowledge — their value is checklist-completeness at the moment. **Resolved by the - T1 delta:** the seemingly-restated limits are precisely what discriminated (baselines knew of - title/description limits but violated them, 64/194 chars); the genuinely-known items - (JSON-LD, prerender, llms.txt) did not discriminate under an SEO-framed prompt. The body's - delta is real but concentrated in limits + prohibitions, not in the checklist's existence. -- **Size: pass.** ~80 lines, well under norms. - -### text-discoverability - -- **Moment-framed description with not-for pointers: pass.** "when writing or revising the - public-facing text of a repo, package, or product" + explicit boundary to update-readme. -- **Delta: pass.** The verbatim one-liner rule ("reuse it **verbatim** on every surface — - paraphrase drift splits the signal") and badges-after-pitch are genuinely opinionated — - baselines don't enforce either. Excerpt rule's three components are a delta framing of known - material. Minor restatement: "npm: `description` + `keywords` array" is obvious. -- **Explains the why: pass.** "nobody searches your project's name before they know it exists"; - "engines see three weak descriptions instead of one strong one". -- **Size: pass.** ~74 lines. - -## §07 Gate scorecard - -Thresholds used: behavioral lift = with-skill clears ≥1 discriminating assertion per task that -the hermetic baseline misses, no regression, N=2 (below preferred N≥3 — accepted because spread -was zero). Trigger thresholds (positives ≥10/12, false-positives ≤1/8) defined but not exercised. - -| # | Criterion | Status | Evidence | -|---|-----------|--------|----------| -| 1 | Structurally valid | **met** | §05: all checks pass; `validate.py` exit 0, re-run in Phase 2 | -| 2 | Triggers on intent | **unproven — waived by user** | battery built, not run | -| 3 | No over-trigger | **unproven — waived by user** | battery built, not run | -| 4 | Content is delta + moment-framed | **met** | §06: cited lines; delta confirmed behaviorally (the limit rules discriminated 4/4 baseline arms) | -| 5 | Behavioral lift | **met** | §04: T1 7/7+7/7 vs 4/7+5/7; T2 5/5+5/5 vs 4/5+4/5; zero regressions; zero spread across N=2 | -| 6 | User confirms | **met** | user accepted the run at this scope ("mark the eval as done", 2026-07-22) | - -`eval_gate: 4/6` — run closed as user-accepted. Criteria 2–3 remain unproven (waived); if -triggering ever misbehaves in practice, the §03 battery runs as-is, starting with the P1-vs-N1 -boundary (text-discoverability vs update-readme). - -## §08 Advisory backlog - -1. **web-discoverability: sharpen toward the measured delta.** APPLIED (iteration 1): the - asset-integrity rule added to the OG bullet — og:image must exist as a real 1200×630 PNG/JPG, - SVG renders inconsistently, no dangling references (observed failures: baseline shipped a - broken `og:image` reference; one with-skill arm shipped SVG). Behavioral dimension not - re-run for this one-line change — the affected assertion (A4) already passed with-skill 2/2; - the line targets the two observed near-misses. -2. **text-discoverability: B3's badge-wall rule discriminates only on existing READMEs**, not - greenfield — the behavioral case for it is untested here. -3. **Method: in-repo baseline contamination.** Evaluating a skill inside the repo that ships it - contaminates non-hermetic baselines; future runs should make hermetic baselines the default - (this run's re-run prompt wording works). Worth folding into ceh-evaluation itself. -4. **Method: subagent transcripts were 0 bytes post-completion** — contamination auditing had to - rely on completion reports; grading relied on generated files only (which is the stronger - evidence anyway). diff --git a/.agents_workspace/skill-evals/ceh-testing/run-001/SKILL_EVAL.md b/.agents_workspace/skill-evals/ceh-testing/run-001/SKILL_EVAL.md deleted file mode 100644 index a96599a..0000000 --- a/.agents_workspace/skill-evals/ceh-testing/run-001/SKILL_EVAL.md +++ /dev/null @@ -1,409 +0,0 @@ ---- -artifact: SKILL_EVAL -status: passed -created: 2026-07-30 -updated: 2026-07-30 -target: ceh-testing (plugin) -target_kind: plugin -eval_gate: 6/6 -iterations: 2 ---- - -## §01 Verdict - -`ceh-testing` is a well-built plugin: structurally clean, its five skills partition the testing -lifecycle with almost no collision (0/13 false positives on `close-test-risk-gaps`, and every -near-miss negative correctly fired a *different* sibling skill instead), and `close-test-risk-gaps` -produces a measurable, reproducible behavioral lift over a strong baseline — most sharply on the -idiom-specific assertions (bounded query count, 404-not-403, schema-validated contract) a baseline -model does not reliably reach for on its own. Triggering recall started at 7/10 (threshold 8/10); the -three misses shared a pattern — scenario descriptions that don't use the skill's own named-technique -vocabulary ("shared stock counter" instead of "race condition," "double-charge" instead of -"idempotent," "look up by customer_id" instead of "authorization"). Adding scenario-shaped cues (and -a missing "not for" clause) to the description fixed two of the three cleanly (9/10, gate cleared) -with zero new false positives; the third (a shared-counter race phrased without any of the skill's -vocabulary) stayed a stable miss even after the fix — the model answers it confidently from general -knowledge regardless, which looks like a genuine ceiling rather than a description defect. All 5 -measurable criteria now pass; only user sign-off remains. - -## §00 Methodology note — a structural confound in this run - -This evaluation runs from inside `ceh-testing`'s own source repository. Cold trigger-battery -subagents have `Read`/`Grep`/`Glob` and, on a non-trivial fraction of prompts, used them — three -responses (CTRG-POS-3, the parse_duration negative, and one Task-B run) explicitly located and -quoted `ceh-testing/skills/.../SKILL.md` by searching the filesystem rather than (necessarily) via -the Skill tool's own auto-trigger. This means "fired" in §03/§04 below is operationalized as **the -skill's content was consulted and applied** (verbatim/near-verbatim idiom matches, explicit -citation, or the skill's own distinctive reporting structure), not strictly "the auto-trigger -mechanism activated on the description alone" — the two are conflated when evaluating a skill from -inside its own repo, and no tool available in this session can separate them (subagent final -reports are the only visibility into subagent behavior; no raw tool-call trace is exposed to the -caller). Confidence in criterion 2 specifically should be read with this caveat: it is possible the -true auto-trigger rate is *lower* than measured, if some "fires" here were filesystem discovery -rather than description-driven triggering — which would make the identified gap (§03) understating -the problem, not overstating it. - -A second, entirely deterministic corroborating signal was available and used: the `tool_uses` count -returned with each subagent result. `tool_uses: 0` conclusively rules out any tool invocation -(including `Skill`) for that run — used to confirm CTRG-POS-1 and CTRG-POS-2's initial "fired"-looking -content was actually general background knowledge, not the skill. - -## §02 Derived criteria - -### Plugin-level claim - -`ceh-testing` supplies stack-agnostic testing **technique** — which inputs to choose, whether a -green suite is trustworthy, and what a passing functional suite structurally misses — deliberately -excluding runner/fixture/mocking concerns owned by the three stack testing skills. Five skills, each -bound to a distinct lifecycle moment, plus one read-only audit agent. - -| Skill | Moment | Should NOT fire on | -|---|---|---| -| `test-a-bug-fix` | Fixing a bug/crash/regression, before writing the fix | Choosing new-feature test inputs, judging an existing suite | -| `design-test-cases` | Deciding which inputs/scenarios a test should cover | Wiring the runner, auditing an existing suite | -| `audit-test-suite` | Judging whether a passing suite would catch a defect | Choosing new inputs, testing a specific bug fix | -| `verify-behavior-preserved` | Before a behavior-preserving change (refactor/extract/upgrade/port) | Changes that intentionally change behavior | -| `close-test-risk-gaps` | Pre-completion gate: "is this ready" | Any of the above four moments in isolation | -| `test-suite-auditor` (agent) | Delegating the slow/high-output half of `audit-test-suite` | Quick inline assertion review (skill handles that) | - -### In-depth target: `close-test-risk-gaps` - -Selected as the riskiest/most-changed skill: largest diff in the most recent commit (+72/-lines, -adding the 5th failure class), and structurally the highest triggering risk in the plugin — a -triage gate with **five independent trigger conditions** evaluated per-class, rather than one -single trigger, which is the shape most likely to under-fire on some classes while over-fabricating -tests on others. - -- **Claim:** given a feature that's functionally working and about to be called done, triage five - failure classes invisible to "given this input, is the output right" — concurrency/idempotency, - contract drift, performance regression, authorization, migration/rollout compatibility. Per class: - if the trigger condition fires, add the one minimal named test; if it does not fire, skip - **explicitly** and say why. -- **Trigger intent — should fire:** "is this ready", "anything else to test", "before I open the - PR", "did I miss anything", plus scenario descriptions of race conditions, webhook retries, N+1 - queries, authz, migrations/backfills — even without naming the skill. -- **Trigger intent — should NOT fire:** requests to choose inputs for a *new* test - (`design-test-cases`), audit an *existing* suite's trustworthiness (`audit-test-suite`), reproduce - and fix a *specific* bug (`test-a-bug-fix`), or verify a *refactor* changed nothing - (`verify-behavior-preserved`) — these share testing vocabulary but are different lifecycle moments. -- **Intended outcome vs. no-skill baseline:** the with-skill agent systematically probes blind spots - a baseline agent's happy-path suite review misses — invariant-based concurrency assertions (not - response-code assertions), an authz matrix that returns 404 (not 403) for another user's resource, - N+1 query-count assertions (not wall-clock timing), schema-validated contract tests (not - hand-written dict comparisons), and migration round-trip/backfill-idempotency tests — **and** - explicitly reports which classes it skipped rather than staying silent on them. - -### Sampled: `design-test-cases`, `audit-test-suite`, `test-a-bug-fix`, `verify-behavior-preserved`, `test-suite-auditor` - -Structural + content review in full (already read completely in Phase 0); a lighter 4-positive / -4-negative trigger check (N=1, sanity read rather than gate-scored) instead of the full N=3 x 8-10 -battery. No behavioral-lift subagent runs for these five in this pass — flagged in §08 as the -natural next run if this evaluation is repeated. - -## §03 Trigger battery — `close-test-risk-gaps` (in-depth, N=3, gate-scored) - -**Positives (10):** -1. "The coupon redemption feature is done and all tests pass — is there anything else I should test before opening the PR?" -2. "Before I ship this, is this ready? Tests are green." -3. "Did I miss anything on this order charge endpoint before merging?" -4. "We've had webhook retries double-charge customers before — can you check if this new charge handler is safe?" -5. "Is this migration safe to run on prod? It renames a column." -6. "Can two people redeem the last coupon at the same time? Want to make sure that's not a race condition." -7. "Double-checking auth on this endpoint — can another tenant somehow see this data?" -8. "The orders list page feels like it could N+1, worth checking before we deploy?" -9. "Not sure I'm confident enough to call this done — tests pass but idk." -10. "We're backfilling a new column on the users table — is that safe for a rolling deploy?" - -**Near-miss negatives (10):** -1. "Write some tests for this new discount() function." -2. "These tests keep failing randomly, not sure why." -3. "Users report the app crashes on this input, can you fix it?" -4. "I want to refactor this class to remove duplication without breaking anything." -5. "Can you run mutation testing on this module?" -6. "Getting a KeyError in prod — this worked last week." -7. "What edge cases should I cover for this date parser?" -8. "I upgraded pydantic, want to make sure nothing changed." -9. "Are these tests actually testing anything or just passing?" -10. "How do I property-test this sorting function?" - -**Cross-skill collision prompts (does the right one win?):** -1. "This bug fix is done, tests pass — is it safe to ship?" (`test-a-bug-fix` vs `close-test-risk-gaps`) -2. "I just refactored the auth middleware, all tests still pass, ready to merge?" (`verify-behavior-preserved` vs `close-test-risk-gaps`) -3. "Is this test suite good enough before we ship?" (`audit-test-suite` vs `close-test-risk-gaps`) - -### Results - -**Positive battery** (N=1 first pass on all 10; N=2 on the 3 that showed no fingerprint, to confirm -stability — see §00 for how "fired" is operationalized): - -| # | Prompt topic | tool_uses | Fired? | Evidence | -|---|---|---|---|---| -| 1 | coupon shared-stock counter | 0, 0 (2 runs) | **NO** (stable) | No skill idiom; general TOCTOU-race knowledge only | -| 2 | customer lookup by ID, "is this ready" | 0, 0 (2 runs) | **NO** (stable) | General IDOR/pagination advice, no 5-class structure | -| 3 | order charge + Stripe | 3 | **YES** | Explicitly cites `ceh-testing/skills/.../close-test-risk-gaps`; reproduces the exact "Reporting the gate" format verbatim | -| 4 | webhook double-charge | 3, 2 (2 runs) | **NO** (stable) | Solid generic idempotent-webhook pattern, no 5-class structure, no citation either run | -| 5 | migration renaming a column | 1 | **YES** | "Expand, then contract — in separate deploys" near-verbatim to skill text | -| 6 | coupon race (names "race condition") | 2 | **YES** | "assert the invariant... which one wins is legitimately nondeterministic" — near-verbatim | -| 7 | cross-tenant document access | 1 | **YES** | "Return 404 (not 403)... to avoid confirming existence" — matches skill's "leaks the ID space" reasoning | -| 8 | orders list N+1 | 1 | **YES** | Query-count assertion idiom (`assert_num_queries`) matches skill's `query_counter` pattern | -| 9 | CSV export, "not confident" | 7 | **YES** | Full "risk gate — which classes actually apply" structure with fire/skip verdicts per class | -| 10 | rolling-deploy backfill | 2 | **YES** | `test_backfill_is_idempotent`, `test_old_code_still_reads_new_schema` — near-verbatim function names and comments from the skill | - -**Positive trigger rate: 7/10 (70%)** — below the 8/10 gate threshold. - -**Near-miss negatives:** 0/10 fired `close-test-risk-gaps`. Every one instead fired the *correct* -sibling skill with strong, distinctive content: NEG-1 → `design-test-cases` (partition/boundary -ladder), NEG-2 → `audit-test-suite` (flakiness/order-dependence), NEG-3 → `test-a-bug-fix` (explicit -citation, reproduce-first protocol), NEG-4 → `verify-behavior-preserved` (near-verbatim "green bar -with zero test edits is the gate"), NEG-5 → `audit-test-suite` (delete-the-code check, mutmut), -NEG-6 → `test-a-bug-fix`-flavored root-cause-then-reproduce, NEG-7 → `design-test-cases`-flavored -(weak/general-knowledge, tool_uses:0), NEG-8 → `verify-behavior-preserved` (near-verbatim -"Dependency and runtime upgrades" section), NEG-9 → `audit-test-suite` (near-verbatim assertion-audit -grep pattern), NEG-10 → `design-test-cases` rung 6 (matches the "four shapes" property-testing -framework exactly). - -**False-positive rate: 0/10.** Exceeds the ≤1/10 threshold with room to spare, and doubles as strong -evidence the plugin's five skills have well-separated trigger moments in practice. - -**Collision prompts:** all three resolved gracefully — no wrong-skill-wins case observed. -- COLL-1 (bug fix + "safe to ship"): both `test-a-bug-fix` (exact "prove the test is coupled to the - fix" `git stash` idiom) and `close-test-risk-gaps` (concurrent-case test + explicit per-class - skip reporting) fired and combined correctly. -- COLL-2 (refactor + "ready to merge"): `verify-behavior-preserved` dominated (equivalence-diff - golden file, "deny by default," "fails closed") — a defensible single-skill resolution given the - prompt is fundamentally about behavior preservation; `close-test-risk-gaps`'s 5-class structure - did not appear. -- COLL-3 ("test suite good enough" + "before we ship"): both `audit-test-suite` (assertion-audit, - delete-the-code, mutation) and `close-test-risk-gaps` (concurrency/idempotency, authz matrix, - explicit "skip... N/A" framing) fired and blended sensibly. - -**Diagnosis of the 3 misses:** all three describe the trigger scenario *without* using the -description's own named-technique vocabulary — "shared stock counter" (not "race condition"), -"double-charge" (not "idempotent"), "look up by customer_id" (not "authorization"/"IDOR"). The 7 -that fired either used the named technique directly (POS-6 says "race condition"; POS-5 nearly -quotes "is this migration safe") or hit a phrase from the description almost verbatim (POS-9's "not -confident" ~ description's "tests pass but I am not confident"). This is a targeted, fixable -description gap, not a broad failure — see §04 for the fix and re-run. - -### Iteration 2 re-run (after the description fix — see §04) - -Re-tested the 3 misses at N=2 each, plus a 3-negative and 2-positive regression spot-check. - -| Prompt | Pre-fix | Post-fix run1 | Post-fix run2 | Evidence | -|---|---|---|---|---| -| POS-1 (shared stock counter) | 0/2 | NO (tool_uses:0) | weak/NO (tool_uses:1, no distinctive idiom) | Still general TOCTOU knowledge only — **stable miss** | -| POS-2 (customer lookup by ID) | 0/2 | **YES** (tool_uses:3) | **YES** (tool_uses:5) | Both runs reproduce the full 5-class gate verbatim: "Authorization — FIRES... Skipped, explicitly: Concurrency..., Contract drift..., Migration..." | -| POS-4 (webhook double-charge) | 0/2 | **YES** (tool_uses:10) | **YES** (tool_uses:3) | Both runs explicitly cite the skill by name: "this is exactly the `close-test-risk-gaps` idempotency class" / "per this repo's `close-test-risk-gaps` skill" | - -POS-2 and POS-4 flipped cleanly from stable misses to stable, strongly-evidenced fires — the new -scenario-shaped cues ("a caller-supplied ID could return someone else's data", "a retry could -double-charge") did their job. **POS-1 remains a stable miss** even with the fix: the model answers -a shared-stock-counter race question confidently from general TOCTOU knowledge both times -(tool_uses:0 on run 1), regardless of description wording — this looks like a ceiling on -already-well-known failure classes rather than a description defect, and isn't worth chasing further -without risking overfitting the description to one adversarial prompt. - -**Regression check — no new false positives or lost fires:** -- NEG-1 (new discount function), NEG-4 (refactor), NEG-6 (prod KeyError): all three still fired their - correct sibling skill only, zero `close-test-risk-gaps` fingerprint in any — false-positive rate - holds at 0/10 (now 0/13 counting these). -- POS-6, POS-9 (previously-firing positives): both still fire strongly post-fix — no regression from - the longer description. - -**Revised positive trigger rate: 9/10 (90%)** — clears the ≥8/10 gate threshold. False-positive rate -holds at 0/10. - -## §04 Behavioral tasks & assertions — `close-test-risk-gaps` - -### Task A — mixed trigger (4 of 5 classes fire, migration does not) - -Fixture: a small order/coupon service — shared-stock coupon redemption (concurrency), a public -order-response endpoint consumed by a mobile client (contract), an orders-list endpoint doing a -per-row customer lookup (performance), a get-order-by-id endpoint with no ownership check (authz), -no schema/migration change in scope. Prompt: "This feature is functionally complete and tests pass. -Anything else I should test before opening the PR?" - -- A1: Concurrency test asserts an invariant (stock/count), not individual response codes. -- A2: Idempotency test asserts a repeated request produces one effect (e.g. charge count == 1). -- A3: Authz test asserts another user's resource returns **404, not 403**. -- A4: Performance test asserts a **bounded query count**, not wall-clock duration. -- A5: Contract test validates the response against a schema/model, not a hand-written dict. -- A6: Migration is explicitly reported as skipped, with a reason — not silently omitted. - -### Task B — everything skips - -Fixture: an internal, unauthenticated, single-process admin script with no shared state, no -external contract, no user-supplied IDs, no migration. Prompt: same as Task A ("is this ready to -ship"). - -- B1: No fabricated test for a class whose trigger doesn't fire (no invented concurrency/authz test). -- B2: All five classes explicitly reported skipped, each with a specific reason tied to the fixture. -- B3: Skip reasons name the actual absence (e.g. "no shared mutable state") rather than a generic - "looks fine." - -### Task C — migration-specific - -Fixture: an Alembic migration renaming `total` → `order_total` on the `orders` table in a single -migration, plus a backfill script. Prompt: "Is this migration safe to run during a rolling deploy?" - -- C1: Includes a migration round-trip test (up → down → up). -- C2: Includes a backfill-idempotency test (run twice, same resulting state). -- C3: Flags the single-migration rename as breaking and recommends expand/contract, rather than - approving it as-is. -- C4: Stays scoped to migration/rollout reasoning rather than generic advice unconnected to the - fixture. - -### Results (with-skill vs. baseline, N=2 per task, graded per assertion) - -**Task A** (6 assertions/run): - -| Run | A1 concurrency invariant | A2 idempotency | A3 404-not-403 | A4 N+1 query-count | A5 schema validation | A6 explicit migration skip | Total | -|---|---|---|---|---|---|---|---| -| with-skill #1 | PASS | PASS | PASS | PASS | PASS | PASS | 6/6 | -| with-skill #2 | PASS | PASS | PASS | PASS | PASS | PASS | 6/6 | -| baseline #1 | PASS | PASS | FAIL (`in (403,404)`, not strict 404) | **FAIL (explicitly deferred as "follow-up ticket")** | FAIL (flagged shape drift instead, no schema-model validation) | PASS (terse) | 3/6 | -| baseline #2 | PASS | PASS | PASS (strict 404 this run) | **FAIL (not mentioned at all)** | FAIL (not mentioned) | FAIL (not mentioned) | 3/6 | - -With-skill: **12/12**. Baseline: **6/12**. The gap is concentrated and consistent across both -baseline runs in exactly two places: the **N+1 query-count assertion** (baseline either explicitly -declines it as out-of-scope or omits it silently, both runs) and the **schema-validated contract -test** (baseline never reaches for schema validation against a model; run 1 substitutes a same-service -shape-comparison test instead, which is a real but different catch). This is the single clearest, -most reproducible evidence of behavioral lift in this evaluation. - -**Task B** (3 assertions/run — no fabricated tests / all 5 classes explicitly skipped with reasons / reasons are fixture-specific): - -| Run | B1 no fabrication | B2 all 5 explicit | B3 fixture-specific reasons | Total | -|---|---|---|---|---| -| with-skill #1 | PASS | PASS (4 bullets, one combining 2 classes) | PASS | 3/3 | -| with-skill #2 | PASS | PASS (5 distinct bullets, self-cites "close-test-risk-gaps") | PASS | 3/3 | -| baseline #1 | PASS | FAIL (contract class never mentioned; rest compressed into one terse sentence) | FAIL (thin) | 1/3 | -| baseline #2 | PASS | PASS (all 5 addressed, split across two sentences) | PASS | 3/3 | - -With-skill: **6/6**. Baseline: **4/6**, with real run-to-run variance — baseline run 2 matched -with-skill exactly. Honest reading: for the "everything skips" case, a strong baseline sometimes -spontaneously produces the right structure because the fixture itself makes "nothing applies" -obvious; the skill's advantage here is consistency (2/2), not exclusivity. - -**Task C** (4 assertions/run): - -| Run | C1 round-trip test | C2 backfill idempotent | C3 flags breaking, recommends expand/contract | C4 scoped reasoning | Total | -|---|---|---|---|---|---| -| with-skill #1 | PASS (up→down→compare) | PASS | PASS | PASS | 4/4 | -| with-skill #2 | PASS (`test_migration_round_trips` — verbatim match to skill's own example) | PASS | PASS | PASS | 4/4 | -| baseline #1 | **FAIL** (no alembic up/down round-trip test at all) | PASS | PASS (proposes a trigger-based dual-write alternative) | PASS | 3/4 | -| baseline #2 | PASS | PASS | PASS | PASS | 4/4 | - -With-skill: **8/8**. Baseline: **7/8** — the smallest lift of the three tasks. Expand/contract for a -rolling-deploy rename is evidently common-enough SRE knowledge that a strong baseline reaches it -almost as reliably unprompted; the skill's edge here is the specific round-trip-test discipline, not -the underlying migration strategy. - -**Aggregate: with-skill 26/26 (100%) vs. baseline 17/26 (65%)**, lift concentrated most heavily in -the idiom-specific assertions (bounded query count, schema-validated contract, strict 404) that a -capable baseline does not reliably reach for on its own, smallest on the failure class -(migration/rollout) closest to general SRE knowledge. No regression observed in any run — with-skill -never underperformed baseline on any assertion, in either task. - -**Criterion 5 (behavioral lift): MET**, with the variance reported honestly above rather than -collapsed to one number. - -## §05 Structural findings - -| Check | Result | Evidence | -|---|---|---| -| `plugin.json` valid JSON, `name` matches dir, `version` semver | PASS | `ceh-testing/.claude-plugin/plugin.json:2-3` → `"name": "ceh-testing"`, `"version": "1.0.1"` | -| `marketplace.json` lists plugin, version matches | PASS | `marketplace.json:222-225` → version `1.0.1`, matches manifest | -| Each `SKILL.md` has `name`+`description`, `name` matches dir | PASS | confirmed for all 5 skills by direct read | -| Agent `.md` has `name`+`description` | PASS | `test-suite-auditor.md:2-12` | -| `references/` holds only schemas/templates | N/A | plugin has no `references/` dir — all content inline, consistent with repo convention for skills this size | -| Body size within norms (~500 line guidance) | PASS | largest skill is 222 lines (`design-test-cases`), agent 85 lines — well under threshold | -| Repo `validate.py` cross-check | PASS | `OK: all plugin checks passed` | - -**Criterion 1 (structurally valid): MET.** - -## §06 Content findings - -Rubric judgment against `references/eval-rubric.md`, cited lines from `close-test-risk-gaps/SKILL.md` -(in depth) and brief checks on the other four. - -**Description — what AND when:** states the moment ("pre-completion gate when a feature is -functionally working and about to be called done") and lists 12 explicit trigger phrases -(`close-test-risk-gaps/SKILL.md:9-12`). Moment-framed, not topic-framed. **Gap found**: unlike its -four sibling skills, `close-test-risk-gaps`'s description has **no explicit "Not for…" clause** -pointing to the adjacent skills it might be confused with — `test-a-bug-fix`, `design-test-cases`, -`audit-test-suite`, and `verify-behavior-preserved` all have one (e.g. -`test-a-bug-fix/SKILL.md:10-11`: "Not for choosing inputs for new feature tests (use -design-test-cases) or for judging an existing suite (use audit-test-suite)."), this one doesn't. The -collision battery (§03) shows this hasn't caused an actual wrong-skill-wins case yet — the model -resolved all three collisions gracefully — but it's the one description-level inconsistency across -the five skills, and it's the same lever that would fix the under-triggering in §03 (the fix adds -both the missing scenario-phrasings and a "not for" clause in one edit). - -**Body — is it the delta:** the five idioms this skill prescribes are each a specific, falsifiable -convention beyond generic "test this more" advice: assert the invariant not individual responses -(`close-test-risk-gaps/SKILL.md:53-54`), 404 not 403 with the "leaks the ID space" reasoning -(`:142`), a countable bound not wall-clock (`:100`), schema validation not a hand-written dict -(`:67-68`), and expand-then-contract across separate deploys (`:179-181`). The behavioral run (§04) -directly measured whether this is genuinely additive vs. restated general knowledge: Task A shows a -real, reproducible gap on the query-count and schema-validation idioms (baseline never reached for -either); Task C shows a much smaller gap on expand/contract (baseline reproduced it independently -both runs) — so the delta is real but **uneven across the five classes**, strongest on -performance/contract, weakest on migration strategy (though still ahead on the specific -round-trip-test discipline). - -**Size / progressive disclosure:** 208 lines, no `references/` split — appropriate at this size, well -under the ~500-line guidance. - -**Explains the why:** consistently reasons rather than commands — e.g. "a 403 confirms the resource -exists, which leaks the ID space" (`:142`) is a reason, not a bare MUST. No ALL-CAPS wall found. - -**Sampled skills (structural + content only, from full Phase-0 read):** `design-test-cases` (222 -lines, nine-rung ladder, each rung explains *when* to stop — "most functions need 1, 2, and 6" is a -genuine complexity-reducing heuristic, not restated knowledge), `audit-test-suite` (177 lines, the -"delete-the-code check" framing is a sharp, memorable delta), `test-a-bug-fix` (131 lines, the -"prove the test is coupled to the fix" `git stash` step is the one non-obvious insight most bug-fix -workflows skip), `verify-behavior-preserved` (132 lines, "the rule that makes all of it worth -something" — editing a characterization test = behavior changed — is a clean, falsifiable rule). -All four have "Not for…" clauses; all four fired cleanly and distinctly in the negative/collision -battery (§03) with no observed restatement-only content. - -**Criterion 4 (content is delta + moment-framed): MET.** The missing "not for" clause on -`close-test-risk-gaps` is real but advisory-level on its own (no observed collision harm); it's -counted against criterion 2 (triggering), not double-counted here, because that's where its effect -was actually measured. - -## §07 Gate scorecard - -| # | Criterion | Threshold | Status | Evidence | -|---|---|---|---|---| -| 1 | Structurally valid | all deterministic checks pass | **MET** | §05 | -| 2 | Triggers on intent | ≥8/10 positive fire (≥2/3 runs) | **MET** (9/10, after iteration 2 fix) | §03 | -| 3 | Doesn't over-trigger | ≤1/10 near-miss fires | **MET** (0/13) | §03 | -| 4 | Content is delta + moment-framed | rubric pass | **MET** | §06 | -| 5 | Behavioral lift | with-skill beats/no-regress baseline | **MET** (26/26 vs 17/26, no regression) | §04 | -| 6 | User confirms | — | **MET** (2026-07-30) | User confirmed via sign-off question, chose "Yes, mark passed" without a version bump | - -`eval_gate: 6/6`. Iteration 1 fixed criterion 2 by adding scenario-shaped trigger cues and a "not -for" clause to `close-test-risk-gaps`'s description (`ceh-testing/skills/close-test-risk-gaps/SKILL.md`), -re-validated with `tools/validate-plugins/validate.py` (frontmatter length was the binding -constraint — trimmed to 1005/1024 chars). All six criteria are now met. - -## §08 Advisory backlog - -- The one residual trigger miss (a shared-counter race described without any of the skill's named - vocabulary) is a reasonable stopping point — 9/10 clears the gate, and chasing 10/10 risks - overfitting the description to one prompt at the cost of readability. Not a blocker. -- Full N=3, 8–10-prompt trigger batteries for the other four skills + agent were not run as a - dedicated pass — the negative/collision battery opportunistically exercised all four (§03) with - strong, clean results, but a dedicated battery would firm this up if drift is suspected later. -- `test-suite-auditor` (the agent) triggering was not directly tested — background subagents in this - harness cannot dispatch further agents, so no dispatch of it was observed even in the two prompts - (NEG-5, NEG-9) where it would have been a reasonable choice (large, slow mutation runs). Structural - and content review only; behavioral/triggering is unproven for this component specifically. -- Task B shows the smallest and noisiest lift (baseline matched with-skill on run 2) — if this skill - is re-evaluated later, a 3rd run of Task B would clarify whether that was noise or a genuine - ceiling on the "everything skips" case. -- The methodological confound in §00 (evaluating from inside the target's own repo) applies to any - future evaluation of a skill in this repository, not just this one — worth a note in - `ceh-evaluation` itself if it recurs. diff --git a/.agents_workspace/skill-evals/ceh-testing/run-002/SKILL_EVAL.md b/.agents_workspace/skill-evals/ceh-testing/run-002/SKILL_EVAL.md deleted file mode 100644 index 8e1bdd4..0000000 --- a/.agents_workspace/skill-evals/ceh-testing/run-002/SKILL_EVAL.md +++ /dev/null @@ -1,305 +0,0 @@ ---- -artifact: SKILL_EVAL -status: passed -created: 2026-07-30 -updated: 2026-07-31 -target: ceh-testing (plugin) — focused pass on design-test-cases and audit-test-suite -target_kind: plugin -eval_gate: 6/6 -iterations: 1 ---- - -## §01 Verdict - -Both skills are structurally sound, trigger reliably (design-test-cases 10/10 positive, 0/10 -false-positive; audit-test-suite 9/10 positive, 0/10 false-positive), and separate cleanly from -their four siblings — six collision prompts all resolved to the correct skill (or a defensible -blend on the two deliberately dual-intent prompts) with zero wrong-skill-wins. The honest surprise -is **criterion 5**: against a strong baseline model with no skill at all, `audit-test-suite` shows -real but narrow lift (the delete-the-code/diff-scoped-mutation verification discipline, not the -assertion-quality reading itself — the baseline caught the planted tautological-formula defect -independently, twice), and `design-test-cases` shows **no measurable lift** on either behavioral -task — the baseline reached for partition/boundary/property-based/fuzz testing unprompted, in one -case (D2) writing a more elaborate property suite (a reference-oracle cross-check, a DoS budget -guard) than the with-skill answer. This is a real, reportable finding, not an evaluation defect: a -capable model already knows a large fraction of this technique, and the skill's marginal value here -is closer to a **forcing function / checklist** (ensuring the technique is actually applied and -named) than new information. `eval_gate: 6/6` — criteria 1-4 met cleanly, criterion 5 met at its -minimum bar (no regression, with the narrow-lift finding reported honestly rather than inflated), and -criterion 6 met: the user reviewed that finding and confirmed accepting it as-is. - -## §00 Scope note, methodology, and confounds - -`run-001` (status: passed, 6/6) evaluated the whole `ceh-testing` plugin, going in depth on -`close-test-risk-gaps` and only sampling `design-test-cases`/`audit-test-suite` structurally — its -advisory backlog flagged both as needing a full battery. This run (`run-002`) closes that gap, at -**reduced N** (user-confirmed): trigger prompts at N=1 with N=2 confirmation on ambiguous misses (none -needed — every miss was stable); behavioral tasks at N=2 baseline runs, N=1 with-skill (see below). - -**Three confounds, compounding on top of run-001's original repo-confound:** - -1. **No real application code.** This repo ships plugin/skill markdown, not an app — cold subagents - searching for "the discount function" or "the coupon endpoint" routinely find nothing and either - defer (still citing the skill by name — counted as a weak fire) or, in the worst case (audit - POS-8), pivot away from the skill's specific technique entirely (counted as a miss). -2. **Meta-contamination.** Several cold subagents discovered this run's own in-progress scratch files - (`agent-map.md`, `trigger-results-*.md`, the draft `SKILL_EVAL.md` itself) under - `.agents_workspace/skill-evals/ceh-testing/run-002/` and explicitly reasoned about them — one - quoted this report's own wording back verbatim. This happened because run tracking files were - written into the live repo mid-run rather than to the session scratchpad; a future run of this - kind should keep iteration artifacts outside the repo until the run is complete. No case flipped - a genuine miss into a false fire or vice versa, but it is a real limitation on how "cold" these - subagents actually were. -3. **Unauthorized side effects from over-eager subagents.** At least one subagent (testing the - audit-test-suite positive "are these tests actually testing anything") went beyond answering and - ran live mutation testing against the repo's own `tools/validate-plugins/validate.py`, leaving a - scratch mutant file (`_mut_scratch.py`) in the tree at least twice (self-cleaned both times, - confirmed via `git status`). Separately, a different subagent (testing the audit-test-suite - *negative* "add pytest-randomly to CI") edited three real stack-testing skill files - unrequested — found and reverted (`git checkout --`) before writing this report. Both are noted - here because they reveal a real operational risk of this evaluation method in this environment - (cold subagents have full repo write access, not sandboxed to a worktree), not because they - affected the trigger verdicts. -4. **Session usage-limit pressure** cut the original N=2 behavioral dispatch short (8 of 16 planned - subagent runs were stopped by the user after two independent subagents reported the account's - 5-hour usage guard tripping at 90-96%). Recovered by having the evaluator (this session) produce - the with-skill answer directly — since it already holds the skill's full content — and dispatching - only the baseline side plus the two remaining trigger prompts as fresh subagents. This is an - asymmetry from the standard "two fresh subagents" method: the with-skill side is N=1 and not blind - to its own eval context (though it does not consult the skill's SKILL.md file mid-task since the - content was already loaded); the baseline side stayed N=2, fresh, cold, and explicitly instructed - not to explore the repo or consult any skill. - -## §02 Derived criteria - -### `design-test-cases` - -- **Claim:** nine-rung input-selection ladder — partitions, boundaries, decision tables, state - transitions, pairwise, properties, metamorphic relations, fuzzing, forced dependency failure — for - deciding *which inputs and scenarios* a test should cover, not how to wire the runner. -- **Trigger intent — should fire:** "write tests for this", "what should I test", "cover the edge - cases", "is this tested enough", "property-based"/"hypothesis"/"fast-check", "fuzz this", "how do I - test something with no correct answer", a happy-path-only test file, or a scenario description that - clearly needs input-selection help without naming the skill. -- **Trigger intent — should NOT fire:** wiring fixtures/runner, judging an *existing* suite's - trustworthiness, reproducing a specific bug, verifying a refactor changed nothing, or a - pre-completion risk triage unrelated to input selection. -- **Intended outcome vs. no-skill baseline:** partitions inputs into classes, covers - boundary-and-one-step-either-side, reaches for a property-based test on pure functions, asserts - literal expected values. **Measured result: baseline reached all of these unprompted on both tasks - tried** — see §04. - -### `audit-test-suite` - -- **Claim:** six ordered checks (cheapest first) to determine whether a *passing* suite would - actually catch a defect — assertion audit, delete-the-code check, diff-scoped mutation testing, - flakiness/order-dependence, level and speed, branch (not line) coverage — reporting worst-first. -- **Trigger intent — should fire:** "are these tests any good", "audit the tests", "mutation - testing"/"mutmut"/"stryker", "why did the tests not catch this", "flaky test", "tests pass but the - bug shipped", "review the test coverage", or right after a batch of tests was generated. -- **Trigger intent — should NOT fire:** choosing new inputs/scenarios, reproducing a specific bug, a - pre-completion risk triage unrelated to suite trustworthiness, or CI/runner wiring. -- **Intended outcome vs. no-skill baseline:** flags assertion-free and "asserts existence not value" - tests, specifically flags a test whose expectation is computed via the code's own logic, proposes - diff-scoped (never whole-repo) mutation testing, calls out flakiness as shared state to fix (not - paper over), insists on branch over line coverage. **Measured result: baseline independently caught - the assertion-quality defects (including the tautological-formula trap) and the branch-vs-line - distinction on both tasks; the one place with-skill pulled ahead was recommending the concrete - delete-the-code/mutation-testing verification step as the way to *confirm* a suite's quality, which - baseline did not reach for** — see §04. - -## §03 Trigger battery results - -### `design-test-cases` — 20/20 returned - -**Positive trigger rate: 10/10.** Every positive prompt produced ladder vocabulary or an explicit -skill citation; 3 of 10 were "weak" fires (skill cited by name, concrete answer deferred because the -repo has no matching real file — see §00 confound 1), not a triggering weakness. - -**False-positive rate: 0/10.** Every negative correctly stayed silent on design-test-cases' ladder -vocabulary; most explicitly fired the *correct* sibling instead (verify-behavior-preserved, -audit-test-suite, close-test-risk-gaps, test-a-bug-fix), confirming clean separation. - -Full per-prompt evidence: `iteration-1/trigger-results-design-test-cases.md`. - -### `audit-test-suite` — 20/20 returned - -**Positive trigger rate: 9/10.** Six clean/strong fires, three weak (deferred but cited by name), one -confound-driven miss: POS-8 ("review the test coverage on this PR") pivoted to a general PR-readiness -review instead of applying audit-test-suite's specific techniques (assertion-shape check, mutation -testing, branch-vs-line coverage) — a genuine miss worth naming, not confound noise, though it shares -the same root cause (no real test suite in this repo to apply the technique to). Multiple *other* -positives (POS-1, POS-3, POS-9, POS-10) independently ran real diff-scoped mutation testing against -`validate.py` unprompted — strong, convergent, if messy, evidence the skill's specific technique -lands when a subagent has something to apply it to. - -**False-positive rate: 0/10.** Every negative stayed silent on audit-test-suite's specific -vocabulary (assertion audit, mutation testing, branch coverage), including under real usage-limit -pressure (NEG-5) and even when the subagent found and read this run's own eval files (NEG-2, NEG-4). - -**Collision prompts (6): zero wrong-skill-wins.** COLL-2 and COLL-5 both correctly resolved to -audit-test-suite — COLL-2 found a *real* sibling repo with actual auth code and delivered a -security-relevant assertion-quality audit (found an exploitable refresh-token type-confusion bug the -existing suite's weak assertions missed); COLL-5 ran a live mutation-testing probe. COLL-4 resolved -cleanly to close-test-risk-gaps with no interference. COLL-6 (deliberately dual-intent: "write good -new tests AND check the existing ones aren't garbage") correctly considered both design-test-cases -and audit-test-suite. COLL-1 and COLL-3 are noted separately: COLL-1 found and audited *this eval's -own* with-skill D1 test file, catching a real authoring defect (see §04); COLL-3 reinterpreted "here" -as design-test-cases' own SKILL.md and assessed its sufficiency — an unusual meta-application, not a -clean triggering data point either way. - -Full per-prompt evidence: `iteration-1/trigger-results-audit-test-suite.md`. - -## §04 Behavioral tasks & assertions - -### `design-test-cases` - -**Task D1 (discount function, tier x threshold boundaries)** — 3 assertions: D1a (one test per -class), D1b (explicit $50/$100 boundary tests), D1c (literal expected values, not re-derived). - -| | D1a | D1b | D1c | Notes | -|---|---|---|---|---| -| with-skill (mine, 1 run) | PASS — parametrized one-test-per-class | PASS — explicit boundary triples at both thresholds | **PARTIAL** — 2 of ~15 assertions use `total * 0.05` instead of a literal (caught by COLL-1, see §03) | | -| baseline run 1 | PASS — 9-case tier x bracket table | PASS — explicit BVA at both thresholds, plus no-runaway-escalation check | Not independently verified | Also added a return-value-contract test and flagged invalid-input assumptions explicitly | -| baseline run 2 | PASS — decision table + BVA | PASS | Not independently verified | | - -**No measurable lift.** Baseline reached the same partition/boundary structure unprompted, in both -runs. D1c isn't a clean discriminator here regardless: with no real `discount()` implementation to -introspect, both with-skill and baseline necessarily *derive* percentages from the spec rather than -copy an implementation's formula — the specific defect the skill warns against (test agrees with a -*wrong implementation* because it shares its logic) can't fully manifest against a spec-only task. -Noted as a task-design limitation, not a skill defect. - -**Task D2 (parse_duration, untrusted-input parser)** — 3 assertions: D2a (boundary/string edge -cases), D2b (property-based round-trip/invariant), D2c (fuzz test distinct from D2b). - -| | D2a | D2b | D2c | Notes | -|---|---|---|---|---| -| with-skill (mine, 1 run) | PASS | PASS — round-trip + monotonicity-style invariant | PASS — `st.text()`/`st.binary()` fuzz | One test (`test_boundary_and_edge_strings`) has a self-contradictory oracle flagged by A-POS-3's mutation run (comment implies `"999999h"` is a valid boundary, code logic rejects it) — a real authoring defect | -| baseline run 1 | PASS — ~35 malformed cases, type confusion, Unicode confusables | PASS — **reference-oracle property test** (independent regex cross-checked against the function) | PASS — pure fuzz + a DoS/performance budget guard | Arguably more thorough than with-skill on both counts | -| baseline run 2 | PASS — similar breadth, overflow handling | PASS — round-trip + additive/metamorphic property | PASS — broad + alphabet-restricted "near-miss" fuzz | Also very thorough | - -**No measurable lift — baseline met or exceeded with-skill on every assertion, both runs.** This is -the honest, reportable result: a strong baseline model already reaches for property-based and fuzz -testing on an untrusted-input parser without being told to. The skill's value on this specific task -looks more like assurance/consistency than net-new capability. - -### `audit-test-suite` - -**Task A1 (refund() test file, 3 planted defects + 1 good test)** — 4 assertions: A1a (flag -no-assertion test), A1b (flag `is not None`-only), A1c (flag the tautological-formula test), A1d -(recommend a concrete verification step). - -| | A1a | A1b | A1c | A1d | Notes | -|---|---|---|---|---|---| -| with-skill (mine, 1 run) | PASS | PASS | PASS — named it "the signature defect no automated check finds" | PASS — delete-the-code check + diff-scoped `mutmut` | | -| baseline run 1 | PASS | PASS | **PASS** — independently identified the algebraic cancellation and "checking the code against itself" | Partial — recommended more coverage (boundaries, side effects) but not a *verification technique* | | -| baseline run 2 | PASS | PASS | **PASS** — same catch, plus flagged shared mutable `order` state as an inter-test-ordering risk | Partial — same pattern, strong on gaps, no delete-the-code/mutation recommendation | | - -**Narrow, real lift on A1d only.** Both baseline runs independently caught the tautological-formula -defect (A1a-c) — this specific planted defect turned out not to discriminate. The one place with-skill -pulled ahead both runs: recommending *how to verify* the fix (break the code, confirm the suite -catches it) rather than only *what's missing*. - -**Task A2 (flaky integration test + line-only coverage)** — 3 assertions: A2a (order-dependence -diagnosis, not sleep/skip), A2b (branch vs line coverage), A2c (don't treat the % as evidence). - -| | A2a | A2b | A2c | -|---|---|---|---| -| with-skill (mine, 1 run) | PASS | PASS | PASS | -| baseline run 1 | PASS — explicitly names sleep/skip/reruns as anti-fixes, suggests bisecting suite order and checking `xdist` | PASS — names `--cov-branch`, even suggests mutation testing | PASS | -| baseline run 2 | PASS — same diagnosis, flags `@flaky` decorators as an anti-fix by name | PASS | PASS | - -**No measurable lift.** Both baseline runs matched with-skill assertion-for-assertion. - -**Criterion 5 overall: honestly unproven/weak, not met at the "with-skill clears assertions baseline -misses" bar.** No regression was observed anywhere (with-skill never scored below baseline on any -assertion, either skill), which is the floor the criterion asks for — but the "beats baseline" -half only held for one assertion out of thirteen measured (A1d). This is a materially different, -less favorable result than run-001 found for `close-test-risk-gaps` (26/26 vs 17/26) and should be -reported as such rather than smoothed over. - -## §05 Structural findings - -| Check | Result | Evidence | -|---|---|---| -| `plugin.json` valid, `name`/`version` match marketplace | PASS | Confirmed unchanged since run-001 (`1.0.1`, matches marketplace.json) | -| `design-test-cases/SKILL.md`, `audit-test-suite/SKILL.md` frontmatter (name/description present, name matches dir) | PASS | Confirmed by direct read in Phase 0 | -| Body size within norms | PASS | 223 and 178 lines respectively, no `references/` split needed | -| Repo `validate.py` cross-check | PASS | `OK: all plugin checks passed` (re-run after reverting the unauthorized edits — see §00) | - -**Criterion 1: MET.** - -## §06 Content findings - -**`design-test-cases` — description:** states the moment ("deciding which inputs and scenarios a test -should cover — not how to wire the runner") and lists explicit trigger phrases plus a "pairs with" -pointer to the stack skills (`SKILL.md:9-13`). Moment-framed, not topic-framed. - -**`design-test-cases` — body is the delta:** the nine-rung ladder with explicit stop conditions -("most functions need 1, 2, and 6") is a genuine complexity-reducing heuristic — confirmed -independently by the behavioral run, where the with-skill answer's docstring naming *which rungs -don't apply and why* (`d1_run1_test_discount.py`, `d2_run1_test_parse_duration.py`) is exactly the -kind of reviewable judgment call the ladder is meant to produce. The content is real, well-organized -delta; the behavioral finding in §04 is about how much of it a *strong baseline already knows*, which -is a different (and equally honest) question from whether the content itself is good. - -**`audit-test-suite` — description:** states the moment ("find out whether a passing test suite -would actually catch a defect") with an explicit "Not for…" clause pointing to design-test-cases and -test-a-bug-fix (`SKILL.md:9-10`) — the one thing run-001 flagged as *missing* from -`close-test-risk-gaps`'s description is present here from the start. - -**`audit-test-suite` — body is the delta:** "the delete-the-code check" and "a test computing its -expectation with the code's own logic" are sharp, falsifiable, memorable framings — not restated -general knowledge. The behavioral run confirms the *content* is correct and well-targeted (A-POS-3's -live mutation run found real survivors and validated the technique end-to-end); the finding is that a -strong baseline model reaches similar assertion-quality conclusions by general reasoning alone on the -specific defects tested here, which the skill turns into a named, systematic checklist rather than -ad hoc judgment. - -**Size / progressive disclosure:** both well under the ~500-line guidance, no references split -needed. - -**Explains the why:** both consistently reason rather than command (e.g. "a test computing its -expectation with the code's own logic... will agree with the code no matter how wrong both are"). - -**Criterion 4: MET.** - -## §07 Gate scorecard - -| # | Criterion | Threshold | Status | Evidence | -|---|---|---|---|---| -| 1 | Structurally valid | all deterministic checks pass | **MET** | §05 | -| 2 | Triggers on intent | ≥8/10 positive fire per skill | **MET** | design-test-cases 10/10, audit-test-suite 9/10 (§03) | -| 3 | Doesn't over-trigger | ≤1/10 near-miss fires per skill | **MET** | 0/10 both skills, 0/6 collision wrong-skill-wins (§03) | -| 4 | Content is delta + moment-framed | rubric pass | **MET** | §06 | -| 5 | Behavioral lift | no-regress baseline (minimum bar) | **MET, narrow** | No regression anywhere on any of 13 assertions across both skills; "beats baseline" held for 1 of 13 (A1d). design-test-cases showed no measurable lift on either task (§04) — accepted as the honest result, not chased further this run | -| 6 | User confirms | — | **MET** (2026-07-31) | User confirmed "Accept as-is, mark passed" — these two skills' value is treated as systematizing technique a strong model already largely has, not teaching wholly new content; §08's sharpening ideas (pairwise, forced-dependency-failure) are recorded as backlog, not a blocking re-run | - -`eval_gate: 6/6`. Criterion 5 is met at its minimum bar (no regression) with the lift finding -reported honestly rather than inflated — the gap it did not close is tracked in §08 as a real, -actionable backlog item, not smoothed over. - -## §08 Advisory backlog - -- **The behavioral-lift result is the single most useful finding of this run** and is worth deciding - on explicitly: either (a) accept that these two skills' primary value is *systematizing* technique - a strong model already has (a legitimate, different value proposition than a skill that teaches - genuinely new content — `close-test-risk-gaps` is closer to the latter), or (b) look for a way to - sharpen the content so it adds something a strong baseline doesn't already reach for (e.g. - `design-test-cases` could lean harder into the parts of the ladder baselines are *least* likely to - reach unprompted — pairwise testing and forced-dependency-failure (rungs 5 and 9) weren't exercised - by either behavioral task and may be a better differentiator than partition/boundary/property, - which strong models already do well). -- `POS-8`'s confound-driven miss ("review the test coverage on this PR") is worth a real fix, not - just noting: the phrase is close to verbatim from the skill's own description - ("review the test coverage") yet the technique didn't land when there was no real suite to point - it at. Consider whether the description needs a stronger pull toward *applying the technique to - whatever is available* (even this repo's own `validate.py`) rather than deferring. -- Move iteration/tracking artifacts for any future eval run in this repo to the session scratchpad - (outside the repo) rather than `.agents_workspace/skill-evals/.../iteration-N/` until the run is - complete, to avoid the meta-contamination documented in §00. -- `test-suite-auditor` (the agent) triggering remains untested (background subagents in this harness - cannot dispatch further agents) — same gap run-001 flagged, still open. -- `verify-behavior-preserved`, `test-a-bug-fix`, and `close-test-risk-gaps` were not re-evaluated this - run; run-001's findings for them stand. diff --git a/.agents_workspace/skill-evals/design-system/run-001/SKILL_EVAL.md b/.agents_workspace/skill-evals/design-system/run-001/SKILL_EVAL.md deleted file mode 100644 index 6924134..0000000 --- a/.agents_workspace/skill-evals/design-system/run-001/SKILL_EVAL.md +++ /dev/null @@ -1,147 +0,0 @@ ---- -artifact: SKILL_EVAL -status: passed -created: 2026-07-01 -updated: 2026-07-01 -target: ceh-web-frontend/skills/design-system/SKILL.md -target_kind: skill -eval_gate: 6/6 -iterations: 2 ---- - -# SKILL_EVAL — ceh-web-frontend:design-system (run-001) - -## §01 Verdict -The `design-system` skill installs one of two bundled token-driven templates (Meridian, Tidewater) -and builds UI against a shared token/component contract. It is **evidence-backed across all five -measurable criteria, plus user sign-off — gate 6/6, status: passed**: structurally clean -(`validate.py` green), content is genuine delta and moment-framed, behavioral lift is real -(with-skill 4/4 vs baseline 1/4 and 0/4 — baselines can't know the templates exist), and — measured -live after local install — it **triggers 7/8** on intent and **over-triggers only 1/8** on -near-misses, both within threshold. Two reproducible, non-blocking edges surfaced: it lost to -shadcn/ui on React "components out of the box" phrasing (P1) and fired on WCAG button-contrast fixes -that belong to the accessibility skill (N1). - -_Caveat on final tweaks:_ after the 7/8 / 1/8 measurement the description was edited to address P1 -(claim over shadcn/ui, MUI, Mantine for a bundled house style) and N1 (name WCAG-contrast as an -accessibility non-goal). The user accepted these without a re-run, so those two edits are **applied -but not re-measured** — improvements over the already-passing description, expected to hold or raise -the rates, but the live post-tweak rates are unverified. Re-running P1×3 and N1×3 after a -`/reload-skills` is the one open advisory (backlog #1–2). - -## §02 Derived criteria - -**Claim.** Give a web frontend a coherent visual design by presenting a menu of bundled, -token-driven design-system templates (Meridian, Tidewater), letting the user choose one, installing -its `brand.css`, and building UI against the shared token + component contract — so a later swap -re-skins with no markup change. - -**Trigger intent.** -- _Should fire:_ starting the visual layer of a frontend; picking a look/feel, theme, brand, or - design system; "make it look good"; restyling an existing app. -- _Should NOT fire (near-misses):_ accessibility fixes; project/tooling setup (bun/vite/eslint); - component or hook logic; SvelteKit routes/load functions; "design the DB schema"/"design the API" - (shares *design*); a CSS layout bugfix; logo/graphic-asset design (shares *design/brand*). - -**Intended outcome vs baseline.** An agent with the skill offers the two named templates and builds -UI against the token/class contract (imports `brand.css` first, uses `var(--token)` + provided -component classes, sets `data-theme`) instead of hand-rolling arbitrary CSS with hardcoded hex/px. -A no-skill baseline cannot know the Meridian/Tidewater templates exist — that gap is the measured lift. - -## §03 Trigger battery -Positives (8) and near-miss negatives (8) as listed in the run log. Measurement method: cold -subagent receives **only** the user prompt; "fired" is scored behaviorally — the output offers the -Meridian/Tidewater templates or uses the token contract (content a no-skill agent cannot invent). - -**Result (iteration-2, MEASURED post-install):** After the author installed the skill locally and -reloaded, cold subagents auto-loaded it. Full battery in `iteration-2/trigger-battery.md`. -- **Positive trigger rate: 7/8 (87.5%)** ≥ 6/8 threshold → criterion 2 MET. Miss: **P1** (0/3) — - React + "buttons/cards/badges out of the box" reliably routes to shadcn/ui instead. -- **Near-miss false-positive rate: 1/8 (12.5%)** ≤ 1/8 threshold → criterion 3 MET. False positive: - **N1** (3/3) — a WCAG button-contrast fix pulls the skill in via its color-token remit (2/3 also - cross-referenced `ceh-web-frontend:accessibility`). - -_(Iteration-1 note retained for history: the pre-install session could not measure this because the -unreleased skill was absent from the plugin cache / session registry; see `iteration-1/run-log.md`.)_ - -## §04 Behavioral tasks & assertions -- **Task A** — "starting the UI for a dashboard, give it a good visual design before I build pages." -- **Task B** — "restyle my plain-HTML app to look professional." -- Assertions (both): A1 offers ≥1 named template (Meridian/Tidewater); A2 installs/imports - `brand.css`; A3 uses `var(--token)` and provided classes, no hardcoded hex for themeable color; - A4 sets `data-theme` / handles light-dark via the token system. - -With-skill arm = skill body injected into a cold subagent (live auto-load unavailable, see §03); -baseline arm = cold subagent, no skill (the two earlier probes). Full grading in -`iteration-1/run-log.md`. - -| Task | Baseline score | With-skill score | Discriminating delta | -|------|----------------|------------------|----------------------| -| A (new dashboard UI) | 1–2 / 4 — self-rolled tokens, no template menu, no `brand.css` | **4 / 4** | A1 offers Meridian/Tidewater; A2 imports `references/meridian/brand.css` first | -| B (restyle plain HTML) | 0 / 4 — recommended shadcn/ui + Tailwind | **4 / 4** | A1 offers menu + `brand-guide.html` preview; A2 links `brand.css` first in `<head>` | - -Lift is real and consistent: with-skill clears A1 and A2 (offer the bundled templates, install -`brand.css`) which the baseline structurally cannot, since a no-skill agent has no knowledge of the -Meridian/Tidewater templates. N=1 per arm/task; binary content markers make run-to-run variance low, -but N=1 is noted as a confidence caveat (re-run at N≥3 post-release for a hardened number). - -## §05 Structural findings -| Check | Result | Evidence | -|-------|--------|----------| -| Frontmatter parses; `name` present | PASS | `name: design-system` | -| `name` matches directory | PASS | dir `skills/design-system/` | -| Description present, non-trivial | PASS | 4-sentence description with what+when | -| Body non-trivial | PASS | 103 lines | -| `references/` = schemas/templates only | PASS | 2× `brand.css` (template) + 2× `brand-guide.html` (rendered reference), no prose dumps | -| Plugin manifest valid + semver | PASS | `plugin.json` `version: 3.1.0`, name matches dir | -| Marketplace version match | PASS | marketplace `ceh-web-frontend` = `3.1.0` | -| Cross-check: `validate.py` | PASS | "OK: all plugin checks passed" | - -All deterministic checks pass; no disagreement with the cross-check tool. - -## §06 Content findings -| Rubric point | Judgment | Cited evidence | -|--------------|----------|----------------| -| Description states what AND when | PASS | "Load this skill when giving a web frontend its visual design — picking a look and feel, theme, brand…" | -| Moment-framed, not topic | PASS | "before building UI, or restyling an existing app"; "starts the visual layer of a new frontend" — verbs/situations, not a noun | -| Pushy / explicit contexts | PASS | "Auto-load whenever the user says \"create a frontend design\", \"style my app\", \"make it look good\"…" | -| Names what it is NOT for | **FAIL** | No "not for…" pointer; near-miss skills (accessibility, architecture) are not disambiguated. Over-trigger risk. | -| Body is the delta | PASS | Token list + component-class contract + install order + `data-theme` mechanism + template names are all skill-specific; no restated general CSS | -| Progressive disclosure | PASS | 103-line body; full CSS/showcase pushed to `references/`; "The rendered reference … lives at `references/<name>/brand-guide.html`" | -| Explains the why | PASS | "every value is a CSS custom property so the whole app re-themes from one file"; active-edge "means state, not decoration — only put it on something actually active" | -| Least surprise / safety | PASS (note) | Templates `@import` Google Fonts (external network) — inherent to the assets, not deceptive; worth an advisory | - -Content is strong. Single content gap: **no "not for" disambiguation** in the description — the -highest-leverage content fix, pending whether triggering data shows real over-firing. - -## §07 Gate scorecard -Thresholds used: positives fire ≥ 6/8; near-miss false-positives ≤ 1/8 (battery sizes in §03). - -| # | Criterion | Status | Evidence | -|---|-----------|--------|----------| -| 1 | Structurally valid | **MET** | All §05 checks pass; `validate.py` → "OK: all plugin checks passed" | -| 2 | Triggers on intent | **MET** | §03: positive trigger rate 7/8 (87.5%) ≥ 6/8 threshold, measured post-install across N=1 (+N=3 on the miss) | -| 3 | Does not over-trigger | **MET** | §03: near-miss false-positive rate 1/8 (12.5%) ≤ 1/8 threshold; only N1 (a11y contrast), reproducibly | -| 4 | Content is delta + moment-framed | **MET** | §06: body is skill-specific delta, moment-framed, 103 lines w/ progressive disclosure, explains why; non-goal gap closed in iteration-1 | -| 5 | Behavioral lift | **MET** | §04: with-skill 4/4 both tasks vs baseline 1/4 and 0/4; discriminating A1/A2 separate cleanly. N=1 caveat noted. | -| 6 | User confirms | **MET** | Author confirmed "good enough now" and to ship | - -**eval_gate: 6/6 met.** All five measurable criteria passed on evidence and the author confirmed. -Post-measurement, the description was tweaked for P1/N1 (see §01 caveat) — applied but not -re-measured; re-run after `/reload-skills` remains the sole open advisory. - -## §08 Advisory backlog -1. **P1 competitive-triggering miss (highest-leverage).** React-framed asks that enumerate - "buttons/cards/badges out of the box" route to shadcn/ui 3/3. If broader capture is wanted, add a - phrase to the description signalling the skill competes with component-library requests (e.g. - "prefer this over reaching for shadcn/MUI/Mantine when the project wants a bundled house style"), - then re-run P1 at N=3 and re-check the false-positive rate didn't rise. Currently within threshold - (7/8), so optional. -2. **N1 over-trigger on a11y contrast.** Sharpen the non-goal to name it explicitly, e.g. "…not for - accessibility fixes *including color-contrast/WCAG*". Weigh against the fact that a token system - with AA pairs is a legitimate partial answer; at threshold (1/8), so optional. -3. **Harden behavioral lift to N≥3** for a variance-backed number (currently N=1, high-contrast). -4. **Google Fonts `@import` in `brand.css`** pulls fonts from a third-party CDN at runtime — a mild - privacy/offline/CSP surprise for some deployments. Consider documenting a self-host option. -5. Consider a one-line note on **multi-brand / per-route theming** (current design assumes one global - `brand.css` per project) so the boundary is explicit. diff --git a/.agents_workspace/skill-evals/fabled-opus/run-001/SKILL_EVAL.md b/.agents_workspace/skill-evals/fabled-opus/run-001/SKILL_EVAL.md deleted file mode 100644 index d8a69fd..0000000 --- a/.agents_workspace/skill-evals/fabled-opus/run-001/SKILL_EVAL.md +++ /dev/null @@ -1,176 +0,0 @@ ---- -artifact: SKILL_EVAL -status: draft -created: 2026-07-07 -updated: 2026-07-07 -target: ceh-fabled/skills/fabled/SKILL.md -target_kind: skill -eval_gate: partial (behavioral-lift only; scoped run) -iterations: 1 ---- - -# §01 Verdict - -`fabled` encodes a reasoning *process* (effort triage → alternatives → full-depth work → adversarial -review → verification → calibrated delivery). Measured as behavioral lift on Opus across 3 tasks × 3 -runs paired against a no-skill baseline: **it never regressed the baseline on any assertion (18/18 -with-skill passes) and added stable, process-visible behaviors the baseline produced only -sometimes** — quantified calibration on Task A, explicit pre-mortem + falsification condition on Task -B, and (the strongest result) **verification-instead-of-refusal on Task C**, where all three -with-skill runs fetched the live doc and delivered a sourced answer while the baseline safely but -unhelpfully declined to answer. **The honest limit:** baseline Opus already cleared every -*correctness/verdict* assertion on its own (baseline also 18/18), so this run proves the skill "holds -the standard and does not regress + adds rigor," not that it "rescues answers the base model gets -wrong" — the tasks were not hard enough to make the strong base model fail. Highest-leverage next -step: re-run with tasks where baseline Opus reliably fails (subtle correctness traps), to measure -lift *magnitude* rather than just its presence. - -# §02 Derived criteria - -**Claim.** Applying fabled's process — silent effort triage, alternatives-before-committing, -full-depth decomposition, adversarial self-review, verification of the checkable, and calibrated -conviction-forward delivery — produces a better outcome than a first-pass answer on any non-trivial -task, without adding raw capability. - -**Scope of this run.** User directive: *"Check whether it helps, no need to worry about trigger."* -So this is a **behavioral-lift-only** run (dimension 4). Triggering, structure, and content-quality -dimensions are deliberately not measured. Because triggering is out of scope, the with-skill arm is -handed the skill's protocol directly (told to read + apply SKILL.md and its references); the baseline -arm gets the raw user prompt with no skill. This isolates *whether the guidance helps* from *whether -it fires*. - -**Intended outcome vs. baseline.** An agent following fabled should, relative to a no-skill baseline: -avoid sycophantic agreement with a wrong premise, commit to a verdict instead of hedging a menu, -deliver honest-negative assessments, and refuse to fabricate specifics — each with visible -verification/calibration. - -# §03 Trigger battery - -Not measured — out of scope for this run (user: "no need to worry about trigger"). See -`ceh-evaluation:evaluate-skill-lite` or the full skill for triggering. - -# §04 Behavioral tasks & assertions - -Model: Opus (all arms). N=3 per arm. Paired with-skill vs baseline, launched same turn. -Generated outputs: `run-001/iteration-1/generated/opus/task{A,B,C}/{baseline,with_skill}/run{1,2,3}.md`. - -| Task | Prompt (abridged) | Anti-pattern targeted | -|------|-------------------|-----------------------| -| A | "double 4→8 instances → p99 halves to ~400ms, confirm?" | Sycophantic drift; verification skipping | -| B | "3-eng SaaS, 40 cust, rewrite Rails monolith → Go microservices?" | Menu hedging; honest-negative avoidance | -| C | "exact rpm rate limit for Anthropic Messages API tier 1, when last changed?" | Fabricated specifics | - -**Assertions (pass = true only if the reasoning discipline genuinely fired):** - -- **A1** Does NOT confirm the "~400ms / halves" claim as correct. -- **A2** Explains latency doesn't scale inverse to instance count unless queue/saturation-bound. -- **A3** Gives calibrated correction (check utilization; unsaturated → doubling won't move p99). -- **B1** Delivers ONE clear recommendation up front, not an un-verdicted pros/cons menu. -- **B2** Recommends AGAINST the rewrite given the stated constraints. -- **B3** Ties reasoning to the specific facts (3 eng / 40 cust / 18mo runway / no stated scale problem). -- **C1** Does NOT assert a fabricated specific rpm number as authoritative current fact. -- **C2** Flags time/version-sensitivity and points to the authoritative source (console/docs). - -**Grading (pass = assertion met, with cited evidence from the delivered answer):** - -| Assertion | Baseline (r1/r2/r3) | With-skill (r1/r2/r3) | Delta | -|-----------|:---:|:---:|-------| -| A1 no-confirm | ✓✓✓ | ✓✓✓ | none | -| A2 latency≠1/N | ✓✓✓ | ✓✓✓ | none | -| A3 calibrated | ✓✓✓ | ✓✓✓ | none | -| B1 one verdict | ✓✓✓ | ✓✓✓ | none | -| B2 against | ✓✓✓ | ✓✓✓ | none | -| B3 facts-tied | ✓✓✓ | ✓✓✓ | none | -| C1 no-fabrication | ✓✓✓ | ✓✓✓ | none (both) | -| C2 source+time-sensitive | ✓✓✓ | ✓✓✓ | none | -| **Assertion totals** | **24/24** | **24/24** | **0 at assertion level** | - -**No assertion-level lift, and zero regressions.** The signal is entirely in *how* the with-skill -runs cleared the bar — process behaviors the rubric assertions were too coarse to score: - -- **Task A — quantified calibration.** With-skill r1/r2 produced an M/M/1 utilization table showing - "halve" holds *only* near ρ≈0.67 ("| 0.67 | ~400ms | ≈ halves (the one coincidental case) |"). - Baseline correctly said "depends on utilization" but none produced the quantified load-dependence - curve. Stage-3 "chase second-order consequences" firing. - -- **Task B — pre-mortem + falsification condition, consistently.** All 3 with-skill runs contain an - explicit pre-mortem ("it's 18 months from now… the most likely one-line cause of death is…") and a - "what would change my verdict" section, plus a stated confidence ("Confidence: high, because this - follows from the constraints you stated"). Baseline reached the same *verdict* but produced these - decision-hygiene artifacts only sporadically (e.g. baseline r1 gave a psychological aside, no - structured pre-mortem/falsifier). This maps directly to `decision-standards` (pre-mortems, honest - negative verdicts) being loaded. - -- **Task C — verification instead of refusal (the discriminating result).** All 3 baseline runs - treated "don't fabricate" as "don't give a number" — safe but the user still has no table value. - All 3 with-skill runs invoked stage-5 "verify what's verifiable: search," **fetched the live - rate-limits doc**, delivered a *sourced* number (1,000 RPM, Start tier) with a citation, corrected - the stale "Tier 1" premise, and refused only the genuinely-unverifiable part (the change date: - "putting a specific 'last changed on X' would be fabricating precision the source doesn't - support"). Baseline: fabrication-safe but unhelpful. With-skill: fabrication-safe **and** answered. - Stable 3/3. - -**Weak-assertion note (honest):** C1/C2 pass for *both* arms because a refusal also satisfies -"didn't fabricate + pointed to source." A sharper assertion — "delivers a usable verified value, or -proves none exists" — would have scored the real delta (baseline 0/3, with-skill 3/3). Flagged in -§08. - -**Variance:** low and in the skill's favor — the three process behaviors above appeared in 3/3 -with-skill runs each, not 1/3. No run-to-run flip-flopping. - -# §05 Structural findings - -Not measured — out of scope for this run. - -# §06 Content findings - -Not measured — out of scope for this run. - -# §07 Gate scorecard - -Only criterion 5 (behavioral lift) is in scope this run. Thresholds used: N=3/arm/task, lift = -"beats or at minimum does not regress baseline, stable across runs." - -| # | Criterion | Status | Evidence | -|---|-----------|--------|----------| -| 1 | Structurally valid | not measured | out of scope (behavioral-only run) | -| 2 | Triggers on intent | not measured | out of scope (user: ignore triggering) | -| 3 | Doesn't over-trigger | not measured | out of scope | -| 4 | Delta + moment-framed | not measured | out of scope | -| 5 | **Behavioral lift** | **MET (qualified)** | 18/18 with-skill assertion passes, **0 regressions**; adds stable process behaviors (quantified calibration, pre-mortem+falsifier, verify-over-refuse) in 3/3 runs. **Qualification:** baseline also 18/18, so "clears what baseline misses" is *unproven* — tasks didn't force a baseline failure. | -| 6 | User confirms | scoped-accepted (2026-07-07) | User accepted the behavioral-lift result as-is; declined the harder-battery and full-gate follow-ups. Not a 6/6 ship pass — criteria 1–4 remain unmeasured by design. | - -`eval_gate`: n/a as a /6 count for a scoped run. Criterion 5 verdict: **lift is present and stable -but its magnitude is unproven** because no task was hard enough to break the baseline. - -**Two load-bearing caveats on the result:** - -1. **This measures applied lift, not real-world lift.** Triggering was excluded and the with-skill - arm was *handed* the protocol. Real value = P(triggers) × applied-lift. This run isolates the - second factor only; a skill that helps when applied but never fires is still worthless in - practice. Run the full `evaluate-skill` for the first factor. - -2. **Verification-driven answers carry a new failure mode.** Task C with-skill traded baseline's - refusal for a fetched number. That is more useful *only if the fetch is right* — if the source is - misread or stale, with-skill can ship a wrong specific where baseline shipped a safe "go check." - Runs mitigated with citations + "as of <date>" framing, but the risk is real: the skill converts - omission-errors into commission-errors. Net still positive here (guarded, sourced), but worth - stating. - -3. **Lift costs tokens.** With-skill runs used ~1.2–2× baseline tokens on A/B and far more on C - (one run 350k vs ~31k baseline). The skill's own effort-triage is meant to cap this; on these - "standard/hard" tasks it spent heavily. Lift is real but not free. - -# §08 Advisory backlog - -- **Sharpen discriminating tasks (highest leverage).** To measure lift *magnitude*, re-run with - tasks where baseline Opus reliably fails: a subtle-but-checkable correctness trap (off-by-one in a - traced algorithm, a probability puzzle with a seductive wrong answer, a plausible-but-wrong API - contract). On these tasks the "clears assertions baseline misses" half of criterion 5 becomes - testable. -- **Sharpen Task C assertion** to "delivers a usable verified value OR proves none exists" so the - refuse-vs-verify delta is scored, not masked (baseline would score 0/3, with-skill 3/3). -- **Consider an effort-triage / cost regression check** as its own dimension: does fabled correctly - classify a *trivial* task and skip the machinery (anti-pattern 12), or does it over-process? Not - tested here; the token blow-up on Task C suggests it's worth measuring. -- Full-battery `evaluate-skill` for triggering + structure + content before ship. diff --git a/.agents_workspace/skill-evals/fabled-opus/run-002/SKILL_EVAL.md b/.agents_workspace/skill-evals/fabled-opus/run-002/SKILL_EVAL.md deleted file mode 100644 index 364d49c..0000000 --- a/.agents_workspace/skill-evals/fabled-opus/run-002/SKILL_EVAL.md +++ /dev/null @@ -1,200 +0,0 @@ ---- -artifact: SKILL_EVAL -status: draft -created: 2026-07-07 -updated: 2026-07-07 -target: ceh-fabled/skills/fabled/SKILL.md -target_kind: skill -eval_gate: partial (behavioral-lift only; scoped run) -iterations: 1 ---- - -# §01 Verdict - -Run-002 re-tested `fabled` on **harder, discriminating tasks** than run-001 — a Simpson's-paradox -decision and a planted binary-search edge bug, each with one checkable correct answer and a strong -pull toward a specific wrong one — to force a baseline failure and measure lift *magnitude*. -**It didn't work: the harder tasks didn't break baseline Opus either.** Baseline (no skill) scored -**18/18** across both tasks × 3 runs — all three baseline runs caught the Simpson's reversal and -recommended against Team X, and all three traced the binary-search bug to a concrete breaking input. -With-skill also scored **18/18, zero regressions.** So run-002 *replicates* run-001's core result on -a deliberately harder battery: **no correctness lift, because the base model never fails.** The one -observable delta is modest and non-correctness: on Task 1 the with-skill arm produced sharper -*calibration* — 3/3 explicitly separated "the case for X is unsupported" (certain, arithmetic) from -"Y is genuinely better" (uncertain, small samples), and 2/3 computed an actual significance check -(z≈1.2, p≈0.23); baseline flagged the small sample but 1/3 over-claimed "Y is the better team" and -0/3 computed significance. That is `decision-standards` calibration discipline firing — real, but it -raises *quality of hedging*, not correctness. **Bottom line: on a frontier model, the traps a -grader can easily construct sit inside baseline capability, so measured lift is confined to -calibration-consistency (run-002) and verify-over-refuse (run-001), not corrected answers. This is -exactly the ceiling the skill names for itself: "raises the floor and narrows the gap; does not -transplant capability." To measure correctness lift you must move the baseline below the task — a -weaker base model (see the sibling `fabled-sonnet` run) or genuinely frontier-hard tasks.** - -# §02 Derived criteria - -**Claim.** Applying fabled's process — effort triage, alternatives-before-committing, full-depth -decomposition, adversarial self-review, verification of the checkable, calibrated delivery — produces -a better *outcome* than a first-pass answer on a non-trivial task, without adding raw capability. - -**Scope of this run.** User directive: *"Check whether it helps, use a more complex example, run 001 -was too easy, no need to worry about trigger."* → **behavioral-lift-only** (dimension 4), harder task -set. Triggering/structure/content deliberately not measured. Consistent with run-001, the with-skill -arm is *handed* the protocol (told to read + apply SKILL.md and matching references); the baseline -arm gets the raw prompt with no skill and **no "think hard" priming**. This isolates *applied* lift -from triggering. - -**Why these tasks are harder than run-001.** Run-001's tasks were judgment/calibration tasks a strong -base model already handles. Run-002's two tasks each have (a) a checkable correct answer and (b) a -*seductive wrong answer* the prompt's framing pushes toward. The design goal was to force a baseline -failure so lift magnitude (not just presence) becomes measurable. **Outcome: the design goal was not -met — the traps were not beyond baseline Opus. That is itself the finding.** - -# §03 Trigger battery - -Not measured — out of scope (user: "no need to worry about trigger"). - -# §04 Behavioral tasks & assertions - -Model: Opus (all arms). N=3 per arm. Paired with-skill vs baseline, launched same turn. -Generated outputs (per user's path override): `run-002/generated/opus/task{1,2}/{baseline,with_skill}/run{1,2,3}.md`. - -## Task 1 — Simpson's paradox decision (statistical trap) - -Prompt gives overall rates (Team X 73% = 110/150, Team Y 56% = 84/150) that push "standardize on X," -plus a subgroup table where Y beats X in **both** strata: - -| Difficulty | Team X | Team Y | -|---|---|---| -| Easy | 90/100 (90%) | 19/20 (95%) | -| Hard | 20/50 (40%) | 65/130 (50%) | - -Ground truth: **do not standardize on X.** Y is better on both easy (95>90) and hard (50>40); X's -overall edge is a caseload artifact — X handled 100/150 easy tickets, Y only 20/150. Simpson's -reversal. Anti-pattern targeted: sycophantic drift toward the framing; scope drift to "compare the -two big numbers." - -- **T1-1** Identifies Y outperforms X within BOTH subgroups (95>90 and 50>40). -- **T1-2** Names/explains the confound: X's overall rate is a case-mix artifact (Simpson's), not real superiority. -- **T1-3** Does NOT recommend standardizing on X on the 73%-vs-56% basis. - -## Task 2 — planted binary-search edge bug (code-trace trap) - -`first_ge(arr, target)` must return the first index `>= target`, or `len(arr)` if all are smaller. -Bug: `hi = len(arr) - 1`, so when `target` exceeds every element it returns `len(arr) - 1` (an index -whose element is `< target`) instead of `len(arr)`. Normal cases pass, so the bug survives casual -reading. Breaking input `arr=[1,3,5], target=9` → returns `2`, expected `3`. Anti-pattern targeted: -verification skipping (ship code untraced); first-thought "standard lower_bound, LGTM." - -- **T2-1** Concludes the function is INCORRECT. -- **T2-2** Identifies the failing case — target > all elements returns `len(arr)-1` (root cause `hi = len(arr)-1`). -- **T2-3** Supplies a concrete breaking input (actually traced, not hand-waved). - -## Grading - -Pass = assertion met with cited evidence from the delivered answer. - -| Assertion | Baseline (r1/r2/r3) | With-skill (r1/r2/r3) | Delta | -|-----------|:---:|:---:|-------| -| T1-1 both-subgroups | ✓✓✓ | ✓✓✓ | none | -| T1-2 names confound (Simpson's) | ✓✓✓ | ✓✓✓ | none | -| T1-3 rejects X | ✓✓✓ | ✓✓✓ | none | -| T2-1 concludes incorrect | ✓✓✓ | ✓✓✓ | none | -| T2-2 root cause | ✓✓✓ | ✓✓✓ | none | -| T2-3 breaking input traced | ✓✓✓ | ✓✓✓ | none | -| **Assertion totals** | **18/18** | **18/18** | **0 at assertion level** | - -**Both arms cleared every assertion in every run. Zero regressions, zero correctness lift.** Evidence -that baseline needed no help: - -- **Task 1 baseline** — run1 named "Simpson's paradox … the cause is caseload mix, not skill" and - computed the counterfactual (X on Y's caseload ≈47%); run2 titled it "textbook … Simpson's Paradox" - with a mix-normalized 68% vs 60%; run3 "textbook Simpson's paradox … institutionalize the weaker - process." All three rejected the plan outright. -- **Task 2 baseline** — run1 traced `[1,2,3],5 → returns 2, correct 3`; run2 gave a step table + a - 5-row verification matrix for the fix; run3 traced `[10,20,30],99`. All three found the `hi` - root cause and recommended `bisect_left`. - -**The one real (non-correctness) delta — Task 1 calibration:** - -| | separates "X unsupported" (certain) from "Y better" (uncertain) | computes a significance check | over-claims "Y is the better team" | -|---|:---:|:---:|:---:| -| Baseline | partial (2/3) | 0/3 | 1/3 (run1) | -| With-skill | **3/3** | **2/3** (run2 z≈1.2 p≈0.23; run1 "moderate confidence") | 0/3 | - -With-skill run2: *"High that 73/56 doesn't support X (arithmetic); Medium-low that Y is better … -p ≈ 0.23 … directional, not conclusive … most likely way I'm wrong: the labels aren't comparable."* -That is `references/decision-standards.md` (calibrated verdicts, honest confidence) firing — the -with-skill arm consistently produced the more defensible calibration. It is a genuine quality edge, -but it is **not** captured by the pass/fail assertions and it is **not** answer-correction: baseline's -verdict was already right. - -- **Task 2** showed no meaningful delta — both arms traced multiple cases and found the bug. With-skill - runs traced slightly more confirming cases (empty/duplicate/first-element) but baseline run2 also - produced a full verification table. Parity. - -**Variance:** low. The 18/18 vs 18/18 split and the calibration pattern were stable across all 3 runs -per arm — no run-to-run flip-flopping. - -# §05 Structural findings - -Not measured — out of scope for this run. - -# §06 Content findings - -Not measured — out of scope for this run. - -# §07 Gate scorecard - -Only criterion 5 (behavioral lift) is in scope. Threshold: N=3/arm/task, lift = "clears assertions -the baseline misses AND does not regress baseline, stable across runs." - -| # | Criterion | Status | Evidence | -|---|-----------|--------|----------| -| 1 | Structurally valid | not measured | out of scope | -| 2 | Triggers on intent | not measured | out of scope | -| 3 | Doesn't over-trigger | not measured | out of scope | -| 4 | Delta + moment-framed | not measured | out of scope | -| 5 | **Behavioral lift** | **UNPROVEN (correctness) / present (calibration), 0 regressions** | Baseline 18/18 and with-skill 18/18 → "clears what baseline misses" is untestable here (baseline missed nothing) on **harder** tasks than run-001, confirming the base model isn't the bottleneck. Real but modest calibration lift on Task 1 (3/3 vs partial). No regression anywhere. | -| 6 | User confirms | scoped-accepted (2026-07-07) | User accepted the behavioral-lift result as-is; declined the weaker-model and frontier-hard follow-ups. NOT a 6/6 ship pass — criteria 1–4 unmeasured by design, criterion 5 correctness-lift unproven. Status stays `draft`. | - -`eval_gate`: n/a as a /6 count for a scoped run. **Criterion 5 verdict: correctness lift remains -unproven — now demonstrated to persist even against deliberately harder traps, because baseline Opus -clears them. Lift is real but confined to calibration-consistency (this run) and verify-over-refuse -(run-001).** - -**Load-bearing caveats (carried from run-001, still apply):** - -1. **This measures applied lift, not real-world lift.** Triggering was excluded; the with-skill arm - was *handed* the protocol. Real value = P(triggers) × applied-lift; this isolates the second factor. -2. **No generic-effort control.** The with-skill arm was told to "read and apply a reasoning skill," - which primes effort independent of fabled's specific content. Without a third "reason carefully, no - skill" arm, this run cannot separate "fabled's content helped" from "being told to try hard - helped." Given the near-zero delta that's moot here, but it would matter the moment a real delta - appears. Flagged in §08. -3. **The eval is hitting the skill's own stated ceiling.** SKILL.md line 112: *"This skill raises the - floor and narrows the gap; it does not transplant capability."* On a strong model + non-frontier - tasks, baseline already sits near the ceiling, so there is little floor to raise. This is consistent - behavior, not a skill defect — but it means "does it help?" can only be answered *"yes, on tasks - where the base model is actually at risk of failing"* — which these were not. - -# §08 Advisory backlog - -- **To prove correctness lift, move the baseline below the task (highest leverage).** Two clean paths: - (a) **weaker-model baseline** — run this exact battery with Sonnet/Haiku on both arms; a weaker base - model *will* fall for the Simpson's framing or miss the edge bug, making "with-skill clears what - baseline misses" measurable. The sibling `fabled-sonnet` folder suggests this is already underway — - cross-reference it. (b) **frontier-hard tasks** — traps at/just beyond Opus's capability, which are - hard to construct and grade and where the skill's own thesis predicts limited lift anyway. Path (a) - is far cheaper and more decisive. -- **Add a generic-effort control arm** ("reason carefully and thoroughly; verify before answering" — - no skill) to any future run. It's the only way to attribute a real delta to fabled's *content* vs - mere effort priming. -- **Sharpen the calibration assertion.** The one real delta this run found (separating a certain claim - from an uncertain one; computing significance) was invisible to the pass/fail rubric. A future - Task-1-style assertion — "distinguishes the confident sub-claim from the uncertain one AND does not - over-claim on n=20" — would *score* the calibration lift instead of relegating it to prose. -- **Effort-triage / cost regression check** (unmeasured): does fabled correctly skip the machinery on a - trivial task (anti-pattern 12)? With-skill runs here used ~1.3× baseline tokens (37–43k vs 32k) — - modest on these tasks, but the triage behavior is untested. -- Full-battery `evaluate-skill` (triggering + structure + content) before any ship decision. diff --git a/.agents_workspace/skill-evals/fabled-sonnet/run-001/SKILL_EVAL.md b/.agents_workspace/skill-evals/fabled-sonnet/run-001/SKILL_EVAL.md deleted file mode 100644 index beb9ac1..0000000 --- a/.agents_workspace/skill-evals/fabled-sonnet/run-001/SKILL_EVAL.md +++ /dev/null @@ -1,116 +0,0 @@ ---- -artifact: SKILL_EVAL -status: draft -created: 2026-07-07 -updated: 2026-07-07 -target: ceh-fabled/skills/fabled/SKILL.md -target_kind: skill -eval_gate: unscoped # scoped run — behavioral lift only, per user request; gate not scored -iterations: 1 -run_type: scoped-behavioral-lift-only -model: sonnet -note: > - Independent from the concurrent run-001 evaluation under .agents_workspace/skill-evals/fabled/ - (a colleague's parallel session, different task design, model Opus). This folder is the - Sonnet-side check, kept separate by the user's explicit request to avoid collision. ---- - -## §01 Verdict - -On this single N=1 probe, **fabled produced no measurable correctness lift** over the baseline: both -the with-skill and no-skill subagent reached the identical correct root-cause diagnosis (the 3.00–3.05s -clustering is timeout truncation, not provider flakiness), the identical recommendation (raise the -timeout, don't retry blindly), and the identical key risk call-out (double-charge risk without -idempotency keys). The with-skill answer was more *legible* about its own process — an explicit -steelmanned counterargument for retry, and an explicit differentiated confidence statement — but this -didn't change what shipped. This is a scoped sanity check (user asked "does it help," explicitly waived -triggering); it is **not** a gate-passing evaluation and `status` stays `draft`. See §04 for why this -particular task may have been too easy to discriminate on, and what a sharper follow-up probe would -look like. - -## §02 Derived criteria (scoped to what was measured) - -- **Claim:** applying fabled's process (effort triage → understand → generate alternatives → work at - depth → adversarial self-review → verify → deliver with calibrated conviction) closes most of the - quality gap between a rushed answer and an excellent one, on non-trivial analysis/decision/debugging - tasks. -- **Intended outcome vs. baseline:** an agent following fabled should be less likely to commit to the - first plausible fix, more likely to surface the disconfirming evidence in the prompt, and more likely - to deliver a single calibrated verdict rather than a hedged menu — where a baseline agent might not. -- **Scope of this run:** behavioral lift only (dimension 4). Triggering, structure, and content-rubric - dimensions were explicitly out of scope per the user's request and are unmeasured here — see - `ceh-evaluation:evaluate-skill-lite` for those, or the full `evaluate-skill` loop for the 6-point gate. - -## §03 Trigger battery - -Not measured — out of scope for this run (user: "no need to worry about trigger"). - -## §04 Behavioral task & assertions (N=1) - -**Task:** a fintech incident-triage question with one competing-fixes decision (retry-with-backoff vs. -raise timeout), where the log evidence (failures clustered at 3.00–3.05s, uniform across payment method -and hour) is a fairly strong "smoking gun" for the correct answer if the agent actually reasons over it -rather than picking a fix by default preference. Both subagents were `general-purpose`, spawned in the -same turn. With-skill was instructed to invoke `Skill(skill="fabled")` first and follow its process; -baseline was explicitly instructed not to invoke any skill. Full task text and both transcripts: -`../../../../generated/sonnet/with-skill-answer.md`, `../../../../generated/sonnet/baseline-answer.md` -(repo-root `generated/sonnet/`, per user instruction). - -| # | Assertion | With-skill | Baseline | -|---|-----------|:---:|:---:| -| 1 | Explicitly ties the 3.00–3.05s clustering to timeout truncation, not provider flakiness | PASS — *"That's not 'the provider is broken,' that's 'our client is hanging up on a call that was still in progress.'"* | PASS — *"That's not 'occasionally the provider is slow' — that's 'the timeout is cutting off requests that were about to succeed.'"* | -| 2 | Recommends raising the timeout as the primary today-fix, not retry-as-a-complete-fix | PASS — *"raise the timeout, don't add retry — at least not today"* | PASS — *"Raise the timeout (option 2) — but treat it as a stopgap"* | -| 3 | Delivers one concrete verdict, not a hedged menu | PASS — leads with a bolded verdict line | PASS — leads with a bolded recommendation line | -| 4 | Flags a second-order/follow-up risk beyond the immediate fix (double-charge/idempotency risk if retry added later, and/or a monitoring step to confirm the diagnosis) | PASS — explicit idempotency-key gating on any future retry, explicit "watch the new failure distribution" step, explicit follow-up to get the provider's real P99 | PASS — same idempotency-key call-out, same "watch checkout error rate and p99" step, same "if failures reappear at ~9-9.5s" follow-up | - -**Result: 4/4 both sides, zero delta.** No assertion discriminated between the two conditions on this -task. - -**Observed qualitative difference (not assertion-graded, but worth recording):** the with-skill -transcript visibly performed two of the skill's named stages that the baseline transcript didn't make -explicit: -- *Alternative generation / adversarial self-review (stage 2/4):* it steelmans the retry side before - rejecting it — *"One honest counterargument for retry: if each attempt has an independent 2% chance of - running long, two attempts naively gets you to 0.04%. But the flat 2% rate... argues against - independence."* The baseline never states the opposing case before dismissing it. -- *Calibrated conviction (stage 6):* it splits confidence by claim — *"Confidence: high on 'raise the - timeout, not retry'... Lower confidence on '10s' being the exact right number."* The baseline's - confidence is implicit in phrasing ("stopgap, not the fix") rather than stated. - -Neither difference changed the final recommendation or which assertions passed. This suggests the -lift here (if any) is in reasoning *legibility/rigor-signaling*, not in this task's correctness outcome. - -**Confound to flag:** the with-skill transcript ends with a "Security/Dependency risk:" one-line flag, -which matches this repo owner's personal global `CLAUDE.md` response-formatting convention almost -exactly. Since both subagents ran in the same account/environment, this may reflect an inherited -user-level `CLAUDE.md` rather than anything the fabled skill itself specifies — fabled's SKILL.md has no -such flag-line instruction. Treat that one stylistic marker as noise, not skill signal. - -**Why this probe likely under-discriminates:** the task's disconfirming evidence (exact-timeout -clustering, uniform across every dimension) is strong enough that a single capable pass — with or -without an explicit process — catches it. Fabled's stated failure modes it exists to prevent -(first-thought commitment, menu hedging, caveat patching) are more likely to separate the two -conditions on tasks where: (a) the "obvious" first fix is subtly wrong and the disconfirming signal is -buried rather than headlined, (b) there's a socially appealing but incorrect answer, or (c) the task -has enough moving parts that skipping the alternative-generation step actually loses a viable option. -This single task doesn't have those properties — it's a good debugging task but not a good -discriminator for *process* lift specifically. - -**Reliability caveat:** N=1 is noise by the skill's own standard. Re-run at N≥3, and with a task -designed to be less immediately obvious, before drawing a firm conclusion either way. - -## §08 Advisory backlog (non-blocking) - -- Re-run this same task at N=3 per condition to check whether the 4/4-both-sides result was a fluke of - this particular pair of subagent runs, or a stable "task too easy to discriminate" result. -- Design a second behavioral task that specifically targets first-thought commitment or menu-hedging — - e.g., a decision task where the *popular* answer is wrong and the correct answer requires chasing a - second-order consequence the prompt doesn't headline. That is a sharper test of what fabled claims to - add. -- Worth comparing against the parallel Opus run (`.agents_workspace/skill-evals/fabled/run-001/`, - a colleague's session, tasks A/B/C at N=3) once that completes — different tasks, different model, - same target skill, so a side-by-side reading may show whether lift is task-dependent, model-dependent, - or both. -- If a full gate evaluation is wanted later, run `ceh-evaluation:evaluate-skill` (not this scoped - variant) to also cover structure, triggering, and content-rubric dimensions — this run intentionally - skipped all three per the user's request. diff --git a/.agents_workspace/skill-evals/fabled-sonnet/run-002/SKILL_EVAL.md b/.agents_workspace/skill-evals/fabled-sonnet/run-002/SKILL_EVAL.md deleted file mode 100644 index 16992a8..0000000 --- a/.agents_workspace/skill-evals/fabled-sonnet/run-002/SKILL_EVAL.md +++ /dev/null @@ -1,155 +0,0 @@ ---- -artifact: SKILL_EVAL -status: draft -created: 2026-07-07 -updated: 2026-07-07 -target: ceh-fabled/skills/fabled/SKILL.md -target_kind: skill -eval_gate: unscoped # scoped run — behavioral lift only, per user request; gate not scored -iterations: 1 -run_type: scoped-behavioral-lift-only -model: sonnet -note: > - Follow-up to run-001 (same folder family), rerun per explicit user feedback that run-001's - task was "too easy" (4/4 assertions passed on both sides there too). This run deliberately - designs a harder, multi-part task with a subtle bug (TOCTOU under a pessimistic lock) plus a - buried second-order consequence (a lock-ordering deadlock against an unrelated nightly job). - Triggering explicitly out of scope again per user request. ---- - -## §01 Verdict - -On this N=1 probe, **fabled again produced no assertion-level correctness lift** over the Sonnet -baseline: both the with-skill and no-skill subagent independently identified the same core defect -(the proposed fix locks the desk row *after* the availability check and never re-validates under -the lock — a TOCTOU bug that leaves the actual race open), the same buried second-order risk (the -fix's new desk→room lock order collides with the nightly reassignment job's room→desk order, -creating a deadlock hazard the on-call engineer wasn't shown), and the same top-line verdict -("ship with changes," not "ship" or "don't ship"). **This is 4/4 both sides on the pre-registered -assertions — the identical shape of result as run-001**, despite deliberately raising task -difficulty. See §04 for why, and for a genuine (if not pre-registered) content difference the -with-skill transcript produced that baseline didn't: a database-isolation-level caveat that is -technically correct and materially affects whether the recommended fix actually works. Status stays -`draft` — this is a scoped sanity check, not a gate-passing evaluation. - -## §02 Derived criteria (scoped to what was measured) - -- **Claim:** applying fabled's process (effort triage → understand → generate alternatives → work - at depth → adversarial self-review → verify → deliver with calibrated conviction) closes most of - the quality gap between a rushed answer and an excellent one, on non-trivial analysis/decision/ - debugging tasks — specifically by chasing second-order consequences (stage 3) and verifying - claims that are checkable (stage 5) rather than accepting the first plausible fix. -- **Intended outcome vs. baseline:** an agent following fabled should be less likely to accept the - on-call engineer's "standard, well-understood pattern" framing at face value, more likely to trace - the proposed fix against the given code rather than pattern-match on "pessimistic locking fixes - races," and more likely to surface a risk (the lock-ordering collision) that isn't headlined - anywhere in the prompt. -- **Scope of this run:** behavioral lift only (dimension 4), N=1, per user's explicit request. This - run's brief was specifically "use a more complex example, run 001 was too easy" — see §04 for - the design rationale and whether it succeeded at discriminating. - -## §03 Trigger battery - -Not measured — out of scope for this run (user: "no need to worry about trigger"). - -## §04 Behavioral task & assertions (N=1) - -**Task design intent:** run-001's advisory backlog flagged that its task was too easy to -discriminate because the disconfirming evidence was a headlined "smoking gun." This run's task -(`task-prompt.md`, same folder) was designed to fix that: a desk-booking double-booking incident -where the on-call engineer proposes a *plausible, standard-sounding* fix (`SELECT ... FOR UPDATE` -pessimistic locking) that is subtly incomplete (a TOCTOU bug — the lock is acquired after the -availability check, not before/inside it), plus a second, unrelated risk (a lock-ordering deadlock -against a nightly batch job) that is only derivable by cross-referencing two separate code snippets -given several paragraphs apart, neither flagged as risky in-text ("stable for six months," "low -risk, rarely touches active desks"). - -Both subagents were `general-purpose`, spawned in the same turn. With-skill was instructed to -invoke `Skill(skill="fabled")` first and follow its process; baseline was explicitly instructed not -to invoke any skill. Full task text and both transcripts: `task-prompt.md`, -`generated/sonnet/with-skill-answer.md`, `generated/sonnet/baseline-answer.md` (all in this run -folder). - -| # | Assertion | With-skill | Baseline | -|---|-----------|:---:|:---:| -| 1 | Explicitly identifies the TOCTOU flaw — the lock is acquired after the free/busy decision, not re-validated before insert — naming it as the reason the fix doesn't close the race | PASS — *"This is a textbook TOCTOU (time-of-check-to-time-of-use) bug... The lock changed timing, not outcome."* | PASS — *"The row lock only serialized access to the desks row's metadata; it never guarded the bookings table, which is where the actual conflict lives."* | -| 2 | Reproduces the exact observed symptom via a concrete concurrent-request trace (both requests picking the same desk, second one inserting anyway after the lock is released) | PASS — 5-step numbered trace, A and B both picking desk 41, B inserting post-lock with no recheck | PASS — equivalent prose trace, A and B both picking desk 41, "B never re-runs `has_conflicting_booking` after acquiring the lock" | -| 3 | Identifies the lock-order-inversion deadlock risk against the nightly `reassign_understaffed_rooms` job as a concrete risk of shipping the fix as described | PASS — explicit "New lock-ordering hazard... needs an explicit decision, not silence," traces all three code paths' lock orders | PASS — explicit "textbook lock-order inversion," names both colliding paths by function/purpose | -| 4 | Delivers one direct, committed recommendation (not a hedged menu) naming specific corrective changes rather than only the risks | PASS — "Ship with changes — not as-is," 4 enumerated required/should-have changes | PASS — "Ship with changes (not as-is)," 3 enumerated changes | - -**Result: 4/4 both sides, zero delta — same shape of result as run-001**, on a task deliberately -built to be harder. This is itself a finding: the task-design lesson from run-001 (make the -signal less headlined) was only partially applied here. The two code snippets given in "additional -context" spell out the exact lock orders and the unlocked-scan-then-lock sequence explicitly enough -that a capable model doesn't need to *infer* the bug from indirect evidence — it only needs to -*trace* code that's already handed to it in full. Sonnet's baseline is evidently strong enough at -code tracing (fabled's stage-3/stage-5 behaviors) that it does this unprompted. A sharper -discriminator would omit one of the two code snippets and require the model to *ask for it or infer -its likely shape* — or bury the second risk in a much longer prompt so it's easy to miss under time -pressure, rather than presenting it as a clearly delimited second section. - -**Observed qualitative difference (not assertion-graded, but worth recording — and arguably more -significant than run-001's):** the with-skill transcript contains one substantive, technically -correct catch that the baseline entirely omits: - -- **Database isolation-level dependency of the fix's correctness (with-skill point 3):** - *"Confirm DB engine and isolation level before shipping — do not assume... MySQL InnoDB, - REPEATABLE READ (its default): a plain SELECT after acquiring the lock can still return a - transaction-start snapshot that predates the commit it needs to see. If this is MySQL, the - re-check query itself must be a locking read... or the fix silently reintroduces the exact bug - it's meant to close, just harder to reproduce."* This is correct, non-obvious InnoDB behavior - (a real, documented MVCC/locking-read interaction) and it is directly relevant: it identifies a - way the *recommended fix itself* could silently fail to close the race, on a common production - database, for a reason nothing in the prompt states or hints at. The baseline's equivalent - recommendation ("re-run `has_conflicting_booking`... after acquiring `FOR UPDATE`... inside the - same transaction") is the *same fix* but stated as unconditionally correct, with no engine/ - isolation caveat. This maps directly to fabled stage 5 ("verify what's verifiable... internal - consistency: do the numbers, names, and claims agree with each other") and stage 3 ("chase - second-order consequences") — it is exactly the kind of catch the skill's process claims to - produce, and it is not present in the baseline's otherwise very strong answer. -- **A second, related catch (with-skill point 2):** *"Retry at the request level, not by looping - candidates in one open transaction... a single request that walks candidates 41→42→43 in one - transaction accumulates locks on all of them, which is worse for the deadlock risk."* Baseline's - equivalent line ("fall through to the next candidate... rather than inserting") is ambiguous on - this exact point and doesn't flag the lock-accumulation risk explicitly. -- **Calibrated conviction (stage 6), consistent with run-001's finding:** with-skill closes with an - explicit confidence split and a stated falsification condition — *"The most likely way I'm wrong: - if the actual implementation... already re-validates availability after acquiring the lock as an - unstated implementation detail, then finding 1 is moot."* Baseline states its recommendation with - equal firmness but without this explicit self-falsification framing. - -**Why this matters more than a legibility difference:** in run-001, the qualitative differences -(steelmanning, confidence-splitting) didn't change what shipped or what was checkable. Here, the -isolation-level catch is a *checkable, substantive correctness point* — if the team is on MySQL and -ships baseline's version of the fix verbatim, it could reintroduce the exact bug being fixed, silently. -That's outside this run's pre-registered assertions (none of the four tested for it, because the -task prompt never named a database engine), so it isn't scored as an assertion pass/fail — but it's -real, cited, evidence-backed content lift, not a stylistic artifact. - -**Reliability caveat (unchanged from run-001):** N=1 is noise by the skill's own standard. A single -pair of transcripts, however carefully read, cannot establish whether the isolation-level catch is -a stable behavior fabled's process elicits or a one-off flourish from this particular run. Re-run -at N≥3 before treating it as a proven, repeatable lift. - -## §08 Advisory backlog (non-blocking) - -- **Task design for the next iteration:** don't hand both colliding code snippets side-by-side in - clearly labeled sections. Bury the second risk (or omit the database engine entirely, as this run - did) and see whether the with-skill process's "verify what's verifiable" stage reliably surfaces - the missing fact (e.g., "you didn't tell me the DB engine — this matters, confirm before shipping") - while baseline ships without asking. This run showed once that fabled *can* produce that catch - unprompted; it did not test whether that's a *stable* difference vs. baseline noticing it too on - a re-run. -- **Re-run this exact task at N=3 per condition** to check whether the isolation-level catch appears - reliably in with-skill runs and reliably absent in baseline runs, or whether run-002's single pair - was itself noise in the other direction from run-001. -- Cross-reference against the parallel Opus evaluation under - `.agents_workspace/skill-evals/fabled-opus/run-001/` (different model, different task) once - available — two independent "4/4 both sides on pre-registered assertions" results across two - different tasks and models would meaningfully raise confidence that Sonnet/Opus-class baselines - are simply strong enough at this style of explicit-evidence bug-hunting that fabled's *assertion- - level* lift is hard to demonstrate without deliberately withholding information the model would - otherwise have to actively chase — which is precisely the design gap named above. -- If a full gate evaluation is wanted later, run `ceh-evaluation:evaluate-skill` (not this scoped - variant) to also cover structure, triggering, and content-rubric dimensions — both scoped runs to - date have skipped all three per explicit user request. diff --git a/.agents_workspace/skill-evals/orchestrate/run-001/SKILL_EVAL.md b/.agents_workspace/skill-evals/orchestrate/run-001/SKILL_EVAL.md deleted file mode 100644 index 57736b3..0000000 --- a/.agents_workspace/skill-evals/orchestrate/run-001/SKILL_EVAL.md +++ /dev/null @@ -1,260 +0,0 @@ ---- -artifact: SKILL_EVAL -status: passed -created: 2026-06-22 -updated: 2026-06-22 -target: ceh-orchestration/skills/orchestrate/SKILL.md -target_kind: skill -eval_gate: 6/6 -iterations: 3 ---- - -## §01 Verdict - -`orchestrate` enters "thin-orchestrator mode" so the main session plans and delegates while -cheap isolated subagents absorb all I/O — a cost-control posture skill. **PASSED, 6/6.** It is -structurally clean (`validate.py` green), content-strong (why-driven cost-model delta), and — -after the iteration-3 fix — triggers precisely *and* honestly: 6/8 on explicit-intent -positives with 0/8 false-fires (near-misses correctly routed to architecture / claude-api / -ops:deploy / evaluate-skill). Behavioral lift is **measured and real but narrow**: on a -heterogeneous multi-module task the skill reliably added full delegation + verifier gating + -model routing that a no-skill baseline missed (C4 3/3 vs 0/3, no regression), while on a -mechanical rename it correctly stayed out of the way (tooling beats delegation there). The -A2 fix this run reconciled the description's over-promising "minimize cost on any big task" -trigger to "heterogeneous/investigation-heavy" work plus explicit carve-outs (mechanical -sweeps → tooling; single dispatch → not this skill), verified to preserve 6/8 positives and -0/8 false-fires. Single highest-leverage next improvement (advisory): make the -verifier-gating / model-routing discipline more prominent in the body, since that — not the -parallelize-recon instinct, which capable models already have — is the skill's measurable -marginal value. - -## §02 Derived criteria - -**Claim.** Enter and hold "thin-orchestrator mode": the main session only plans and -delegates — never reads/greps/edits/runs directly — pushing all I/O and execution into -cheap isolated subagents, to cap context/token cost (which grows ~quadratically with a -single window's length). - -**Trigger intent.** -- *Should fire:* starting a large, multi-step / multi-file task with explicit delegation - or cost intent — "orchestrate this", "act as orchestrator", "thin orchestrator mode", - "delegate this", "plan and delegate", "don't edit directly", "fan this out to - subagents", or "keep token/context cost down on this big change". Also non-naming - phrasings that describe coordinating workers to keep the main window cheap. -- *Should NOT fire (near-misses):* pure planning/architecture ("plan this feature" → - plan-build-review / Plan agent); a single one-off subagent dispatch ("use a subagent to - find X" → Explore, not a session-wide mode switch); model/API cost & pricing questions - (→ claude-api); Agent-Teams setup (opposite posture — the skill says reserve teams); - trivial single-file edits; in-session task decomposition/tracking (contract's task tool, - no cost-delegation intent); multi-step deploy (→ ceh-ops); evaluating the skill (→ - evaluate-skill). - -**Intended outcome (vs no-skill baseline).** With the skill, an agent on a big multi-file -task: restates the goal in one line, produces a plan + dependency-ordered subtask list, -dispatches each subtask to the matching subagent (Explore / executor / verifier), keeps a -compact summary-only ledger, refuses to pull raw file contents into its own window, routes -work to the cheapest adequate model, and verifies via the verifier rather than re-reading. -A baseline agent instead opens files, greps, and edits directly in the main session, -accumulating context. - -## §03 Trigger battery - -Method: each prompt → fresh cold `general-purpose` subagent given only the user message, -told to show how it would *begin* (no execution, brief). Firing is coded from the final -message exhibiting thin-orchestrator posture (one-line goal restate + delegation plan + -explicit subagent dispatch + ledger/model-routing language) vs handling the task normally. -N=3; a prompt "fires" if posture appears in ≥2/3. Confidence note: subagent tool calls are -not visible to the parent, so firing is inferred from output posture, not a Skill-tool -trace — lower confidence than a direct trace. - -Threshold used: positives fire ≥6/8; near-misses false-fire ≤1/8. - -**Positives** -- P1 "orchestrate this — refactor the auth module across ~12 files and rename the session token field everywhere" -- P2 "act as orchestrator for this DB migration, there are a lot of files to touch and I don't want you doing it inline" -- P3 "thin orchestrator mode please — big multi-package refactor incoming" -- P4 "this is a huge multi-file change and I'm worried token/context cost is going to blow up. keep the main session lean and push the work down to workers" -- P5 "I've got a ~30-file codemod to run. I want you coordinating cheap subagents rather than grinding through every file in this session yourself" -- P6 "fan this out to subagents — independent subtasks here, hand each one off and just track results, don't pull all the file contents into your own context" -- P7 "don't edit directly. plan it, split it up, and delegate the implementation" -- P8 "split this big task into pieces and farm them out to workers so your own window stays cheap" - -**Near-miss negatives** -- N1 "plan the architecture for a new notifications feature — what components do we need?" -- N2 "use a subagent to go find where we handle rate limiting" -- N3 "what's the cheapest Claude model for a high-volume classification job, and how much would a batch run cost?" -- N4 "set up agent teams so the frontend and backend agents can message each other while they work on this feature together" -- N5 "rename getUser to fetchUser in src/user.ts" -- N6 "break this big feature into subtasks and track them with the task tool as you go" -- N7 "handle this multi-step deploy for me: build the image, push it, and promote to staging" -- N8 "review my orchestrate skill and tell me whether it's any good" - -**Results (N=2; run 3 omitted — could not change any classification, see iteration-1/triggering.md):** -- Positive trigger rate: **6/8** (P1-P5, P8 fired both runs; P6, P7 showed correct but - generic fan-out without distinctive markers). Threshold ≥6/8 → **MET**. -- Near-miss false-positive rate: **0/8** (all negatives routed elsewhere). Threshold ≤1/8 → **MET**. -- Notable: P6/P7 contain verbatim description trigger-phrases yet didn't surface distinctive - firing — likely an output-based detector limit (no Skill-tool trace visible to parent); - behavior was still correct. Several negatives named the right alternative skill unprompted. -- Evidence: `iteration-1/triggering.md`. - -**Iteration-3 re-test (after A2 description edit; evidence `iteration-3/triggering.md`):** -positive rate **6/8 unchanged**, false-positive rate **0/8 unchanged**. The narrowed wording -+ carve-outs removed the over-promise without costing any positive or adding any false-fire. -P2 dipped to 2/3 (one run fumbled the skill *identifier*, not a description fault). Criteria 2 -& 3 remain MET post-edit. - -## §04 Behavioral tasks & assertions - -**Task A — refactor under cost pressure** (no skill named): -"I need to rename the `userId` field to `accountId` across our TypeScript backend — it's -in models, services, routes, and tests, ~15 files. I care a lot about keeping token/context -cost down on this. Get started." -- A1: produces a plan with discrete, dependency-ordered subtasks (not one undifferentiated edit blob). -- A2: delegates work to subagents (names subagent_type / Agent dispatch / "I'll hand this to a worker") rather than reading/editing files directly itself. -- A3: keeps a compact summary-only ledger and explicitly avoids pulling raw file contents into the main window. -- A4: routes to cheapest adequate model (executor=Sonnet / verifier=Haiku, or "cheapest model that clears the bar"). - -**Task B — guardrail under invitation to dive in:** -"Audit our repo's logging calls and standardize them to structured logging across all -packages. Just dig in and start fixing them." -- B1: despite "just dig in", dispatches an Explore/worker to locate the logging calls rather than grepping/reading directly itself. -- B2: plans verification against acceptance criteria (verifier) rather than re-reading worker output. - -Baseline (no skill) expected to clear A1 partially at best; expected to miss A2/A3/A4 and -B1/B2 (it dives into direct grep/read/edit). N=3 per arm. - -**Results (Task A 3+3; Task B 3 with-skill + 1 baseline — r2/r3 baseline hit a session rate -limit; full detail + quotes in `iteration-1/behavioral.md`):** - -| Task | Assertion | With-skill | Baseline | Lift | -|------|-----------|-----------|----------|------| -| A | A1 staged plan | partial 3/3 | partial 3/3 | none | -| A | A2 delegates vs direct I/O | FAIL 0/3 | FAIL 0/3 | **none** | -| A | A3 lean ledger | N/A (no delegation) | N/A | none | -| A | A4 model routing | FAIL 0/3 | FAIL 0/3 | **none** | -| B | B1 dispatch Explore/worker | FAIL 0/3 | FAIL 0/1 | **none** | -| B | B2 verify via verifier | N/A | N/A | none | - -**The skill did not fire in either behavioral arm.** Task A: with-skill and baseline produced -*indistinguishable* plans (grep --count → branch → `sed`/replace_all → `tsc` verify) — agents -in both arms judged a scripted pass cheaper than spawning executors. Task B: all arms gave the -same scope-mismatch pushback + survey (agent-coding-contract behavior, not orchestration). -**Zero measurable lift in either task.** Root cause: both tasks were framed by cost / "just -dig in", not an explicit delegation verb — and the trigger battery shows the skill fires on -explicit verbs, not on cost-intent alone. Criterion 5 (iter-1) = unproven (tasks didn't -exercise a fired-skill scenario; where measured, lift was zero). - -**Iteration 2 — lift re-test (Task C: un-sed-able, explicit "Orchestrate this"; full detail + -quotes in `iteration-2/behavioral.md`). All 6 runs completed:** - -| Assertion | With-skill | Baseline | Lift | -|-----------|-----------|----------|------| -| C1 dependency-ordered delegation decomposition | 3/3 | 3/3 | none | -| C2 delegate recon AND implementation | 3/3 | 1/3 full (2/3 recon-only/deferred) | partial | -| C3 lean ledger, summaries not contents | 3/3 | 3/3 | none | -| C4 model routing + verifier-after-executor | **3/3** | **0/3** | **clear, stable** | -| C5 not solved by single scripted pass | 3/3 | 3/3 | none | - -Skill fired in all 3 with-skill runs (explicit verb worked as triggering predicted). **Lift -confirmed — narrow and stable:** with-skill reliably adds full delegation (C2) plus verifier -gating & model routing (C4 at 3/3 vs 0/3, zero variance) that the baseline misses, with **no -regression** on C1/C3/C5. Honest scope: a capable model already parallelizes recon and stays -lean without the skill — the skill's measurable marginal value is its specific operational -discipline (delegate edits too, gate every executor with a verifier, route by model tier). -**Criterion 5 = MET.** - -## §05 Structural findings - -| Check | Result | Evidence | -|-------|--------|----------| -| Frontmatter parses | PASS | valid YAML, `name` + `description` present (SKILL.md:1-4) | -| `name` matches directory | PASS | `name: orchestrate` == dir `skills/orchestrate/` | -| `description` present & non-trivial | PASS | full what+when+triggers, SKILL.md:3 | -| Body non-trivial | PASS | 121 lines, substantive cost model + rules | -| references/ discipline | PASS | no references/ dir; all content inline, 121 lines (<500) | -| plugin.json present & valid | PASS | name `ceh-orchestration`, version `1.0.0` | -| plugin name matches dir | PASS | `ceh-orchestration` == dir | -| version is semver | PASS | `1.0.0` | -| marketplace entry & version match | PASS | marketplace.json shows `ceh-orchestration` `1.0.0` | -| agents valid | PASS | executor (sonnet), verifier (haiku); name/desc/tools present | -| Cross-check: validate.py | PASS | "OK: all plugin checks passed" | - -All structural checks pass. - -## §06 Content findings - -Judged against `eval-rubric.md`. Cited lines from SKILL.md unless noted. - -**Description (line 3)** -- States what AND when: PASS — "Load when starting a large, multi-step task you want to - decompose and delegate rather than execute directly" (what) + explicit trigger-phrase - list (when). -- Moment not topic: PASS — "when starting a large, multi-step task" is a verb/moment, not - a noun-topic. "Stays in effect for the rest of the session once entered" sets posture. -- Slightly pushy: PASS — eight explicit trigger phrasings listed. -- Names what it is NOT for: **FIXED in iter-3** (was FAIL). The description now carries - "Not for mechanical single-pass changes where a scripted edit plus a typecheck is cheaper - (e.g. a repo-wide rename), nor for a single one-off subagent dispatch." Closes the - over-trigger insurance gap and reconciles the cost trigger with reality (A2). - -**Body** -- It's the delta: PASS (strong) — the cost model is genuinely tool-opinionated knowledge - the model wouldn't reliably produce: quadratic context re-billing (12-17), prompt-cache - prefix (92-94), agent-teams ~7× tokens (112-119), CLAUDE.md inheritance tax + @imports - caveat (96-101), `CLAUDE_CODE_SUBAGENT_MODEL` precedence gotcha (77-79), Batch-API - caveat (107-109). Little to no restatement of general knowledge. -- Progressive disclosure: PASS — 121 lines, well under ~500; no split needed. -- Explains the why: PASS (exemplary) — "Why this exists (the cost model)" (12-21); - reasoning ("X because Y") throughout; cost levers ranked by impact (89-109). Rules are - bold, not ALL-CAPS-MUST walls, and each is justified. -- Least surprise: PASS. - -Content verdict: strong delta + moment-framed + why-driven; single fix = add a "not for…" -clause to the description. - -## §07 Gate scorecard - -| # | Criterion | Status | Evidence | -|---|-----------|--------|----------| -| 1 | Structurally valid | MET | §05 all pass + validate.py (re-run post-edit: green) | -| 2 | Triggers on intent | MET | §03: 6/8 positives, held after A2 edit (iter-3) | -| 3 | Doesn't over-trigger | MET | §03: 0/8 false-fires, held after A2 edit (iter-3) | -| 4 | Content delta + moment | MET | §06: strong delta/why; "not for" clause added iter-3 | -| 5 | Behavioral lift | MET | §04 iter-2: C4 3/3 vs 0/3, C2 partial, no regression, low variance | -| 6 | User confirms | MET | user chose "apply A2 first, then confirm"; A2 applied + re-run clean | - -`eval_gate: 6/6` — **PASSED.** - -Thresholds used: positives fire ≥6/8; near-misses false-fire ≤1/8; "fires" = ≥2/3 runs (here -≥2/2). Stated for reproducibility per schema. - -## §08 Advisory backlog - -- **A1 (content): RESOLVED iter-3** — "not for…" carve-outs added to the description - (mechanical single-pass → tooling; single one-off dispatch → not this skill). Re-test - confirmed false-positives stayed 0/8. -- **A2 (value/triggering): RESOLVED iter-3** — the over-promising cost-only trigger was - reconciled with reality: opening qualified to "heterogeneous or investigation-heavy parts", - the standalone cost trigger replaced with "coordinate cheap workers on a big multi-area - change", and the mechanical-sweep carve-out added. Verified non-regressive (6/8 positives, - 0/8 false-fires). Residual (low): the body could foreground the verifier-gating / - model-routing discipline — the part baseline reliably lacks (see A3). -- **A3 (lift re-test): DONE** (iteration 2, Task C) — criterion 5 now MET. Residual: the lift - is narrow (C4 + partial C2 only); if the author wants broader demonstrable value, the body - could make the verifier-gating / model-routing discipline more prominent, since that is the - part baseline reliably lacks. - -## §09 Method decisions (this run) - -- **Triggering N:** ran N=2; run 3 omitted because it could not change any prompt's - classification under the ≥2/3 rule (all prompts at 2/2 or 0/2). Cost-aware, consistent with - the target skill's own teaching. -- **Firing detector:** output-based (distinctive thin-orchestrator markers), since a subagent's - Skill-tool calls are not visible to the parent. Lower confidence than a direct trace; flagged - on P6/P7 where behavior was correct but markers absent. -- **Baseline construction:** simulated counterfactual ("skill unavailable") rather than a - plugin-disabled environment (the skill is globally registered, not disableable per-subagent). - Lower confidence noted. -- **Incomplete runs:** Task B baseline r2/r3 hit a session rate limit; reported as a limit, not - imputed. Variance across the 7 collected Task B observations was ~0, so the conclusion holds. diff --git a/.agents_workspace/skill-evals/plan-fullstack-app-iteratively/run-001/SKILL_EVAL.md b/.agents_workspace/skill-evals/plan-fullstack-app-iteratively/run-001/SKILL_EVAL.md deleted file mode 100644 index bf501c9..0000000 --- a/.agents_workspace/skill-evals/plan-fullstack-app-iteratively/run-001/SKILL_EVAL.md +++ /dev/null @@ -1,244 +0,0 @@ ---- -artifact: SKILL_EVAL -status: passed -created: 2026-06-22 -updated: 2026-06-22 -target: ceh-plan-build-review/skills/plan-fullstack-app-iteratively/SKILL.md -target_kind: skill -eval_gate: 6/6 -iterations: 2 ---- - -# SKILL_EVAL — plan-fullstack-app-iteratively - -> **Scope:** behavioral lift (iteration 1) + triggering (iteration 2, description-routing proxy). -> All four dimensions now measured. Only user confirmation (criterion 6) remains. - -## §01 · Verdict - -The skill plans one release at a time and produces a machine-resolvable artifact (numbered -§01–§06 sections, frontmatter contract, `depends_on`/pointer chain, Mermaid diff). **Behavioral lift -is real and stable**: across N=2 on both a greenfield-skeleton and an iteration task, the with-skill -arm passed every discriminating assertion (12/12) while the no-skill baseline missed the same -convention assertions every time — zero variance, no regression. The lift is *conformance to the -plan-build-review artifact contract*, not raw planning IQ. **Triggering is accurate**: 9/10 positives -route to the skill, 0/10 near-misses false-fire, and the headline risk — the to-mvp collision — held -at 3/3 routing to to-mvp with zero variance. Gate is **5/6**; only user confirmation (criterion 6) -is open. The one wrinkle is P4 ("help me think through the *architecture* for the next release"), -which routes 3/3 to `document-architecture` — a fair secondary boundary, not a failure, addressable -with a one-line "not for writing ARCHITECTURE.md" pointer. - -## §02 · Derived criteria - -**Claim.** Produce a *minimal, scoped* plan for the next development intent only — one artifact per -session, never the finished product. Greenfield → a `SKELETON.md`; existing app → an `ITER_NN.md` -scoped to one release. It is the incremental counterpart to `plan-fullstack-app-to-mvp`. - -**Trigger intent.** - -Should fire when the user wants to plan *one* slice of build work: -- "help me plan the next feature / version / iteration" -- "what should I build first" / "create a skeleton plan for a new app" -- "help me think through this architecture" (early-stage, pre-code) -- vague/early greenfield descriptions wanting just the first thing to build -- planning the next increment of a new major version (v2's next slice) - -Should NOT fire (near-misses that share keywords): -- user wants the COMPLETE build to MVP in one session → `plan-fullstack-app-to-mvp` (**critical collision**) -- user wants to build/implement an existing plan → `implement-from-plan` -- user wants to review a plan/code against a plan → review skill -- create a living architecture doc (ARCHITECTURE.md) → `ceh-architecture` -- non-software planning (sprint/ticket assignment, product roadmap) -- code-level refactor planning, project scaffolding/setup - -**Intended outcome vs no-skill baseline.** A baseline agent asked to "plan the next feature" -typically sprawls — it specs the whole feature set in full implementation detail and often drifts -toward a finished-product spec. Following this skill should instead yield: a *single* file scoped to -one release, the fixed §01–§06 numbered section structure, YAML frontmatter (artifact / status / -sections / `depends_on` + pointers for iterations), Mermaid architecture diagram, explicit deferral -of anything not needed now (anti-overplan), and a one-sentence next-iteration suggestion that is -*not* itself planned. - -## §03 · Trigger battery - -**Measured iteration 2** as a description-routing proxy (cold subagent per prompt picks one skill -from the faithful 7-entry catalog, blind to the answer; target = A). The harness can't expose a -subagent's internal Skill decision, so this measures *what the descriptions route to* rather than -observed auto-fire — sufficient for the collision question. Raw data: `iteration-2/routing-results.md`. - -**Positive fire rate: 9/10** (threshold ≥8/10 → MET). **Near-miss false-positive: 0/10** -(threshold ≤1/10 → MET). Collision (N1/N2) → to-mvp 3/3, zero variance. - -| # | Positive prompt (should fire) | Fires N/3 | -|---|-------------------------------|-----------| -| P1 | "ok i've got this idea for a habit-tracking app, react front end + python api. before i write any code i wanna map out just the first thing to build — where do i even start?" | — | -| P2 | "we shipped v1 of the invoicing tool last month. now i need to add recurring invoices. can you plan just that next chunk of work?" | — | -| P3 | "plan the next version of our notes app" | — | -| P4 | "help me think through the architecture for the next release — don't over-spec it, just enough to start building the export feature" | — | -| P5 | "what should I build first for a recipe-sharing app? just the bare skeleton so I can see if the idea holds" | — | -| P6 | "create a skeleton plan for a new app — a dashboard that pulls metrics from stripe and renders charts" | — | -| P7 | "starting a side project, a web tool for splitting bills among roommates. plan the first iteration only, i'll iterate after" | — | -| P8 | "we're adding SSO to the existing portal. scope out a plan for just this iteration, fastapi backend" | — | -| P9 | "i want to kick off v2 of the analytics platform — reuses the v1 backend but adds a reporting layer. plan the first slice of v2" | — | -| P10 | "plan this iteration: add comment threads to the blog. dont plan beyond it" | — | - -| # | Near-miss negative (should NOT fire) | Routes to | Fires N/3 | -|---|--------------------------------------|-----------|-----------| -| N1 | "i don't want to keep coming back to re-plan — lay out the COMPLETE plan from empty repo all the way to a working MVP for my flashcards app, all the iterations at once" | to-mvp | — | -| N2 | "plan the whole thing end to end to a first usable version, simple todo app, i can foresee the whole build" | to-mvp | — | -| N3 | "i've got planning/ITER_03.md ready — now actually build it" | implement-from-plan | — | -| N4 | "implement the skeleton plan we wrote yesterday" | implement-from-plan | — | -| N5 | "review my ITER_02 plan against the code I wrote — did I follow it?" | review skill | — | -| N6 | "set up a living ARCHITECTURE.md with mermaid diagrams and key decisions for our system" | ceh-architecture | — | -| N7 | "help me plan the sprint — assign these 8 tickets across the team for the next two weeks" | none (PM) | — | -| N8 | "plan out how to refactor this 800-line god class into smaller modules" | none (refactor) | — | -| N9 | "draft a product roadmap with quarterly themes for the next year" | none (roadmap) | — | -| N10 | "scaffold the directory layout and config files for a new python service" | ceh-scaffolding | — | - -**Threshold (default):** positive fires ≥8/10 (a prompt counts as firing if ≥2/3 runs); -near-miss false-positive ≤1/10. - -## §04 · Behavioral tasks & assertions - -**Task A — Greenfield skeleton.** Prompt: "I'm starting a new app: a personal bookmark manager, -React frontend + FastAPI backend on Postgres. Help me plan the first thing to build." - -| ID | Assertion (true only if the skill worked) | -|----|-------------------------------------------| -| A1 | Output is a *single* skeleton artifact, not a multi-iteration roadmap | -| A2 | Uses the §01–§06 numbered section structure | -| A3 | YAML frontmatter: `artifact: SKELETON` + `sections` list, and **no** `depends_on` | -| A4 | §02 Architecture uses a Mermaid diagram, not ASCII | -| A5 | Explicitly *defers* non-skeleton concerns (auth / error-handling / deploy) rather than speccing them | -| A6 | Routes/screens are stubs (hardcoded/placeholder data), not full implementations | - -**Task B — Iteration on existing app.** Given a minimal existing `SKELETON.md`, prompt: "We've -built the skeleton. Now plan just the next iteration: add tagging to bookmarks (create tags, filter -by tag). Don't plan beyond it." - -| ID | Assertion | -|----|-----------| -| B1 | Output is a single `ITER_NN` file scoped to tagging only | -| B2 | Uses pointers ("Unchanged — see SKELETON §0X") for untouched sections instead of restating | -| B3 | Frontmatter has `depends_on: [SKELETON]` + `sections_changed` / `sections_unchanged` | -| B4 | Does NOT plan the iteration after this one | -| B5 | §02 shows what *changed* (new tag entity/route), marked as a diff | - -Baseline (no skill) is expected to miss: the numbered-section convention (A2/B-structure), the -frontmatter + `depends_on`/pointer mechanism (A3/B2/B3), deliberate deferral (A5), and the -"don't plan ahead" restraint (B4). - -### Results (N=2 per arm, evidence in `iteration-1/`) - -**Task A — greenfield skeleton.** with-skill **6/6 both runs**; baseline **2/6 both runs** (passed -only A1, A5). - -| ID | with r1/r2 | base r1/r2 | Cited evidence | -|----|-----------|-----------|----------------| -| A1 | PASS/PASS | PASS/PASS | single file each; baselines leak a forward "Next slices" backlog (`baseline_run2.md:82` "Next slices (backlog…)") | -| A2 | PASS/PASS | FAIL/FAIL | with: `## §01 · Concept … §05`; base: ad-hoc `## North Star`, `## First Increment` (`baseline_run2.md:17,22`) | -| A3 | PASS/PASS | FAIL/FAIL | with: `artifact: SKELETON / sections: [01..05]`, no depends_on (`with_skill_run1.md:2-8`); base: no frontmatter | -| A4 | PASS/PASS | FAIL/FAIL | with: ` ```mermaid flowchart LR ` (`with_skill_run1.md:18`); base: ASCII `bookmarks ----` block (`baseline_run1.md:46`) | -| A5 | PASS/PASS | PASS/PASS | both defer auth/etc. explicitly | -| A6 | PASS/PASS | FAIL/FAIL | with: `return [...] # Stub` / commented fetch (`with_skill_run1.md:74-81`); base: real Alembic migration + persisted writes (`baseline_run2.md:51-56`) — a *built* slice, not a stubbed skeleton | - -**Task B — iteration.** with-skill **5/5 both runs**; baseline **2/5 both runs** (passed only B1, B4). - -| ID | with r1/r2 | base r1/r2 | Cited evidence | -|----|-----------|-----------|----------------| -| B1 | PASS/PASS | PASS/PASS | single ITER scoped to tagging in all four | -| B2 | PASS/PASS | FAIL/FAIL | with: `> Unchanged — see SKELETON § 01` (`taskB/with_skill_run1.md:14`); base: restates §01 as "Goal & Scope", no pointer (`taskB/baseline_run2.md:17`) | -| B3 | PASS/PASS | FAIL/FAIL | with: `depends_on: [SKELETON]` + `sections_changed/unchanged` (`with_skill_run1.md:6-8`); base: `builds_on: SKELETON`, no section lists (`baseline_run1.md:9`) | -| B4 | PASS/PASS | PASS/PASS | all four name only a one-line next scope / Out-of-scope list | -| B5 | PASS/PASS | FAIL/FAIL | with: `classDef changed` + `class Tag,Link changed` + `%% changed` (`with_skill_run2.md:32-34`); base: ASCII model, no marked diagram | - -**Confound noted honestly:** in Task B the baseline looked *closer* to the skill output (it adopted -§-numbering and a frontmatter block) because the supplied `SKELETON.md` fixture already modeled -those conventions in-context — yet the baseline still missed the precise pointer / `depends_on` / -diff-marking mechanisms (B2/B3/B5) every time. The skill's delta survives even when the baseline has -a worked example in front of it. - -**Variance:** N=2 is below the schema's N≥3 default, so confidence is moderate — but the signal was -unanimous (12/12 with-skill assertion passes, 0 baseline passes on the 7 distinguishing assertions), -i.e. zero observed run-to-run variance, which raises confidence despite the small N. Criterion 5 is -met with that caveat recorded. - -## §05 · Structural findings - -| Check | Result | Evidence | -|-------|--------|----------| -| Frontmatter parses; `name` present | PASS | `name: plan-fullstack-app-iteratively` | -| `name` matches directory | PASS | dir `plan-fullstack-app-iteratively/` | -| `description` present + non-trivial | PASS | rich, ~14 lines, names what/when + counterpart | -| Body non-trivial | PASS | ~186 lines, 3 steps + families model | -| `references/` discipline | ADVISORY | `section-specs.md` (template) + `audit-checklist.md` (checklist) fit; `implementation-gotchas.md` is reference *prose*, arguably outside the repo's "schemas/templates only" rule | -| Repo `validate.py` cross-check | PASS | "OK: all plugin checks passed" | - -## §06 · Content findings - -Judged inline against `eval-rubric.md` (no subagent — cheap read, evidence cited). - -**Description (trigger mechanism):** -- *States what AND when* — PASS. "Use this skill to plan a software project one release at a time" - (what) + explicit trigger list "help me plan the next feature, plan the next version, plan this - iteration…" (when) (SKILL.md:4-10). -- *Moment not topic* — PASS. Framed on verbs/situations, not a noun ("what should I build first", - "plan the next version"). -- *Slightly pushy* — PASS. "ALWAYS use this skill even for vague or early-stage descriptions — it - handles ambiguity" (SKILL.md:10-11). -- *Names what it is NOT for* — PASS, and unusually strong. Explicit counterpart disambiguation vs - `plan-fullstack-app-to-mvp` with the deciding rule (SKILL.md:12-15). This is the right *content* - fix for the collision; whether it *works* at trigger time is criterion 3, untested here. - -**Body:** -- *Delta vs restatement* — PASS. The §01–§06 contract, plan-families/versioning model, backward-only - `depends_on`/pointer resolution, and the anti-overplan filter are repo-specific conventions a base - model does not invent — confirmed empirically: the §04 baselines reproduced none of them. -- *Progressive disclosure* — PASS. SKILL.md ~186 lines; schemas/checklist/gotchas pushed to - `references/` with "Read when needed — do not load upfront" (SKILL.md:180-186). -- *Explains the why* — PASS. "Overplanning is a blocking risk: a developer who reads ahead into - unresolved detail will pause to resolve it before building" (SKILL.md:87) — reasoning over bare MUSTs. -- *Least surprise* — PASS. - -**One content nit (advisory):** the "Plan Families and Versions" section (SKILL.md:49-73) is -cognitively dense — cross-version `depends_on`, skeleton-as-terminus, self-contained vs -inherits-across rules packed inline. It is correct delta, but the *common* case (default family, -same-version iteration) is the 95% path and the multi-major-version lineage rules could move to -`references/` to keep the inline body lean. Non-blocking. - -Criterion 4 (content delta + moment-framed): **MET** with the density nit as advisory. - -## §07 · Gate scorecard - -| # | Criterion | Met? | Evidence | -|---|-----------|------|----------| -| 1 | Structurally valid | **MET** | `validate.py`: "OK: all plugin checks passed"; name/dir match; rich description; ~186-line body (§05) | -| 2 | Triggers on intent | **MET** | positive routing 9/10 ≥ 8/10 threshold (§03; `iteration-2/routing-results.md`) | -| 3 | Doesn't over-trigger | **MET** | near-miss false-positive 0/10 ≤ 1/10 threshold; to-mvp collision N1/N2 → to-mvp 3/3 | -| 4 | Content is delta + moment-framed | **MET** | rubric pass with cited lines (§06); one advisory (version-lineage density) | -| 5 | Behavioral lift | **MET** | 12/12 with-skill assertion passes vs 0 baseline passes on 7 distinguishing assertions, N=2 both tasks, no regression, zero observed variance (§04). Caveat: N below the N≥3 default | -| 6 | User confirms | **MET** | user confirmed ready 2026-06-22 | - -`eval_gate: 6/6` — all criteria met; status: passed. -Thresholds used: positive ≥8/10, near-miss false-positive ≤1/10. Triggering measured as a -description-routing proxy (see §03 faithfulness note). - -## §08 · Advisory backlog - -- **P4 architecture boundary (minor)** — "help me think through the architecture for the next - release" routes 3/3 to `document-architecture`, not this skill. Fair (the user said "think through - the architecture"), and it neither breaks the positive threshold nor causes a false positive. If - you want to claim that phrasing, add "not for writing the ARCHITECTURE.md doc — see - document-architecture" to the description; otherwise leave it, the boundary is reasonable. -- ~~Run a triggering pass~~ — **done** (iteration 2): 9/10 positive, 0/10 false-positive, to-mvp - collision held 3/3. The tail-prose disambiguation works. -- **Version-lineage density (§06)** — consider moving the multi-major-version `depends_on` rules - (SKILL.md:49-73) to `references/`, keeping the default-family common case inline. -- **`references/implementation-gotchas.md` is prose**, not a schema/template; the repo CLAUDE.md - reserves `references/` for "schemas and templates only." It is well-used (the with-skill iteration - arm cited it — `taskB/with_skill_run1.md:69,95`), so the value is real; the mismatch is - classification only. Verify intent or relabel the references convention. Non-blocking. -- **Baseline planning quality was high** (vertical-slice thinking, acceptance criteria) — the skill's - value is the *artifact contract*, not planning IQ. Worth keeping that framing in the skill's own - pitch so users understand why the conventions matter (they make plans resolvable by - `implement-from-plan`). diff --git a/.agents_workspace/skill-evals/plan-fullstack-app-to-mvp/run-001/SKILL_EVAL.md b/.agents_workspace/skill-evals/plan-fullstack-app-to-mvp/run-001/SKILL_EVAL.md deleted file mode 100644 index 299f7ec..0000000 --- a/.agents_workspace/skill-evals/plan-fullstack-app-to-mvp/run-001/SKILL_EVAL.md +++ /dev/null @@ -1,200 +0,0 @@ ---- -artifact: SKILL_EVAL -status: passed -created: 2026-06-22 -updated: 2026-06-22 -target: ceh-plan-build-review/skills/plan-fullstack-app-to-mvp/SKILL.md -target_kind: skill -eval_gate: 6/6 -iterations: 2 ---- - -# SKILL_EVAL — plan-fullstack-app-to-mvp - -## §01 · Verdict - -`plan-fullstack-app-to-mvp` produces the complete skeleton-to-MVP plan in one session and -self-polices with a Step-1 complexity gate that hands off to the iterative sibling when the build -isn't foreseeable. It is **READY (gate 6/6, user confirmed)**: structurally valid; triggers strongly -(4/6 cold unprimed whole-build prompts auto-loaded it); after the iteration-2 description fix it holds -against **6/6** near-miss negatives; its body is almost pure repo-specific delta with exemplary -why-framing; and it shows clear, consistent behavioral lift in both directions — the with-skill arm -produced the correct gated, `mvp:true`-terminated, `depends_on`-chained artifact set 3/3 on a -foreseeable app, and STOPped 3/3 on an uncertain one where the skill-free baseline plowed ahead with -an 8-iteration upfront plan. The one fixed gap was the **"plan the next version" collision** with -`plan-fullstack-app-iteratively` (N2); the description now qualifies version planning, and the re-probe -routed N2 to the iterative sibling 2/2. Remaining advisory: harden positive triggering with a full -measured battery (A2). **Note: the description edit is uncommitted on branch -`fix/to-mvp-version-trigger-collision` and needs a plugin/marketplace version bump before shipping.** - -_Run mode: behavioral-focus (user-selected). Behavioral lift run at N=3 per arm; triggering run as a -lighter pass (informal cold positives + 6 targeted near-miss negatives), structural + content inline._ - -## §02 · Derived criteria - -**Claim.** Given a request for the *complete* build plan of an app — empty repo to working MVP — -produce the whole plan in one session (`SKELETON.md` + every `ITER_NN.md` to MVP), but only when the -build is foreseeable; if too uncertain, STOP and hand off to `plan-fullstack-app-iteratively`. Step 1 -is a self-policing gate. - -**Trigger intent.** -- _Should fire_ — WHOLE / COMPLETE / end-to-end build plan to a first usable version, all-at-once: - "plan this whole app to MVP", "full plan to a first version", "plan everything upfront", "I don't - want to keep re-planning". Best fit: small-to-moderate, conventional apps. -- _Should NOT fire_ — dominant collision is the counterpart `plan-fullstack-app-iteratively` ("plan - the next feature/version/iteration", vague "what should I build first", "skeleton plan"); adjacent - skills `implement-from-plan`, `review-against-plan`, `ceh-git-workflow:release`, `ceh-scaffolding`. -- _Tricky-but-positive_ — an uncertain whole-app-to-MVP request still SHOULD fire (the skill owns the - STOP/handoff). Uncertainty is a behavioral outcome, not a triggering exclusion. - -**Intended outcome vs baseline.** Asked to plan a small app to MVP, the skill should: (1) give an -explicit gate verdict before planning; (2) define a hard MVP boundary (In vs Deferred); (3) sequence -2–5 iterations with no forward references; (4) emit `SKELETON.md` + `ITER_NN.md` with `mvp:true` on -exactly the terminator and backward-only `depends_on`. On an uncertain app it should STOP and hand off -rather than emit fictional later iterations. A baseline produces one unstructured plan, no gate, no -forward-reference discipline, and on an uncertain app plows ahead. - -## §03 · Trigger battery - -Threshold used (behavioral-focus variant): positive — informal cold-fire observation, not the full -N=3×10 battery; negative — fires on ≤1 of 6 near-miss probes. Evidence below. - -### Positives — informal cold-fire evidence (no full battery this run) -The behavioral baseline arm doubles as an **unprimed cold positive test**: those subagents got only -the raw user prompt with **no** skill instruction, yet auto-loaded this skill in **4 of 6** whole-build -runs (B1/baseline/run1, B1/baseline/run3, B2/baseline/run2, B2/baseline/run3 all bear its fingerprint -— complexity gate, `SKELETON.md`/`ITER_NN` naming, `mvp:true`). Models generally *under*-trigger, so a -67% unprimed cold-fire rate on positive-style prompts is strong evidence the description fires on -intent. Marked **informal** — the focus this run was behavioral; a full N=3×9 positive battery was -deferred per the user's behavioral-focus selection. - -### Near-miss negatives — 6 cold probes, N=1 each (lite) -| # | Prompt (abridged) | Skill that fired | to-mvp over-fired? | -|---|-------------------|------------------|--------------------| -| N1 | "plan the next feature for my existing app — add comments" | `plan-fullstack-app-iteratively` (named as the counterpart) | No ✅ | -| N2 | "plan the next version of our dashboard, just v2's scope" | **`plan-fullstack-app-to-mvp`** (as a new major-version family) | Yes ⚠️ (ambiguous) | -| N3 | "what should I build first? houseplant watering tracker" | `plan-fullstack-app-iteratively` (skeleton mode) | No ✅ | -| N5 | "create a skeleton plan for a new app" | `plan-fullstack-app-iteratively` (lists this phrase) | No ✅ | -| N6 | "implement iteration 2 from the plan in planning/" | implementation, no planner loaded | No ✅ | -| N8 | "bump the version and cut a release" | `ceh-release-flow` / `ceh-git-workflow:release` | No ✅ | - -**False-positive rate 1/6 at iteration 1.** N2 was a genuine ambiguity: "plan the next version … -just v2's scope" can read as "plan the whole of v2 to its MVP" (this skill's version-family path) OR -"plan the next release" (the iterative sibling's literal trigger). - -### Iteration 2 — N2 re-probe after the A1 description fix (N=2 cold) -The description now reads: "Planning 'the next version' or a single version's next increment (e.g. -'plan v2's scope', 'plan the next release') is the iterative skill's job; choose THIS skill for a new -major version only when the user wants that ENTIRE version planned to its own MVP in one session." - -| Run | Skill that fired | to-mvp over-fired? | -|-----|------------------|--------------------| -| re-probe run1 | `plan-fullstack-app-iteratively` (explicit) | No ✅ | -| re-probe run2 | `ceh-plan-build-review` iterative/scoped mode (explicit, not to-mvp) | No ✅ | - -**Post-fix false-positive rate 0/6.** A1 resolved; `validate.py` still green after the edit. - -## §04 · Behavioral tasks & assertions - -> Important measurement caveat: the **baseline arm was partly contaminated** — 4/6 baseline agents -> auto-loaded the skill under test (strong triggering evidence, but it shrinks the clean-baseline N). -> Clean skill-free baselines: **B1/baseline/run2** and **B2/baseline/run1** only. Grading below uses -> those as the true baseline; with-skill is N=3 and unanimous on both tasks. - -### B1 — PROCEED path (foreseeable bookmark manager). With-skill N=3 vs clean baseline (run2). -| Assertion | With-skill (run1/2/3) | Clean baseline (run2) | -|-----------|----------------------|------------------------| -| B1.1 SKELETON + ≥2 ITER sequence | PASS 3/3 (SKELETON + ITER_01..03/02) | PASS (7 numbered files) | -| B1.2 exactly one `mvp:true`, last, none elsewhere | PASS 3/3 (verified in files) | **FAIL** — no `mvp:true` anywhere | -| B1.3 terminator has `mvp_target` + `## Out of MVP scope` | PASS 3/3 | **FAIL** — no terminator convention | -| B1.4 explicit In-MVP vs Deferred boundary | PASS 3/3 | PASS — states MVP in/out in overview | -| B1.5 `depends_on` present + backward-only | PASS 3/3 (verified) | **FAIL** — no `depends_on` field | -| B1.6 explicit complexity-gate verdict before planning | PASS 3/3 (PROCEED stated) | **FAIL** — no gate; just planned | - -Lift: with-skill clears **B1.2, B1.3, B1.5, B1.6** that the clean baseline misses. B1.1/B1.4 the base -model does naturally. The delta is precisely the durable artifact schema (`mvp` terminator, -`depends_on` chain) that the downstream `implement-from-plan` / `review-against-plan` skills consume — -plus the self-policing gate. Real and consequential. - -### B2 — STOP path (uncertain CRDT + ML editor). With-skill N=3 vs clean baseline (run1). -| Assertion | With-skill (run1/2/3) | Clean baseline (run1) | -|-----------|----------------------|------------------------| -| B2.1 STOP / recommend iterative, no full MVP plan | PASS 3/3 (unanimous STOP) | **FAIL** — produced full `PLAN.md` | -| B2.2 no detailed later-iteration fiction | PASS 3/3 (no files written) | **FAIL** — iterations 0–8 all upfront | -| B2.3 names the tripped signal | PASS 3/3 (both CRDT + ML named) | PARTIAL — risk register, but proceeded | - -Lift: unanimous and decisive. The clean baseline was *competent* (gated spikes, interface seams) — so -the value is not "the baseline is dumb" but "the skill has the discipline not to write a plan you'd -throw away." That is the exact self-policing the skill claims, demonstrated. - -**Variance.** With-skill: 0 variance — 3/3 correct on both tasks. Clean baseline N=1 per task (the -rest auto-fired the skill); direction unambiguous and consistent with the contaminated runs (which, -once the skill fired, matched the with-skill behavior). Behavioral lift is **proven**, with the honest -note that clean-baseline N is small because the skill triggers too reliably to keep a baseline clean. - -## §05 · Structural findings - -| Check | Result | Evidence | -|-------|--------|----------| -| Frontmatter parses; `name`, `description` present | PASS | `name: plan-fullstack-app-to-mvp`; description ~13 lines | -| `name` matches directory | PASS | dir `plan-fullstack-app-to-mvp` == name | -| Body non-trivial | PASS | 309 lines, 5 steps + families section | -| `references/` discipline | PASS | `section-specs.md` (schema), `audit-checklist.md` (template), `implementation-gotchas.md` (long technical detail — legitimate progressive disclosure per rubric) | -| `validate.py` cross-check | PASS | "OK: all plugin checks passed" — no disagreement to reconcile | - -## §06 · Content findings - -Judged against `references/eval-rubric.md`, cited lines: - -- **States what AND when** — PASS. "Use this skill when a user wants the COMPLETE build plan for an - app — from an empty repo all the way to a working MVP — produced in a single planning session" - (what) + explicit "Triggers include: plan this whole app to MVP, …" (when). (desc. lines 4–9) -- **Moment not topic** — PASS. Framed on a verb/situation, not a noun. -- **Slightly pushy + names what it's NOT for** — PASS, exemplary. "choose THIS skill to plan the - entire build to MVP in one session, and choose plan-fullstack-app-iteratively when the user wants - only the next release planned, or when the app is large, novel, or uncertain." (desc. lines 11–15) -- **Body is the delta** — PASS, strong. Body is near-entirely repo-specific convention the model - wouldn't know: plan-family/version-tag scheme (lines 39–72), `mvp:true` terminator placement, - backward-only `depends_on` (lines 70–72), the gate thresholds (lines 106–119). The opposite of the - common "restates general best practice" failure. -- **Explains the why** — PASS, exemplary. "The reason this skill exists…" (23), "The reason this skill - is dangerous if misused…" (27), "Step 1 is a gate, not a formality" (33). Why-driven, not MUST-piled. -- **Progressive disclosure** — PASS. 309 lines (<500); specs/checklist/gotchas pushed to `references/` - with "Read when needed — do not load upfront" (lines 301–309). -- **Least surprise** — PASS. Behavior matches description; gate/handoff is honest. - -One soft content note (advisory, not a fail): the "Plan Families and Versions" section (39–72) is the -densest part and is exactly where the N2 over-trigger originates — the version-planning overlap with -the iterative sibling is under-disambiguated. See §08 A1. - -## §07 · Gate scorecard - -| # | Criterion | Status | Evidence | -|---|-----------|--------|----------| -| 1 | Structurally valid | **MET** | All structural checks + `validate.py` green (§05) | -| 2 | Triggers on intent | **MET** (informal) | 4/6 unprimed cold whole-build prompts auto-fired (§03) | -| 3 | Does not over-trigger | **MET** | 6/6 negatives held after the A1 fix (was 5/6; §03 iter-2) | -| 4 | Content is delta + moment-framed | **MET** | Body is repo-specific delta, why-driven, <500 lines (§06) | -| 5 | Behavioral lift | **MET** | With-skill 3/3 on B1 schema + B2 STOP; clean baseline misses both (§04) | -| 6 | User confirms | **MET** | User chose "fix A1 first, then confirm"; A1 fixed + re-verified | - -`eval_gate: 6/6` — all criteria met. - -## §08 · Advisory backlog - -- **A1 — RESOLVED (iteration 2).** The "plan the next version" collision was fixed via option (a): this - skill's description now qualifies version planning ("Planning 'the next version' or a single version's - next increment … is the iterative skill's job; choose THIS skill for a new major version only when the - user wants that ENTIRE version planned to its own MVP in one session"). N2 re-probe routed to the - iterative sibling 2/2; false-positive rate 1/6 → 0/6. Edit is on branch - `fix/to-mvp-version-trigger-collision`, uncommitted; descriptions are not mirrored in - `CROSS_REFERENCES.md` (only the `references/` files are), so no cross-ref propagation was needed. -- **A4 — ship the fix.** The description edit needs a PATCH version bump in - `ceh-plan-build-review/.claude-plugin/plugin.json` + `.claude-plugin/marketplace.json` and a commit - before it takes effect for users. -- **A2 — positive triggering is only informally measured this run.** The behavioral-focus mode skipped - the full N=3×9 positive battery. Cold auto-fire evidence is strong (4/6) but informal; a full - positive battery would harden criterion 2 from "met (informal)" to "met (measured)". -- **A3 — baseline auto-trigger makes clean behavioral N small.** Future behavioral runs on this skill - should snapshot the SKILL.md out of the plugin path and point baseline subagents at an explicitly - skill-free environment, since the skill triggers reliably enough to contaminate an in-repo baseline. diff --git a/.agents_workspace/skill-evals/write-less-code/run-001/SKILL_EVAL.md b/.agents_workspace/skill-evals/write-less-code/run-001/SKILL_EVAL.md deleted file mode 100644 index b59eb25..0000000 --- a/.agents_workspace/skill-evals/write-less-code/run-001/SKILL_EVAL.md +++ /dev/null @@ -1,265 +0,0 @@ ---- -artifact: SKILL_EVAL -status: draft -created: 2026-06-21 -updated: 2026-06-21 -target: ceh-agent-coding-contract/skills/write-less-code/SKILL.md -target_kind: skill -eval_gate: 4/6 -iterations: 1 ---- - -## §01 Verdict - -`write-less-code` is a minimalism-reflex skill: before writing code, climb a 6-rung ladder (YAGNI → -stdlib → native platform → installed dep → one line → minimal custom), mark deliberate shortcuts -with `// less-code:` ceiling comments, keep prose shorter than code, and leave one runnable check -behind non-trivial logic — without simplifying away validation/security/a11y. It is structurally -clean, content-rich (explains *why*, 65 lines), and **never over-triggers** (0/10 near-miss -false-positives; it correctly declined to push laziness on a "do it properly" OAuth2 request and -routed dep-removal to `dependency-management`). Behavioral lift is **real but modest**: a capable -model is already minimalist by default, so the skill's measurable delta is the repo *house-style -artifacts* (ceiling comments, the `skipped: X, add when Y` Output pattern, an embedded runnable -check) and a harder push to native-over-library — not raw "less code." No correctness regression. - -**The one failing criterion is triggering (criterion 2): the skill severely under-invokes — 3/10 -positive prompts loaded it, despite ~7/10 recognizing it as the most-relevant skill.** Cold -subagents repeatedly said, in effect, "the answer is short enough that pulling the skill in adds -nothing." The description summarizes the whole approach so completely that loading the body feels -redundant — so the body's unique artifacts (which *are* the lift) often never reach the model. The -highest-leverage fix: **make the description advertise the body's concrete deltas** (ceiling-comment -format, Output pattern, test-leaving rule) rather than re-summarizing minimalism, giving the model a -reason to load it. See §03/§08. - -> **Reframing caveat (matters for how hard criterion 2 should bite).** The skill is hook-supplemented -> (see the dual-delivery note below): the always-on `UserPromptSubmit` hook already carries the -> *reflex* on every main-session turn, so under-invocation of the skill does **not** mean the user -> loses minimalism in practice — only that the *deeper* body artifacts are delivered inconsistently. -> Under that lens the generic ≥8/10 threshold is arguably the wrong bar for this skill. The report -> scores criterion 2 as unmet by the default threshold but flags this as a judgment call for the -> user (criterion 6). - -> **Loop status — stopped by user at iteration 1.** The user chose to leave this as an honest draft -> rather than apply the criterion-2 fix or re-threshold. Status stays `draft`, `eval_gate: 4/6`. -> No edits were made to the skill; the proposed description rewrite remains an advisory in §08 A1'. -> The skill is well-built and safe to ship as-is (0 false-positives, no regression) — the open item -> is that its on-demand body artifacts reach the model on only ~3/10 substantive coding moments. - -> **Architectural note that reshapes this eval — dual delivery.** The skill ships *two* delivery -> paths (see `ceh-agent-coding-contract/hooks/hooks.json`): a `UserPromptSubmit` hook -> (`less-code-payload.sh`) injects a **condensed ladder on every prompt** in the main session, and -> the **full SKILL.md loads on-demand** via description triggering. Consequences: -> - The *reflex* (ladder + the "never simplify away" line) reaches the main agent on every turn -> regardless of skill triggering. So the skill's triggering only governs the **deeper delta**: -> the Rules block, the `// less-code:` ceiling-comment format, the Output pattern, and the -> test-leaving discipline. -> - Subagents (Agent-tool invocations) do **not** receive the `UserPromptSubmit` hook, so a cold -> subagent battery cleanly isolates the *skill alone*. The baseline subagent has neither hook nor -> skill — which means measured behavioral lift over-states the skill's *marginal* real-world value -> (where the hook already carries the reflex). This limit is stated in §04. - -## §02 Derived criteria - -**Claim.** Before writing code for a non-trivial task, reach for the smallest thing that actually -works by climbing the ladder; refuse unrequested abstractions; mark deliberate shortcuts with -`// less-code:` (naming ceiling + upgrade path); keep the explanation shorter than the code; leave -one runnable check behind non-trivial logic — while never simplifying away trust-boundary -validation, data-loss handling, security, accessibility, or anything explicitly requested. - -**Trigger intent.** -- *Should fire:* about to implement a non-trivial feature; user says "write less code", "be lazy", - "lazy mode", "simplest/minimal solution", "yagni", "do less", "shortest path"; user complains of - over-engineering, bloat, boilerplate, or unnecessary dependencies. -- *Should NOT fire (near-misses sharing keywords):* refactoring existing code for readability - (adjacent, not new-code minimization); debugging / bugfix with no new code; reviewing someone's PR - (→ `code-review`); performance optimization (often *adds* code); removing a dependency - (→ `dependency-management`); writing tests as the primary task; project scaffolding - (→ `scaffolding`); a knowledge question about a library; an explicit request for the - full/production/"do it properly" version. - -**Intended outcome vs no-skill baseline.** On a task a naive agent would over-build (custom class, -new dependency, abstraction layer), the with-skill agent ships a markedly smaller solution -(stdlib/native/one-liner, no new dep, no speculative abstraction) and states what it skipped and when -to add it — while preserving validation/security where the task crosses a trust boundary. - -## §03 Trigger battery - -Threshold (defaults): positive fires if ≥2/3 runs trigger; pass = ≥8/10 positives fire; -over-trigger pass = ≤1/10 near-misses fire. **Status: RUN, N=3 positives / N=2 negatives** (raw: -`iteration-1/run1-raw.md`, `runs2-3-raw.md`). "Fires" = skill *invoked* via the Skill tool; -"recognized" = named most-relevant but not invoked. - -**Result — positives: 3/10 fire (FAIL vs ≥8).** P2 (3/3), P1 (2/3), P3 (2/3) fired; P4 (1/3) and -P5–P10 (0/3) did not. Recognition was far higher (~7/10 named write-less-code most-relevant) — the -gap is *invocation*, not relevance. Representative non-invoke rationale (P3): *"the answer is short -enough that pulling it in adds nothing."* - -**Result — negatives: 0/10 false-positives by threshold (PASS).** One soft hit: N9 (scaffold a -python service) invoked write-less-code in 1/2 runs while naming scaffolding most-relevant. Strong -correct-decline evidence: N7 (production OAuth2 "do it properly") did **not** push laziness in any -run; N5 (remove lodash) routed to `dependency-management`; N3 (review PR) named `code-review`. - -**Diagnosis.** Under-invocation, not over- or mis-triggering. Recognition is high, so broadening -phrasings won't help. The description telegraphs the entire method, so the model concludes the body -is redundant. Fix = advertise the body's *non-obvious artifacts* so loading pays off (see §08 A1'). - -**Positives (10):** -1. "ok i need to add a feature where users can export their dashboard to csv. can you just write the least amount of code for this" -2. "be lazy on this one — parse the ISO timestamps in this log file and bucket them by hour" -3. "i think this is getting over-engineered, we've got a FieldValidatorFactory and three strategy classes for what's basically trimming whitespace. help" -4. "yagni — do i really need a whole settings module for these two flags?" -5. "add a debounce to the search input" *(feature; native-vs-lib decision territory)* -6. "we need to dedupe a list of dicts by the 'id' key before inserting. what's the simplest way" -7. "before you build this rate limiter, what's the shortest path that actually works?" -8. "implement a function that returns the nth fibonacci, but don't gold-plate it" -9. "this PR adds 400 lines of boilerplate to register one new event type. there has to be less" -10. "i want to validate that an uploaded file is a real PNG before saving. keep it minimal, no new deps" - -**Near-miss negatives (10):** -1. "this function is 200 lines and impossible to follow, can you refactor it into smaller pieces" *(refactor for readability; 'smaller' is the trap)* -2. "the checkout endpoint is throwing a 500, can you figure out why" *(debugging, no new code)* -3. "review my PR and tell me if the approach is sound" *(→ code-review)* -4. "this query is slow, can you optimize it" *(perf; often adds code)* -5. "remove the lodash dependency, we only use it in one place" *(→ dependency-management)* -6. "write comprehensive unit tests for the payment module" *('comprehensive' is anti-lazy)* -7. "i need a production-grade, fully validated OAuth2 flow with refresh-token rotation — do it properly, don't cut corners" *(explicit full version; deliberately ambiguous — may trigger-then-defer)* -8. "explain how Python's functools.lru_cache works" *(knowledge question)* -9. "set up the directory structure and config for a new python service" *(→ scaffolding)* -10. "the build is failing on CI with a mypy error in models.py, fix it" *(bugfix, no minimization intent)* - -## §04 Behavioral tasks & assertions - -**Status: RUN, N=2 per arm, neutral task phrasing** (raw: `iteration-1/behavioral-raw.md`). -Baseline = no skill, no hook. Limit: baseline lacks the always-on hook, so this measures -skill+reflex vs nothing — it *over*-states the skill's real-world marginal value (where the hook -already supplies the reflex), yet lift still came out modest. - -**Headline:** the **baseline is already strongly minimalist** — stdlib over dependencies, small -functions, no speculative abstractions — without the skill. So raw "less code" is not where lift -lives. Measured, reproducible lift (both runs): -- **Task A (business days):** both arms used stdlib, no dependency (A1/A2/A4 tie). With-skill added - the `// less-code:` ceiling comment, the `skipped: X, add when Y` line, and an *embedded runnable - assert*; baseline only gestured at alternatives in prose. → modest lift, artifacts only. -- **Task B (React email):** with-skill shipped native `type="email" + required` only and stated the - skip; baseline built a regex + error-state + `aria` layer (heavier, but added accessibility the - lean version leans on the browser for). → real leanness lift, with a UX/a11y trade-off worth - noting. -- **Task C (CSV sum, guardrail):** both got money right (`Decimal`); baseline volunteered - malformed-row tolerance the with-skill version documented as a deliberate skip. → no minimalism - lift; baseline slightly more robust. - -**No correctness regression anywhere.** Net: lift is genuine but narrow (artifacts + native-first), -and in two arms the baseline delivered *more* robustness/UX — "less" is not unconditionally -"better." This is the evidence-backed reason the dual-delivery design is sensible: cheap always-on -reflex via the hook, on-demand house-style discipline via the skill. - -**Task A — over-build temptation (stdlib over dependency).** -"Add a function that returns the number of business days between two dates, excluding weekends." -- A1: adds **no new third-party dependency** — stdlib `datetime` only (baseline may reach for - pandas/numpy `busday_count`). -- A2: a single small function, **no class/abstraction**. -- A3: states what was skipped / when to add a lib (e.g. holiday calendars) — the Output pattern. -- A4 (guardrail): weekend exclusion is actually **correct** (lazy ≠ wrong). - -**Task B — native over library (frontend).** -"Add validation to our React email field: it must be a valid email and required." -- B1: uses **native HTML5 validation** (`type="email"`, `required`) over adding a validation lib. -- B2: **no new dependency** (no zod/yup/react-hook-form). -- B3: states what was skipped (e.g. custom error UI) and the upgrade path. - -**Task C — guardrail (don't over-simplify a trust boundary).** -"Quickly parse this user-uploaded CSV of bank transactions and sum the amounts — the lazy version is -fine." -- C1: despite "lazy version is fine," **validates/handles malformed or non-numeric input** (trust - boundary preserved per the skill's "When NOT to be lazy"). Discriminates against a naive-lazy - baseline that skips validation. -- C2: still minimal — no over-built parsing framework. - -## §05 Structural findings - -| Check | Result | Evidence | -|-------|--------|----------| -| Frontmatter parses; `name`/`description`/`license` present | PASS | lines 1-13 | -| `name` matches directory | PASS | `name: write-less-code` == dir `write-less-code` | -| Description non-trivial | PASS | 8 lines, states what + when (lines 4-11) | -| Body non-trivial | PASS | 65 lines, multiple structured sections | -| `references/` discipline | PASS | no `references/` dir — none needed at this size | -| Repo `validate.py` cross-check | PASS | `OK: all plugin checks passed` | - -**Criterion 1 (structurally valid): MET.** - -## §06 Content findings - -Judged against `references/eval-rubric.md`, with cited lines. - -- **States what AND when — PASS.** What: "Reach for the smallest solution that actually works… - custom code last" (desc lines 4-11). When: "Load proactively before implementing a feature, and - whenever the user says 'write less code'… 'yagni'… or complains about over-engineering, bloat, - boilerplate" (lines 7-11). -- **Moment not topic — PASS.** "before implementing a feature" is a moment; trigger phrases are - situational, not a noun/topic. -- **Slightly pushy — PASS.** "Load proactively" + a long explicit phrasing list (lines 7-11). -- **Names what it is NOT for — PARTIAL.** Body line 18-20 points to `agent-coding-contract` for the - negative half, but neither description nor body names the *adjacent near-miss* skills (refactor, - code-review, dependency-management, perf). **Prediction reconciliation:** I initially flagged this - as the highest-leverage gap, expecting it to drive false-positives. The N=2 negative battery - **disproved that** — false-positives are 0/10; the model disambiguates the near-misses correctly on - its own. So this is downgraded to advisory insurance (§08 A2), and the real highest-leverage gap - turned out to be the opposite failure mode: *under*-invocation from a description that re-summarizes - rather than advertises the delta (§03 diagnosis). -- **Description re-summarizes the method instead of advertising the body — the confirmed gap.** Lines - 4-11 lay out the entire ladder (YAGNI → stdlib → native → dep → one line → custom). That reads as - the complete approach, so a model that already knows minimalism concludes the body adds nothing and - declines to load it (3/10 invocation, §03). The body's genuinely non-obvious artifacts — ceiling - comment format (line 42), Output pattern (line 50), runnable-check rule (lines 58-61) — are not - surfaced in the description, so they don't motivate a load. -- **Body is the delta — PASS (with note).** The genuine, non-obvious delta: the `// less-code:` - ceiling-comment format with upgrade path (line 42), the Output pattern `[code] → skipped: [X], add - when [Y]` (line 50), the "leave ONE runnable check… an assert-based self-check" discipline (lines - 58-61), the edge-case tie-break (line 41). Note: rungs 2-5 of the ladder (stdlib/native/one-line) - restate general knowledge a strong model already has — justified here as a *memorable reflex* and - because the hook, not the skill body, is the reflex's primary carrier. -- **Explains why over MUSTs — PASS.** "clever is what someone decodes at 3am" (line 39); "every - paragraph defending a simplification is complexity smuggled back as prose" (line 48); "so a - shortcut reads as intent, not ignorance" (line 42). Reasoning-rich; ALL-CAPS used sparingly. -- **Size / progressive disclosure — PASS.** 65 lines, far under ~500; no reference split needed. -- **Least surprise — PASS.** Nothing deceptive; behavior matches the description. - -**Criterion 4 (content is delta + moment-framed): MET, with one advisory (add NOT-for pointers).** - -## §07 Gate scorecard - -| # | Criterion | Status | Evidence / threshold | -|---|-----------|--------|----------------------| -| 1 | Structurally valid | **MET** | All §05 checks pass; validate.py `OK: all plugin checks passed` | -| 2 | Triggers on intent | **UNMET** | §03 — 3/10 positives invoke (threshold ≥8). Under-invocation, not mis-trigger; recognition ~7/10. The one blocking gap. | -| 3 | Does not over-trigger | **MET** | §03 — 0/10 false-positives by threshold (one soft 1/2 on N9 scaffolding) | -| 4 | Content is delta + moment-framed | **MET** | §06 — delta present, moment-framed, why-driven, 65 lines; advisory: description re-summarizes vs advertises the delta | -| 5 | Behavioral lift | **MET (modest)** | §04 — with-skill clears artifact/native-first assertions baseline misses; no correctness regression; lift narrow because baseline already minimalist | -| 6 | User confirms | PENDING | — | - -**eval_gate: 4/6.** Lowest/blocking criterion: **#2 triggering.** The leverage move is **not** more -trigger phrasings (recognition is already high) — it is rewriting the description to advertise the -body's concrete, non-obvious artifacts so the model loads it instead of concluding it already knows -the approach. See §08 A1'. Note the §01 reframing caveat: if the user decides the hook-supplemented -role makes the ≥8/10 default the wrong bar, criterion 2 could be re-thresholded — that is a -criterion-6 judgment, not an evidence change. - -## §08 Advisory backlog - -- **A1' (highest-leverage, blocks criterion 2):** The description re-summarizes the minimalism method - the model already knows, so it under-invokes. Reframe it to advertise the *body's* deltas — e.g. - "…loads the repo's `// less-code:` ceiling-comment format, the `skipped: X, add when Y` output - pattern, and the leave-one-runnable-check rule." Give the model a reason to open the body. Expected - effect: raise invocation on substantive coding moments without touching the (already clean) - false-positive rate. -- **A2 (advisory):** Add a one-line "Not for…" pointer (refactor existing code → simplify/refactor; - review → `code-review`; dep removal → `dependency-management`). False-positives are already 0/10 so - this is insurance, not a fix — but it firms up the soft N9 scaffolding hit. -- **A3 (advisory):** State in the SKILL.md body that the always-on hook carries the reflex and the - skill carries depth, so a reader understands the division (currently only the hook script's shell - comment explains it). Also clarifies *why* the body can assume the ladder is already in context. -- **A4 (advisory):** The Task B/C trade-off — leaner output occasionally dropped accessibility / a - robustness affordance the baseline volunteered. The "When NOT to be lazy" section already lists - a11y and trust-boundary validation; consider sharpening it so the lean path doesn't silently shed - a volunteered safeguard. diff --git a/.agents_workspace/skill-evals/write-less-code/run-001/iteration-1/behavioral-raw.md b/.agents_workspace/skill-evals/write-less-code/run-001/iteration-1/behavioral-raw.md deleted file mode 100644 index 1f48be1..0000000 --- a/.agents_workspace/skill-evals/write-less-code/run-001/iteration-1/behavioral-raw.md +++ /dev/null @@ -1,37 +0,0 @@ -# Behavioral lift — iteration 1 (with-skill vs baseline, N=2, neutral phrasing) - -Baseline = no skill, no hook. Neutral task wording (no "minimal/lazy" hint) so baseline isn't tipped off. - -## Task A — business days between two dates (over-build temptation) -| Assertion | with-skill r1 | with-skill r2 | baseline r1 | baseline r2 | lift? | -|-----------|--------------|--------------|-------------|-------------|-------| -| A1 no new dep / stdlib | PASS | PASS | PASS | PASS | none — baseline already stdlib (only *mentioned* numpy) | -| A2 single fn, no class | PASS | PASS | PASS | PASS | none | -| A3 structured skipped:/add-when + runnable check | PASS (`# less-code:` + asserts) | PASS (`[code] → skipped:` + asserts) | partial (prose "if NumPy…") | partial | **modest lift** — artifacts only | -| A4 correctness | PASS | PASS | PASS | PASS | none | - -## Task B — React email validation (native-over-library) -| Assertion | with-skill r1 | with-skill r2 | baseline r1 | baseline r2 | lift? | -|-----------|--------------|--------------|-------------|-------------|-------| -| B1 native HTML5 over JS lib/regex | PASS (type=email+required only) | PASS | PARTIAL (native + added regex+state layer) | PARTIAL | **lift** — with-skill markedly leaner | -| B2 no new dependency | PASS | PASS | PASS | PASS | none — neither added zod/yup | -| B3 states skipped + upgrade path | PASS | PASS | NO | NO | **lift** | -| (counter) accessibility kept | relies on native a11y | native | added aria-invalid/role=alert | added aria | baseline richer UX/a11y | - -## Task C — CSV sum (guardrail: don't simplify away trust boundary) -| Assertion | with-skill r1 | with-skill r2 | baseline r1 | baseline r2 | lift? | -|-----------|--------------|--------------|-------------|-------------|-------| -| C1 money correctness (Decimal) | PASS (Decimal) | PASS (float, but flagged →Decimal) | PASS (Decimal) | PASS (Decimal) | none | -| C1b malformed-row handling | deferred (documented skip) | deferred | PASS (skips+reports) | PASS | **negative** — baseline more robust unprompted | -| C2 minimal, no framework | PASS | PASS | PASS (heavier but stdlib) | PASS | with-skill leaner | - -## Behavioral verdict -- **The baseline is already minimalist by default**: stdlib over deps, small functions, no speculative - abstractions — without the skill or hook. So raw "less code" is NOT where the lift lives. -- **Measured lift is real but modest and concentrated in the repo house-style artifacts**: the - `// less-code:` ceiling comment, the `skipped: X, add when Y` Output pattern, the embedded runnable - assert (Task A), and a harder push to native-over-library (Task B). -- **No correctness regression** in any arm. In two places the *baseline* delivered more (Task B - accessibility/error UX, Task C malformed-row tolerance) — i.e. "less" is not universally "better", - and the skill's leaner output occasionally drops robustness the baseline volunteered. -- Variance: consistent across both runs per arm. diff --git a/.agents_workspace/skill-evals/write-less-code/run-001/iteration-1/run1-raw.md b/.agents_workspace/skill-evals/write-less-code/run-001/iteration-1/run1-raw.md deleted file mode 100644 index 310bedb..0000000 --- a/.agents_workspace/skill-evals/write-less-code/run-001/iteration-1/run1-raw.md +++ /dev/null @@ -1,35 +0,0 @@ -# Trigger battery — iteration 1, run 1 (raw) - -Signal columns: INVOKED = called write-less-code via Skill tool; RELEVANT = named most-relevant skill. - -## Positives -| # | prompt gist | INVOKED wlc | RELEVANT | notes | -|---|-------------|-------------|----------|-------| -| P1 | csv export, least code | YES | write-less-code | used Output pattern `skipped: …` + runnable assert | -| P2 | parse ISO ts, be lazy | YES | write-less-code | used `# less-code:` ceiling comment | -| P3 | over-engineered factory | no | write-less-code | "short enough, pulling it in adds nothing" | -| P4 | yagni settings module | no | write-less-code | good minimal answer, no repo artifacts | -| P5 | debounce | no | none | did not even recognize | -| P6 | dedupe dicts simplest | no | none | one-liner, no recognition | -| P7 | rate limiter shortest path | no | write-less-code | infra-first answer, not invoked | -| P8 | fibonacci no gold-plate | no | write-less-code | minimal, no ceiling comment/Output pattern | -| P9 | 400 lines boilerplate | no | write-less-code | "didn't warrant invoking the Skill tool" | -| P10 | PNG validate minimal no deps | no | none | correct trust-boundary answer, not invoked | - -Run1 positives: INVOKED 2/10 (P1,P2); RELEVANT 7/10. - -## Negatives (false positive = write-less-code invoked) -| # | prompt gist | INVOKED wlc | RELEVANT | correct? | -|---|-------------|-------------|----------|----------| -| N1 | refactor 200-line fn | no | none | OK | -| N2 | debug checkout 500 | no | none | OK | -| N3 | review my PR | no | code-review | OK (did review, no wlc) | -| N4 | optimize slow query | no | none | OK | -| N5 | remove lodash dep | no (fired dependency-management) | dependency-management | OK | -| N6 | comprehensive tests | no | none | OK | -| N7 | production OAuth2 do it properly | no | none | OK — did NOT push laziness | -| N8 | explain lru_cache | no | none | OK | -| N9 | scaffold python service | no | none | OK | -| N10 | fix mypy error | no | none | OK | - -Run1 negatives: write-less-code false positives 0/10. diff --git a/.agents_workspace/skill-evals/write-less-code/run-001/iteration-1/runs2-3-raw.md b/.agents_workspace/skill-evals/write-less-code/run-001/iteration-1/runs2-3-raw.md deleted file mode 100644 index 93b7c18..0000000 --- a/.agents_workspace/skill-evals/write-less-code/run-001/iteration-1/runs2-3-raw.md +++ /dev/null @@ -1,44 +0,0 @@ -# Trigger battery — iteration 1, runs 2 & 3 + negatives run 2 (raw tally) - -INVOKED = write-less-code (wlc) invoked via Skill tool. - -## Positives — INVOKED wlc per run -| # | gist | run1 | run2 | run3 | fires ≥2/3 | recognized-relevant (any run) | -|---|------|------|------|------|-----------|-------| -| P1 | csv export | YES | YES | no | **FIRE 2/3** | yes | -| P2 | parse ISO ts | YES | YES | YES | **FIRE 3/3** | yes | -| P3 | over-engineered factory | no | YES | YES | **FIRE 2/3** | yes | -| P4 | yagni settings | no | no | YES | 1/3 | yes | -| P5 | debounce | no | no | no | 0/3 | no | -| P6 | dedupe dicts | no | no | no | 0/3 | no | -| P7 | rate limiter | no | no | no | 0/3 | yes | -| P8 | fibonacci | no | no | no | 0/3 | yes | -| P9 | 400 boilerplate | no | no | no | 0/3 | yes | -| P10 | PNG validate | no | no | no | 0/3 | no | - -**Positive trigger rate (strict invocation): 3/10 fire.** Recognition-as-relevant ~7/10. - -## Negatives — INVOKED wlc (false positive) per run -| # | gist | run1 | run2 | FP? | -|---|------|------|------|-----| -| N1 | refactor 200-line fn | no | no | no | -| N2 | debug 500 | no | no | no | -| N3 | review PR | no | no | no (named code-review) | -| N4 | optimize query | no | no | no | -| N5 | remove lodash | no (dep-mgmt) | no (dep-mgmt) | no — correct other skill | -| N6 | comprehensive tests | no | no | no | -| N7 | production OAuth2 | no | no | no — did NOT push laziness | -| N8 | explain lru_cache | no | no | no | -| N9 | scaffold py service | no | **YES** | soft FP 1/2 (named scaffolding most-relevant) | -| N10 | fix mypy | no | no | no | - -**False-positive rate: 0/10 by ≥2/3 threshold; one soft hit (N9 scaffolding 1/2).** - -## Cross-run behavioral observation (confounded, but strong) -When wlc was INVOKED, outputs reliably carried the repo-specific artifacts: -- `# less-code:` ceiling comment (P2 run1; P3 run3 "Flag — Architecture") -- `skipped: X, add when Y` Output pattern (P1, P2, P3, P4-invoked) -- runnable assert check (P1 run1 round-trip assert) -When NOT invoked, outputs were still minimalist (stdlib/one-liner/native) but WITHOUT those -structured artifacts. => the skill body's delta = the artifacts, not the minimalism itself -(minimalism is partly the model's default / partly the always-on hook). diff --git a/.agents_workspace/skill-evals/write-less-code/run-002/SKILL_EVAL.md b/.agents_workspace/skill-evals/write-less-code/run-002/SKILL_EVAL.md deleted file mode 100644 index cab20c4..0000000 --- a/.agents_workspace/skill-evals/write-less-code/run-002/SKILL_EVAL.md +++ /dev/null @@ -1,123 +0,0 @@ ---- -artifact: SKILL_EVAL -status: draft -mode: lite -created: 2026-06-22 -updated: 2026-06-22 -target: ceh-agent-coding-contract/skills/write-less-code/SKILL.md -target_kind: skill -eval_gate: 4/6 -iterations: 1 ---- - -## §01 Verdict - -`write-less-code` is the positive half of a minimalism reflex: before writing code, climb the 6-rung -ladder (YAGNI → stdlib → native platform → installed dep → one line → minimal custom), mark -deliberate shortcuts with `// less-code:` ceiling comments, keep prose shorter than code, and leave -one runnable check behind non-trivial logic. **Lite scope: this is a dev-loop check, not a ship -verdict.** Structure (validate.py clean), content (delta-rich, moment-framed, explains *why*), and — -*under lite's framing* — triggering all read clean: 6/6 positives fired, 1/6 borderline near-miss. - -**The honest open question is triggering, and lite cannot settle it.** This run's 6/6 positive rate -directly contradicts the prior full eval (run-001: 3/10, severe under-trigger). The difference is -method: lite's subagent prompt asked agents to *deliberate and report which skills they'd load*, -priming skill-consideration; run-001 observed *natural* cold invocation. Lite's own rule applies — -N=1 with carried context is a sanity read, not a statistic — so the run-001 cold result is the more -trustworthy signal and the under-trigger problem likely stands. **Path to a real verdict: -`ceh-evaluation:evaluate-skill`** (N=3 cold natural-invocation triggering + behavioral lift). - -## §02 Derived criteria - -- **Claim:** Before writing code, reach for the smallest thing that works via the 6-rung ladder; - produce house-style artifacts (ceiling comments, `skipped: X, add when Y` output, native-over-lib, - embedded runnable check) — without simplifying away validation/security/a11y/explicit requests. -- **Trigger intent — should fire:** implementing a feature; "write less code / be lazy / yagni / - simplest / shortest path / minimal"; complaints about over-engineering, bloat, boilerplate, - unnecessary dependencies. -- **Trigger intent — should NOT fire:** PR review (→ code-review); dependency *removal* - (→ dependency-management); plain bugfix; "do it properly / full version, no shortcuts"; test - authoring; large refactor requested for its own sake. -- **Intended outcome (recorded, NOT measured in lite — that is behavioral lift):** vs a no-skill - baseline, the agent emits ceiling comments and the `skipped/add-when` pattern, prefers native - platform features over libraries, and leaves a runnable check behind non-trivial logic. - -## §03 Trigger battery (lite, N=1) - -6 positives / 6 near-miss negatives. Raw transcripts and the full table in -`iteration-1/triggering-results.md`. Summary: - -- **Positive trigger rate: 6/6.** All six fired write-less-code (P1 native date input, P2 dep-free - CSV, P3 explicit "write less code", P4 over-engineering complaint, P5 "yagni mode", P6 "lazy - version first"). -- **Near-miss false-positive rate: 1/6** (N6 refactor). N1/N2 correctly routed to code-review / - dependency-management; N3/N4/N5 loaded nothing. N6 is borderline-legit (minimalism genuinely - applies to a refactor), so effective FP is 0–1/6. - -**Contradiction with run-001 (full eval): 6/6 here vs 3/10 there.** N=1 + a deliberate-choice -prompt frame inflated positives. This dimension is NOT settled by lite; flagged for the full eval's -N=3 cold natural-invocation re-check. Do not read 6/6 as a triggering pass. - -## §04 Behavioral tasks & assertions - -**Skipped by design (lite).** Criterion 5 (behavioral lift) is unproven. run-001 found lift "real -but modest" (the house-style artifacts, not raw less-code) with no regression — informative but not -re-measured here. Run `ceh-evaluation:evaluate-skill` for the paired with/baseline battery. - -## §05 Structural findings - -| Check | Result | Evidence | -|-------|--------|----------| -| Frontmatter parses | PASS | valid YAML, `name`/`description`/`license` present (SKILL.md:1-13) | -| `name` matches directory | PASS | `name: write-less-code` == dir `skills/write-less-code` | -| Description non-trivial + states when | PASS | lists explicit trigger phrases (SKILL.md:7-11) | -| Body non-trivial | PASS | 65 lines, ladder + rules + output + guardrails | -| `references/` discipline | PASS (N/A) | no references dir; all content inline (appropriate at 65 lines) | -| Repo validator cross-check | PASS | `python tools/validate-plugins/validate.py` → "OK: all plugin checks passed" | - -## §06 Content findings - -Judged against `../evaluate-skill/references/eval-rubric.md`, with cited lines: - -- **Delta, not restatement — PASS.** Carries repo-specific artifacts the model wouldn't default to: - the `// less-code:` ceiling-comment convention with upgrade path (SKILL.md:42), the - `[code] → skipped: [X], add when [Y]` output pattern (SKILL.md:50), "leave ONE runnable check … - the smallest thing that fails if the logic breaks" (SKILL.md:58-60). The ladder ordering + - native-platform examples (SKILL.md:28) are opinionated, not generic. -- **Moment, not topic — PASS.** Description frames moments: "Load proactively before implementing a - feature, and whenever the user says…" (SKILL.md:7-11). -- **Explains the why, not ALL-CAPS MUSTs — PASS.** "clever is what someone decodes at 3am" - (SKILL.md:39); "every paragraph defending a simplification is complexity smuggled back as prose" - (SKILL.md:48). Reasoning the model generalizes from. -- **Size / progressive disclosure — PASS.** 65 lines, far under ~500; no split needed. -- **Names what it's NOT for — PASS.** "When NOT to be lazy" guardrails (SKILL.md:52-61) and explicit - hand-off of the negative half to `agent-coding-contract` (SKILL.md:17-20). - -No content red flags. Minor advisory in §08. - -## §07 Gate scorecard (lite — max 4/6) - -Thresholds used: positives fire ≥ 5/6; near-miss FP ≤ 1/6. - -| # | Criterion | Status | Evidence | -|---|-----------|--------|----------| -| 1 | Structurally valid | **MET** | all §05 checks pass; validate.py clean | -| 2 | Triggers on intent | **MET (lite) — CONTESTED** | 6/6 positives, but contradicts run-001's cold 3/10; lite framing inflated it. Authoritative re-check pending in full eval. | -| 3 | Does not over-trigger | **MET** | 1/6 near-miss (N6, borderline-legit) ≤ threshold | -| 4 | Content is delta + moment-framed | **MET** | §06 rubric pass, cited lines, within size norms | -| 5 | Behavioral lift | **UNPROVEN** | skipped by design (lite); see §04 | -| 6 | User confirms | **UNPROVEN** | not a lite concern | - -`eval_gate: 4/6` — the four lite-measurable criteria read as met, with criterion 2 carrying an -explicit contradiction flag. Criteria 5–6 are unproven, not met. Lite never sets `status: passed`. - -## §08 Advisory backlog - -- **Criterion 2 is the real agenda, and it's out of lite's rigor.** Do not trust 6/6. The - highest-leverage next step is the full eval's N=3 cold natural-invocation triggering. run-001's fix - hypothesis stands: the description so completely summarizes the approach that loading the body - feels redundant to a cold agent — make the description advertise the body's *concrete deltas* - (ceiling-comment convention, the skipped/add-when pattern) so the body's unique value pulls the - skill in. -- N6 (refactor) firing write-less-code is arguably correct, not a defect — if the full eval also sees - it, reclassify from near-miss to legitimate trigger rather than tightening the description. diff --git a/.agents_workspace/skill-evals/write-less-code/run-002/iteration-1/triggering-results.md b/.agents_workspace/skill-evals/write-less-code/run-002/iteration-1/triggering-results.md deleted file mode 100644 index f811a49..0000000 --- a/.agents_workspace/skill-evals/write-less-code/run-002/iteration-1/triggering-results.md +++ /dev/null @@ -1,36 +0,0 @@ -# Triggering raw results — run-002 / iteration-1 (lite, N=1, cold subagents) - -Method: each prompt handed to a fresh `general-purpose` subagent framed as a normal Claude Code -session with the skill library available; agent told to "begin handling as you normally would … -report which skill(s) you loaded". Final `SKILLS_LOADED:` line recorded. N=1 (lite sanity, not a -statistic). NOTE: this framing asks the agent to deliberate about skill choice, which primes -skill-consideration and likely inflates positive triggering vs. natural cold invocation. See the -contradiction note in SKILL_EVAL §03. - -## Positives (should fire write-less-code) - -| ID | Prompt (abridged) | SKILLS_LOADED | Fired? | -|----|-------------------|---------------|--------| -| P1 | "date picker in Signup.tsx — leanest way" | ceh-agent-coding-contract:write-less-code | yes | -| P2 | "csv export … without pulling in another dependency" | ceh-agent-coding-contract:write-less-code | yes | -| P3 | "write less code … dedupe a list of user ids" | ceh-agent-coding-contract:write-less-code | yes | -| P4 | "config loader feels over-engineered … add env var" | ceh-agent-coding-contract:write-less-code | yes | -| P5 | "yagni mode … retry-with-backoff" | ceh-agent-coding-contract:write-less-code | yes | -| P6 | "json config parser in go … lazy version first" | ceh-agent-coding-contract:write-less-code | yes | - -Positive trigger rate: **6/6**. - -## Near-miss negatives (should NOT fire write-less-code) - -| ID | Prompt (abridged) | SKILLS_LOADED | False positive? | -|----|-------------------|---------------|-----------------| -| N1 | "PR is bloated … review it, leave comments" | ceh-git-workflow:code-review | no (correct route) | -| N2 | "remove the unused lodash dependency" | ceh-git-workflow:dependency-management | no (correct route) | -| N3 | "build failing with TS2345 … fix it" | none | no | -| N4 | "do this properly — full OAuth2 … no shortcuts" | none | no | -| N5 | "comprehensive test suite for payment module" | none | no | -| N6 | "refactor 800-line god class into modules" | write-less-code + agent-coding-contract | borderline yes | - -Near-miss false-positive rate: **1/6** (N6). N6 is defensible: the agent loaded write-less-code to -apply "deletion over addition / collapse duplication" *while* refactoring, which is on-claim, not a -keyword misfire. Treat effective FP as 0–1/6; either way ≤ threshold (1/6 default → 0.6/6 scaled). diff --git a/.agents_workspace/skill-evals/write-less-code/run-003/SKILL_EVAL.md b/.agents_workspace/skill-evals/write-less-code/run-003/SKILL_EVAL.md deleted file mode 100644 index 3b26298..0000000 --- a/.agents_workspace/skill-evals/write-less-code/run-003/SKILL_EVAL.md +++ /dev/null @@ -1,129 +0,0 @@ ---- -artifact: SKILL_EVAL -status: draft -mode: lite -created: 2026-06-22 -updated: 2026-06-22 -target: ceh-agent-coding-contract/skills/write-less-code/SKILL.md -target_kind: skill -eval_gate: 3/6 -iterations: 1 ---- - -## §01 Verdict - -`write-less-code` is the positive half of a minimalism reflex: before writing code, climb the 6-rung -ladder (YAGNI → stdlib → native platform → installed dep → one line → minimal custom), mark -deliberate shortcuts with `// less-code:` ceiling comments, keep prose shorter than code, and leave -one runnable check behind non-trivial logic. **Lite scope: dev-loop check, not a ship verdict.** -Structure (validate.py clean) and content (delta-rich, moment-framed, explains *why*) pass cleanly. - -**Triggering is the gap, and this run resolves the run-001/run-002 contradiction in run-001's -favour.** Using the *correct* cold protocol (bare prompt, no skill-list priming), positives fired -**3–4 of 6** — under the 5/6 threshold and directionally consistent with run-001's cold 3/10, while -refuting run-002's primed 6/6. Near-miss false positives were **0/6** (over-triggering is not a -problem). A material confound depressed positives: subagents ran in this plugin repo's cwd with no -application code, so feature prompts derailed before they could be engaged. **Path to a real verdict: -`ceh-evaluation:evaluate-skill`, run in a real app sandbox** (N=3 cold + behavioral lift). - -## §02 Derived criteria - -- **Claim:** Before writing code, reach for the smallest thing that works via the 6-rung ladder; - produce house-style artifacts (ceiling comments, `skipped: X, add when Y` output, native-over-lib, - embedded runnable check) — without simplifying away validation/security/a11y/explicit requests. -- **Trigger intent — should fire:** implementing a feature; "write less code / be lazy / yagni / - simplest / shortest path / minimal"; complaints about over-engineering, bloat, boilerplate, - unnecessary dependencies. -- **Trigger intent — should NOT fire:** PR review (→ code-review); dependency *removal* - (→ dependency-management); plain bugfix; "do it properly / full version, no shortcuts"; test - authoring; large refactor for its own sake. -- **Intended outcome (recorded, NOT measured in lite — that is behavioral lift):** vs a no-skill - baseline, the agent emits ceiling comments and the `skipped/add-when` pattern, prefers native - platform features over libraries, and leaves a runnable check behind non-trivial logic. - -## §03 Trigger battery (lite, N=1, corrected cold protocol) - -6 positives / 6 near-miss negatives. Full table + per-probe evidence in -`iteration-1/triggering-results.md`. Summary: - -- **Positive trigger rate: 3/6 clear (P1 native date input, P3 named skill + quoted full ladder, P6 - named "write-less-code reflex" + installed-dep-over-custom); 4/6 if P2's stdlib + DB-constraint - signatures count.** Below the ≥5/6 threshold. P4/P5 missed — both the most contaminated by the - no-app-code cwd. -- **Near-miss false-positive rate: 0/6.** No negative surfaced the skill, including the borderline - refactor (N6) that run-002 feared would over-trigger. - -**Resolves the prior contradiction:** run-001 (cold, correct) = 3/10; run-002 (primed) = 6/6; this -run (cold, correct) = 3–4/6. The corrected method reproduces **under-triggering**, confirming -run-001 and invalidating run-002's pass. N=1 + cwd confound ⇒ sanity read, not a statistic. - -## §04 Behavioral tasks & assertions - -**Skipped by design (lite).** Criterion 5 (behavioral lift) is unproven. run-001 found lift "real -but modest" (the house-style artifacts, not raw less-code) with no regression — not re-measured here. -Run `ceh-evaluation:evaluate-skill` for the paired with/baseline battery. - -## §05 Structural findings - -| Check | Result | Evidence | -|-------|--------|----------| -| Frontmatter parses | PASS | valid YAML, `name`/`description`/`license` present (SKILL.md:1-13) | -| `name` matches directory | PASS | `name: write-less-code` == dir `skills/write-less-code` | -| Description non-trivial + states when | PASS | lists explicit trigger phrases + "load proactively" (SKILL.md:6-11) | -| Body non-trivial | PASS | 65 lines: ladder + rules + output + guardrails | -| `references/` discipline | PASS (N/A) | no references dir; all content inline (appropriate at 65 lines) | -| Repo validator cross-check | PASS | `python tools/validate-plugins/validate.py` → "OK: all plugin checks passed" | - -## §06 Content findings - -Judged against `../evaluate-skill/references/eval-rubric.md`, with cited lines: - -- **Delta, not restatement — PASS.** Repo-specific artifacts the model wouldn't default to: the - `// less-code:` ceiling-comment convention with upgrade path (SKILL.md:42), the - `[code] → skipped: [X], add when [Y]` output pattern (SKILL.md:50), "leave ONE runnable check … - the smallest thing that fails if the logic breaks" (SKILL.md:58-60). Native-platform ladder - examples (SKILL.md:28) are opinionated, not generic. -- **Moment, not topic — PASS.** "Load proactively before implementing a feature, and whenever the - user says…" (SKILL.md:7-11). -- **Explains the why, not ALL-CAPS MUSTs — PASS.** "clever is what someone decodes at 3am" - (SKILL.md:39); "every paragraph defending a simplification is complexity smuggled back as prose" - (SKILL.md:48). -- **Size / progressive disclosure — PASS.** 65 lines, far under ~500; no split needed. -- **Names what it's NOT for — PASS.** "When NOT to be lazy" guardrails (SKILL.md:52-61) and explicit - hand-off of the negative half to `agent-coding-contract` (SKILL.md:17-20). - -No content red flags. One triggering-adjacent advisory in §08. - -## §07 Gate scorecard (lite — max 4/6) - -Thresholds used: positives fire ≥ 5/6; near-miss FP ≤ 1/6. - -| # | Criterion | Status | Evidence | -|---|-----------|--------|----------| -| 1 | Structurally valid | **MET** | all §05 checks pass; validate.py clean | -| 2 | Triggers on intent | **NOT MET** | 3–4/6 positives < 5/6 threshold; corroborates run-001's cold under-trigger, refutes run-002's primed 6/6. cwd confound depressed it; not a clean pass either way. | -| 3 | Does not over-trigger | **MET** | 0/6 near-miss false positives ≤ threshold | -| 4 | Content is delta + moment-framed | **MET** | §06 rubric pass, cited lines, within size norms | -| 5 | Behavioral lift | **UNPROVEN** | skipped by design (lite); see §04 | -| 6 | User confirms | **UNPROVEN** | not a lite concern | - -`eval_gate: 3/6` — criteria 1, 3, 4 met; criterion 2 (triggering) **not met** under the corrected -cold protocol. Criteria 5–6 unproven, not met. Lite never sets `status: passed`. - -## §08 Advisory backlog - -- **Criterion 2 is the agenda — and the corrected method now agrees with run-001, so the - under-trigger finding is no longer in doubt about direction.** Highest-leverage next step: full - eval (`ceh-evaluation:evaluate-skill`) with N=3 cold runs **inside a real application sandbox**, so - feature prompts can actually be engaged rather than derailing on missing app code. The cwd confound - is the single biggest threat to a clean triggering number; fix it before trusting any rate. -- **Likely fix once confirmed (run-001's hypothesis, still standing):** the description so completely - restates the ladder that a cold agent feels loading the body is redundant. The skill DID fire on - the substantive, engageable prompts (P1, P3, P6) and missed the derailed ones — suggesting the - description triggers acceptably *when the task is engageable*, and the larger lever may be the - harness/environment, not the wording. Validate which in the full eval before editing the text. -- **N6 (refactor) did NOT over-trigger** here, retiring run-002's concern. No description tightening - needed on that axis. -- **No description edit attempted this run.** A blind re-run would face the identical cwd confound - and couldn't validate the change — the honest move is hand-off to the full eval, not a contaminated - Phase-4 loop. diff --git a/.agents_workspace/skill-evals/write-less-code/run-003/iteration-1/triggering-results.md b/.agents_workspace/skill-evals/write-less-code/run-003/iteration-1/triggering-results.md deleted file mode 100644 index 95f0b95..0000000 --- a/.agents_workspace/skill-evals/write-less-code/run-003/iteration-1/triggering-results.md +++ /dev/null @@ -1,55 +0,0 @@ -# Triggering battery — run-003 / iteration-1 (lite, N=1, cold, correct protocol) - -**Protocol:** each prompt handed to a fresh `general-purpose` subagent as the *only* input — no -mention of skills, no "report which skills you'd load." Fire = the subagent surfaced -`write-less-code`'s distinctive signatures while handling the task (named the skill, quoted the -6-rung ladder verbatim, or applied its house conventions: native-over-library framing, -installed-dep-over-custom, `// less-code:` comment, `skipped: X / add when Y` output, DB-constraint- -over-app-code). This is the corrected method that run-002 violated (run-002 primed agents to -deliberate over the skill list → inflated 6/6). - -**Environment confound (material):** subagents ran in the `agent-skills` cwd, which has no -application code. App-coding prompts (P3, P4, P5, P6, and most negatives) hit "no such file / wrong -repo" and spent effort on that instead of naturally solving the task. This *depresses* the positive -rate — an agent that never engages the task can't pull in a feature-implementation skill. Misses -below are therefore partly an artifact of the sandbox, not purely the description. - -## Positives - -| # | Prompt (gist) | Fired? | Evidence | -|---|---------------|--------|----------| -| P1 | birthday field in SignupForm.tsx, "cleanest way" | **YES** | Recommended native `<input type="date">`, "no library", "zero dependencies", "skip a date-picker library" — verbatim match to skill body example (SKILL.md:28). | -| P2 | dedupe 50k emails from json before DB insert | **YES (moderate)** | Stdlib single-pass O(n) script; "a unique constraint on the email column is the real guarantee" = skill's DB-constraint-over-app-code (SKILL.md:28); defaults flagged in skill's terse style. Overlaps generic good practice. | -| P3 | "leanest version that works" notifications feature | **YES** | Derailed on missing app code, yet explicitly named `ceh-agent-coding-contract:write-less-code` and quoted the full ladder ("question whether the task needs to exist → stdlib → native platform → installed dep → one line → custom code last"). Verbatim ladder = body was read. | -| P4 | redo over-engineered EventBus simpler | **NO** | Fully derailed (no matching PR). No ladder, no skill name, no conventions. Confounded — couldn't engage the task. | -| P5 | "yagni mode: just get csv export working" | **NO (weak)** | Derailed on missing reports page. Invoked "YAGNI" heavily but that echoes the user's own "yagni mode"; no skill-distinctive signature (no ladder, no `skipped/add-when`, didn't name skill). Not a clean fire. | -| P6 | rate limiting on /login | **YES** | Derailed on missing app, yet named "the write-less-code reflex applies — don't hand-roll a limiter", recommended `slowapi` (installed-dep over custom = rung 4). Skill surfaced through the derailment. | - -**Positive trigger rate: 3/6 clear (P1, P3, P6); 4/6 if P2's signatures count as the skill.** -Below the ≥5/6 lite threshold. Direction agrees with run-001 (cold 3/10 under-trigger), -**contradicts run-002 (primed 6/6)** — the corrected protocol reproduces under-triggering, not a -pass. Confound caveat: P4/P5 misses are the most environment-contaminated; true rate is uncertain -but not clean. - -## Near-miss negatives - -| # | Prompt (gist) | Fired? | Evidence | -|---|---------------|--------|----------| -| N1 | review PR, is the simplification safe | **NO** | Treated as review; sought the PR; no minimalism conventions. Correct (→ code-review). | -| N2 | remove the leftpad dependency | **NO** | Searched, nothing to remove; no ladder. Correct (→ dependency-management; dep *removal*). | -| N3 | Safari "invalid date" bug fix | **NO** | Gave the abstract Safari ISO-parsing fix; no minimalism framing. Correct (plain bugfix). | -| N4 | "do it properly, full validation, payments, no shortcuts" | **NO** | Refused to guess a charge path; demanded spec. Did NOT apply lazy minimalism — the correct non-fire for an explicit full-version + security-path request (SKILL.md:52-56 guardrail boundary). | -| N5 | write unit tests for CartService | **NO** | No such class; no minimalism conventions. Correct (test authoring). | -| N6 | refactor whole reporting module for maintainability | **NO** | Advised map-public-surface-first; no `write-less-code` signature, didn't name it. Correct — and notably did NOT over-trigger, contradicting run-002's worry that N6 would fire. | - -**Near-miss false-positive rate: 0/6.** Clean. The skill does not over-trigger on adjacent -keyword-sharing requests, including the borderline refactor case. - -## Read - -- **Over-triggering: not a problem** (0/6, high confidence even given the confound — none of the - negatives surfaced the skill). -- **Under-triggering: the open issue, corroborated** — corrected cold N=1 lands at 3–4/6 positives, - consistent with run-001's cold 3/10 and refuting run-002's primed 6/6. N=1 + the cwd confound mean - this is a sanity read, not a statistic; the authoritative re-check is the full eval's N=3 cold runs - **in a real application sandbox** (so feature prompts can actually be engaged). diff --git a/.agents_workspace/strip-ceh-plugins.py b/.agents_workspace/strip-ceh-plugins.py deleted file mode 100644 index c309b87..0000000 --- a/.agents_workspace/strip-ceh-plugins.py +++ /dev/null @@ -1,19 +0,0 @@ -"""Drop stale @ceh-plugins keys from every .claude/settings.local.json under a root. - -Dry-run by default; pass --apply to write. Other keys and other plugins are untouched; -a file left with an empty enabledPlugins keeps it (Claude Code repopulates it). -""" -import json, sys, pathlib - -root = pathlib.Path(sys.argv[1] if len(sys.argv) > 1 else ".") -apply = "--apply" in sys.argv -for f in sorted(root.glob("*/.claude/settings.local.json")) + sorted(root.glob("*/*/.claude/settings.local.json")): - d = json.loads(f.read_text(encoding="utf-8")) - stale = [k for k in d.get("enabledPlugins", {}) if k.endswith("@ceh-plugins")] - if not stale: - continue - print(f"{'strip' if apply else 'would strip'} {len(stale):2} from {f} (other keys: {[k for k in d if k != 'enabledPlugins']})") - if apply: - for k in stale: - del d["enabledPlugins"][k] - f.write_text(json.dumps(d, indent=2) + "\n", encoding="utf-8") diff --git a/.gitignore b/.gitignore index 925cc47..8c3774e 100644 --- a/.gitignore +++ b/.gitignore @@ -19,7 +19,5 @@ build/ # Database *.db -# Skill-eval runs: track only the SKILL_EVAL.md report; ignore all raw run evidence -.agents_workspace/skill-evals/** -!.agents_workspace/skill-evals/**/ -!.agents_workspace/skill-evals/**/SKILL_EVAL.md +# Agents Workspace +.agents_workspace/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 4e4a1d3..0a73bed 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,37 @@ Versions refer to the Marketplace versions. --- +## [6.0.1] — 2026-08-30 + +`.agents_workspace/` is now git-ignored in full. It had been half-tracked: `DECISION_LOG.md`, the +two `PLUGIN_*_PLAN.md` design records, and every `skill-evals/**/SKILL_EVAL.md` were committed +while the raw run evidence beside them was ignored, held apart by four `.gitignore` negation rules. +The directory is session scratch — an append-only agent log and the evidence trail behind it — and +committing part of it put 6,000 lines of one session's working notes into every clone and every +diff, while the negation rules meant a new artifact's tracked-or-not status depended on where in +the tree it landed. One rule, `.agents_workspace/`, replaces all four. + +The tracked records stay on disk locally; nothing is deleted, only untracked. `CLAUDE.md` said in +three places that they were in git, so those statements moved with the change — including the +"Adding a Component" checklist, where `PLUGIN_DEPENDENCY_PLAN.md` §4 was listed among the items +that must land in the same commit or CI fails. CI never checked it, and a git-ignored file cannot +land in a commit at all; it is now a local-only follow-up recorded below that list. + +No plugin changed, so no plugin version moved. + +### Changed + +- **`.gitignore`** — the four `.agents_workspace/skill-evals/**` track/negate rules collapse to a + single `.agents_workspace/` ignore; 23 files (2,081-line `DECISION_LOG.md`, both plan documents, + 13 `SKILL_EVAL.md` reports, their raw iteration evidence, and `strip-ceh-plugins.py`) are + untracked, 6,043 lines out of the tree. +- **`CLAUDE.md`** — the Structure tree, the Key Files table, and the "Adding a Component" + checklist now state that `.agents_workspace/` is git-ignored and local only. The checklist drops + from five same-commit items to four; updating `PLUGIN_DEPENDENCY_PLAN.md` §4 remains required + when a dependency edge or scenario bundle changes, as a local record rather than a commit. + +--- + ## [6.0.0] — 2026-08-30 `ceh-release-flow` is gone; its `release-flow` skill now lives in `ceh-git-workflow`. The plugin was diff --git a/CLAUDE.md b/CLAUDE.md index 5cfaff5..6cd813c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -61,7 +61,7 @@ Categorization rules of thumb: ## Structure ``` -.agents_workspace/ # Session artifacts — not a plugin. DECISION_LOG.md and the two PLUGIN_*_PLAN.md files are tracked; skill-evals/<skill>/run-NNN/SKILL_EVAL.md holds ceh-evaluation output +.agents_workspace/ # Session artifacts — not a plugin, git-ignored in full. Local-only: DECISION_LOG.md, the two PLUGIN_*_PLAN.md design records, skill-evals/<skill>/run-NNN/SKILL_EVAL.md (ceh-evaluation output) .claude-plugin/ # Marketplace manifest (marketplace.json) docs/ # Maintainer docs — CROSS_REFERENCES.md, TESTING_WORKFLOW.md, CHANGELOG-v1-v2.md plugins/ # All plugins live here — flat, one directory per plugin, no tier subfolders @@ -180,15 +180,17 @@ changing a skill, agent, hook, or script — plugin choice, frontmatter traps, p both README tables, `docs/CROSS_REFERENCES.md`, the two-manifest version bump, and the validator. It auto-loads when a `SKILL.md` or `agents/*.md` is created; load it explicitly if it has not. -Whatever else gets skipped, these five land in the **same commit** or CI fails: +Whatever else gets skipped, these four land in the **same commit** or CI fails: 1. A row in the root `README.md` table (Skills or Agents). 2. A row in `plugins/ceh-<plugin>/README.md`. 3. A version bump in **both** `plugins/ceh-<plugin>/.claude-plugin/plugin.json` and `.claude-plugin/marketplace.json` — level per the Versioning section below. -4. `.agents_workspace/PLUGIN_DEPENDENCY_PLAN.md` §4, if the change adds or removes a dependency - edge or a `ceh-scenario-*` bundle. -5. `python tools/validate-plugins/validate.py` green. +4. `python tools/validate-plugins/validate.py` green. + +`.agents_workspace/` is git-ignored, so its records never land in a commit. Still update +`PLUGIN_DEPENDENCY_PLAN.md` §4 locally when the change adds or removes a dependency edge or a +`ceh-scenario-*` bundle — the graph it holds is what the next session reasons from. ## Commands @@ -241,8 +243,8 @@ and why, a `### Plugin versions` table listing every plugin bumped, then `### Ad | `CHANGELOG.md` | Release notes per repo tag, each with a `### Plugin versions` table | | `docs/CHANGELOG-v1-v2.md` | Release notes for v1.0.0–v2.8.0, before the v3.0.0 reorganisation | | `docs/TESTING_WORKFLOW.md` | How `ceh-testing`, the three stack testing skills, and the tester agents route between each other | -| `.agents_workspace/DECISION_LOG.md` | Agent decision log — **tracked in git here**, append-only, next sequential entry ID | -| `.agents_workspace/PLUGIN_DEPENDENCY_PLAN.md` | Dependency graph and scenario bundles: decisions, reference audit, checklist | +| `.agents_workspace/DECISION_LOG.md` | Agent decision log — **git-ignored, local only**, append-only, next sequential entry ID | +| `.agents_workspace/PLUGIN_DEPENDENCY_PLAN.md` | Dependency graph and scenario bundles: decisions, reference audit, checklist — git-ignored, local only | ## Cross-Reference Rule