From 86295e9ff9adea8eb745aed82df08af858787d99 Mon Sep 17 00:00:00 2001 From: Jon Langevin Date: Tue, 30 Jun 2026 05:40:26 -0400 Subject: [PATCH 01/16] Design brainstorm visualization --- .../0005-visual-companion-instructions.md | 33 +++++++ ...6-06-30-brainstorm-visualization-design.md | 96 +++++++++++++++++++ 2 files changed, 129 insertions(+) create mode 100644 decisions/0005-visual-companion-instructions.md create mode 100644 docs/plans/2026-06-30-brainstorm-visualization-design.md diff --git a/decisions/0005-visual-companion-instructions.md b/decisions/0005-visual-companion-instructions.md new file mode 100644 index 0000000..5f39011 --- /dev/null +++ b/decisions/0005-visual-companion-instructions.md @@ -0,0 +1,33 @@ +# 0005. Add visual companion instructions + +**Status:** Accepted +**Date:** 2026-06-30 +**Decision-makers:** ADK maintainer via issue #78; implementing agent +**Related:** docs/plans/2026-06-30-brainstorm-visualization-design.md, https://github.com/GoCodeAlone/autonomous-dev-kit/issues/78 + +## Context + +Issue #78 asks ADK to add a brainstorm/design visualization companion similar to the updated `obra/superpowers` brainstorming skill. ADK skills must stay host-neutral and portable across supported agents. This repo does not currently include the upstream browser companion server or its scripts. + +## Decision + +We will add host-neutral visual-companion guidance to `skills/brainstorming/SKILL.md`, backed by a local `skills/brainstorming/visual-companion.md` guide, because this satisfies the requested workflow without introducing an unproven runtime dependency. + +**Alternatives considered and rejected:** + +- **Copy upstream browser server contract verbatim** — rejected because ADK does not ship those scripts and host process semantics differ. +- **Add diagrams unconditionally to every design** — rejected because many brainstorm questions are textual; mandatory diagrams add noise and token cost. + +## Consequences + +**Positive:** + +- Brainstorming can offer diagrams/mockups just-in-time when visual reasoning helps. +- Guidance remains portable and can use host-native rendering such as Mermaid before any future server exists. + +**Negative:** + +- This does not add a browser event loop or click-state capture. +- Future server support will need a separate design and runtime validation. + +**Reversibility:** revert the brainstorming skill edits and companion guide; no migrations or generated runtime state are involved. diff --git a/docs/plans/2026-06-30-brainstorm-visualization-design.md b/docs/plans/2026-06-30-brainstorm-visualization-design.md new file mode 100644 index 0000000..0c7bff9 --- /dev/null +++ b/docs/plans/2026-06-30-brainstorm-visualization-design.md @@ -0,0 +1,96 @@ +# Brainstorm Visualization Design + +**Issue:** #78 — Brainstorm visualization +**Date:** 2026-06-30 +**Status:** Approved by user instruction; design recorded before implementation +**ADR:** `decisions/0005-visual-companion-instructions.md` + +## Original Ask + +> At https://github.com/obra/superpowers, their skill has been updated to add a visualization companion for brainstorms/designs, with diagrams, etc. This is quite handy, we should add something similar. + +## Global Design Guidance + +`Guidance: none found as docs/design-guidance.md; constraints cited from repo canon.` + +| source | guidance | design response | +|---|---|---| +| `AGENTS.md` | Skills use host-gated content; forbidden host-specific tokens only in host blocks; recognized hosts include `zed-agent`. | Add host-neutral visual guidance; avoid new host-specific tool names outside host blocks. | +| `README.md` Cross-LLM/plugin positioning | ADK bundles portable skills/hooks/agents/commands for multiple hosts. | Prefer Mermaid/markdown and generic visual artifacts over an upstream-only browser server dependency. | +| `skills/brainstorming/SKILL.md` | Brainstorming already has adaptive batching, assumptions, self-challenge, adversarial review, and design docs. | Insert visual companion as a just-in-time optional aid without weakening existing gates. | + +## Approaches Considered + +| option | summary | trade-off | +|---|---|---| +| A. Copy upstream browser companion verbatim | Add upstream `Visual Companion` text and server instructions. | Fast but references scripts/tools ADK does not ship; likely host-portability regressions. | +| B. Host-neutral progressive companion | Add just-in-time visual-offer rules, diagram/mockup decision test, host-native fallback, and companion guide. | Solves issue now; does not provide browser click capture. Recommended. | +| C. Build full browser companion runtime | Add scripts, frame template, event state, and docs. | Larger runtime feature; requires multi-host process validation beyond issue scope. | + +**Decision:** B. See `decisions/0005-visual-companion-instructions.md`. + +## Design + +- Modify `skills/brainstorming/SKILL.md`: + - Add checklist item after project guidance: offer visual companion just-in-time, not upfront. + - Update process-flow diagram to include the visual-companion decision. + - Add process bullets explaining when to offer/use visuals. + - Add `## Visual Companion` section with per-question decision rules. +- Add `skills/brainstorming/visual-companion.md`: + - Covers when visuals help, artifact types, lightweight Mermaid examples, mockup guidance, accessibility/plain-text fallback, and host limitations. + - Explicitly says visuals are a tool, not a mode; textual questions stay in chat. +- Add regression assertions to `tests/pipeline-evidence-doc-sync.sh` for #78: + - `skills/brainstorming/SKILL.md` mentions `Visual Companion`. + - The skill requires `just-in-time` offering. + - The guide exists and mentions `Mermaid`. + +## Security Review + +| area | assessment | +|---|---| +| Auth/authz | No auth boundary changes. | +| Secrets/PII | Visual guidance must not expose secrets/PII; guide will require redacted examples. | +| Dependency/trust | No new dependency or network service. | +| Abuse case | Avoid instructing agents to open browsers/servers by default; visuals are opt-in and can be represented in markdown. | +| Least privilege | No filesystem writes except intentional design artifacts during brainstorming. | + +## Infrastructure Impact + +| resource | impact | +|---|---| +| Cloud/network | None. | +| Local runtime | None; no server added. | +| Storage | Adds one skill guide markdown file plus design/ADR/plan artifacts. | +| Cost/scale | Slight token cost only when visual companion is used. | +| Release/rollback | Revert markdown/test changes; no migrations. | + +## Multi-Component Validation + +| boundary | proof | +|---|---| +| Skill text ↔ content rules | `bash tests/skill-content-grep.sh` → host-neutral token guard passes. | +| Skill refs ↔ repo files | `bash tests/skill-cross-refs.sh` → references resolve. | +| #78 regression | `bash tests/pipeline-evidence-doc-sync.sh` → visual companion assertions pass. | +| Full repo contract | Run AGENTS.md test list before PR. | + +## Assumptions + +| id | assumption | challenge | fallback | +|---|---|---|---| +| A1 | Issue #78 wants workflow guidance, not necessarily a shipped browser server. | User may expect click-driven browser companion parity. | ADR scopes this to host-neutral instructions; future server runtime can be separate issue. | +| A2 | Mermaid/markdown diagrams are available or acceptable across enough hosts. | Some hosts render only plain text. | Guide requires plain-text summary/fallback for every visual. | +| A3 | Adding one companion guide is consistent with skill docs shape. | Existing skills mostly keep logic in SKILL.md. | Keep SKILL.md authoritative; guide is referenced for detailed examples only. | + +## Self-Challenge + +| doubt | response | +|---|---| +| Laziest solution: paste upstream section only. | Rejected: it references scripts ADK does not ship and weakens host neutrality. | +| Fragile assumption: no full browser runtime required. | Mitigated by explicit scope and ADR; issue says "something similar," not exact parity. | +| YAGNI risk: guide becomes too broad. | Keep examples generic; no runtime implementation, no click-event schema. | +| Failure mode: agent overuses diagrams. | Just-in-time/per-question rules say never offer upfront and use visuals only when clearer than text. | +| Repo pattern conflict: skill process gates. | Visual step is inserted before questions and does not skip project guidance, assumptions, review, or planning gates. | + +## Rollback + +Revert commits modifying `skills/brainstorming/SKILL.md`, `skills/brainstorming/visual-companion.md`, and `tests/pipeline-evidence-doc-sync.sh`; rerun AGENTS.md test list. No runtime state, migrations, or version pins. From c2415de0beeda46df54d1b7087bbd7ebf7966c8a Mon Sep 17 00:00:00 2001 From: Jon Langevin Date: Tue, 30 Jun 2026 05:49:22 -0400 Subject: [PATCH 02/16] Address brainstorm visualization design review --- ...-brainstorm-visualization-design-review.md | 43 +++++++ ...6-06-30-brainstorm-visualization-design.md | 105 +++++++++++++----- 2 files changed, 119 insertions(+), 29 deletions(-) create mode 100644 docs/plans/2026-06-30-brainstorm-visualization-design-review.md diff --git a/docs/plans/2026-06-30-brainstorm-visualization-design-review.md b/docs/plans/2026-06-30-brainstorm-visualization-design-review.md new file mode 100644 index 0000000..73e2fda --- /dev/null +++ b/docs/plans/2026-06-30-brainstorm-visualization-design-review.md @@ -0,0 +1,43 @@ +### Adversarial Review Report +**Phase:** design +**Artifact:** docs/plans/2026-06-30-brainstorm-visualization-design.md +**Status:** FAIL + +**Findings (Critical):** +- None. + +**Findings (Important):** +- `D1` [Repo-precedent conflicts / Multi-component validation] [`docs/plans/2026-06-30-brainstorm-visualization-design.md:42-45`, `:67-75`]: The validation strategy is grep-only for a behavior-changing skill edit. Repo precedent says skill edits follow TDD for process docs: "NO SKILL WITHOUT A FAILING TEST FIRST" and "This applies to NEW skills AND EDITS to existing skills" (`skills/writing-skills/SKILL.md:461-480`), with application/retrieval scenarios for technique/reference skills (`:482-529`). The design only asserts strings like `Visual Companion`, `just-in-time`, and `Mermaid`, which does not prove agents offer the companion at the right time, avoid offering it upfront, choose text vs visual per question, or load/use the guide. Recommendation: add a RED/GREEN skill-behavior scenario or pressure test for brainstorming before implementation, then prove post-change behavior with a representative transcript/eval; keep grep checks as regression guards, not primary proof. +- `D2` [Declared integration proof / User-intent drift / Existence-runtime-validity] [`docs/plans/2026-06-30-brainstorm-visualization-design.md:24-30`, `:78-82`]: The design replaces upstream’s browser-based visual companion with host-native Markdown/Mermaid guidance but does not define a host integration matrix or prove the real consumer surface. Upstream’s current `skills/brainstorming/SKILL.md` describes a browser companion that opens a tab, starts a server, and records browser events; upstream README also documents visual companion telemetry. ADK’s `docs/cross-llm-coverage.md:7-20` has no visual/Mermaid rendering capability row, and `tests/cross-llm-coverage.md:13` only tracks brainstorming’s question-tool host conditionals. Recommendation: explicitly state the parity boundary as an accepted non-goal, add a host-by-host matrix marking Markdown/Mermaid/browser behavior as `runtime-integrated`, `config-only`, or `deferred`, and require at least one real-host rendering proof or mark rendering as best-effort text-only fallback. +- `D3` [Missing failure modes / Assumptions under attack] [`docs/plans/2026-06-30-brainstorm-visualization-design.md:39-45`, `:80-82`, `:90-91`]: The design covers “agent overuses diagrams” and plain-text fallback, but misses likely first failures: invalid Mermaid syntax, hosts that display Mermaid as raw code, diagrams that contradict the text design, stale visual artifacts after the conversation moves on, and inaccessible visuals where the fallback is lower-fidelity than the decision being made. Recommendation: add explicit failure rules: text remains the source of truth, every visual must include a concise text equivalent, agents must fall back when rendering is unavailable or invalid, and visual artifacts must be updated or retired when decisions change. + +**Findings (Minor):** +- `D4` [Artifact-class precedent / YAGNI violations / Simpler alternative not considered] [`docs/plans/2026-06-30-brainstorm-visualization-design.md:39-41`, `:78-83`]: The design adds `skills/brainstorming/visual-companion.md` without justifying that it is heavy reference material. Skill-authoring precedent says supporting files are “Only if needed” (`skills/writing-skills/SKILL.md:74-82`), keep principles/concepts and code patterns under 50 lines inline (`:86-93`), and use heavy reference files when material is too large (`:451-459`). Recommendation: either make the initial change a short inline `## Visual Companion` section, or bound the guide’s role/size and explain why it is a heavy reference rather than SKILL.md content. +- `D5` [Repo-precedent conflicts] [`docs/plans/2026-06-30-brainstorm-visualization-design.md:36-40`]: The design says the guide will include Mermaid examples while existing skill process diagrams use `dot`, and `skills/writing-skills/SKILL.md:377-405` gives Graphviz/dot flowchart guidance plus render tooling for skill diagrams. If Mermaid is only for user-facing generated artifacts, that distinction is not explicit. Recommendation: state that SKILL.md process flow remains `dot`; Mermaid examples are only optional output artifacts shown to users where supported. +- `D6` [Infrastructure impact / Artifact-class precedent] [`docs/plans/2026-06-30-brainstorm-visualization-design.md:42-45`, `tests/pipeline-evidence-doc-sync.sh:1-4`, `:68-91`]: `tests/pipeline-evidence-doc-sync.sh` is currently framed as a regression guard for pipeline evidence issues #69/#70/#71/#72 and later #79, not a general feature-behavior test bucket. Adding #78 visual assertions there may blur the script’s contract and make future maintenance less obvious. Recommendation: either update the script header/sectioning to include #78 explicitly or create a focused skill-behavior/content regression test for brainstorming visualization. + +**Bug-class scan transcript:** +| Class | Result | Note | +|---|---|---| +| Project-guidance conflicts | Clean | No `docs/design-guidance.md` exists per prompt; the design cites repo canon from `AGENTS.md`, `README.md`, and `skills/brainstorming/SKILL.md`, which is an acceptable substitute here. | +| Assumptions under attack | Finding | `D2` and `D3`: the assumptions that “workflow guidance” is enough and Mermaid/text fallback is acceptable are load-bearing and insufficiently proven. | +| Repo-precedent conflicts | Finding | `D1` conflicts with `skills/writing-skills/SKILL.md` TDD-for-skill-edits precedent; `D5` risks conflicting with dot/Graphviz skill-diagram precedent. | +| Artifact-class precedent | Finding | `D4` questions the unproven need for a sibling guide file; `D6` questions using `pipeline-evidence-doc-sync.sh` for this feature’s regression assertions. | +| YAGNI violations | Finding | `D4`: a separate companion guide may be more structure than needed for the first host-neutral version. | +| Missing failure modes | Finding | `D3`: invalid Mermaid, non-rendering hosts, stale/contradictory visuals, and accessibility degradation are not designed through. | +| Security/privacy architecture | Clean | Current scoped design avoids browser server, network service, dependencies, and says visual examples must not expose secrets/PII; future browser parity would need a new privacy decision, especially because upstream documents telemetry. | +| Infrastructure impact | Finding | `D6`: no cloud/runtime infra is added, but the test-suite impact is under-specified for a script with an existing narrower contract. | +| Multi-component validation | Finding | `D1` and `D2`: grep checks do not prove the skill behavior or host rendering boundary. | +| Declared integration proof | Finding | `D2`: host-native Markdown/Mermaid rendering is treated as available/acceptable without a host integration matrix or real-host proof. | +| Contributed UI rendering proof | Clean | The design does not register a UI route/widget into a host shell; this class is not directly applicable unless browser companion/runtime work is reintroduced. | +| Rollback story | Clean | `docs/plans/2026-06-30-brainstorm-visualization-design.md:94-97` gives a concrete markdown/test revert path with no migrations or runtime state. | +| Simpler alternative not considered | Finding | `D4`: the design considers copying upstream and building a runtime, but not a minimal inline-only section that avoids a new sidecar guide. | +| User-intent drift | Finding | `D2`: the user pointed at upstream’s visualization companion; the design may underdeliver by shipping chat/Markdown guidance unless the reduced parity boundary is explicitly accepted. | +| Existence/runtime-validity | Finding | `D2`: the design does not verify the emitted visual artifacts target a real rendering/consumer surface across ADK-supported hosts. | + +**Options the author may not have considered:** +1. **Inline-only MVP:** Add a concise `## Visual Companion` section directly to `skills/brainstorming/SKILL.md` with just-in-time offer rules, per-question visual/text decision rules, and plain-text fallback. This is smaller, follows the “keep inline” precedent, and can be behavior-tested before introducing a heavier guide. +2. **Two-tier parity design:** Ship host-neutral Markdown/Mermaid guidance now, but explicitly add a deferred “browser companion parity” track with acceptance criteria copied from upstream: browser launch, session security, event capture, telemetry policy, cleanup, and multi-host process behavior. This prevents the current PR from being mistaken for full upstream parity. +3. **Capability-matrix-first approach:** Before changing brainstorming behavior, update `docs/cross-llm-coverage.md` and `tests/cross-llm-coverage.md` with visual-rendering capabilities per host. Then make the skill branch on that matrix: rendered Mermaid where proven, Markdown/text fallback elsewhere. + +**Verdict reasoning:** The design is directionally reasonable but not ready to pass adversarial review because its validation does not prove the behavioral skill change, and its scope reduction from upstream’s browser companion to Markdown/Mermaid guidance is not backed by a real host capability matrix or explicit accepted parity boundary. These are Important findings because they can cause the implementation to pass grep tests while failing the actual user-visible goal: agents reliably offering and using useful brainstorm/design visuals across ADK-supported hosts. diff --git a/docs/plans/2026-06-30-brainstorm-visualization-design.md b/docs/plans/2026-06-30-brainstorm-visualization-design.md index 0c7bff9..98e2b00 100644 --- a/docs/plans/2026-06-30-brainstorm-visualization-design.md +++ b/docs/plans/2026-06-30-brainstorm-visualization-design.md @@ -2,7 +2,7 @@ **Issue:** #78 — Brainstorm visualization **Date:** 2026-06-30 -**Status:** Approved by user instruction; design recorded before implementation +**Status:** Approved by user instruction; revised after adversarial review cycle 1 **ADR:** `decisions/0005-visual-companion-instructions.md` ## Original Ask @@ -16,16 +16,40 @@ | source | guidance | design response | |---|---|---| | `AGENTS.md` | Skills use host-gated content; forbidden host-specific tokens only in host blocks; recognized hosts include `zed-agent`. | Add host-neutral visual guidance; avoid new host-specific tool names outside host blocks. | -| `README.md` Cross-LLM/plugin positioning | ADK bundles portable skills/hooks/agents/commands for multiple hosts. | Prefer Mermaid/markdown and generic visual artifacts over an upstream-only browser server dependency. | -| `skills/brainstorming/SKILL.md` | Brainstorming already has adaptive batching, assumptions, self-challenge, adversarial review, and design docs. | Insert visual companion as a just-in-time optional aid without weakening existing gates. | +| `README.md` Cross-LLM/plugin positioning | ADK bundles portable skills/hooks/agents/commands for multiple hosts. | Prefer host-neutral Markdown plus optional Mermaid where rendered; never require an upstream-only browser server. | +| `skills/brainstorming/SKILL.md` | Brainstorming has adaptive batching, assumptions, self-challenge, adversarial review, and design docs. | Insert visual companion as a just-in-time optional aid without weakening gates. | +| `skills/writing-skills/SKILL.md` | Skill edits require failing tests first and supporting files only when useful. | Add a focused RED/GREEN shell test for visual-companion behavior markers; keep SKILL.md authoritative and the companion guide bounded to examples/fallback rules. | + +## Parity Boundary + +| upstream capability | ADK issue #78 scope | rationale | +|---|---|---| +| Browser tab companion | Deferred | ADK does not ship the server/scripts; adding them needs runtime/process/security validation across hosts. | +| Click/event capture | Deferred | Requires browser session state and privacy policy; out of scope for this markdown-only PR. | +| Diagrams/mockups during brainstorm/design | In scope | Add just-in-time visual guidance using host-native chat/markdown artifacts. | +| Text fallback | Required | Host rendering differs; text remains source of truth. | + +## Host Visual Capability Matrix + +| host | Markdown text | Mermaid/rendered diagrams | Browser companion | +|---|---|---|---| +| `zed-agent` | runtime-integrated: markdown chat output | runtime-integrated: Zed renders Mermaid in assistant markdown when supported; must include text fallback | deferred | +| `claude-code` | config-only: markdown output | config-only/best-effort: render support varies by surface | deferred | +| `codex` | config-only: markdown output | config-only/best-effort | deferred | +| `opencode` | config-only: markdown output | config-only/best-effort | deferred | +| `cursor` | config-only: markdown output | config-only/best-effort | deferred | +| `hermes-agent` | config-only: markdown output | config-only/best-effort | deferred | + +`config-only` rows mean the skill can emit markdown/diagram code, but this PR does not prove host rendering beyond plain text. ∴ every visual must have a concise textual equivalent and decisions must be recorded in text. ## Approaches Considered | option | summary | trade-off | |---|---|---| -| A. Copy upstream browser companion verbatim | Add upstream `Visual Companion` text and server instructions. | Fast but references scripts/tools ADK does not ship; likely host-portability regressions. | -| B. Host-neutral progressive companion | Add just-in-time visual-offer rules, diagram/mockup decision test, host-native fallback, and companion guide. | Solves issue now; does not provide browser click capture. Recommended. | -| C. Build full browser companion runtime | Add scripts, frame template, event state, and docs. | Larger runtime feature; requires multi-host process validation beyond issue scope. | +| A. Copy upstream browser companion verbatim | Add upstream `Visual Companion` text and server instructions. | Fast but references scripts/tools ADK does not ship; host-portability and security gaps. | +| B. Host-neutral progressive companion | Add just-in-time offer rules, diagram/mockup decision test, failure rules, host matrix, and bounded guide. | Solves issue now; browser click capture remains deferred. Recommended. | +| C. Inline-only MVP | Put all guidance in `SKILL.md`, no guide file. | Smallest patch but clutters an already-long skill and lacks room for examples/fallbacks. | +| D. Full browser companion runtime | Add scripts, frame template, event state, and docs. | Larger runtime feature; requires multi-host launch validation beyond issue scope. | **Decision:** B. See `decisions/0005-visual-companion-instructions.md`. @@ -33,26 +57,37 @@ - Modify `skills/brainstorming/SKILL.md`: - Add checklist item after project guidance: offer visual companion just-in-time, not upfront. - - Update process-flow diagram to include the visual-companion decision. - - Add process bullets explaining when to offer/use visuals. - - Add `## Visual Companion` section with per-question decision rules. -- Add `skills/brainstorming/visual-companion.md`: - - Covers when visuals help, artifact types, lightweight Mermaid examples, mockup guidance, accessibility/plain-text fallback, and host limitations. - - Explicitly says visuals are a tool, not a mode; textual questions stay in chat. -- Add regression assertions to `tests/pipeline-evidence-doc-sync.sh` for #78: - - `skills/brainstorming/SKILL.md` mentions `Visual Companion`. - - The skill requires `just-in-time` offering. - - The guide exists and mentions `Mermaid`. + - Keep process-flow diagram in `dot` to match existing skill precedent. + - Add process bullets: offer visuals only when a question is genuinely clearer shown than described. + - Add `## Visual Companion` section: per-question decision rules, text-source-of-truth, stale/invalid visual fallback, and guide reference. +- Add bounded `skills/brainstorming/visual-companion.md`: + - Role: reference examples and failure/accessibility rules only; `SKILL.md` remains authoritative. + - Covers diagrams/mockups, Mermaid examples, visual/text decision test, no-secrets rule, stale visual retirement, and plain-text fallback. +- Add focused regression test `tests/brainstorm-visual-companion.sh`: + - RED before implementation: `skills/brainstorming/SKILL.md` lacks `Visual Companion`, `just-in-time`, and guide reference; guide file missing. + - GREEN after implementation: asserts just-in-time offer, not-upfront rule, per-question decision, text fallback/source of truth, guide existence, and guide mentions Mermaid plus accessibility/fallback. +- Keep `tests/pipeline-evidence-doc-sync.sh` unchanged; do not overload its issue-specific contract. + +## Failure Rules + +| failure | required behavior | +|---|---| +| Mermaid invalid or unsupported | Continue with plain text; never claim rendered proof unless observed. | +| Visual contradicts text | Text design/plan is source of truth; fix or withdraw visual before proceeding. | +| Visual becomes stale after decision changes | Retire/update the visual; do not leave obsolete artifact as active guidance. | +| Accessibility or remote-host limitation | Include concise text equivalent for every visual and accept text-only feedback. | +| User declines visual companion | Continue text-only; do not offer again unless user raises visuals. | +| Sensitive data in visual | Redact or use synthetic examples; no secrets/PII in diagrams/mockups. | ## Security Review | area | assessment | |---|---| | Auth/authz | No auth boundary changes. | -| Secrets/PII | Visual guidance must not expose secrets/PII; guide will require redacted examples. | -| Dependency/trust | No new dependency or network service. | -| Abuse case | Avoid instructing agents to open browsers/servers by default; visuals are opt-in and can be represented in markdown. | -| Least privilege | No filesystem writes except intentional design artifacts during brainstorming. | +| Secrets/PII | Guide requires redacted/synthetic visual examples; no secrets/PII. | +| Dependency/trust | No dependency, network service, telemetry, or browser session state. | +| Abuse case | Avoid opening browsers/servers by default; visuals are optional and text-backed. | +| Least privilege | No filesystem writes except normal design artifacts; no runtime process. | ## Infrastructure Impact @@ -60,7 +95,7 @@ |---|---| | Cloud/network | None. | | Local runtime | None; no server added. | -| Storage | Adds one skill guide markdown file plus design/ADR/plan artifacts. | +| Storage | Adds one bounded skill guide, one focused test, plus design/ADR/plan artifacts. | | Cost/scale | Slight token cost only when visual companion is used. | | Release/rollback | Revert markdown/test changes; no migrations. | @@ -68,29 +103,41 @@ | boundary | proof | |---|---| +| Skill behavior markers | `bash tests/brainstorm-visual-companion.sh` → asserts just-in-time/not-upfront/per-question/text fallback/guide contract. | | Skill text ↔ content rules | `bash tests/skill-content-grep.sh` → host-neutral token guard passes. | | Skill refs ↔ repo files | `bash tests/skill-cross-refs.sh` → references resolve. | -| #78 regression | `bash tests/pipeline-evidence-doc-sync.sh` → visual companion assertions pass. | | Full repo contract | Run AGENTS.md test list before PR. | +| Runtime rendering | Deferred except Zed markdown/Mermaid support where observed by host; all other hosts use text fallback and are marked config-only/best-effort. | ## Assumptions | id | assumption | challenge | fallback | |---|---|---|---| -| A1 | Issue #78 wants workflow guidance, not necessarily a shipped browser server. | User may expect click-driven browser companion parity. | ADR scopes this to host-neutral instructions; future server runtime can be separate issue. | -| A2 | Mermaid/markdown diagrams are available or acceptable across enough hosts. | Some hosts render only plain text. | Guide requires plain-text summary/fallback for every visual. | -| A3 | Adding one companion guide is consistent with skill docs shape. | Existing skills mostly keep logic in SKILL.md. | Keep SKILL.md authoritative; guide is referenced for detailed examples only. | +| A1 | Issue #78 accepts host-neutral visual guidance as "similar" to upstream. | User may expect full browser parity. | Parity boundary marks browser/event capture deferred; future server runtime requires separate issue/design. | +| A2 | Markdown/Mermaid diagrams are useful even where rendered as code. | Some hosts show raw code only. | Text source-of-truth + concise text equivalent required. | +| A3 | A bounded guide file is justified. | Existing skill guidance prefers inline when short. | SKILL.md remains authoritative; guide is limited to examples/failure/accessibility details that would bloat SKILL.md. | ## Self-Challenge | doubt | response | |---|---| | Laziest solution: paste upstream section only. | Rejected: it references scripts ADK does not ship and weakens host neutrality. | -| Fragile assumption: no full browser runtime required. | Mitigated by explicit scope and ADR; issue says "something similar," not exact parity. | -| YAGNI risk: guide becomes too broad. | Keep examples generic; no runtime implementation, no click-event schema. | +| Fragile assumption: no full browser runtime required. | Mitigated by parity boundary, ADR, and deferred browser row. | +| YAGNI risk: guide becomes too broad. | Bound guide role; no runtime implementation, telemetry, session keys, or click-event schema. | | Failure mode: agent overuses diagrams. | Just-in-time/per-question rules say never offer upfront and use visuals only when clearer than text. | -| Repo pattern conflict: skill process gates. | Visual step is inserted before questions and does not skip project guidance, assumptions, review, or planning gates. | +| Repo pattern conflict: test script misuse. | Use focused `tests/brainstorm-visual-companion.sh`; keep pipeline evidence test unchanged. | + +## Review Cycle 1 Resolutions + +| finding | resolution | +|---|---| +| D1 | Added focused RED/GREEN behavior-marker test and made grep checks secondary. | +| D2 | Added parity boundary and host visual capability matrix; browser companion deferred. | +| D3 | Added explicit failure rules and text-source-of-truth behavior. | +| D4 | Accepted with mitigation: guide bounded to reference examples/failure rules; SKILL.md authoritative. | +| D5 | Resolved: process flow remains `dot`; Mermaid only for optional generated visuals. | +| D6 | Resolved: use new focused test, not `pipeline-evidence-doc-sync.sh`. | ## Rollback -Revert commits modifying `skills/brainstorming/SKILL.md`, `skills/brainstorming/visual-companion.md`, and `tests/pipeline-evidence-doc-sync.sh`; rerun AGENTS.md test list. No runtime state, migrations, or version pins. +Revert commits modifying `skills/brainstorming/SKILL.md`, `skills/brainstorming/visual-companion.md`, `tests/brainstorm-visual-companion.sh`, and planning/ADR artifacts; rerun AGENTS.md test list. No runtime state, migrations, or version pins. From 13a4f5b2690abe0ff142d3a2a547164578113352 Mon Sep 17 00:00:00 2001 From: Jon Langevin Date: Tue, 30 Jun 2026 06:00:49 -0400 Subject: [PATCH 03/16] Add behavior proof to visualization design --- ...-brainstorm-visualization-design-review.md | 63 ++++++++++------- ...6-06-30-brainstorm-visualization-design.md | 68 +++++++++++++------ 2 files changed, 86 insertions(+), 45 deletions(-) diff --git a/docs/plans/2026-06-30-brainstorm-visualization-design-review.md b/docs/plans/2026-06-30-brainstorm-visualization-design-review.md index 73e2fda..1c58a90 100644 --- a/docs/plans/2026-06-30-brainstorm-visualization-design-review.md +++ b/docs/plans/2026-06-30-brainstorm-visualization-design-review.md @@ -1,4 +1,5 @@ ### Adversarial Review Report + **Phase:** design **Artifact:** docs/plans/2026-06-30-brainstorm-visualization-design.md **Status:** FAIL @@ -7,37 +8,49 @@ - None. **Findings (Important):** -- `D1` [Repo-precedent conflicts / Multi-component validation] [`docs/plans/2026-06-30-brainstorm-visualization-design.md:42-45`, `:67-75`]: The validation strategy is grep-only for a behavior-changing skill edit. Repo precedent says skill edits follow TDD for process docs: "NO SKILL WITHOUT A FAILING TEST FIRST" and "This applies to NEW skills AND EDITS to existing skills" (`skills/writing-skills/SKILL.md:461-480`), with application/retrieval scenarios for technique/reference skills (`:482-529`). The design only asserts strings like `Visual Companion`, `just-in-time`, and `Mermaid`, which does not prove agents offer the companion at the right time, avoid offering it upfront, choose text vs visual per question, or load/use the guide. Recommendation: add a RED/GREEN skill-behavior scenario or pressure test for brainstorming before implementation, then prove post-change behavior with a representative transcript/eval; keep grep checks as regression guards, not primary proof. -- `D2` [Declared integration proof / User-intent drift / Existence-runtime-validity] [`docs/plans/2026-06-30-brainstorm-visualization-design.md:24-30`, `:78-82`]: The design replaces upstream’s browser-based visual companion with host-native Markdown/Mermaid guidance but does not define a host integration matrix or prove the real consumer surface. Upstream’s current `skills/brainstorming/SKILL.md` describes a browser companion that opens a tab, starts a server, and records browser events; upstream README also documents visual companion telemetry. ADK’s `docs/cross-llm-coverage.md:7-20` has no visual/Mermaid rendering capability row, and `tests/cross-llm-coverage.md:13` only tracks brainstorming’s question-tool host conditionals. Recommendation: explicitly state the parity boundary as an accepted non-goal, add a host-by-host matrix marking Markdown/Mermaid/browser behavior as `runtime-integrated`, `config-only`, or `deferred`, and require at least one real-host rendering proof or mark rendering as best-effort text-only fallback. -- `D3` [Missing failure modes / Assumptions under attack] [`docs/plans/2026-06-30-brainstorm-visualization-design.md:39-45`, `:80-82`, `:90-91`]: The design covers “agent overuses diagrams” and plain-text fallback, but misses likely first failures: invalid Mermaid syntax, hosts that display Mermaid as raw code, diagrams that contradict the text design, stale visual artifacts after the conversation moves on, and inaccessible visuals where the fallback is lower-fidelity than the decision being made. Recommendation: add explicit failure rules: text remains the source of truth, every visual must include a concise text equivalent, agents must fall back when rendering is unavailable or invalid, and visual artifacts must be updated or retired when decisions change. +- `D1` [Repo-precedent conflicts / Multi-component validation] [`docs/plans/2026-06-30-brainstorm-visualization-design.md:66-68`, `:102-109`; `skills/writing-skills/SKILL.md:461-529`, `:620-649`; `skills/writing-skills/testing-skills-with-subagents.md:7-12`, `:30-39`, `:82-95`]: Still not materially resolved. The revised design upgrades from generic grep checks to `tests/brainstorm-visual-companion.sh`, but the described GREEN state still asserts text markers (`Visual Companion`, `just-in-time`, guide reference, Mermaid/accessibility/fallback) rather than a representative skill-behavior scenario. Repo precedent for skill edits requires RED/GREEN around how agents behave: baseline failure, same scenario with skill, pressure/application evidence. This is a behavior-changing process-skill edit; a shell marker test can prove the instructions exist, not that agents offer visuals at the right time, avoid offering them upfront, choose text vs visual per question, or consult the guide only when useful. Recommendation: add one explicit skill-behavior proof to the design/plan: e.g. a pressure/application transcript or fixture where brainstorming handles (a) a conceptual question text-only, (b) a layout/architecture question with a just-in-time visual offer in its own message, (c) a declined offer without re-offering, and (d) an accepted visual path with text fallback. Keep the shell test as a regression guard, not the primary proof. **Findings (Minor):** -- `D4` [Artifact-class precedent / YAGNI violations / Simpler alternative not considered] [`docs/plans/2026-06-30-brainstorm-visualization-design.md:39-41`, `:78-83`]: The design adds `skills/brainstorming/visual-companion.md` without justifying that it is heavy reference material. Skill-authoring precedent says supporting files are “Only if needed” (`skills/writing-skills/SKILL.md:74-82`), keep principles/concepts and code patterns under 50 lines inline (`:86-93`), and use heavy reference files when material is too large (`:451-459`). Recommendation: either make the initial change a short inline `## Visual Companion` section, or bound the guide’s role/size and explain why it is a heavy reference rather than SKILL.md content. -- `D5` [Repo-precedent conflicts] [`docs/plans/2026-06-30-brainstorm-visualization-design.md:36-40`]: The design says the guide will include Mermaid examples while existing skill process diagrams use `dot`, and `skills/writing-skills/SKILL.md:377-405` gives Graphviz/dot flowchart guidance plus render tooling for skill diagrams. If Mermaid is only for user-facing generated artifacts, that distinction is not explicit. Recommendation: state that SKILL.md process flow remains `dot`; Mermaid examples are only optional output artifacts shown to users where supported. -- `D6` [Infrastructure impact / Artifact-class precedent] [`docs/plans/2026-06-30-brainstorm-visualization-design.md:42-45`, `tests/pipeline-evidence-doc-sync.sh:1-4`, `:68-91`]: `tests/pipeline-evidence-doc-sync.sh` is currently framed as a regression guard for pipeline evidence issues #69/#70/#71/#72 and later #79, not a general feature-behavior test bucket. Adding #78 visual assertions there may blur the script’s contract and make future maintenance less obvious. Recommendation: either update the script header/sectioning to include #78 explicitly or create a focused skill-behavior/content regression test for brainstorming visualization. +- `D7` [Repo-precedent conflicts / Declared integration proof / Artifact-class precedent] [`docs/plans/2026-06-30-brainstorm-visualization-design.md:32-43`; `README.md:238-239`; `docs/cross-llm-coverage.md:7-20`; `tests/cross-llm-coverage.md:3-13`]: The design now has a host visual capability matrix, but only inside the transient design artifact. Repo docs call `docs/cross-llm-coverage.md` the full capability matrix and `tests/cross-llm-coverage.md` says it is updated whenever a skill changes; the revised design does not add a task to update either durable artifact for visual/Mermaid/browser behavior or to explain why this capability is intentionally local-only. Recommendation: either add durable coverage-doc updates for the visual capability rows, or explicitly state that this PR does not change canonical host capability coverage because rendering remains best-effort text fallback except where separately proven. +- `D8` [Infrastructure impact / Existence-runtime-validity / Multi-component validation] [`docs/plans/2026-06-30-brainstorm-visualization-design.md:66-69`, `:102-109`; `AGENTS.md:21-37`]: The design adds `tests/brainstorm-visual-companion.sh`, but does not say whether the new test becomes part of the documented pre-commit test suite. `AGENTS.md` is the repo’s current durable test list; without updating it (or another runner), the new regression guard is easy to pass once during implementation and then omit in future changes. Recommendation: add `bash tests/brainstorm-visual-companion.sh` to `AGENTS.md`’s skill-content/checks list, or create/update a documented skill-test aggregator that includes it. +- `D9` [YAGNI violations / Missing failure modes / Infrastructure impact] [`docs/plans/2026-06-30-brainstorm-visualization-design.md:63-65`, `:98-99`, `:126-127`; upstream `skills/brainstorming/SKILL.md` Visual Companion section fetched 2026-06-30]: The bounded sidecar guide is now justified, but the design still does not specify lazy-loading semantics for the guide. Upstream only reads `skills/brainstorming/visual-companion.md` after the user accepts the companion; ADK’s design says the guide exists and is referenced, while claiming token cost only when used. Without an explicit “load/read this guide only after visual companion acceptance or when composing a visual artifact” rule, agents may preload the guide in every brainstorming session, increasing token cost and nudging overuse. Recommendation: add a lazy-load rule to the planned `SKILL.md` section and include it in the regression test markers. + +**Cycle 1 Important resolution assessment:** +| finding | materially resolved? | evidence / residual | +|---|---|---| +| `D1` | No | Test design changed from broad grep to focused marker assertions, but still lacks RED/GREEN behavior proof required by `skills/writing-skills/SKILL.md` and `testing-skills-with-subagents.md`. Kept open as Important. | +| `D2` | Yes | Parity boundary now explicitly defers browser/event capture (`docs/plans/2026-06-30-brainstorm-visualization-design.md:23-43`), ADR records host-neutral guidance over upstream runtime (`decisions/0005-visual-companion-instructions.md:12-31`). Residual canonical coverage-doc concern is downgraded to `D7` Minor. | +| `D3` | Yes | Failure rules now cover invalid/unsupported Mermaid, text source-of-truth, stale/contradictory visuals, accessibility/remote limits, decline behavior, and secrets/PII (`docs/plans/2026-06-30-brainstorm-visualization-design.md:71-80`). | + +**Prior minor resolution assessment:** +| finding | status | +|---|---| +| `D4` | Resolved/accepted with mitigation: design bounds the guide to examples/failure/accessibility rules and keeps `SKILL.md` authoritative (`docs/plans/2026-06-30-brainstorm-visualization-design.md:63-65`, `:118`). | +| `D5` | Resolved: process flow remains `dot`; Mermaid is scoped to optional user-facing visuals (`docs/plans/2026-06-30-brainstorm-visualization-design.md:60`, `:65`). | +| `D6` | Resolved: design uses a new focused test and keeps `tests/pipeline-evidence-doc-sync.sh` unchanged (`docs/plans/2026-06-30-brainstorm-visualization-design.md:66-69`). | **Bug-class scan transcript:** | Class | Result | Note | |---|---|---| -| Project-guidance conflicts | Clean | No `docs/design-guidance.md` exists per prompt; the design cites repo canon from `AGENTS.md`, `README.md`, and `skills/brainstorming/SKILL.md`, which is an acceptable substitute here. | -| Assumptions under attack | Finding | `D2` and `D3`: the assumptions that “workflow guidance” is enough and Mermaid/text fallback is acceptable are load-bearing and insufficiently proven. | -| Repo-precedent conflicts | Finding | `D1` conflicts with `skills/writing-skills/SKILL.md` TDD-for-skill-edits precedent; `D5` risks conflicting with dot/Graphviz skill-diagram precedent. | -| Artifact-class precedent | Finding | `D4` questions the unproven need for a sibling guide file; `D6` questions using `pipeline-evidence-doc-sync.sh` for this feature’s regression assertions. | -| YAGNI violations | Finding | `D4`: a separate companion guide may be more structure than needed for the first host-neutral version. | -| Missing failure modes | Finding | `D3`: invalid Mermaid, non-rendering hosts, stale/contradictory visuals, and accessibility degradation are not designed through. | -| Security/privacy architecture | Clean | Current scoped design avoids browser server, network service, dependencies, and says visual examples must not expose secrets/PII; future browser parity would need a new privacy decision, especially because upstream documents telemetry. | -| Infrastructure impact | Finding | `D6`: no cloud/runtime infra is added, but the test-suite impact is under-specified for a script with an existing narrower contract. | -| Multi-component validation | Finding | `D1` and `D2`: grep checks do not prove the skill behavior or host rendering boundary. | -| Declared integration proof | Finding | `D2`: host-native Markdown/Mermaid rendering is treated as available/acceptable without a host integration matrix or real-host proof. | -| Contributed UI rendering proof | Clean | The design does not register a UI route/widget into a host shell; this class is not directly applicable unless browser companion/runtime work is reintroduced. | -| Rollback story | Clean | `docs/plans/2026-06-30-brainstorm-visualization-design.md:94-97` gives a concrete markdown/test revert path with no migrations or runtime state. | -| Simpler alternative not considered | Finding | `D4`: the design considers copying upstream and building a runtime, but not a minimal inline-only section that avoids a new sidecar guide. | -| User-intent drift | Finding | `D2`: the user pointed at upstream’s visualization companion; the design may underdeliver by shipping chat/Markdown guidance unless the reduced parity boundary is explicitly accepted. | -| Existence/runtime-validity | Finding | `D2`: the design does not verify the emitted visual artifacts target a real rendering/consumer surface across ADK-supported hosts. | +| Project-guidance conflicts | Clean | No `docs/design-guidance.md` or `docs/PORTFOLIO.md` exists; the design cites repo canon from `AGENTS.md`, `README.md`, and skill files, which matches the project-guidance fallback. | +| Assumptions under attack | Finding | `D1`: assumes instruction-marker tests prove behavior; `D9`: assumes token cost stays low without a lazy-load rule. | +| Repo-precedent conflicts | Finding | `D1` conflicts with skill-edit RED/GREEN behavior-testing precedent; `D7` conflicts with durable cross-LLM coverage-doc precedent. | +| Artifact-class precedent | Finding | `D7`: host capability matrices already live in `docs/cross-llm-coverage.md` / `tests/cross-llm-coverage.md`; `D8`: documented test-suite membership lives in `AGENTS.md`. | +| YAGNI violations | Finding | `D9`: a sidecar guide can become unnecessary session overhead unless lazy-loaded only when visual work actually happens. | +| Missing failure modes | Finding | `D9`: eager guide loading / visual overuse remains an unaddressed failure mode; prior visual-rendering failures from `D3` are resolved. | +| Security/privacy architecture | Clean | Scope avoids browser server, telemetry, network service, new dependency, auth boundary, and persistent session state; design requires redacted/synthetic examples and no secrets/PII. | +| Infrastructure impact | Finding | `D8`: adding a standalone test without wiring it into the documented suite creates maintenance/validation drift. | +| Multi-component validation | Finding | `D1`: marker tests do not prove the skill-consumer behavior boundary; `D8`: the test exists in design but is not integrated into the repo’s durable validation surface. | +| Declared integration proof | Finding | `D7`: design-local host matrix exists, but declared visual capability is not reflected in the repo’s canonical coverage artifacts. | +| Contributed UI rendering proof | Clean | No host shell route/widget/page is contributed; browser companion runtime and click capture are explicitly deferred. | +| Rollback story | Clean | Rollback is concrete markdown/test revert with no runtime state, migrations, or version pins (`docs/plans/2026-06-30-brainstorm-visualization-design.md:141-143`). | +| Simpler alternative not considered | Clean | Revised design explicitly considered inline-only MVP and rejected it as too cramped for examples/fallbacks (`docs/plans/2026-06-30-brainstorm-visualization-design.md:45-54`). | +| User-intent drift | Clean | Browser parity risk is now explicit: upstream browser/event capture is deferred, diagrams/mockups during brainstorm/design remain in scope, and ADR records the trade-off. | +| Existence/runtime-validity | Finding | `D8`: emitted test artifact is named, but the design does not prove the repo’s documented test consumer will pick it up; `D7`: canonical coverage consumers are not updated. | **Options the author may not have considered:** -1. **Inline-only MVP:** Add a concise `## Visual Companion` section directly to `skills/brainstorming/SKILL.md` with just-in-time offer rules, per-question visual/text decision rules, and plain-text fallback. This is smaller, follows the “keep inline” precedent, and can be behavior-tested before introducing a heavier guide. -2. **Two-tier parity design:** Ship host-neutral Markdown/Mermaid guidance now, but explicitly add a deferred “browser companion parity” track with acceptance criteria copied from upstream: browser launch, session security, event capture, telemetry policy, cleanup, and multi-host process behavior. This prevents the current PR from being mistaken for full upstream parity. -3. **Capability-matrix-first approach:** Before changing brainstorming behavior, update `docs/cross-llm-coverage.md` and `tests/cross-llm-coverage.md` with visual-rendering capabilities per host. Then make the skill branch on that matrix: rendered Mermaid where proven, Markdown/text fallback elsewhere. +1. **Behavior fixture before shell guard:** Keep `tests/brainstorm-visual-companion.sh`, but add a small scenario transcript/eval artifact under `skills/brainstorming/test-fixtures/visual-companion/` that captures the RED/GREEN behavior the shell script cannot express. Trade-off: more artifact maintenance, but it matches `writing-skills` precedent for process-skill edits. +2. **Canonical matrix update instead of design-only matrix:** Move the host visual capability matrix rows into `docs/cross-llm-coverage.md` and summarize them in the design. Trade-off: broader doc change, but future host-portability reviews won’t have to rediscover a matrix buried in a dated plan. +3. **Inline-only until behavior proof exists:** Delay the sidecar guide and ship only a concise `SKILL.md` section plus behavior proof. Trade-off: fewer examples initially, but removes lazy-loading/token-risk and can be expanded once agents demonstrate the rule works. -**Verdict reasoning:** The design is directionally reasonable but not ready to pass adversarial review because its validation does not prove the behavioral skill change, and its scope reduction from upstream’s browser companion to Markdown/Mermaid guidance is not backed by a real host capability matrix or explicit accepted parity boundary. These are Important findings because they can cause the implementation to pass grep tests while failing the actual user-visible goal: agents reliably offering and using useful brainstorm/design visuals across ADK-supported hosts. +**Verdict reasoning:** The revised design materially resolves the browser-parity boundary (`D2`) and visual failure-rules gap (`D3`), and the prior minor concerns `D4`-`D6` are adequately mitigated. It still fails because `D1` remains an Important validation flaw: the proposed RED/GREEN test is still marker-based, not behavior-based, despite repo precedent requiring skill edits to prove agent behavior under realistic scenarios. The remaining new findings are Minor, but `D1` is tangible enough to keep design review at FAIL until the design adds a representative skill-behavior proof or explicitly records why this skill edit is exempt from that precedent. diff --git a/docs/plans/2026-06-30-brainstorm-visualization-design.md b/docs/plans/2026-06-30-brainstorm-visualization-design.md index 98e2b00..0420bd7 100644 --- a/docs/plans/2026-06-30-brainstorm-visualization-design.md +++ b/docs/plans/2026-06-30-brainstorm-visualization-design.md @@ -2,7 +2,7 @@ **Issue:** #78 — Brainstorm visualization **Date:** 2026-06-30 -**Status:** Approved by user instruction; revised after adversarial review cycle 1 +**Status:** Approved by user instruction; revised after adversarial review cycles 1-2 **ADR:** `decisions/0005-visual-companion-instructions.md` ## Original Ask @@ -15,10 +15,11 @@ | source | guidance | design response | |---|---|---| -| `AGENTS.md` | Skills use host-gated content; forbidden host-specific tokens only in host blocks; recognized hosts include `zed-agent`. | Add host-neutral visual guidance; avoid new host-specific tool names outside host blocks. | +| `AGENTS.md` | Skills use host-gated content; forbidden host-specific tokens only in host blocks; recognized hosts include `zed-agent`; test scripts are documented pre-commit checks. | Add host-neutral visual guidance; update documented checks for the new focused test. | | `README.md` Cross-LLM/plugin positioning | ADK bundles portable skills/hooks/agents/commands for multiple hosts. | Prefer host-neutral Markdown plus optional Mermaid where rendered; never require an upstream-only browser server. | +| `docs/cross-llm-coverage.md` / `tests/cross-llm-coverage.md` | Capability and skill coverage matrices record host-portability intent. | Add a visual-companion capability row and brainstorming note so the host matrix is durable, not design-local. | | `skills/brainstorming/SKILL.md` | Brainstorming has adaptive batching, assumptions, self-challenge, adversarial review, and design docs. | Insert visual companion as a just-in-time optional aid without weakening gates. | -| `skills/writing-skills/SKILL.md` | Skill edits require failing tests first and supporting files only when useful. | Add a focused RED/GREEN shell test for visual-companion behavior markers; keep SKILL.md authoritative and the companion guide bounded to examples/fallback rules. | +| `skills/writing-skills/SKILL.md` | Skill edits require failing tests first and behavior proof, not marker-only tests. | Add a focused shell regression guard plus a behavior pressure-test fixture and subagent pressure run. | ## Parity Boundary @@ -33,7 +34,7 @@ | host | Markdown text | Mermaid/rendered diagrams | Browser companion | |---|---|---|---| -| `zed-agent` | runtime-integrated: markdown chat output | runtime-integrated: Zed renders Mermaid in assistant markdown when supported; must include text fallback | deferred | +| `zed-agent` | runtime-integrated: markdown chat output | runtime-integrated where Zed renders Mermaid in assistant markdown; must include text fallback | deferred | | `claude-code` | config-only: markdown output | config-only/best-effort: render support varies by surface | deferred | | `codex` | config-only: markdown output | config-only/best-effort | deferred | | `opencode` | config-only: markdown output | config-only/best-effort | deferred | @@ -47,7 +48,7 @@ | option | summary | trade-off | |---|---|---| | A. Copy upstream browser companion verbatim | Add upstream `Visual Companion` text and server instructions. | Fast but references scripts/tools ADK does not ship; host-portability and security gaps. | -| B. Host-neutral progressive companion | Add just-in-time offer rules, diagram/mockup decision test, failure rules, host matrix, and bounded guide. | Solves issue now; browser click capture remains deferred. Recommended. | +| B. Host-neutral progressive companion | Add just-in-time offer rules, diagram/mockup decision test, failure rules, host matrix, behavior proof, and bounded guide. | Solves issue now; browser click capture remains deferred. Recommended. | | C. Inline-only MVP | Put all guidance in `SKILL.md`, no guide file. | Smallest patch but clutters an already-long skill and lacks room for examples/fallbacks. | | D. Full browser companion runtime | Add scripts, frame template, event state, and docs. | Larger runtime feature; requires multi-host launch validation beyond issue scope. | @@ -59,14 +60,33 @@ - Add checklist item after project guidance: offer visual companion just-in-time, not upfront. - Keep process-flow diagram in `dot` to match existing skill precedent. - Add process bullets: offer visuals only when a question is genuinely clearer shown than described. - - Add `## Visual Companion` section: per-question decision rules, text-source-of-truth, stale/invalid visual fallback, and guide reference. + - Add `## Visual Companion` section: per-question decision rules, text-source-of-truth, stale/invalid visual fallback, lazy-load guide rule, and guide reference. - Add bounded `skills/brainstorming/visual-companion.md`: - - Role: reference examples and failure/accessibility rules only; `SKILL.md` remains authoritative. + - Role: reference examples and failure/accessibility rules only; load only after visual companion acceptance or while composing a visual artifact. - Covers diagrams/mockups, Mermaid examples, visual/text decision test, no-secrets rule, stale visual retirement, and plain-text fallback. - Add focused regression test `tests/brainstorm-visual-companion.sh`: - - RED before implementation: `skills/brainstorming/SKILL.md` lacks `Visual Companion`, `just-in-time`, and guide reference; guide file missing. - - GREEN after implementation: asserts just-in-time offer, not-upfront rule, per-question decision, text fallback/source of truth, guide existence, and guide mentions Mermaid plus accessibility/fallback. -- Keep `tests/pipeline-evidence-doc-sync.sh` unchanged; do not overload its issue-specific contract. + - RED before implementation: required skill markers/guide/fixture/coverage rows absent. + - GREEN after implementation: asserts just-in-time offer, not-upfront rule, per-question decision, text fallback/source of truth, lazy-load rule, guide existence, guide mentions Mermaid plus accessibility/fallback, behavior fixture exists, and coverage/docs checks include visual companion. +- Add behavior fixture `skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md`: + - Records pressure scenario and expected compliant behavior for four paths: conceptual text-only, visual offer in its own message, declined offer/no re-offer, accepted visual with text fallback. + - Includes RED baseline notes against pre-change brainstorming: no visual companion rule exists, so the expected just-in-time offer and fallback paths are absent. +- Run behavior proof with a subagent after implementation: + - Prompt a reviewer with the fixture and current `skills/brainstorming/SKILL.md`. + - Expected: reviewer reports PASS for all four paths and cites the exact skill/guide lines that force the behavior. + - This is the primary behavior proof; shell test is the regression guard. +- Update durable docs: + - `AGENTS.md`: add `bash tests/brainstorm-visual-companion.sh` to skill content checks. + - `docs/cross-llm-coverage.md`: add visual companion capability row. + - `tests/cross-llm-coverage.md`: update brainstorming notes with visual companion host-neutral/best-effort behavior. + +## Behavior Fixture Expected Paths + +| path | compliant behavior | +|---|---| +| Conceptual text-only question | Do not offer visual companion; ask/answer in text because the answer is conceptual. | +| Visual question arises | Send only the just-in-time offer message; no bundled clarifying question. | +| User declines | Continue text-only and do not offer again unless user raises visuals. | +| User accepts | Use the guide only when composing visuals; include text summary/fallback; record final decision in text. | ## Failure Rules @@ -78,6 +98,7 @@ | Accessibility or remote-host limitation | Include concise text equivalent for every visual and accept text-only feedback. | | User declines visual companion | Continue text-only; do not offer again unless user raises visuals. | | Sensitive data in visual | Redact or use synthetic examples; no secrets/PII in diagrams/mockups. | +| Guide loaded eagerly | Violation; companion guide loads only after acceptance or while composing a visual artifact. | ## Security Review @@ -95,17 +116,20 @@ |---|---| | Cloud/network | None. | | Local runtime | None; no server added. | -| Storage | Adds one bounded skill guide, one focused test, plus design/ADR/plan artifacts. | -| Cost/scale | Slight token cost only when visual companion is used. | +| Storage | Adds one bounded skill guide, one fixture, one focused test, plus design/ADR/plan artifacts. | +| Test suite | `AGENTS.md` documented checks gain one shell test. | +| Cost/scale | Slight token cost only when visual companion is used; lazy-load rule avoids every-session overhead. | | Release/rollback | Revert markdown/test changes; no migrations. | ## Multi-Component Validation | boundary | proof | |---|---| -| Skill behavior markers | `bash tests/brainstorm-visual-companion.sh` → asserts just-in-time/not-upfront/per-question/text fallback/guide contract. | +| Skill behavior | Subagent pressure proof against `expected-behavior.md` → PASS for four paths. | +| Skill behavior markers | `bash tests/brainstorm-visual-companion.sh` → asserts just-in-time/not-upfront/per-question/text fallback/lazy-load/guide/fixture/coverage contract. | | Skill text ↔ content rules | `bash tests/skill-content-grep.sh` → host-neutral token guard passes. | | Skill refs ↔ repo files | `bash tests/skill-cross-refs.sh` → references resolve. | +| Coverage docs | `bash tests/brainstorm-visual-companion.sh` checks visual rows/notes exist. | | Full repo contract | Run AGENTS.md test list before PR. | | Runtime rendering | Deferred except Zed markdown/Mermaid support where observed by host; all other hosts use text fallback and are marked config-only/best-effort. | @@ -115,7 +139,8 @@ |---|---|---|---| | A1 | Issue #78 accepts host-neutral visual guidance as "similar" to upstream. | User may expect full browser parity. | Parity boundary marks browser/event capture deferred; future server runtime requires separate issue/design. | | A2 | Markdown/Mermaid diagrams are useful even where rendered as code. | Some hosts show raw code only. | Text source-of-truth + concise text equivalent required. | -| A3 | A bounded guide file is justified. | Existing skill guidance prefers inline when short. | SKILL.md remains authoritative; guide is limited to examples/failure/accessibility details that would bloat SKILL.md. | +| A3 | A bounded guide file is justified. | Existing skill guidance prefers inline when short. | SKILL.md remains authoritative; guide is lazy-loaded and limited to examples/failure/accessibility details. | +| A4 | Subagent pressure proof is acceptable for behavior validation. | It is not CI-enforced. | Shell regression guard preserves contract markers; PR body includes behavior proof transcript. | ## Self-Challenge @@ -123,21 +148,24 @@ |---|---| | Laziest solution: paste upstream section only. | Rejected: it references scripts ADK does not ship and weakens host neutrality. | | Fragile assumption: no full browser runtime required. | Mitigated by parity boundary, ADR, and deferred browser row. | -| YAGNI risk: guide becomes too broad. | Bound guide role; no runtime implementation, telemetry, session keys, or click-event schema. | +| YAGNI risk: guide becomes too broad. | Bound + lazy-load guide; no runtime implementation, telemetry, session keys, or click-event schema. | | Failure mode: agent overuses diagrams. | Just-in-time/per-question rules say never offer upfront and use visuals only when clearer than text. | -| Repo pattern conflict: test script misuse. | Use focused `tests/brainstorm-visual-companion.sh`; keep pipeline evidence test unchanged. | +| Repo pattern conflict: marker-only testing. | Add fixture + subagent pressure proof as primary; shell test only guards regressions. | -## Review Cycle 1 Resolutions +## Review Cycle Resolutions | finding | resolution | |---|---| -| D1 | Added focused RED/GREEN behavior-marker test and made grep checks secondary. | +| D1 | Added behavior fixture + subagent pressure proof as primary validation. | | D2 | Added parity boundary and host visual capability matrix; browser companion deferred. | | D3 | Added explicit failure rules and text-source-of-truth behavior. | -| D4 | Accepted with mitigation: guide bounded to reference examples/failure rules; SKILL.md authoritative. | +| D4 | Accepted with mitigation: guide bounded/lazy-loaded; SKILL.md authoritative. | | D5 | Resolved: process flow remains `dot`; Mermaid only for optional generated visuals. | | D6 | Resolved: use new focused test, not `pipeline-evidence-doc-sync.sh`. | +| D7 | Add durable coverage doc updates. | +| D8 | Add focused test to documented `AGENTS.md` checks. | +| D9 | Add lazy-load rule to SKILL.md/guide/test. | ## Rollback -Revert commits modifying `skills/brainstorming/SKILL.md`, `skills/brainstorming/visual-companion.md`, `tests/brainstorm-visual-companion.sh`, and planning/ADR artifacts; rerun AGENTS.md test list. No runtime state, migrations, or version pins. +Revert commits modifying `skills/brainstorming/SKILL.md`, `skills/brainstorming/visual-companion.md`, `skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md`, `tests/brainstorm-visual-companion.sh`, `docs/cross-llm-coverage.md`, `tests/cross-llm-coverage.md`, `AGENTS.md`, and planning/ADR artifacts; rerun AGENTS.md test list. No runtime state, migrations, or version pins. From 03297c533e3ff548b5241428db5835fc98fac38e Mon Sep 17 00:00:00 2001 From: Jon Langevin Date: Tue, 30 Jun 2026 06:07:27 -0400 Subject: [PATCH 04/16] Pass brainstorm visualization design review --- ...-brainstorm-visualization-design-review.md | 66 +++++++++---------- ...6-06-30-brainstorm-visualization-design.md | 20 ++++-- 2 files changed, 43 insertions(+), 43 deletions(-) diff --git a/docs/plans/2026-06-30-brainstorm-visualization-design-review.md b/docs/plans/2026-06-30-brainstorm-visualization-design-review.md index 1c58a90..bd19022 100644 --- a/docs/plans/2026-06-30-brainstorm-visualization-design-review.md +++ b/docs/plans/2026-06-30-brainstorm-visualization-design-review.md @@ -2,55 +2,49 @@ **Phase:** design **Artifact:** docs/plans/2026-06-30-brainstorm-visualization-design.md -**Status:** FAIL +**Status:** PASS **Findings (Critical):** - None. **Findings (Important):** -- `D1` [Repo-precedent conflicts / Multi-component validation] [`docs/plans/2026-06-30-brainstorm-visualization-design.md:66-68`, `:102-109`; `skills/writing-skills/SKILL.md:461-529`, `:620-649`; `skills/writing-skills/testing-skills-with-subagents.md:7-12`, `:30-39`, `:82-95`]: Still not materially resolved. The revised design upgrades from generic grep checks to `tests/brainstorm-visual-companion.sh`, but the described GREEN state still asserts text markers (`Visual Companion`, `just-in-time`, guide reference, Mermaid/accessibility/fallback) rather than a representative skill-behavior scenario. Repo precedent for skill edits requires RED/GREEN around how agents behave: baseline failure, same scenario with skill, pressure/application evidence. This is a behavior-changing process-skill edit; a shell marker test can prove the instructions exist, not that agents offer visuals at the right time, avoid offering them upfront, choose text vs visual per question, or consult the guide only when useful. Recommendation: add one explicit skill-behavior proof to the design/plan: e.g. a pressure/application transcript or fixture where brainstorming handles (a) a conceptual question text-only, (b) a layout/architecture question with a just-in-time visual offer in its own message, (c) a declined offer without re-offering, and (d) an accepted visual path with text fallback. Keep the shell test as a regression guard, not the primary proof. +- None. **Findings (Minor):** -- `D7` [Repo-precedent conflicts / Declared integration proof / Artifact-class precedent] [`docs/plans/2026-06-30-brainstorm-visualization-design.md:32-43`; `README.md:238-239`; `docs/cross-llm-coverage.md:7-20`; `tests/cross-llm-coverage.md:3-13`]: The design now has a host visual capability matrix, but only inside the transient design artifact. Repo docs call `docs/cross-llm-coverage.md` the full capability matrix and `tests/cross-llm-coverage.md` says it is updated whenever a skill changes; the revised design does not add a task to update either durable artifact for visual/Mermaid/browser behavior or to explain why this capability is intentionally local-only. Recommendation: either add durable coverage-doc updates for the visual capability rows, or explicitly state that this PR does not change canonical host capability coverage because rendering remains best-effort text fallback except where separately proven. -- `D8` [Infrastructure impact / Existence-runtime-validity / Multi-component validation] [`docs/plans/2026-06-30-brainstorm-visualization-design.md:66-69`, `:102-109`; `AGENTS.md:21-37`]: The design adds `tests/brainstorm-visual-companion.sh`, but does not say whether the new test becomes part of the documented pre-commit test suite. `AGENTS.md` is the repo’s current durable test list; without updating it (or another runner), the new regression guard is easy to pass once during implementation and then omit in future changes. Recommendation: add `bash tests/brainstorm-visual-companion.sh` to `AGENTS.md`’s skill-content/checks list, or create/update a documented skill-test aggregator that includes it. -- `D9` [YAGNI violations / Missing failure modes / Infrastructure impact] [`docs/plans/2026-06-30-brainstorm-visualization-design.md:63-65`, `:98-99`, `:126-127`; upstream `skills/brainstorming/SKILL.md` Visual Companion section fetched 2026-06-30]: The bounded sidecar guide is now justified, but the design still does not specify lazy-loading semantics for the guide. Upstream only reads `skills/brainstorming/visual-companion.md` after the user accepts the companion; ADK’s design says the guide exists and is referenced, while claiming token cost only when used. Without an explicit “load/read this guide only after visual companion acceptance or when composing a visual artifact” rule, agents may preload the guide in every brainstorming session, increasing token cost and nudging overuse. Recommendation: add a lazy-load rule to the planned `SKILL.md` section and include it in the regression test markers. +- `D10` [Declared integration proof / Multi-component validation] [`docs/plans/2026-06-30-brainstorm-visualization-design.md`]: Host matrix originally risked overclaiming Zed Mermaid runtime rendering. Recommendation: mark rendered diagrams config-only/best-effort unless explicitly observed. _Resolution: fixed in design cycle 3._ +- `D11` [Missing failure modes / Repo-precedent conflicts] [`docs/plans/2026-06-30-brainstorm-visualization-design.md`]: Visual-companion offer asks the user a question; batch-budget interaction needed definition. Recommendation: count the offer as one question batch. _Resolution: fixed in design cycle 3._ +- `D12` [Missing failure modes / Multi-component validation] [`docs/plans/2026-06-30-brainstorm-visualization-design.md`]: Behavior fixture needed at least one negative/failure-mode pressure path. Recommendation: add stale/contradictory visual case. _Resolution: fixed in design cycle 3._ +- `D13` [User-intent drift / Rollback story] [`docs/plans/2026-06-30-brainstorm-visualization-design.md`]: Browser/event-capture parity is deferred; durable follow-up needed. Recommendation: add `docs/FOLLOWUPS.md` entry. _Resolution: fixed in design cycle 3._ -**Cycle 1 Important resolution assessment:** -| finding | materially resolved? | evidence / residual | +**Prior Important resolution assessment:** +| finding | status | assessment | |---|---|---| -| `D1` | No | Test design changed from broad grep to focused marker assertions, but still lacks RED/GREEN behavior proof required by `skills/writing-skills/SKILL.md` and `testing-skills-with-subagents.md`. Kept open as Important. | -| `D2` | Yes | Parity boundary now explicitly defers browser/event capture (`docs/plans/2026-06-30-brainstorm-visualization-design.md:23-43`), ADR records host-neutral guidance over upstream runtime (`decisions/0005-visual-companion-instructions.md:12-31`). Residual canonical coverage-doc concern is downgraded to `D7` Minor. | -| `D3` | Yes | Failure rules now cover invalid/unsupported Mermaid, text source-of-truth, stale/contradictory visuals, accessibility/remote limits, decline behavior, and secrets/PII (`docs/plans/2026-06-30-brainstorm-visualization-design.md:71-80`). | - -**Prior minor resolution assessment:** -| finding | status | -|---|---| -| `D4` | Resolved/accepted with mitigation: design bounds the guide to examples/failure/accessibility rules and keeps `SKILL.md` authoritative (`docs/plans/2026-06-30-brainstorm-visualization-design.md:63-65`, `:118`). | -| `D5` | Resolved: process flow remains `dot`; Mermaid is scoped to optional user-facing visuals (`docs/plans/2026-06-30-brainstorm-visualization-design.md:60`, `:65`). | -| `D6` | Resolved: design uses a new focused test and keeps `tests/pipeline-evidence-doc-sync.sh` unchanged (`docs/plans/2026-06-30-brainstorm-visualization-design.md:66-69`). | +| `D1` marker-only validation did not prove behavior | Resolved | Design now requires `expected-behavior.md` fixture plus subagent pressure proof as primary validation; shell test is only a regression guard. | +| `D2` browser-parity boundary missing | Resolved | Design includes parity boundary, host matrix, and deferred browser/event capture. | +| `D3` visual failure modes missing | Resolved | Design includes invalid/unsupported rendering, stale/contradictory visuals, accessibility fallback, decline behavior, secrets/PII, and lazy-load rules. | **Bug-class scan transcript:** | Class | Result | Note | |---|---|---| -| Project-guidance conflicts | Clean | No `docs/design-guidance.md` or `docs/PORTFOLIO.md` exists; the design cites repo canon from `AGENTS.md`, `README.md`, and skill files, which matches the project-guidance fallback. | -| Assumptions under attack | Finding | `D1`: assumes instruction-marker tests prove behavior; `D9`: assumes token cost stays low without a lazy-load rule. | -| Repo-precedent conflicts | Finding | `D1` conflicts with skill-edit RED/GREEN behavior-testing precedent; `D7` conflicts with durable cross-LLM coverage-doc precedent. | -| Artifact-class precedent | Finding | `D7`: host capability matrices already live in `docs/cross-llm-coverage.md` / `tests/cross-llm-coverage.md`; `D8`: documented test-suite membership lives in `AGENTS.md`. | -| YAGNI violations | Finding | `D9`: a sidecar guide can become unnecessary session overhead unless lazy-loaded only when visual work actually happens. | -| Missing failure modes | Finding | `D9`: eager guide loading / visual overuse remains an unaddressed failure mode; prior visual-rendering failures from `D3` are resolved. | -| Security/privacy architecture | Clean | Scope avoids browser server, telemetry, network service, new dependency, auth boundary, and persistent session state; design requires redacted/synthetic examples and no secrets/PII. | -| Infrastructure impact | Finding | `D8`: adding a standalone test without wiring it into the documented suite creates maintenance/validation drift. | -| Multi-component validation | Finding | `D1`: marker tests do not prove the skill-consumer behavior boundary; `D8`: the test exists in design but is not integrated into the repo’s durable validation surface. | -| Declared integration proof | Finding | `D7`: design-local host matrix exists, but declared visual capability is not reflected in the repo’s canonical coverage artifacts. | -| Contributed UI rendering proof | Clean | No host shell route/widget/page is contributed; browser companion runtime and click capture are explicitly deferred. | -| Rollback story | Clean | Rollback is concrete markdown/test revert with no runtime state, migrations, or version pins (`docs/plans/2026-06-30-brainstorm-visualization-design.md:141-143`). | -| Simpler alternative not considered | Clean | Revised design explicitly considered inline-only MVP and rejected it as too cramped for examples/fallbacks (`docs/plans/2026-06-30-brainstorm-visualization-design.md:45-54`). | -| User-intent drift | Clean | Browser parity risk is now explicit: upstream browser/event capture is deferred, diagrams/mockups during brainstorm/design remain in scope, and ADR records the trade-off. | -| Existence/runtime-validity | Finding | `D8`: emitted test artifact is named, but the design does not prove the repo’s documented test consumer will pick it up; `D7`: canonical coverage consumers are not updated. | +| Project-guidance conflicts | Clean | No `docs/design-guidance.md`, `docs/PROJECTS.md`, or `docs/PORTFOLIO.md` exists; design uses repo canon from `AGENTS.md`, `README.md`, coverage docs, and skill files. | +| Assumptions under attack | Clean | Browser parity, Mermaid rendering, guide scope, and behavior-proof limitations are explicit assumptions with fallbacks. | +| Repo-precedent conflicts | Clean | Skill-edit TDD precedent is addressed with behavior fixture + subagent proof; process diagrams remain `dot`; coverage/docs artifacts are updated. | +| Artifact-class precedent | Clean | New test is documented in `AGENTS.md`; host capability matrix changes are planned in durable coverage docs. | +| YAGNI violations | Clean | Full browser runtime is deferred; guide is bounded and lazy-loaded. | +| Missing failure modes | Clean | Core visual control-flow and negative stale/contradictory visual path are covered; text remains source of truth. | +| Security/privacy architecture | Clean | No browser server, telemetry, network service, auth boundary, or persistent session state; guide requires synthetic/redacted examples. | +| Infrastructure impact | Clean | No cloud/network/runtime infrastructure; storage/test-suite impacts are explicit and reversible. | +| Multi-component validation | Clean | Skill behavior proof, shell regression guard, content guards, cross-ref guard, and coverage doc checks are planned. | +| Declared integration proof | Clean | Rendered diagrams are config-only/best-effort unless observed; browser companion is deferred. | +| Contributed UI rendering proof | Clean | No host UI route/widget/page is contributed. | +| Rollback story | Clean | Rollback lists concrete file classes to revert; no runtime state, migrations, or version pins. | +| Simpler alternative not considered | Clean | Copy-upstream, host-neutral, inline-only, and full-runtime alternatives are considered. | +| User-intent drift | Clean | Markdown/Mermaid companion satisfies #78 scope; browser/event capture parity is tracked as follow-up. | +| Existence/runtime-validity | Clean | Existing artifacts to edit are named; emitted artifacts and consumer tests are specified. | **Options the author may not have considered:** -1. **Behavior fixture before shell guard:** Keep `tests/brainstorm-visual-companion.sh`, but add a small scenario transcript/eval artifact under `skills/brainstorming/test-fixtures/visual-companion/` that captures the RED/GREEN behavior the shell script cannot express. Trade-off: more artifact maintenance, but it matches `writing-skills` precedent for process-skill edits. -2. **Canonical matrix update instead of design-only matrix:** Move the host visual capability matrix rows into `docs/cross-llm-coverage.md` and summarize them in the design. Trade-off: broader doc change, but future host-portability reviews won’t have to rediscover a matrix buried in a dated plan. -3. **Inline-only until behavior proof exists:** Delay the sidecar guide and ship only a concise `SKILL.md` section plus behavior proof. Trade-off: fewer examples initially, but removes lazy-loading/token-risk and can be expanded once agents demonstrate the rule works. +1. **Full runtime now:** implement upstream-like browser/event capture in this PR. Rejected by design as too broad for host-neutral validation. +2. **Inline-only MVP:** avoid the guide file. Rejected because examples/failure/accessibility rules would bloat `SKILL.md`; mitigated by lazy-load. +3. **All diagrams text-only:** avoid Mermaid entirely. Rejected because issue explicitly asks for diagrams; mitigated by text fallback and best-effort rendering labels. -**Verdict reasoning:** The revised design materially resolves the browser-parity boundary (`D2`) and visual failure-rules gap (`D3`), and the prior minor concerns `D4`-`D6` are adequately mitigated. It still fails because `D1` remains an Important validation flaw: the proposed RED/GREEN test is still marker-based, not behavior-based, despite repo precedent requiring skill edits to prove agent behavior under realistic scenarios. The remaining new findings are Minor, but `D1` is tangible enough to keep design review at FAIL until the design adds a representative skill-behavior proof or explicitly records why this skill edit is exempt from that precedent. +**Verdict reasoning:** PASS. The revised design resolves all Critical/Important findings. Remaining concerns were minor and have been folded into the design: no overclaimed Mermaid runtime integration, clear question-batch semantics, negative behavior fixture coverage, and a durable follow-up for deferred browser parity. diff --git a/docs/plans/2026-06-30-brainstorm-visualization-design.md b/docs/plans/2026-06-30-brainstorm-visualization-design.md index 0420bd7..b717893 100644 --- a/docs/plans/2026-06-30-brainstorm-visualization-design.md +++ b/docs/plans/2026-06-30-brainstorm-visualization-design.md @@ -2,7 +2,7 @@ **Issue:** #78 — Brainstorm visualization **Date:** 2026-06-30 -**Status:** Approved by user instruction; revised after adversarial review cycles 1-2 +**Status:** Approved by user instruction; revised after adversarial review cycles 1-3 **ADR:** `decisions/0005-visual-companion-instructions.md` ## Original Ask @@ -34,7 +34,7 @@ | host | Markdown text | Mermaid/rendered diagrams | Browser companion | |---|---|---|---| -| `zed-agent` | runtime-integrated: markdown chat output | runtime-integrated where Zed renders Mermaid in assistant markdown; must include text fallback | deferred | +| `zed-agent` | runtime-integrated: markdown chat output | config-only/best-effort unless rendered output is explicitly observed; must include text fallback | deferred | | `claude-code` | config-only: markdown output | config-only/best-effort: render support varies by surface | deferred | | `codex` | config-only: markdown output | config-only/best-effort | deferred | | `opencode` | config-only: markdown output | config-only/best-effort | deferred | @@ -68,25 +68,27 @@ - RED before implementation: required skill markers/guide/fixture/coverage rows absent. - GREEN after implementation: asserts just-in-time offer, not-upfront rule, per-question decision, text fallback/source of truth, lazy-load rule, guide existence, guide mentions Mermaid plus accessibility/fallback, behavior fixture exists, and coverage/docs checks include visual companion. - Add behavior fixture `skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md`: - - Records pressure scenario and expected compliant behavior for four paths: conceptual text-only, visual offer in its own message, declined offer/no re-offer, accepted visual with text fallback. + - Records pressure scenario and expected compliant behavior for five paths: conceptual text-only, visual offer in its own message, declined offer/no re-offer, accepted visual with text fallback, and stale/contradictory visual after a changed decision. - Includes RED baseline notes against pre-change brainstorming: no visual companion rule exists, so the expected just-in-time offer and fallback paths are absent. - Run behavior proof with a subagent after implementation: - Prompt a reviewer with the fixture and current `skills/brainstorming/SKILL.md`. - - Expected: reviewer reports PASS for all four paths and cites the exact skill/guide lines that force the behavior. + - Expected: reviewer reports PASS for all five paths and cites the exact skill/guide lines that force the behavior. - This is the primary behavior proof; shell test is the regression guard. - Update durable docs: - `AGENTS.md`: add `bash tests/brainstorm-visual-companion.sh` to skill content checks. - `docs/cross-llm-coverage.md`: add visual companion capability row. - `tests/cross-llm-coverage.md`: update brainstorming notes with visual companion host-neutral/best-effort behavior. + - `docs/FOLLOWUPS.md`: record deferred browser/event-capture parity as a follow-up so #78 scope reduction is durable. ## Behavior Fixture Expected Paths | path | compliant behavior | |---|---| | Conceptual text-only question | Do not offer visual companion; ask/answer in text because the answer is conceptual. | -| Visual question arises | Send only the just-in-time offer message; no bundled clarifying question. | +| Visual question arises | Send only the just-in-time offer message; no bundled clarifying question; the offer counts as one question batch. | | User declines | Continue text-only and do not offer again unless user raises visuals. | | User accepts | Use the guide only when composing visuals; include text summary/fallback; record final decision in text. | +| Decision changes after visual | Retire or update stale/contradictory visual; preserve text as source of truth. | ## Failure Rules @@ -131,7 +133,7 @@ | Skill refs ↔ repo files | `bash tests/skill-cross-refs.sh` → references resolve. | | Coverage docs | `bash tests/brainstorm-visual-companion.sh` checks visual rows/notes exist. | | Full repo contract | Run AGENTS.md test list before PR. | -| Runtime rendering | Deferred except Zed markdown/Mermaid support where observed by host; all other hosts use text fallback and are marked config-only/best-effort. | +| Runtime rendering | Mermaid/rendered diagrams are config-only/best-effort unless a host-rendered output is explicitly observed; text fallback is required everywhere. | ## Assumptions @@ -165,7 +167,11 @@ | D7 | Add durable coverage doc updates. | | D8 | Add focused test to documented `AGENTS.md` checks. | | D9 | Add lazy-load rule to SKILL.md/guide/test. | +| D10 | Downgrade Mermaid rendering to config-only/best-effort unless explicitly observed. | +| D11 | Visual-companion offer counts as a question batch. | +| D12 | Add negative stale/contradictory-visual fixture path. | +| D13 | Add `docs/FOLLOWUPS.md` entry for deferred browser/event-capture parity. | ## Rollback -Revert commits modifying `skills/brainstorming/SKILL.md`, `skills/brainstorming/visual-companion.md`, `skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md`, `tests/brainstorm-visual-companion.sh`, `docs/cross-llm-coverage.md`, `tests/cross-llm-coverage.md`, `AGENTS.md`, and planning/ADR artifacts; rerun AGENTS.md test list. No runtime state, migrations, or version pins. +Revert commits modifying `skills/brainstorming/SKILL.md`, `skills/brainstorming/visual-companion.md`, `skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md`, `tests/brainstorm-visual-companion.sh`, `docs/cross-llm-coverage.md`, `tests/cross-llm-coverage.md`, `docs/FOLLOWUPS.md`, `AGENTS.md`, and planning/ADR artifacts; rerun AGENTS.md test list. No runtime state, migrations, or version pins. From 93435fb8cdf0b305e2293077b8491f8c38a1200e Mon Sep 17 00:00:00 2001 From: Jon Langevin Date: Tue, 30 Jun 2026 06:10:22 -0400 Subject: [PATCH 05/16] Plan brainstorm visualization implementation --- .../2026-06-30-brainstorm-visualization.md | 332 ++++++++++++++++++ 1 file changed, 332 insertions(+) create mode 100644 docs/plans/2026-06-30-brainstorm-visualization.md diff --git a/docs/plans/2026-06-30-brainstorm-visualization.md b/docs/plans/2026-06-30-brainstorm-visualization.md new file mode 100644 index 0000000..d2d2ba4 --- /dev/null +++ b/docs/plans/2026-06-30-brainstorm-visualization.md @@ -0,0 +1,332 @@ +# Brainstorm Visualization Implementation Plan + +> **For the implementing agent:** REQUIRED SUB-SKILL: Use autodev:executing-plans to implement this plan task-by-task. + +**Goal:** Add host-neutral brainstorm/design visualization companion guidance for issue #78. + +**Architecture:** Extend `skills/brainstorming/SKILL.md` with just-in-time visual companion rules while keeping existing gates intact. Add a lazy-loaded guide, behavior fixture, focused regression guard, coverage/doc updates, and a follow-up for deferred browser parity. + +**Tech Stack:** Markdown skills/docs; Bash regression tests; GitHub issue/PR workflow. + +**Base branch:** main + +--- + +## Scope Manifest + +**PR Count:** 1 +**Tasks:** 4 +**Estimated Lines of Change:** ~260 + +**Out of scope:** +- Browser server/runtime companion. +- Click/event capture or telemetry. +- Claiming Mermaid rendering is runtime-integrated on hosts where it has not been observed. + +**PR Grouping:** + +| PR # | Title | Tasks | Branch | +|------|-------|-------|--------| +| 1 | Add brainstorm visual companion guidance | Task 1, Task 2, Task 3, Task 4 | issue-78-brainstorm-visualization | + +**Status:** Draft + +--- + +## Task 1: Skill Regression Guard and Behavior Fixture + +**Files:** +- Create: `tests/brainstorm-visual-companion.sh` +- Create: `skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md` + +**Step 1: Write the failing regression guard** + +Create `tests/brainstorm-visual-companion.sh`: + +```bash +#!/usr/bin/env bash +# Regression guard for issue #78 brainstorm visualization companion. +set -uo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +BRAIN="$ROOT/skills/brainstorming/SKILL.md" +GUIDE="$ROOT/skills/brainstorming/visual-companion.md" +FIXTURE="$ROOT/skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md" +AGENTS="$ROOT/AGENTS.md" +CAPS="$ROOT/docs/cross-llm-coverage.md" +SKILL_COVERAGE="$ROOT/tests/cross-llm-coverage.md" +fail=0 +pass(){ printf 'PASS: %s\n' "$1"; } +bad(){ printf 'FAIL: %s\n' "$1" >&2; fail=$((fail+1)); } +has(){ grep -qiF "$2" "$1"; } +hasE(){ grep -qiE "$2" "$1"; } + +has "$BRAIN" "Visual Companion" && pass "brainstorming has Visual Companion section" || bad "brainstorming missing Visual Companion section" +has "$BRAIN" "just-in-time" && pass "visual companion offer is just-in-time" || bad "visual companion offer is not just-in-time" +hasE "$BRAIN" 'not[ -]upfront|NOT upfront' && pass "visual companion is not offered upfront" || bad "visual companion upfront prohibition missing" +has "$BRAIN" "per-question" && pass "visual decision is per-question" || bad "per-question decision rule missing" +has "$BRAIN" "text remains the source of truth" && pass "text source-of-truth rule present" || bad "text source-of-truth rule missing" +has "$BRAIN" "question batch" && pass "visual offer batch-budget rule present" || bad "visual offer batch-budget rule missing" +has "$BRAIN" "lazy-load" && pass "guide lazy-load rule present" || bad "guide lazy-load rule missing" +has "$BRAIN" "visual-companion.md" && pass "guide reference present" || bad "guide reference missing" + +[ -f "$GUIDE" ] && pass "visual companion guide exists" || bad "visual companion guide missing" +if [ -f "$GUIDE" ]; then + has "$GUIDE" "Mermaid" && pass "guide mentions Mermaid" || bad "guide missing Mermaid guidance" + has "$GUIDE" "accessibility" && pass "guide mentions accessibility" || bad "guide missing accessibility guidance" + has "$GUIDE" "fallback" && pass "guide mentions fallback" || bad "guide missing fallback guidance" + has "$GUIDE" "secrets" && pass "guide forbids secrets" || bad "guide missing secrets guidance" +fi + +[ -f "$FIXTURE" ] && pass "behavior fixture exists" || bad "behavior fixture missing" +if [ -f "$FIXTURE" ]; then + for marker in "conceptual text-only" "visual offer" "declines" "accepts" "stale visual" "RED baseline"; do + has "$FIXTURE" "$marker" && pass "fixture covers $marker" || bad "fixture missing $marker" + done +fi + +has "$AGENTS" "tests/brainstorm-visual-companion.sh" && pass "AGENTS documents visual companion test" || bad "AGENTS missing visual companion test" +has "$CAPS" "Visual companion" && pass "capability matrix includes visual companion" || bad "capability matrix missing visual companion" +has "$SKILL_COVERAGE" "visual companion" && pass "skill coverage notes visual companion" || bad "skill coverage missing visual companion note" + +echo ""; echo "Results: $fail failure(s)"; [ "$fail" -eq 0 ] +``` + +**Step 2: Verify RED** + +Run: `bash tests/brainstorm-visual-companion.sh` + +Expected: FAIL with at least these missing-contract messages: + +```text +FAIL: brainstorming missing Visual Companion section +FAIL: visual companion guide missing +FAIL: behavior fixture missing +FAIL: AGENTS missing visual companion test +``` + +**Step 3: Add behavior fixture** + +Create `skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md` with: + +```markdown +# Brainstorm Visual Companion Expected Behavior + +## RED baseline + +Pre-change `skills/brainstorming/SKILL.md` has no Visual Companion section, no just-in-time visual-offer rule, no lazy-load guide rule, and no stale visual rule. A baseline agent has no explicit reason to offer visuals only when useful or to preserve text as source of truth. + +## Pressure scenario + +A user asks an agent to brainstorm a dashboard redesign quickly. The session has limited question budget, mixed conceptual and layout decisions, and the user may decline visuals to save time. + +## Expected compliant paths + +| path | expected behavior | +|---|---| +| conceptual text-only | For conceptual questions such as "what does trust mean for this dashboard?", the agent stays in text and does not offer the visual companion. | +| visual offer | When the next decision is genuinely visual, the agent sends only the just-in-time visual-companion offer in its own message; it does not bundle a clarifying question. | +| declines | If the user declines, the agent continues text-only and does not offer again unless the user raises visuals. | +| accepts | If the user accepts, the agent lazy-loads `visual-companion.md` only while composing visual artifacts, includes a concise text fallback, and records the final decision in text. | +| stale visual | If a visual becomes stale or contradicts a later text decision, the agent retires or updates it and treats text as the source of truth. | +``` + +**Step 4: Commit policy** + +Do not commit yet if the test is still red. Continue Task 2 to make this guard pass. + +--- + +## Task 2: Brainstorming Skill and Visual Companion Guide + +**Files:** +- Modify: `skills/brainstorming/SKILL.md` +- Create: `skills/brainstorming/visual-companion.md` +- Uses tests from Task 1. + +**Step 1: Update `SKILL.md`** + +Modify `skills/brainstorming/SKILL.md`: + +- Checklist: insert after project guidance: + - `Offer the visual companion just-in-time — never upfront; only when a question would genuinely be clearer shown than described.` +- Process flow: add an `Offer visual companion if useful` box between project guidance and clarifying questions. +- Understanding/process bullets: + - Visual offer is its own message. + - It counts as one question batch. + - Use per-question decision: visual only if seeing beats reading. + - If declined, continue text-only and do not re-offer unless the user raises visuals. + - Lazy-load `skills/brainstorming/visual-companion.md` only after acceptance or while composing a visual artifact. +- Add `## Visual Companion` section: + - Visuals are a tool, not a mode. + - Use visuals for mockups, Mermaid diagrams, flows, state diagrams, architecture maps, side-by-side layout comparisons. + - Use terminal/text for requirements, conceptual choices, tradeoff tables, scope decisions. + - Text remains the source of truth. + - Every visual needs a concise text equivalent/fallback. + - Invalid/unsupported Mermaid → fall back to text. + - Stale/contradictory visual → retire/update it. + - No secrets/PII in visuals. + +**Step 2: Add guide** + +Create `skills/brainstorming/visual-companion.md` with bounded reference content: + +- `# Visual Companion Guide` +- `## Lazy-load rule` +- `## When to use visuals` +- `## When to stay text-only` +- `## Mermaid and diagram examples` +- `## Mockups and comparisons` +- `## Accessibility and fallback` +- `## Privacy and safety` +- `## Stale visuals` + +**Step 3: Verify GREEN for skill/guide markers** + +Run: `bash tests/brainstorm-visual-companion.sh` + +Expected: still FAIL only for durable docs not yet updated: + +```text +FAIL: AGENTS missing visual companion test +FAIL: capability matrix missing visual companion +FAIL: skill coverage missing visual companion note +``` + +If any skill/guide/fixture assertion fails, fix before continuing. + +--- + +## Task 3: Durable Docs and Follow-Up + +**Files:** +- Modify: `AGENTS.md` +- Modify: `docs/cross-llm-coverage.md` +- Modify: `tests/cross-llm-coverage.md` +- Create: `docs/FOLLOWUPS.md` + +**Step 1: Update documented checks** + +In `AGENTS.md`, under skill content checks, add: + +```bash +bash tests/brainstorm-visual-companion.sh +``` + +**Step 2: Update capability matrix** + +In `docs/cross-llm-coverage.md`, add a capability row: + +```markdown +| Visual companion output | ⚠️ markdown/Mermaid best-effort; browser deferred | ⚠️ markdown/Mermaid best-effort; browser deferred | ⚠️ markdown/Mermaid best-effort; browser deferred | ⚠️ markdown/Mermaid best-effort; browser deferred | ⚠️ markdown/Mermaid best-effort; browser deferred | ⚠️ markdown plus best-effort Mermaid rendering; browser deferred | +``` + +Add a note: rendered diagrams require text fallback; browser/event capture is deferred. + +**Step 3: Update skill coverage** + +In `tests/cross-llm-coverage.md`, update brainstorming note to mention: + +```text +Visual companion guidance is host-neutral and best-effort; browser/event capture is deferred. +``` + +**Step 4: Add deferred follow-up** + +Create `docs/FOLLOWUPS.md` if absent: + +```markdown +# Follow-ups + +## Brainstorm visual companion browser parity + +- **Source:** issue #78 / `docs/plans/2026-06-30-brainstorm-visualization-design.md` +- **Status:** Deferred +- **Follow-up:** Evaluate whether ADK should ship an upstream-like browser companion with tab launch, click/event capture, session security, cleanup, and host process validation. +``` + +**Step 5: Verify GREEN** + +Run: `bash tests/brainstorm-visual-companion.sh` + +Expected: + +```text +PASS: brainstorming has Visual Companion section +... +Results: 0 failure(s) +``` + +**Step 6: Commit Tasks 1-3** + +Run: + +```bash +git add AGENTS.md docs/cross-llm-coverage.md tests/cross-llm-coverage.md docs/FOLLOWUPS.md tests/brainstorm-visual-companion.sh skills/brainstorming/SKILL.md skills/brainstorming/visual-companion.md skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md +git commit -m "Add brainstorm visual companion guidance" +``` + +--- + +## Task 4: Behavior Proof and Full Validation + +**Files:** +- No production file changes expected unless behavior proof finds a gap. +- PR body must include behavior proof summary and test output. + +**Step 1: Run subagent behavior proof** + +Dispatch a reviewer with this prompt: + +```text +Read `skills/brainstorming/SKILL.md`, `skills/brainstorming/visual-companion.md`, and `skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md`. + +Assess whether the current skill instructions force all expected behavior paths: conceptual text-only, visual offer, declines, accepts, stale visual. Bias toward finding gaps. Return PASS only if each path is explicitly supported by skill/guide text, citing the lines/sections. +``` + +Expected: PASS for all five paths. + +If FAIL: update `SKILL.md` or guide minimally, rerun `bash tests/brainstorm-visual-companion.sh`, rerun behavior proof, and commit fixes. + +**Step 2: Run targeted checks** + +Run: + +```bash +bash tests/brainstorm-visual-companion.sh +bash tests/skill-content-grep.sh +bash tests/skill-cross-refs.sh +bash tests/no-machine-paths.sh +``` + +Expected: all PASS / `Results: 0 failure(s)`. + +**Step 3: Run full documented suite** + +Run AGENTS.md commands: + +```bash +bash tests/skill-content-grep.sh +bash tests/adk-path-canonicalization.sh +bash tests/pipeline-evidence-doc-sync.sh +bash tests/skill-cross-refs.sh +bash tests/brainstorm-visual-companion.sh +bash tests/hook-contracts.sh +bash tests/hook-stdout-discipline.sh +bash tests/plan-scope-check-contracts.sh +bash tests/version-check.sh +``` + +Expected: all PASS, `OK: All version files agree on version 6.6.1`. + +**Step 4: Commit validation fixes if needed** + +If Step 1-3 required edits: + +```bash +git add +git commit -m "Tighten visual companion validation" +``` + +If no edits, no commit. + +**Rollback:** revert the implementation commit(s) and rerun the full documented suite. No runtime state or migrations. From 99d370c78eb3bb5a7b4ccbb34cd4c363c295cf9a Mon Sep 17 00:00:00 2001 From: Jon Langevin Date: Tue, 30 Jun 2026 06:19:29 -0400 Subject: [PATCH 06/16] Address brainstorm visualization plan review --- ...30-brainstorm-visualization-plan-review.md | 53 ++++ .../2026-06-30-brainstorm-visualization.md | 235 ++++++++++-------- 2 files changed, 185 insertions(+), 103 deletions(-) create mode 100644 docs/plans/2026-06-30-brainstorm-visualization-plan-review.md diff --git a/docs/plans/2026-06-30-brainstorm-visualization-plan-review.md b/docs/plans/2026-06-30-brainstorm-visualization-plan-review.md new file mode 100644 index 0000000..b45dffe --- /dev/null +++ b/docs/plans/2026-06-30-brainstorm-visualization-plan-review.md @@ -0,0 +1,53 @@ +### Adversarial Review Report + +**Phase:** plan +**Artifact:** docs/plans/2026-06-30-brainstorm-visualization.md +**Status:** FAIL + +**Findings (Critical):** +- None. + +**Findings (Important):** +- `P1` [Verification-class mismatch / Repo-precedent conflicts]: Plan did not run/capture a RED baseline pressure scenario without the skill; absence-of-text was not behavior evidence. Recommendation: add baseline subject-agent pressure run and record actual missing behavior. +- `P2` [Missing integration proof / Verification-class mismatch / Multi-component validation]: GREEN behavior proof was static reviewer-only and did not show an agent applying the skill under pressure. Recommendation: add subject-agent pressure run plus reviewer scoring. +- `P3` [Over/under-decomposition / Hidden serial dependencies]: Tasks 1-3 modeled serial TDD phases as separate task units and left Task 1 intentionally red. Recommendation: collapse into one integrated TDD task or make each task independently green. +- `P4` [Missing declared integration matrix / Declared integration proof]: Plan did not preserve design host visual capability matrix as an executable declared-integration matrix. Recommendation: add plan-level matrix marking markdown, Mermaid, browser, and click/event capture as config-only/deferred with proof/rationale. + +**Findings (Minor):** +- `P5` [Artifact-class precedent / YAGNI]: Guide scope not capped enough. Recommendation: bound guide to examples/failure/accessibility; keep `SKILL.md` authoritative. +- `P6` [Documentation verification mismatch]: Markdown table/docs verification was mostly string grep. Recommendation: add or justify lightweight structural checks. +- `P7` [Planned-code compile-validity]: Embedded Bash used weaker `set -uo pipefail`. Recommendation: use `set -euo pipefail` or justify omission. + +**Bug-class scan transcript:** +| Class | Result | Note | +|---|---|---| +| Project-guidance conflicts | Finding | `writing-skills` requires true RED baseline pressure evidence and behavior proof (`P1`, `P2`). | +| Assumptions under attack | Finding | Assumed textual coverage maps to behavior (`P2`). | +| Repo-precedent conflicts | Finding | Skill-edit precedent requires pressure scenarios and observed behavior (`P1`, `P2`). | +| Artifact-class precedent | Finding | Guide file needed tighter bounded scope (`P5`). | +| YAGNI violations | Finding | Guide could grow beyond accepted scope (`P5`). | +| Missing failure modes | Clean | Visual failure modes are planned. | +| Security/privacy architecture | Clean | No runtime server/telemetry/auth/session state. | +| Infrastructure impact | Clean | No infra changes. | +| Multi-component validation | Finding | Behavior proof did not exercise actual skill application (`P2`). | +| Declared integration proof | Finding | Plan-level matrix missing (`P4`). | +| Contributed UI rendering proof | Clean | No UI contribution. | +| Rollback story | Clean | Revert markdown/test/docs changes. | +| Simpler alternative not considered | Clean | Alternatives covered in design. | +| User-intent drift | Clean | Browser/event parity deferred and tracked. | +| Existence/runtime-validity | Clean | Existing files named; absent follow-up doc created. | +| Over/under-decomposition | Finding | Serial red/green phases split across tasks (`P3`). | +| Verification-class mismatch | Finding | Marker/static proof mismatch for skill behavior (`P1`, `P2`). | +| Hidden serial dependencies | Finding | Task 2 depended on Task 1 red state (`P3`). | +| Missing rollback wiring | Clean | Rollback wired. | +| Missing integration proof | Finding | Subject-agent proof missing (`P2`). | +| Missing declared integration matrix | Finding | Plan-level matrix missing (`P4`). | +| Identifier/naming-convention match | Clean | Names follow repo conventions. | +| Planned-code compile-validity | Finding | Bash strictness concern (`P7`). | + +**Options the author may not have considered:** +1. One integrated TDD task for skill behavior. +2. Transcript-backed fixture/proof artifact. +3. Inline-only first PR. + +**Verdict reasoning:** FAIL. The plan could pass marker checks without proving behavior, split one serial TDD cycle into misleading tasks, and omitted a plan-level declared integration matrix. diff --git a/docs/plans/2026-06-30-brainstorm-visualization.md b/docs/plans/2026-06-30-brainstorm-visualization.md index d2d2ba4..69591b0 100644 --- a/docs/plans/2026-06-30-brainstorm-visualization.md +++ b/docs/plans/2026-06-30-brainstorm-visualization.md @@ -4,9 +4,9 @@ **Goal:** Add host-neutral brainstorm/design visualization companion guidance for issue #78. -**Architecture:** Extend `skills/brainstorming/SKILL.md` with just-in-time visual companion rules while keeping existing gates intact. Add a lazy-loaded guide, behavior fixture, focused regression guard, coverage/doc updates, and a follow-up for deferred browser parity. +**Architecture:** Extend `skills/brainstorming/SKILL.md` with just-in-time visual companion rules while keeping existing gates intact. Add a lazy-loaded guide, behavior fixture, focused regression guard, coverage/doc updates, follow-up tracking, and transcript-backed pressure proof. -**Tech Stack:** Markdown skills/docs; Bash regression tests; GitHub issue/PR workflow. +**Tech Stack:** Markdown skills/docs; Bash regression tests; subagent pressure testing; GitHub issue/PR workflow. **Base branch:** main @@ -15,8 +15,8 @@ ## Scope Manifest **PR Count:** 1 -**Tasks:** 4 -**Estimated Lines of Change:** ~260 +**Tasks:** 3 +**Estimated Lines of Change:** ~300 **Out of scope:** - Browser server/runtime companion. @@ -27,26 +27,56 @@ | PR # | Title | Tasks | Branch | |------|-------|-------|--------| -| 1 | Add brainstorm visual companion guidance | Task 1, Task 2, Task 3, Task 4 | issue-78-brainstorm-visualization | +| 1 | Add brainstorm visual companion guidance | Task 1, Task 2, Task 3 | issue-78-brainstorm-visualization | **Status:** Draft --- -## Task 1: Skill Regression Guard and Behavior Fixture +## Declared Integration Matrix + +| declared visual capability | host(s) | state | plan proof / rationale | +|---|---|---|---| +| Markdown text visual explanations | all supported hosts | config-only | Skill emits normal markdown/chat text; no host-specific runtime API changes. `tests/brainstorm-visual-companion.sh` asserts text fallback/source-of-truth contract. | +| Mermaid/rendered diagrams | all supported hosts | config-only / best-effort | Skill may emit Mermaid, but rendering is not claimed unless observed in a specific host. Every Mermaid visual requires text fallback. | +| Browser companion tab | all supported hosts | deferred | Tracked in `docs/FOLLOWUPS.md`; needs separate runtime/process/security design. | +| Click/event capture | all supported hosts | deferred | Tracked in `docs/FOLLOWUPS.md`; no browser session state in this PR. | + +--- + +## Task 1: Integrated TDD Implementation **Files:** - Create: `tests/brainstorm-visual-companion.sh` - Create: `skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md` +- Modify: `skills/brainstorming/SKILL.md` +- Create: `skills/brainstorming/visual-companion.md` +- Modify: `AGENTS.md` +- Modify: `docs/cross-llm-coverage.md` +- Modify: `tests/cross-llm-coverage.md` +- Create: `docs/FOLLOWUPS.md` + +**Step 1: RED behavior baseline** + +Dispatch a baseline subject agent against the current pre-change `skills/brainstorming/SKILL.md` with this prompt: -**Step 1: Write the failing regression guard** +```text +You are testing current brainstorming behavior before issue #78 is implemented. +Read `skills/brainstorming/SKILL.md` only. +Scenario: A user is brainstorming a dashboard redesign quickly. First they ask a conceptual question: "what does trust mean for this dashboard?" Later they ask a visual layout choice that would be clearer shown than described. Then they decline visuals to save time. Finally they change a decision after a diagram would have existed. +Return what the current skill explicitly requires for: conceptual text-only, visual offer, declined offer/no re-offer, accepted visual with text fallback, stale visual. Cite skill lines/sections. If the skill lacks a rule, say MISSING. +``` + +Expected RED: baseline output reports MISSING for visual offer, declined/no re-offer, accepted visual/text fallback, and stale visual. Save the summarized baseline in `docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md` after Task 2 GREEN proof. + +**Step 2: Write failing shell regression guard** Create `tests/brainstorm-visual-companion.sh`: ```bash #!/usr/bin/env bash # Regression guard for issue #78 brainstorm visualization companion. -set -uo pipefail +set -euo pipefail ROOT="$(cd "$(dirname "$0")/.." && pwd)" BRAIN="$ROOT/skills/brainstorming/SKILL.md" GUIDE="$ROOT/skills/brainstorming/visual-companion.md" @@ -75,6 +105,7 @@ if [ -f "$GUIDE" ]; then has "$GUIDE" "accessibility" && pass "guide mentions accessibility" || bad "guide missing accessibility guidance" has "$GUIDE" "fallback" && pass "guide mentions fallback" || bad "guide missing fallback guidance" has "$GUIDE" "secrets" && pass "guide forbids secrets" || bad "guide missing secrets guidance" + has "$GUIDE" "SKILL.md remains authoritative" && pass "guide is bounded by SKILL.md authority" || bad "guide authority boundary missing" fi [ -f "$FIXTURE" ] && pass "behavior fixture exists" || bad "behavior fixture missing" @@ -86,27 +117,27 @@ fi has "$AGENTS" "tests/brainstorm-visual-companion.sh" && pass "AGENTS documents visual companion test" || bad "AGENTS missing visual companion test" has "$CAPS" "Visual companion" && pass "capability matrix includes visual companion" || bad "capability matrix missing visual companion" +has "$CAPS" "browser deferred" && pass "capability matrix tracks browser deferral" || bad "capability matrix missing browser deferral" has "$SKILL_COVERAGE" "visual companion" && pass "skill coverage notes visual companion" || bad "skill coverage missing visual companion note" echo ""; echo "Results: $fail failure(s)"; [ "$fail" -eq 0 ] ``` -**Step 2: Verify RED** - Run: `bash tests/brainstorm-visual-companion.sh` -Expected: FAIL with at least these missing-contract messages: +Expected RED: ```text FAIL: brainstorming missing Visual Companion section FAIL: visual companion guide missing FAIL: behavior fixture missing FAIL: AGENTS missing visual companion test +Results: failure(s) ``` **Step 3: Add behavior fixture** -Create `skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md` with: +Create `skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md`: ```markdown # Brainstorm Visual Companion Expected Behavior @@ -130,20 +161,7 @@ A user asks an agent to brainstorm a dashboard redesign quickly. The session has | stale visual | If a visual becomes stale or contradicts a later text decision, the agent retires or updates it and treats text as the source of truth. | ``` -**Step 4: Commit policy** - -Do not commit yet if the test is still red. Continue Task 2 to make this guard pass. - ---- - -## Task 2: Brainstorming Skill and Visual Companion Guide - -**Files:** -- Modify: `skills/brainstorming/SKILL.md` -- Create: `skills/brainstorming/visual-companion.md` -- Uses tests from Task 1. - -**Step 1: Update `SKILL.md`** +**Step 4: Update brainstorming skill** Modify `skills/brainstorming/SKILL.md`: @@ -166,128 +184,129 @@ Modify `skills/brainstorming/SKILL.md`: - Stale/contradictory visual → retire/update it. - No secrets/PII in visuals. -**Step 2: Add guide** +**Step 5: Add bounded guide** -Create `skills/brainstorming/visual-companion.md` with bounded reference content: +Create `skills/brainstorming/visual-companion.md` with these sections only: `Lazy-load rule`, `When to use visuals`, `When to stay text-only`, `Mermaid and diagram examples`, `Mockups and comparisons`, `Accessibility and fallback`, `Privacy and safety`, `Stale visuals`. Include one Mermaid example and one mockup guidance list. State: `SKILL.md remains authoritative`; do not duplicate full brainstorming workflow. -- `# Visual Companion Guide` -- `## Lazy-load rule` -- `## When to use visuals` -- `## When to stay text-only` -- `## Mermaid and diagram examples` -- `## Mockups and comparisons` -- `## Accessibility and fallback` -- `## Privacy and safety` -- `## Stale visuals` +**Step 6: Update durable docs** -**Step 3: Verify GREEN for skill/guide markers** +- `AGENTS.md`: add `bash tests/brainstorm-visual-companion.sh` under skill content checks. +- `docs/cross-llm-coverage.md`: add `Visual companion output` row with markdown/Mermaid best-effort and browser deferred for every host. Add a note that rendered diagrams require text fallback and browser/event capture is deferred. +- `tests/cross-llm-coverage.md`: update brainstorming note: `Visual companion guidance is host-neutral and best-effort; browser/event capture is deferred.` +- Create `docs/FOLLOWUPS.md`: -Run: `bash tests/brainstorm-visual-companion.sh` +```markdown +# Follow-ups -Expected: still FAIL only for durable docs not yet updated: +## Brainstorm visual companion browser parity -```text -FAIL: AGENTS missing visual companion test -FAIL: capability matrix missing visual companion -FAIL: skill coverage missing visual companion note +- **Source:** issue #78 / `docs/plans/2026-06-30-brainstorm-visualization-design.md` +- **Status:** Deferred +- **Follow-up:** Evaluate whether ADK should ship an upstream-like browser companion with tab launch, click/event capture, session security, cleanup, and host process validation. ``` -If any skill/guide/fixture assertion fails, fix before continuing. +**Step 7: Verify GREEN** ---- +Run: `bash tests/brainstorm-visual-companion.sh` -## Task 3: Durable Docs and Follow-Up +Expected: -**Files:** -- Modify: `AGENTS.md` -- Modify: `docs/cross-llm-coverage.md` -- Modify: `tests/cross-llm-coverage.md` -- Create: `docs/FOLLOWUPS.md` +```text +PASS: brainstorming has Visual Companion section +... +Results: 0 failure(s) +``` -**Step 1: Update documented checks** +**Step 8: Commit** -In `AGENTS.md`, under skill content checks, add: +Run: ```bash -bash tests/brainstorm-visual-companion.sh +git add AGENTS.md docs/cross-llm-coverage.md tests/cross-llm-coverage.md docs/FOLLOWUPS.md tests/brainstorm-visual-companion.sh skills/brainstorming/SKILL.md skills/brainstorming/visual-companion.md skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md +git commit -m "Add brainstorm visual companion guidance" ``` -**Step 2: Update capability matrix** +**Rollback:** revert this commit; rerun `bash tests/brainstorm-visual-companion.sh` only if keeping the test, otherwise rerun full suite after revert. + +--- -In `docs/cross-llm-coverage.md`, add a capability row: +## Task 2: Transcript-Backed Behavior Proof -```markdown -| Visual companion output | ⚠️ markdown/Mermaid best-effort; browser deferred | ⚠️ markdown/Mermaid best-effort; browser deferred | ⚠️ markdown/Mermaid best-effort; browser deferred | ⚠️ markdown/Mermaid best-effort; browser deferred | ⚠️ markdown/Mermaid best-effort; browser deferred | ⚠️ markdown plus best-effort Mermaid rendering; browser deferred | +**Files:** +- Create: `docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md` +- Modify implementation files only if proof finds gaps. + +**Step 1: GREEN subject-agent pressure run** + +Dispatch a subject agent with: + +```text +Use `skills/brainstorming/SKILL.md` and, only if the skill instructs you to, `skills/brainstorming/visual-companion.md`. +Scenario: A user is brainstorming a dashboard redesign quickly. First they ask the conceptual question "what does trust mean for this dashboard?" Later they ask a visual layout choice that would be clearer shown than described. Then they decline visuals to save time. Finally imagine they had accepted a visual and then changed a decision so the visual is stale. +Return the exact messages/actions you would take for: conceptual text-only, visual offer, declines, accepts, stale visual. Cite the skill/guide rule that governs each path. ``` -Add a note: rendered diagrams require text fallback; browser/event capture is deferred. +Expected: subject demonstrates all five fixture paths and cites current skill/guide rules. -**Step 3: Update skill coverage** +**Step 2: Reviewer scores the transcript** -In `tests/cross-llm-coverage.md`, update brainstorming note to mention: +Dispatch a reviewer with the subject output plus `skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md`: ```text -Visual companion guidance is host-neutral and best-effort; browser/event capture is deferred. +Score the subject transcript against all expected paths. Bias toward finding failures. PASS only if conceptual text-only, visual offer, declines, accepts, and stale visual paths match the fixture and cite rules. ``` -**Step 4: Add deferred follow-up** +Expected: PASS. -Create `docs/FOLLOWUPS.md` if absent: +**Step 3: Write proof artifact** -```markdown -# Follow-ups +Create `docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md`: -## Brainstorm visual companion browser parity +```markdown +# Brainstorm Visualization Behavior Proof -- **Source:** issue #78 / `docs/plans/2026-06-30-brainstorm-visualization-design.md` -- **Status:** Deferred -- **Follow-up:** Evaluate whether ADK should ship an upstream-like browser companion with tab launch, click/event capture, session security, cleanup, and host process validation. -``` +## RED baseline -**Step 5: Verify GREEN** + -Run: `bash tests/brainstorm-visual-companion.sh` +## GREEN subject run -Expected: + -```text -PASS: brainstorming has Visual Companion section -... -Results: 0 failure(s) -``` +## Reviewer score -**Step 6: Commit Tasks 1-3** +PASS for: conceptual text-only, visual offer, declines, accepts, stale visual. -Run: +## Evidence -```bash -git add AGENTS.md docs/cross-llm-coverage.md tests/cross-llm-coverage.md docs/FOLLOWUPS.md tests/brainstorm-visual-companion.sh skills/brainstorming/SKILL.md skills/brainstorming/visual-companion.md skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md -git commit -m "Add brainstorm visual companion guidance" +- `bash tests/brainstorm-visual-companion.sh` → `Results: 0 failure(s)` ``` ---- +**Step 4: Fix and re-run if needed** -## Task 4: Behavior Proof and Full Validation +If reviewer FAILs: update `skills/brainstorming/SKILL.md` or guide minimally, rerun `bash tests/brainstorm-visual-companion.sh`, rerun subject + reviewer proof, then continue. -**Files:** -- No production file changes expected unless behavior proof finds a gap. -- PR body must include behavior proof summary and test output. +**Step 5: Commit proof/fixes** -**Step 1: Run subagent behavior proof** +Run: -Dispatch a reviewer with this prompt: +```bash +git add docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md skills/brainstorming/SKILL.md skills/brainstorming/visual-companion.md +git commit -m "Prove visual companion behavior" +``` -```text -Read `skills/brainstorming/SKILL.md`, `skills/brainstorming/visual-companion.md`, and `skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md`. +If no implementation fix was needed, commit only the proof artifact. -Assess whether the current skill instructions force all expected behavior paths: conceptual text-only, visual offer, declines, accepts, stale visual. Bias toward finding gaps. Return PASS only if each path is explicitly supported by skill/guide text, citing the lines/sections. -``` +**Rollback:** revert proof/fix commit; no runtime state. -Expected: PASS for all five paths. +--- -If FAIL: update `SKILL.md` or guide minimally, rerun `bash tests/brainstorm-visual-companion.sh`, rerun behavior proof, and commit fixes. +## Task 3: Full Validation and PR Prep -**Step 2: Run targeted checks** +**Files:** +- No production file changes expected unless validation finds a gap. + +**Step 1: Run targeted checks** Run: @@ -300,7 +319,7 @@ bash tests/no-machine-paths.sh Expected: all PASS / `Results: 0 failure(s)`. -**Step 3: Run full documented suite** +**Step 2: Run full documented suite** Run AGENTS.md commands: @@ -318,9 +337,9 @@ bash tests/version-check.sh Expected: all PASS, `OK: All version files agree on version 6.6.1`. -**Step 4: Commit validation fixes if needed** +**Step 3: Commit validation fixes if needed** -If Step 1-3 required edits: +If Step 1-2 required edits: ```bash git add @@ -329,4 +348,14 @@ git commit -m "Tighten visual companion validation" If no edits, no commit. -**Rollback:** revert the implementation commit(s) and rerun the full documented suite. No runtime state or migrations. +**Step 4: PR body evidence** + +PR body must include: + +- Issue: closes #78. +- RED baseline summary from behavior proof. +- GREEN subject/reviewer behavior proof summary. +- Targeted and full-suite command outputs. +- Deferred browser/event-capture parity follow-up path: `docs/FOLLOWUPS.md`. + +**Rollback:** revert implementation/proof commits and rerun the full documented suite. No runtime state or migrations. From 3eeefdd893d2ea7a6f7ed67fceba9655381677b1 Mon Sep 17 00:00:00 2001 From: Jon Langevin Date: Tue, 30 Jun 2026 06:27:19 -0400 Subject: [PATCH 07/16] Make visualization plan host-neutral --- .../2026-06-30-brainstorm-visualization.md | 33 ++++++++++++++----- 1 file changed, 24 insertions(+), 9 deletions(-) diff --git a/docs/plans/2026-06-30-brainstorm-visualization.md b/docs/plans/2026-06-30-brainstorm-visualization.md index 69591b0..15d4d2d 100644 --- a/docs/plans/2026-06-30-brainstorm-visualization.md +++ b/docs/plans/2026-06-30-brainstorm-visualization.md @@ -44,6 +44,17 @@ --- +## Pressure-Proof Execution Protocol + +Behavior proof means an isolated subject applies the skill under pressure; it is not self-review. + +1. Prefer a native subagent/delegation tool when the current host exposes one. +2. If native subagents are unavailable, use a fresh isolated host thread/session in the same worktree and paste the exact subject prompt. +3. Capture the subject output and reviewer score in `docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md`. +4. If no isolated subject can be run, stop before implementation and record the blocker; do not replace the proof with self-review. + +--- + ## Task 1: Integrated TDD Implementation **Files:** @@ -92,7 +103,7 @@ hasE(){ grep -qiE "$2" "$1"; } has "$BRAIN" "Visual Companion" && pass "brainstorming has Visual Companion section" || bad "brainstorming missing Visual Companion section" has "$BRAIN" "just-in-time" && pass "visual companion offer is just-in-time" || bad "visual companion offer is not just-in-time" -hasE "$BRAIN" 'not[ -]upfront|NOT upfront' && pass "visual companion is not offered upfront" || bad "visual companion upfront prohibition missing" +hasE "$BRAIN" 'not[ -]upfront|never[ -]upfront|NOT upfront' && pass "visual companion is not offered upfront" || bad "visual companion upfront prohibition missing" has "$BRAIN" "per-question" && pass "visual decision is per-question" || bad "per-question decision rule missing" has "$BRAIN" "text remains the source of truth" && pass "text source-of-truth rule present" || bad "text source-of-truth rule missing" has "$BRAIN" "question batch" && pass "visual offer batch-budget rule present" || bad "visual offer batch-budget rule missing" @@ -125,7 +136,7 @@ echo ""; echo "Results: $fail failure(s)"; [ "$fail" -eq 0 ] Run: `bash tests/brainstorm-visual-companion.sh` -Expected RED: +Expected RED representative failures (the full output may include additional missing-marker failures such as `just-in-time`, `per-question`, `question batch`, and `lazy-load`): ```text FAIL: brainstorming missing Visual Companion section @@ -236,9 +247,9 @@ git commit -m "Add brainstorm visual companion guidance" - Create: `docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md` - Modify implementation files only if proof finds gaps. -**Step 1: GREEN subject-agent pressure run** +**Step 1: GREEN isolated-subject pressure run** -Dispatch a subject agent with: +Use the Pressure-Proof Execution Protocol above: native subagent if available; otherwise a fresh isolated host thread/session in the same worktree. Give the isolated subject this prompt: ```text Use `skills/brainstorming/SKILL.md` and, only if the skill instructs you to, `skills/brainstorming/visual-companion.md`. @@ -306,7 +317,11 @@ If no implementation fix was needed, commit only the proof artifact. **Files:** - No production file changes expected unless validation finds a gap. -**Step 1: Run targeted checks** +**Step 1: Documentation validation note** + +This repo has no dedicated markdown renderer/spellchecker command. For markdown/doc changes, use the focused shell guard, `skill-cross-refs`, `no-machine-paths`, and manual editor preview where available. + +**Step 2: Run targeted checks** Run: @@ -319,7 +334,7 @@ bash tests/no-machine-paths.sh Expected: all PASS / `Results: 0 failure(s)`. -**Step 2: Run full documented suite** +**Step 3: Run full documented suite** Run AGENTS.md commands: @@ -337,9 +352,9 @@ bash tests/version-check.sh Expected: all PASS, `OK: All version files agree on version 6.6.1`. -**Step 3: Commit validation fixes if needed** +**Step 4: Commit validation fixes if needed** -If Step 1-2 required edits: +If Step 2-3 required edits: ```bash git add @@ -348,7 +363,7 @@ git commit -m "Tighten visual companion validation" If no edits, no commit. -**Step 4: PR body evidence** +**Step 5: PR body evidence** PR body must include: From 165b1b570e242a69590aae40883a2a9aa5f7d92c Mon Sep 17 00:00:00 2001 From: Jon Langevin Date: Tue, 30 Jun 2026 06:36:21 -0400 Subject: [PATCH 08/16] Pass brainstorm visualization plan review --- ...30-brainstorm-visualization-plan-review.md | 82 +++++++++++-------- .../2026-06-30-brainstorm-visualization.md | 27 ++++-- 2 files changed, 67 insertions(+), 42 deletions(-) diff --git a/docs/plans/2026-06-30-brainstorm-visualization-plan-review.md b/docs/plans/2026-06-30-brainstorm-visualization-plan-review.md index b45dffe..ee3979f 100644 --- a/docs/plans/2026-06-30-brainstorm-visualization-plan-review.md +++ b/docs/plans/2026-06-30-brainstorm-visualization-plan-review.md @@ -2,52 +2,64 @@ **Phase:** plan **Artifact:** docs/plans/2026-06-30-brainstorm-visualization.md -**Status:** FAIL +**Status:** PASS **Findings (Critical):** - None. **Findings (Important):** -- `P1` [Verification-class mismatch / Repo-precedent conflicts]: Plan did not run/capture a RED baseline pressure scenario without the skill; absence-of-text was not behavior evidence. Recommendation: add baseline subject-agent pressure run and record actual missing behavior. -- `P2` [Missing integration proof / Verification-class mismatch / Multi-component validation]: GREEN behavior proof was static reviewer-only and did not show an agent applying the skill under pressure. Recommendation: add subject-agent pressure run plus reviewer scoring. -- `P3` [Over/under-decomposition / Hidden serial dependencies]: Tasks 1-3 modeled serial TDD phases as separate task units and left Task 1 intentionally red. Recommendation: collapse into one integrated TDD task or make each task independently green. -- `P4` [Missing declared integration matrix / Declared integration proof]: Plan did not preserve design host visual capability matrix as an executable declared-integration matrix. Recommendation: add plan-level matrix marking markdown, Mermaid, browser, and click/event capture as config-only/deferred with proof/rationale. +- None. **Findings (Minor):** -- `P5` [Artifact-class precedent / YAGNI]: Guide scope not capped enough. Recommendation: bound guide to examples/failure/accessibility; keep `SKILL.md` authoritative. -- `P6` [Documentation verification mismatch]: Markdown table/docs verification was mostly string grep. Recommendation: add or justify lightweight structural checks. -- `P7` [Planned-code compile-validity]: Embedded Bash used weaker `set -uo pipefail`. Recommendation: use `set -euo pipefail` or justify omission. +- `P12` [Verification-class mismatch / Repo-precedent conflicts]: RED baseline should capture natural behavior before explicit rule audit. _Resolution: plan now instructs subject to first answer actual messages/actions, then mark missing requirements._ +- `P13` [Missing integration proof]: Reviewer should verify subject citations against current skill/guide files. _Resolution: reviewer prompt now includes `SKILL.md` and `visual-companion.md` and requires real-rule verification._ +- `P14` [Project-guidance conflicts / Verification-class mismatch]: Task 1 commit should not happen before core content/cross-ref/path checks. _Resolution: Task 1 now runs focused guard, content grep, cross-ref, and no-machine-path checks before commit._ +- `P15` [Missing failure modes / Artifact capture]: RED baseline proof should be captured immediately. _Resolution: Task 1 now creates the proof artifact immediately after RED and Task 2 appends GREEN evidence._ + +**Prior Important resolution assessment:** +| finding | status | assessment | +|---|---|---| +| `P1` RED baseline pressure evidence missing | Resolved | Plan requires isolated baseline subject run and immediate proof capture. | +| `P2` GREEN subject-agent behavior proof missing | Resolved | Plan requires isolated GREEN subject run plus reviewer scoring all five fixture paths against real skill/guide rules. | +| `P3` serial TDD phases split across misleading tasks | Resolved | Plan uses one integrated TDD implementation task, then proof, then validation. | +| `P4` declared integration matrix missing | Resolved | Plan-level matrix marks markdown, Mermaid, browser tab, and click/event capture as config-only/best-effort or deferred with proof/rationale. | +| `P8` host-neutral pressure-proof protocol | Resolved | Plan defines native subagent first, fresh isolated host thread/session fallback, and stop-if-no-isolated-subject behavior. | **Bug-class scan transcript:** | Class | Result | Note | |---|---|---| -| Project-guidance conflicts | Finding | `writing-skills` requires true RED baseline pressure evidence and behavior proof (`P1`, `P2`). | -| Assumptions under attack | Finding | Assumed textual coverage maps to behavior (`P2`). | -| Repo-precedent conflicts | Finding | Skill-edit precedent requires pressure scenarios and observed behavior (`P1`, `P2`). | -| Artifact-class precedent | Finding | Guide file needed tighter bounded scope (`P5`). | -| YAGNI violations | Finding | Guide could grow beyond accepted scope (`P5`). | -| Missing failure modes | Clean | Visual failure modes are planned. | -| Security/privacy architecture | Clean | No runtime server/telemetry/auth/session state. | -| Infrastructure impact | Clean | No infra changes. | -| Multi-component validation | Finding | Behavior proof did not exercise actual skill application (`P2`). | -| Declared integration proof | Finding | Plan-level matrix missing (`P4`). | -| Contributed UI rendering proof | Clean | No UI contribution. | -| Rollback story | Clean | Revert markdown/test/docs changes. | -| Simpler alternative not considered | Clean | Alternatives covered in design. | -| User-intent drift | Clean | Browser/event parity deferred and tracked. | -| Existence/runtime-validity | Clean | Existing files named; absent follow-up doc created. | -| Over/under-decomposition | Finding | Serial red/green phases split across tasks (`P3`). | -| Verification-class mismatch | Finding | Marker/static proof mismatch for skill behavior (`P1`, `P2`). | -| Hidden serial dependencies | Finding | Task 2 depended on Task 1 red state (`P3`). | -| Missing rollback wiring | Clean | Rollback wired. | -| Missing integration proof | Finding | Subject-agent proof missing (`P2`). | -| Missing declared integration matrix | Finding | Plan-level matrix missing (`P4`). | -| Identifier/naming-convention match | Clean | Names follow repo conventions. | -| Planned-code compile-validity | Finding | Bash strictness concern (`P7`). | +| Project-guidance conflicts | Clean | Skill-edit TDD/behavior-proof guidance is represented in tasks. | +| Assumptions under attack | Clean | Browser parity, Mermaid rendering, isolated-subject availability, and behavior-proof limits have explicit fallbacks. | +| Repo-precedent conflicts | Clean | Skill testing uses RED/GREEN pressure proof; guide is bounded/lazy-loaded; docs/tests are validated. | +| Artifact-class precedent | Clean | Fixture, focused shell guard, proof doc, and coverage docs follow repo shapes. | +| YAGNI violations | Clean | Browser runtime/event capture/telemetry/session state deferred; guide stays bounded. | +| Missing failure modes | Clean | Invalid Mermaid, stale/contradictory visuals, declines, text fallback, accessibility, secrets/PII, and lazy-load violations are covered. | +| Security/privacy architecture | Clean | No auth, server, telemetry, network, session, or secret flow added. | +| Infrastructure impact | Clean | No cloud/network/migration/version/runtime loader impact. | +| Multi-component validation | Clean | Plan includes shell guard, isolated subject proof, reviewer scoring, content/cross-ref/path checks, and full suite. | +| Declared integration proof | Clean | Matrix avoids overclaiming runtime rendering; browser/click capture are deferred with follow-up. | +| Contributed UI rendering proof | Clean | No host UI contribution. | +| Rollback story | Clean | Revert commits; no runtime state/migrations. | +| Simpler alternative not considered | Clean | Alternatives handled in design. | +| User-intent drift | Clean | Host-neutral visual companion satisfies #78; browser parity tracked as follow-up. | +| Existence/runtime-validity | Clean | Existing files named; emitted files have consumers in validation commands. | +| Over/under-decomposition | Clean | Three serial tasks match implementation/proof/validation phases. | +| Verification-class mismatch | Clean | Documentation and skill behavior changes have class-appropriate checks and proof. | +| Auth/authz chain composition | Clean | No auth/authz chain. | +| Hidden serial dependencies | Clean | Serial dependencies are explicit. | +| Missing rollback wiring | Clean | Rollback wired per task. | +| Missing integration proof | Clean | Isolated-subject and reviewer proof cover skill→agent behavior boundary. | +| Missing declared integration matrix | Clean | Present in plan. | +| Missing contributed UI route proof | Clean | Not applicable. | +| Infrastructure verification mismatch | Clean | No infrastructure change. | +| Plugin-loader runtime layout | Clean | No plugin process/layout change. | +| Config-validation schema rules | Clean | No schema config. | +| Identifier/naming-convention match | Clean | Names follow repo kebab-case/test-fixture conventions. | +| Planned-code compile-validity | Clean | Embedded Bash uses valid syntax and `set -euo pipefail`; no compiled snippets. | **Options the author may not have considered:** -1. One integrated TDD task for skill behavior. -2. Transcript-backed fixture/proof artifact. -3. Inline-only first PR. +1. Commit RED proof separately before implementation; rejected as extra commit noise, mitigated by immediate proof artifact capture. +2. Inline-only companion; rejected in design, mitigated by bounded lazy-loaded guide. +3. Full browser runtime now; rejected as separate runtime/security/process scope. -**Verdict reasoning:** FAIL. The plan could pass marker checks without proving behavior, split one serial TDD cycle into misleading tasks, and omitted a plan-level declared integration matrix. +**Verdict reasoning:** PASS. The plan now has no Critical or Important findings. Minor proof-capture and validation ambiguities have been resolved in the plan text. diff --git a/docs/plans/2026-06-30-brainstorm-visualization.md b/docs/plans/2026-06-30-brainstorm-visualization.md index 15d4d2d..927afe5 100644 --- a/docs/plans/2026-06-30-brainstorm-visualization.md +++ b/docs/plans/2026-06-30-brainstorm-visualization.md @@ -75,10 +75,10 @@ Dispatch a baseline subject agent against the current pre-change `skills/brainst You are testing current brainstorming behavior before issue #78 is implemented. Read `skills/brainstorming/SKILL.md` only. Scenario: A user is brainstorming a dashboard redesign quickly. First they ask a conceptual question: "what does trust mean for this dashboard?" Later they ask a visual layout choice that would be clearer shown than described. Then they decline visuals to save time. Finally they change a decision after a diagram would have existed. -Return what the current skill explicitly requires for: conceptual text-only, visual offer, declined offer/no re-offer, accepted visual with text fallback, stale visual. Cite skill lines/sections. If the skill lacks a rule, say MISSING. +First answer what messages/actions you would actually take in the scenario using only the current skill. Then separately mark what the current skill explicitly requires for: conceptual text-only, visual offer, declined offer/no re-offer, accepted visual with text fallback, stale visual. Cite skill lines/sections. If the skill lacks a rule, say MISSING. ``` -Expected RED: baseline output reports MISSING for visual offer, declined/no re-offer, accepted visual/text fallback, and stale visual. Save the summarized baseline in `docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md` after Task 2 GREEN proof. +Expected RED: baseline output reports MISSING for visual offer, declined/no re-offer, accepted visual/text fallback, and stale visual. Immediately create `docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md` with the raw or summarized RED baseline output; append GREEN proof in Task 2. **Step 2: Write failing shell regression guard** @@ -228,12 +228,25 @@ PASS: brainstorming has Visual Companion section Results: 0 failure(s) ``` -**Step 8: Commit** +**Step 8: Pre-commit quick checks** Run: ```bash -git add AGENTS.md docs/cross-llm-coverage.md tests/cross-llm-coverage.md docs/FOLLOWUPS.md tests/brainstorm-visual-companion.sh skills/brainstorming/SKILL.md skills/brainstorming/visual-companion.md skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md +bash tests/brainstorm-visual-companion.sh +bash tests/skill-content-grep.sh +bash tests/skill-cross-refs.sh +bash tests/no-machine-paths.sh +``` + +Expected: all PASS / `Results: 0 failure(s)`. + +**Step 9: Commit** + +Run: + +```bash +git add AGENTS.md docs/cross-llm-coverage.md tests/cross-llm-coverage.md docs/FOLLOWUPS.md tests/brainstorm-visual-companion.sh skills/brainstorming/SKILL.md skills/brainstorming/visual-companion.md skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md git commit -m "Add brainstorm visual companion guidance" ``` @@ -261,17 +274,17 @@ Expected: subject demonstrates all five fixture paths and cites current skill/gu **Step 2: Reviewer scores the transcript** -Dispatch a reviewer with the subject output plus `skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md`: +Dispatch a reviewer with the subject output plus `skills/brainstorming/SKILL.md`, `skills/brainstorming/visual-companion.md`, and `skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md`: ```text -Score the subject transcript against all expected paths. Bias toward finding failures. PASS only if conceptual text-only, visual offer, declines, accepts, and stale visual paths match the fixture and cite rules. +Score the subject transcript against all expected paths. Verify each cited rule exists in the current skill or guide. Bias toward finding failures. PASS only if conceptual text-only, visual offer, declines, accepts, and stale visual paths match the fixture and cite real rules. ``` Expected: PASS. **Step 3: Write proof artifact** -Create `docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md`: +Append GREEN proof to `docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md`: ```markdown # Brainstorm Visualization Behavior Proof From fa41c3f7c9c0a3b4b9823ecf4bcaed3b02768eba Mon Sep 17 00:00:00 2001 From: Jon Langevin Date: Tue, 30 Jun 2026 06:37:11 -0400 Subject: [PATCH 09/16] Fix visualization plan task headings --- docs/plans/2026-06-30-brainstorm-visualization.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/plans/2026-06-30-brainstorm-visualization.md b/docs/plans/2026-06-30-brainstorm-visualization.md index 927afe5..c1b14f0 100644 --- a/docs/plans/2026-06-30-brainstorm-visualization.md +++ b/docs/plans/2026-06-30-brainstorm-visualization.md @@ -55,7 +55,7 @@ Behavior proof means an isolated subject applies the skill under pressure; it is --- -## Task 1: Integrated TDD Implementation +### Task 1: Integrated TDD Implementation **Files:** - Create: `tests/brainstorm-visual-companion.sh` @@ -254,7 +254,7 @@ git commit -m "Add brainstorm visual companion guidance" --- -## Task 2: Transcript-Backed Behavior Proof +### Task 2: Transcript-Backed Behavior Proof **Files:** - Create: `docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md` @@ -325,7 +325,7 @@ If no implementation fix was needed, commit only the proof artifact. --- -## Task 3: Full Validation and PR Prep +### Task 3: Full Validation and PR Prep **Files:** - No production file changes expected unless validation finds a gap. From 1bc08a4d73d71af366a8354be7827ffc0904b7c7 Mon Sep 17 00:00:00 2001 From: Jon Langevin Date: Tue, 30 Jun 2026 06:37:51 -0400 Subject: [PATCH 10/16] Lock brainstorm visualization scope --- docs/plans/2026-06-30-brainstorm-visualization.md | 2 +- docs/plans/2026-06-30-brainstorm-visualization.md.scope-lock | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) create mode 100644 docs/plans/2026-06-30-brainstorm-visualization.md.scope-lock diff --git a/docs/plans/2026-06-30-brainstorm-visualization.md b/docs/plans/2026-06-30-brainstorm-visualization.md index c1b14f0..77b0fb8 100644 --- a/docs/plans/2026-06-30-brainstorm-visualization.md +++ b/docs/plans/2026-06-30-brainstorm-visualization.md @@ -29,7 +29,7 @@ |------|-------|-------|--------| | 1 | Add brainstorm visual companion guidance | Task 1, Task 2, Task 3 | issue-78-brainstorm-visualization | -**Status:** Draft +**Status:** Locked 2026-06-30T10:37:25Z --- diff --git a/docs/plans/2026-06-30-brainstorm-visualization.md.scope-lock b/docs/plans/2026-06-30-brainstorm-visualization.md.scope-lock new file mode 100644 index 0000000..c5e02d3 --- /dev/null +++ b/docs/plans/2026-06-30-brainstorm-visualization.md.scope-lock @@ -0,0 +1 @@ +889fc3ebf145472d192fcd78939c5edf14b962cac3a05717110f90cda3893915 From 848a27ddc5ba24380c4a1dcbc52141f441b4e1de Mon Sep 17 00:00:00 2001 From: Jon Langevin Date: Tue, 30 Jun 2026 06:43:59 -0400 Subject: [PATCH 11/16] Add brainstorm visual companion guidance --- AGENTS.md | 1 + docs/FOLLOWUPS.md | 7 +++ docs/cross-llm-coverage.md | 3 + ...brainstorm-visualization-behavior-proof.md | 34 +++++++++++ skills/brainstorming/SKILL.md | 38 ++++++++++--- .../visual-companion/expected-behavior.md | 19 +++++++ skills/brainstorming/visual-companion.md | 57 +++++++++++++++++++ tests/brainstorm-visual-companion.sh | 47 +++++++++++++++ tests/cross-llm-coverage.md | 2 +- 9 files changed, 198 insertions(+), 10 deletions(-) create mode 100644 docs/FOLLOWUPS.md create mode 100644 docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md create mode 100644 skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md create mode 100644 skills/brainstorming/visual-companion.md create mode 100644 tests/brainstorm-visual-companion.sh diff --git a/AGENTS.md b/AGENTS.md index ed3eaec..f865154 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,6 +26,7 @@ bash tests/skill-content-grep.sh bash tests/adk-path-canonicalization.sh bash tests/pipeline-evidence-doc-sync.sh bash tests/skill-cross-refs.sh +bash tests/brainstorm-visual-companion.sh # Hook contract tests bash tests/hook-contracts.sh diff --git a/docs/FOLLOWUPS.md b/docs/FOLLOWUPS.md new file mode 100644 index 0000000..e351c6c --- /dev/null +++ b/docs/FOLLOWUPS.md @@ -0,0 +1,7 @@ +# Follow-ups + +## Brainstorm visual companion browser parity + +- **Source:** issue #78 / `docs/plans/2026-06-30-brainstorm-visualization-design.md` +- **Status:** Deferred +- **Follow-up:** Evaluate whether ADK should ship an upstream-like browser companion with tab launch, click/event capture, session security, cleanup, and host process validation. diff --git a/docs/cross-llm-coverage.md b/docs/cross-llm-coverage.md index fa297ee..eb709ea 100644 --- a/docs/cross-llm-coverage.md +++ b/docs/cross-llm-coverage.md @@ -18,6 +18,7 @@ Host-by-host capability matrix for the Autonomous Dev Kit skills system. | Host declaration for skill conditionals | `Host: claude-code` in CLAUDE.md | `Host: codex` in `~/.codex/AGENTS.md` | `Host: opencode` in `~/.config/opencode/AGENTS.md` | n/a | Auto-detected as `hermes-agent` | `Host: zed-agent` in `~/.config/zed/AGENTS.md` or `%APPDATA%\\Zed\\AGENTS.md` | | Skill discovery path (user scope) | `~/.claude/skills/` (personal skills); autodev installed to `~/.claude/plugins/marketplace/autodev/` via marketplace | `~/.agents/skills/` | `~/.config/opencode/skills/` | via plugin (no manual symlink) | `~/.hermes/skills/` (symlink or `hermes skills install`) | `~/.agents/skills//` (flat; no namespace folder) | | Model tier vocabulary | role names → `haiku`/`sonnet`/`opus` (see `agents/model-tiers.md`) | role names → `gpt-5.4-mini`/`gpt-5.4`/`gpt-5.5` | role names → host-pass-through | role names → host-pass-through | role names → host-pass-through | role names → host-pass-through | +| Visual companion output | ⚠️ markdown/Mermaid best-effort; browser deferred | ⚠️ markdown/Mermaid best-effort; browser deferred | ⚠️ markdown/Mermaid best-effort; browser deferred | ⚠️ markdown/Mermaid best-effort; browser deferred | ⚠️ markdown/Mermaid best-effort; browser deferred | ⚠️ markdown plus best-effort Mermaid rendering; browser deferred | ## Notes @@ -31,6 +32,8 @@ Host-by-host capability matrix for the Autonomous Dev Kit skills system. **Zed Agent:** Zed Skills apply only to Zed Agent, not ACP External Agents or Terminal Threads. Zed requires a flat skills layout, so ADK installs each `skills//` directory directly under `~/.agents/skills/` or `/.agents/skills/`; project-local skills require a trusted worktree. Zed does not expose the Claude/Codex lifecycle hook contract for skill packages. Current Zed task hooks are narrower (`create_worktree` is the documented event), so ADK hook-driven reminders use manual skill fallbacks and optional Zed Tool Permissions on Zed. +**Visual companion output:** Brainstorming may emit markdown, mockups, or Mermaid diagrams where useful, but rendered diagrams are best-effort and every visual needs a text fallback. Browser companion and click/event capture are deferred. + ## Related files - `tests/cross-llm-coverage.md` — per-skill host-conditional vs host-neutral audit diff --git a/docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md b/docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md new file mode 100644 index 0000000..7207ab6 --- /dev/null +++ b/docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md @@ -0,0 +1,34 @@ +# Brainstorm Visualization Behavior Proof + +## RED baseline + +Baseline subject read pre-change `skills/brainstorming/SKILL.md` only. + +### Actual behavior summary + +- Conceptual prompt (`what does trust mean for this dashboard?`) → subject stayed text-only by default using existing clarifying-question workflow. +- Visual layout choice → subject would still present 2–3 text approaches with trade-offs and numbered options. +- Declined visuals → subject would continue text-only, but only by inference because no visual-offer protocol exists. +- Changed decision after diagram → subject would revise design text; no stale-diagram handling exists. + +### Missing explicit requirements + +| path | baseline result | +|---|---| +| conceptual text-only | MISSING as a distinct visual-companion rule; only general clarifying-question/design rules exist. | +| visual offer | MISSING; no diagrams/mockups/visualization offer rule. | +| declines | MISSING; no declined-offer/no-reoffer state rule. | +| accepts | MISSING; no accepted-visual/text-fallback rule. | +| stale visual | MISSING; no stale visual update/retire rule. | + +## GREEN subject run + +Pending implementation. + +## Reviewer score + +Pending implementation. + +## Evidence + +Pending implementation. diff --git a/skills/brainstorming/SKILL.md b/skills/brainstorming/SKILL.md index 9aae96e..c0b1377 100644 --- a/skills/brainstorming/SKILL.md +++ b/skills/brainstorming/SKILL.md @@ -27,14 +27,15 @@ You MUST create a task for each of these items and complete them in order: 1. **Explore project context** — check files, docs, recent commits 2. **Apply project design guidance** — invoke `autodev:project-design-guidance`; read existing guidance or run its Q&A if no durable guidance exists -3. **Ask clarifying questions** — adaptive batching: group related questions to reduce round-trips; use targeted singles for follow-ups -4. **Propose 2-3 approaches** — with trade-offs and your recommendation -5. **List load-bearing assumptions explicitly** — every design rests on assumptions ("upstream API is idempotent", "single-tenant", "user has admin"); write them down so the adversarial reviewer can attack them -6. **Self-challenge round** — before presenting to the user, role-play a skeptic against your own design and surface the top 3 doubts (see "Self-challenge round" below). Cleans up obvious issues before the user sees the design. -7. **Present design** — in sections scaled to their complexity, get user approval after each section. Include the assumption list, project-guidance fit, and the top doubts surfaced by the self-challenge so the user sees them. -8. **Write design doc** — save to `docs/plans/YYYY-MM-DD--design.md` and commit. Use `autodev:condensed-pipeline-writing` for internal density. The doc MUST include `## Global Design Guidance`, `## Security Review`, `## Infrastructure Impact`, `## Multi-Component Validation`, `## Assumptions`, and a `## Rollback` section for change classes that affect runtime (build, deployment, version pins, startup config, migrations, plugin loading paths) — same trigger list as `runtime-launch-validation`. -9. **Adversarial design review** — invoke `adversarial-design-review --phase=design`. On FAIL, revise per tangible Critical/Important findings and re-run until the review converges to no tangible issues (remaining nitpicks become Minor PASS items). On PASS, proceed. -10. **Transition to implementation** — invoke writing-plans skill to create implementation plan +3. **Offer the visual companion just-in-time** — never upfront; only when a question would genuinely be clearer shown than described +4. **Ask clarifying questions** — adaptive batching: group related questions to reduce round-trips; use targeted singles for follow-ups +5. **Propose 2-3 approaches** — with trade-offs and your recommendation +6. **List load-bearing assumptions explicitly** — every design rests on assumptions ("upstream API is idempotent", "single-tenant", "user has admin"); write them down so the adversarial reviewer can attack them +7. **Self-challenge round** — before presenting to the user, role-play a skeptic against your own design and surface the top 3 doubts (see "Self-challenge round" below). Cleans up obvious issues before the user sees the design. +8. **Present design** — in sections scaled to their complexity, get user approval after each section. Include the assumption list, project-guidance fit, and the top doubts surfaced by the self-challenge so the user sees them. +9. **Write design doc** — save to `docs/plans/YYYY-MM-DD--design.md` and commit. Use `autodev:condensed-pipeline-writing` for internal density. The doc MUST include `## Global Design Guidance`, `## Security Review`, `## Infrastructure Impact`, `## Multi-Component Validation`, `## Assumptions`, and a `## Rollback` section for change classes that affect runtime (build, deployment, version pins, startup config, migrations, plugin loading paths) — same trigger list as `runtime-launch-validation`. +10. **Adversarial design review** — invoke `adversarial-design-review --phase=design`. On FAIL, revise per tangible Critical/Important findings and re-run until the review converges to no tangible issues (remaining nitpicks become Minor PASS items). On PASS, proceed. +11. **Transition to implementation** — invoke writing-plans skill to create implementation plan ## Process Flow @@ -42,6 +43,7 @@ You MUST create a task for each of these items and complete them in order: digraph brainstorming { "Explore project context" [shape=box]; "Apply project guidance" [shape=box]; + "Offer visual companion if useful" [shape=box]; "Ask clarifying questions" [shape=box]; "Propose 2-3 approaches" [shape=box]; "List assumptions" [shape=box]; @@ -53,7 +55,8 @@ digraph brainstorming { "Invoke writing-plans skill" [shape=doublecircle]; "Explore project context" -> "Apply project guidance"; - "Apply project guidance" -> "Ask clarifying questions"; + "Apply project guidance" -> "Offer visual companion if useful"; + "Offer visual companion if useful" -> "Ask clarifying questions"; "Ask clarifying questions" -> "Propose 2-3 approaches"; "Propose 2-3 approaches" -> "List assumptions"; "List assumptions" -> "Self-challenge round"; @@ -74,6 +77,9 @@ digraph brainstorming { **Understanding the idea:** - Check out the current project state first (files, docs, recent commits) - Invoke `autodev:project-design-guidance` before finalizing questions or approaches. If no durable guidance exists, ask the guidance Q&A first so the feature-specific brainstorm is not designed in isolation. +- Offer the visual companion just-in-time, not upfront. Only offer when the next question would genuinely be clearer shown than described, and send only the offer in that message; the offer counts as one question batch. If the user declines, continue text-only and do not re-offer unless the user raises visuals. +- Decide per-question whether visuals help. Use visuals for mockups, Mermaid diagrams, flows, state diagrams, architecture maps, or side-by-side layout comparisons. Stay text-only for requirements, conceptual choices, tradeoff tables, and scope decisions. +- If the user accepts, lazy-load `skills/brainstorming/visual-companion.md` only while composing visual artifacts. Text remains the source of truth; every visual needs a concise text fallback. - Ask questions using adaptive batching — group related questions to reduce round-trips: - **First batch:** covers purpose, constraints, scope, and tech choices - **Follow-ups:** Targeted single questions based on interesting or ambiguous answers @@ -124,6 +130,20 @@ If any answer surfaces a real issue, revise the design before presenting it. Oth This is intentionally lightweight; the heavyweight pass is `adversarial-design-review`, which runs after the design is committed. +## Visual Companion + +Visuals are a tool, not a mode. Use them only when seeing the option, flow, relationship, or layout would help the user decide better than prose. + +**Offer rule:** The first visual-companion offer is just-in-time and never upfront. The offer must be its own message, counts as one question batch, and asks whether the user wants a visual aid. Do not bundle another clarifying question into the same message. + +**Use visuals for:** mockups, wireframes, Mermaid diagrams, state flows, entity relationships, architecture maps, spatial comparisons, and side-by-side visual design options. + +**Stay text-only for:** requirements questions, conceptual choices, tradeoff lists, scope decisions, implementation approaches, or any question where the answer is words rather than a visual preference. + +**Fallback and correctness:** text remains the source of truth. Every visual must include a concise text equivalent. If Mermaid is invalid or unsupported, continue in plain text and do not claim rendered proof. If a visual becomes stale or contradicts a later text decision, retire or update it before proceeding. Do not include secrets or PII in diagrams or mockups. + +**Guide loading:** lazy-load `skills/brainstorming/visual-companion.md` only after the user accepts the companion or when composing a visual artifact. `SKILL.md` remains authoritative; the guide provides examples, accessibility rules, and fallback details. + ## Question-batch budget (cost-control gate) A brainstorming session has a **soft cap of 5 question-batches** before the agent must converge. A "batch" is one round-trip — one outgoing message that contains questions (whether one targeted follow-up or four AskUserQuestion-style options). The cap exists so a session cannot spiral indefinitely while the user keeps answering; convergence is a feature, not an accident. diff --git a/skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md b/skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md new file mode 100644 index 0000000..066ec09 --- /dev/null +++ b/skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md @@ -0,0 +1,19 @@ +# Brainstorm Visual Companion Expected Behavior + +## RED baseline + +Pre-change `skills/brainstorming/SKILL.md` has no Visual Companion section, no just-in-time visual-offer rule, no lazy-load guide rule, and no stale visual rule. A baseline agent has no explicit reason to offer visuals only when useful or to preserve text as source of truth. + +## Pressure scenario + +A user asks an agent to brainstorm a dashboard redesign quickly. The session has limited question budget, mixed conceptual and layout decisions, and the user may decline visuals to save time. + +## Expected compliant paths + +| path | expected behavior | +|---|---| +| conceptual text-only | For conceptual questions such as "what does trust mean for this dashboard?", the agent stays in text and does not offer the visual companion. | +| visual offer | When the next decision is genuinely visual, the agent sends only the just-in-time visual-companion offer in its own message; it does not bundle a clarifying question. | +| declines | If the user declines, the agent continues text-only and does not offer again unless the user raises visuals. | +| accepts | If the user accepts, the agent lazy-loads `visual-companion.md` only while composing visual artifacts, includes a concise text fallback, and records the final decision in text. | +| stale visual | If a visual becomes stale or contradicts a later text decision, the agent retires or updates it and treats text as the source of truth. | diff --git a/skills/brainstorming/visual-companion.md b/skills/brainstorming/visual-companion.md new file mode 100644 index 0000000..1adc26b --- /dev/null +++ b/skills/brainstorming/visual-companion.md @@ -0,0 +1,57 @@ +# Visual Companion Guide + +## Lazy-load rule + +Load this guide only after the user accepts the Visual Companion or when composing a visual artifact. `SKILL.md remains authoritative`; this file provides examples, accessibility guidance, and fallback details. + +## When to use visuals + +Use a visual when seeing the structure is clearer than reading prose: + +- UI mockups, wireframes, navigation structures, layout comparisons +- Mermaid flowcharts, sequence diagrams, state diagrams, ER diagrams, timelines +- Architecture maps, data-flow diagrams, relationship maps +- Side-by-side visual options where spatial hierarchy matters + +## When to stay text-only + +Stay in terminal/chat text for conceptual or scope decisions: + +- Requirements and constraints +- Tradeoff tables and approach selection +- Domain language such as "what does trust mean?" +- Questions where the answer is words rather than visual preference + +## Mermaid and diagram examples + +Keep diagrams small and include a text fallback. + +```mermaid +flowchart TD + A[User goal] --> B[Design option] + B --> C{Decision} + C --> D[Record text decision] +``` + +Text fallback: "User goal leads to a design option, then a decision, then the decision is recorded in text." + +## Mockups and comparisons + +For mockups, show only the decision-relevant structure: + +- 2-4 options maximum +- Use synthetic/redacted content +- State what question the visual answers +- Summarize each option in text before asking for feedback + +## Accessibility and fallback + +Every visual needs a concise text fallback. If Mermaid does not render, if the user is remote/headless, or if the visual would be hard to read, continue in text. Accept text-only feedback as primary. + +## Privacy and safety + +Never include secrets, credentials, tokens, private customer data, or PII in visuals. Use placeholders or synthetic examples. + +## Stale visuals + +If the user changes a decision after a visual was shown, retire or update the stale visual. The current text design remains the source of truth. diff --git a/tests/brainstorm-visual-companion.sh b/tests/brainstorm-visual-companion.sh new file mode 100644 index 0000000..153adb4 --- /dev/null +++ b/tests/brainstorm-visual-companion.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +# Regression guard for issue #78 brainstorm visualization companion. +set -euo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +BRAIN="$ROOT/skills/brainstorming/SKILL.md" +GUIDE="$ROOT/skills/brainstorming/visual-companion.md" +FIXTURE="$ROOT/skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md" +AGENTS="$ROOT/AGENTS.md" +CAPS="$ROOT/docs/cross-llm-coverage.md" +SKILL_COVERAGE="$ROOT/tests/cross-llm-coverage.md" +fail=0 +pass(){ printf 'PASS: %s\n' "$1"; } +bad(){ printf 'FAIL: %s\n' "$1" >&2; fail=$((fail+1)); } +has(){ grep -qiF "$2" "$1"; } +hasE(){ grep -qiE "$2" "$1"; } + +has "$BRAIN" "Visual Companion" && pass "brainstorming has Visual Companion section" || bad "brainstorming missing Visual Companion section" +has "$BRAIN" "just-in-time" && pass "visual companion offer is just-in-time" || bad "visual companion offer is not just-in-time" +hasE "$BRAIN" 'not[ -]upfront|never[ -]upfront|NOT upfront' && pass "visual companion is not offered upfront" || bad "visual companion upfront prohibition missing" +has "$BRAIN" "per-question" && pass "visual decision is per-question" || bad "per-question decision rule missing" +has "$BRAIN" "text remains the source of truth" && pass "text source-of-truth rule present" || bad "text source-of-truth rule missing" +has "$BRAIN" "question batch" && pass "visual offer batch-budget rule present" || bad "visual offer batch-budget rule missing" +has "$BRAIN" "lazy-load" && pass "guide lazy-load rule present" || bad "guide lazy-load rule missing" +has "$BRAIN" "visual-companion.md" && pass "guide reference present" || bad "guide reference missing" + +[ -f "$GUIDE" ] && pass "visual companion guide exists" || bad "visual companion guide missing" +if [ -f "$GUIDE" ]; then + has "$GUIDE" "Mermaid" && pass "guide mentions Mermaid" || bad "guide missing Mermaid guidance" + has "$GUIDE" "accessibility" && pass "guide mentions accessibility" || bad "guide missing accessibility guidance" + has "$GUIDE" "fallback" && pass "guide mentions fallback" || bad "guide missing fallback guidance" + has "$GUIDE" "secrets" && pass "guide forbids secrets" || bad "guide missing secrets guidance" + has "$GUIDE" "SKILL.md remains authoritative" && pass "guide is bounded by SKILL.md authority" || bad "guide authority boundary missing" +fi + +[ -f "$FIXTURE" ] && pass "behavior fixture exists" || bad "behavior fixture missing" +if [ -f "$FIXTURE" ]; then + for marker in "conceptual text-only" "visual offer" "declines" "accepts" "stale visual" "RED baseline"; do + has "$FIXTURE" "$marker" && pass "fixture covers $marker" || bad "fixture missing $marker" + done +fi + +has "$AGENTS" "tests/brainstorm-visual-companion.sh" && pass "AGENTS documents visual companion test" || bad "AGENTS missing visual companion test" +has "$CAPS" "Visual companion" && pass "capability matrix includes visual companion" || bad "capability matrix missing visual companion" +has "$CAPS" "browser deferred" && pass "capability matrix tracks browser deferral" || bad "capability matrix missing browser deferral" +has "$SKILL_COVERAGE" "visual companion" && pass "skill coverage notes visual companion" || bad "skill coverage missing visual companion note" + +echo ""; echo "Results: $fail failure(s)"; [ "$fail" -eq 0 ] diff --git a/tests/cross-llm-coverage.md b/tests/cross-llm-coverage.md index acaa5fd..7f0a9d2 100644 --- a/tests/cross-llm-coverage.md +++ b/tests/cross-llm-coverage.md @@ -10,7 +10,7 @@ host-neutral. Updated whenever a skill changes. | post-merge-retrospective | host-conditional | host-conditional | host-conditional | host-conditional | host-conditional | host-conditional | Inline-vs-subagent decision block in ``; inline-only prose includes Zed; Hermes inline block | | recording-decisions | host-neutral | host-neutral | host-neutral | host-neutral | host-neutral | host-neutral | ADR storage protocol; no host-specific tooling | | scope-lock | host-neutral | host-neutral | host-neutral | host-neutral | host-neutral | host-neutral | Pure markdown + shell-script invariant; no host-specific tooling | -| brainstorming | host-conditional | host-conditional | host-conditional | host-conditional | host-conditional | host-conditional | `AskUserQuestion` in ``; numbered-list fallback includes Zed; Hermes uses `clarify` | +| brainstorming | host-conditional | host-conditional | host-conditional | host-conditional | host-conditional | host-conditional | `AskUserQuestion` in ``; numbered-list fallback includes Zed; Hermes uses `clarify`; visual companion guidance is host-neutral and best-effort; browser/event capture is deferred | | dispatching-parallel-agents | host-neutral | host-neutral | host-neutral | host-neutral | host-neutral | host-neutral | generic parallel-dispatch pattern; no tool-specific refs | | executing-plans | host-conditional | host-conditional | host-conditional | host-conditional | host-conditional | host-conditional | tool-use block in ``; prose fallback includes Zed; Hermes uses `todo` | | finishing-a-development-branch | host-neutral | host-neutral | host-neutral | host-neutral | host-neutral | host-neutral | audited clean — no forbidden tokens; bash-based throughout | From bfe3f1bc718bca5f331b13c8b2ddc1c05e8111bd Mon Sep 17 00:00:00 2001 From: Jon Langevin Date: Tue, 30 Jun 2026 06:48:42 -0400 Subject: [PATCH 12/16] Prove visual companion behavior --- ...brainstorm-visualization-behavior-proof.md | 26 ++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-) diff --git a/docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md b/docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md index 7207ab6..5e54820 100644 --- a/docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md +++ b/docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md @@ -23,12 +23,32 @@ Baseline subject read pre-change `skills/brainstorming/SKILL.md` only. ## GREEN subject run -Pending implementation. +Subject used the revised `skills/brainstorming/SKILL.md` and loaded `skills/brainstorming/visual-companion.md` only for the accepted-visual path. + +| path | subject behavior | +|---|---| +| conceptual text-only | Stayed text-only for `what does trust mean for this dashboard?`; did not load or offer the visual companion. | +| visual offer | Sent a standalone just-in-time offer: `This layout choice would be clearer shown than described. Want me to sketch 2–3 quick wireframe options before you choose?` | +| declines | Continued text-only and explicitly did not re-offer unless asked. | +| accepts | Lazy-loaded the guide, composed a small Mermaid-style visual, included a text fallback, and recorded recommendation in text. | +| stale visual | Retired the stale visual and stated the current text decision as source of truth. | ## Reviewer score -Pending implementation. +PASS overall. + +| path | reviewer verdict | cited rule | +|---|---|---| +| conceptual text-only | PASS | `SKILL.md` text-only rule + guide text-only domain-language rule. | +| visual offer | PASS | `SKILL.md` just-in-time, standalone, question-batch offer rule. | +| declines | PASS | `SKILL.md` decline/no-reoffer rule. | +| accepts | PASS | `SKILL.md` lazy-load + fallback rules; guide fallback guidance. | +| stale visual | PASS | `SKILL.md` text source-of-truth/stale visual rule; guide stale visual rule. | + +Reviewer note: accepted-visual transcript used `[Mermaid flowchart]` as shorthand in the reviewer prompt, so it proves behavior selection and fallback, not host-rendered Mermaid output. Rendering remains best-effort/config-only per design. ## Evidence -Pending implementation. +- `bash tests/brainstorm-visual-companion.sh` → `Results: 0 failure(s)` +- RED baseline isolated subject: MISSING for visual offer, declined/no-reoffer, accepted visual with fallback, and stale visual. +- GREEN isolated subject + reviewer: PASS for conceptual text-only, visual offer, declines, accepts, and stale visual. From e653c2ea784ff00273bc24abb511d4f5b1851bb9 Mon Sep 17 00:00:00 2001 From: Jon Langevin Date: Tue, 30 Jun 2026 07:06:27 -0400 Subject: [PATCH 13/16] Tighten visual companion validation --- ...brainstorm-visualization-behavior-proof.md | 77 ++++++++++++++----- .../visual-companion/expected-behavior.md | 2 +- tests/brainstorm-visual-companion.sh | 12 ++- 3 files changed, 67 insertions(+), 24 deletions(-) diff --git a/docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md b/docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md index 5e54820..a1d721b 100644 --- a/docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md +++ b/docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md @@ -4,12 +4,24 @@ Baseline subject read pre-change `skills/brainstorming/SKILL.md` only. -### Actual behavior summary +### Raw RED subject output -- Conceptual prompt (`what does trust mean for this dashboard?`) → subject stayed text-only by default using existing clarifying-question workflow. -- Visual layout choice → subject would still present 2–3 text approaches with trade-offs and numbered options. -- Declined visuals → subject would continue text-only, but only by inference because no visual-offer protocol exists. -- Changed decision after diagram → subject would revise design text; no stale-diagram handling exists. +```text +Actual current-skill behavior in scenario: +1. User asks: "what does trust mean for this dashboard?" + - Treat as brainstorming/design exploration. + - Stay text-only by default: clarify purpose, users, constraints, and success criteria. + - Use adaptive question batching, likely numbered options in Zed. +2. Later visual layout choice would be clearer shown than described + - Still present 2–3 text approaches with trade-offs and a recommendation. + - In Zed, present options as a numbered list and ask the user to pick/reply. +3. User declines visuals to save time + - Since the current skill has no visual-offer protocol, continue text-only. + - May treat "save time/no visuals" as a constraint, but this is inferred, not explicitly required. +4. User changes a decision after a diagram would have existed + - Revise the design text / assumptions / chosen approach as needed. + - No explicit stale-diagram handling exists, because visuals/diagrams are not currently governed. +``` ### Missing explicit requirements @@ -23,29 +35,52 @@ Baseline subject read pre-change `skills/brainstorming/SKILL.md` only. ## GREEN subject run -Subject used the revised `skills/brainstorming/SKILL.md` and loaded `skills/brainstorming/visual-companion.md` only for the accepted-visual path. +Subject used revised `skills/brainstorming/SKILL.md` and loaded `skills/brainstorming/visual-companion.md` only for the accepted-visual path. -| path | subject behavior | -|---|---| -| conceptual text-only | Stayed text-only for `what does trust mean for this dashboard?`; did not load or offer the visual companion. | -| visual offer | Sent a standalone just-in-time offer: `This layout choice would be clearer shown than described. Want me to sketch 2–3 quick wireframe options before you choose?` | -| declines | Continued text-only and explicitly did not re-offer unless asked. | -| accepts | Lazy-loaded the guide, composed a small Mermaid-style visual, included a text fallback, and recorded recommendation in text. | -| stale visual | Retired the stale visual and stated the current text decision as source of truth. | +### Raw GREEN subject output + +```text +1. Conceptual text-only +Action: Do not load visual-companion.md; answer in text. +Message: Trust here means the dashboard helps users believe the data is accurate, fresh, explainable, and safe to act on. For this redesign, which trust signal matters most? 1. Data accuracy/provenance 2. Freshness/status 3. Explainability/drilldown 4. Consistency with existing reports. Reply with a number or adjust the wording. +Rule: SKILL.md says to stay text-only for requirements, conceptual choices, tradeoff tables, and scope decisions; visual-companion.md lists domain language like "what does trust mean?" as text-only. + +2. Visual offer +Action: Offer visuals just-in-time, as the only content in the message. +Message: This layout choice would be clearer shown than described. Want me to sketch 2–3 quick wireframe options before you choose? +Rule: First visual companion offer must be just-in-time, never upfront, its own message, and count as one question batch. + +3. User declines visuals +Action: Continue text-only; do not re-offer unless the user raises visuals. +Message: Got it — we'll stay text-only and optimize for speed. I won't re-offer visuals unless you ask. Text-only layout options: 1. KPI strip + exception queue 2. Narrative summary + drilldowns 3. Role-based sections by team. My recommendation is option 1 if fast trust-building is the goal. +Rule: If the user declines, continue text-only and do not re-offer unless the user raises visuals. + +4. User accepts visuals +Action: Lazy-load visual-companion.md, then compose a small visual with a text fallback. +Message: Great — I'll keep it lightweight and decision-focused. Here are three rough layout directions using synthetic content: [Mermaid flowchart]. Text fallback: Option 1 leads with metrics and trust badges, then highlights exceptions. Option 2 leads with a written summary, then lets users drill down. Option 3 groups content by team ownership. I'd choose option 1 for fastest confidence and scanability. +Rule: Load the guide only after acceptance or while composing a visual; visuals must include concise text equivalents. + +5. Visual becomes stale +Action: Retire or update the visual before proceeding; keep the text decision as source of truth. +Message: That changes the layout decision, so the previous wireframe is now stale. I'm retiring it rather than relying on it. Current text source of truth: we're prioritizing role-based sections over a universal KPI strip. If we need another visual later, I'll redraw it from that decision. +Rule: If a visual becomes stale or contradicts a later text decision, retire or update it before proceeding; current text design remains the source of truth. +``` ## Reviewer score PASS overall. -| path | reviewer verdict | cited rule | -|---|---|---| -| conceptual text-only | PASS | `SKILL.md` text-only rule + guide text-only domain-language rule. | -| visual offer | PASS | `SKILL.md` just-in-time, standalone, question-batch offer rule. | -| declines | PASS | `SKILL.md` decline/no-reoffer rule. | -| accepts | PASS | `SKILL.md` lazy-load + fallback rules; guide fallback guidance. | -| stale visual | PASS | `SKILL.md` text source-of-truth/stale visual rule; guide stale visual rule. | +### Raw reviewer score + +```text +1. Conceptual text-only — PASS. Matches fixture: conceptual/domain-language question stays text-only and does not load/offer visuals. SKILL.md says to stay text-only for requirements/conceptual choices; visual-companion.md lists domain language like "what does trust mean?" as text-only. +2. Visual offer — PASS. Offer is just-in-time, standalone, and does not bundle another clarifying question. SKILL.md requires first visual offer to be just-in-time, its own message, count as one question batch, and not bundle another clarifying question. +3. User declines visuals — PASS. Continues text-only and explicitly avoids re-offering unless user asks. SKILL.md says if the user declines, continue text-only and do not re-offer unless the user raises visuals. +4. User accepts visuals — PASS. Lazy-load action is after acceptance; response includes a visual plus concise text fallback and keeps the decision text-readable. SKILL.md requires lazy-loading only after acceptance/while composing visuals and every visual to include a concise text fallback. Guide says the same. Minor watchpoint: the transcript says "[Mermaid flowchart]" as a placeholder, so this passes only as transcript shorthand, not as an actual rendered artifact. +5. Visual becomes stale — PASS. Correctly retires the stale visual and treats the current text decision as source of truth. SKILL.md says text remains source of truth and stale/contradictory visuals must be retired or updated before proceeding. Guide repeats the stale visual rule. +``` -Reviewer note: accepted-visual transcript used `[Mermaid flowchart]` as shorthand in the reviewer prompt, so it proves behavior selection and fallback, not host-rendered Mermaid output. Rendering remains best-effort/config-only per design. +Reviewer note: accepted-visual transcript used `[Mermaid flowchart]` as shorthand, so it proves behavior selection and fallback, not host-rendered Mermaid output. Rendering remains best-effort/config-only per design. ## Evidence diff --git a/skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md b/skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md index 066ec09..1648a0f 100644 --- a/skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md +++ b/skills/brainstorming/test-fixtures/visual-companion/expected-behavior.md @@ -13,7 +13,7 @@ A user asks an agent to brainstorm a dashboard redesign quickly. The session has | path | expected behavior | |---|---| | conceptual text-only | For conceptual questions such as "what does trust mean for this dashboard?", the agent stays in text and does not offer the visual companion. | -| visual offer | When the next decision is genuinely visual, the agent sends only the just-in-time visual-companion offer in its own message; it does not bundle a clarifying question. | +| visual offer | When the next decision is genuinely visual, the agent sends only the just-in-time visual-companion offer in its own message; it does not bundle a clarifying question, and the offer counts as one question batch. | | declines | If the user declines, the agent continues text-only and does not offer again unless the user raises visuals. | | accepts | If the user accepts, the agent lazy-loads `visual-companion.md` only while composing visual artifacts, includes a concise text fallback, and records the final decision in text. | | stale visual | If a visual becomes stale or contradicts a later text decision, the agent retires or updates it and treats text as the source of truth. | diff --git a/tests/brainstorm-visual-companion.sh b/tests/brainstorm-visual-companion.sh index 153adb4..62ecbba 100644 --- a/tests/brainstorm-visual-companion.sh +++ b/tests/brainstorm-visual-companion.sh @@ -19,7 +19,7 @@ has "$BRAIN" "just-in-time" && pass "visual companion offer is just-in-time" || hasE "$BRAIN" 'not[ -]upfront|never[ -]upfront|NOT upfront' && pass "visual companion is not offered upfront" || bad "visual companion upfront prohibition missing" has "$BRAIN" "per-question" && pass "visual decision is per-question" || bad "per-question decision rule missing" has "$BRAIN" "text remains the source of truth" && pass "text source-of-truth rule present" || bad "text source-of-truth rule missing" -has "$BRAIN" "question batch" && pass "visual offer batch-budget rule present" || bad "visual offer batch-budget rule missing" +has "$BRAIN" "counts as one question batch" && pass "visual offer batch-budget rule present" || bad "visual offer batch-budget rule missing" has "$BRAIN" "lazy-load" && pass "guide lazy-load rule present" || bad "guide lazy-load rule missing" has "$BRAIN" "visual-companion.md" && pass "guide reference present" || bad "guide reference missing" @@ -37,11 +37,19 @@ if [ -f "$FIXTURE" ]; then for marker in "conceptual text-only" "visual offer" "declines" "accepts" "stale visual" "RED baseline"; do has "$FIXTURE" "$marker" && pass "fixture covers $marker" || bad "fixture missing $marker" done + has "$FIXTURE" "counts as one question batch" && pass "fixture covers visual offer batch budget" || bad "fixture missing visual offer batch budget" +fi + +visual_row="$(grep -i '^| Visual companion output |' "$CAPS" || true)" +if [ -n "$visual_row" ]; then + count="$(printf '%s' "$visual_row" | grep -oi 'browser deferred' | wc -l | tr -d ' ')" + [ "$count" -ge 6 ] && pass "visual companion row defers browser for all hosts" || bad "visual companion row missing per-host browser deferral" +else + bad "capability matrix missing Visual companion output row" fi has "$AGENTS" "tests/brainstorm-visual-companion.sh" && pass "AGENTS documents visual companion test" || bad "AGENTS missing visual companion test" has "$CAPS" "Visual companion" && pass "capability matrix includes visual companion" || bad "capability matrix missing visual companion" -has "$CAPS" "browser deferred" && pass "capability matrix tracks browser deferral" || bad "capability matrix missing browser deferral" has "$SKILL_COVERAGE" "visual companion" && pass "skill coverage notes visual companion" || bad "skill coverage missing visual companion note" echo ""; echo "Results: $fail failure(s)"; [ "$fail" -eq 0 ] From 9b80a1c7b22dd89af39b19224254390ae9cd69d6 Mon Sep 17 00:00:00 2001 From: Jon Langevin Date: Tue, 30 Jun 2026 07:16:43 -0400 Subject: [PATCH 14/16] Strengthen visual companion proof --- ...brainstorm-visualization-behavior-proof.md | 23 +++++++++++++++---- tests/brainstorm-visual-companion.sh | 23 +++++++++++++++---- 2 files changed, 36 insertions(+), 10 deletions(-) diff --git a/docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md b/docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md index a1d721b..7119f46 100644 --- a/docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md +++ b/docs/plans/2026-06-30-brainstorm-visualization-behavior-proof.md @@ -39,7 +39,7 @@ Subject used revised `skills/brainstorming/SKILL.md` and loaded `skills/brainsto ### Raw GREEN subject output -```text +````text 1. Conceptual text-only Action: Do not load visual-companion.md; answer in text. Message: Trust here means the dashboard helps users believe the data is accurate, fresh, explainable, and safe to act on. For this redesign, which trust signal matters most? 1. Data accuracy/provenance 2. Freshness/status 3. Explainability/drilldown 4. Consistency with existing reports. Reply with a number or adjust the wording. @@ -57,14 +57,27 @@ Rule: If the user declines, continue text-only and do not re-offer unless the us 4. User accepts visuals Action: Lazy-load visual-companion.md, then compose a small visual with a text fallback. -Message: Great — I'll keep it lightweight and decision-focused. Here are three rough layout directions using synthetic content: [Mermaid flowchart]. Text fallback: Option 1 leads with metrics and trust badges, then highlights exceptions. Option 2 leads with a written summary, then lets users drill down. Option 3 groups content by team ownership. I'd choose option 1 for fastest confidence and scanability. +Message: Great — I'll keep it lightweight and decision-focused. Here are three rough layout directions using synthetic content: + +```mermaid +flowchart TD + A[Option 1: KPI strip] --> B[Exceptions queue] + A --> C[Freshness + source badges] + D[Option 2: Narrative summary] --> E[Key changes] + D --> F[Drilldowns] + G[Option 3: Team sections] --> H[Sales] + G --> I[Ops] + G --> J[Finance] +``` + +Text fallback: Option 1 leads with metrics and trust badges, then highlights exceptions. Option 2 leads with a written summary, then lets users drill down. Option 3 groups content by team ownership. I'd choose option 1 for fastest confidence and scanability. Rule: Load the guide only after acceptance or while composing a visual; visuals must include concise text equivalents. 5. Visual becomes stale Action: Retire or update the visual before proceeding; keep the text decision as source of truth. Message: That changes the layout decision, so the previous wireframe is now stale. I'm retiring it rather than relying on it. Current text source of truth: we're prioritizing role-based sections over a universal KPI strip. If we need another visual later, I'll redraw it from that decision. Rule: If a visual becomes stale or contradicts a later text decision, retire or update it before proceeding; current text design remains the source of truth. -``` +```` ## Reviewer score @@ -76,11 +89,11 @@ PASS overall. 1. Conceptual text-only — PASS. Matches fixture: conceptual/domain-language question stays text-only and does not load/offer visuals. SKILL.md says to stay text-only for requirements/conceptual choices; visual-companion.md lists domain language like "what does trust mean?" as text-only. 2. Visual offer — PASS. Offer is just-in-time, standalone, and does not bundle another clarifying question. SKILL.md requires first visual offer to be just-in-time, its own message, count as one question batch, and not bundle another clarifying question. 3. User declines visuals — PASS. Continues text-only and explicitly avoids re-offering unless user asks. SKILL.md says if the user declines, continue text-only and do not re-offer unless the user raises visuals. -4. User accepts visuals — PASS. Lazy-load action is after acceptance; response includes a visual plus concise text fallback and keeps the decision text-readable. SKILL.md requires lazy-loading only after acceptance/while composing visuals and every visual to include a concise text fallback. Guide says the same. Minor watchpoint: the transcript says "[Mermaid flowchart]" as a placeholder, so this passes only as transcript shorthand, not as an actual rendered artifact. +4. User accepts visuals — PASS. Lazy-load action is after acceptance; response includes an actual Mermaid visual artifact plus concise text fallback and keeps the decision text-readable. SKILL.md requires lazy-loading only after acceptance/while composing visuals and every visual to include a concise text fallback. Guide says the same. 5. Visual becomes stale — PASS. Correctly retires the stale visual and treats the current text decision as source of truth. SKILL.md says text remains source of truth and stale/contradictory visuals must be retired or updated before proceeding. Guide repeats the stale visual rule. ``` -Reviewer note: accepted-visual transcript used `[Mermaid flowchart]` as shorthand, so it proves behavior selection and fallback, not host-rendered Mermaid output. Rendering remains best-effort/config-only per design. +Reviewer note: accepted-visual transcript includes a Mermaid artifact and text fallback, proving artifact composition behavior. Rendering remains best-effort/config-only per design; this proof does not claim host-rendered Mermaid output. ## Evidence diff --git a/tests/brainstorm-visual-companion.sh b/tests/brainstorm-visual-companion.sh index 62ecbba..d0299fa 100644 --- a/tests/brainstorm-visual-companion.sh +++ b/tests/brainstorm-visual-companion.sh @@ -29,6 +29,8 @@ if [ -f "$GUIDE" ]; then has "$GUIDE" "accessibility" && pass "guide mentions accessibility" || bad "guide missing accessibility guidance" has "$GUIDE" "fallback" && pass "guide mentions fallback" || bad "guide missing fallback guidance" has "$GUIDE" "secrets" && pass "guide forbids secrets" || bad "guide missing secrets guidance" + has "$GUIDE" "PII" && pass "guide forbids PII" || bad "guide missing PII guidance" + hasE "$GUIDE" 'synthetic|redacted' && pass "guide requires synthetic or redacted content" || bad "guide missing synthetic/redacted guidance" has "$GUIDE" "SKILL.md remains authoritative" && pass "guide is bounded by SKILL.md authority" || bad "guide authority boundary missing" fi @@ -40,15 +42,26 @@ if [ -f "$FIXTURE" ]; then has "$FIXTURE" "counts as one question batch" && pass "fixture covers visual offer batch budget" || bad "fixture missing visual offer batch budget" fi -visual_row="$(grep -i '^| Visual companion output |' "$CAPS" || true)" -if [ -n "$visual_row" ]; then - count="$(printf '%s' "$visual_row" | grep -oi 'browser deferred' | wc -l | tr -d ' ')" - [ "$count" -ge 6 ] && pass "visual companion row defers browser for all hosts" || bad "visual companion row missing per-host browser deferral" +if awk -F'|' ' + BEGIN { rows = 0; ok = 0 } + /^\| Visual companion output \|/ { + rows++ + row_ok = (NF >= 9) + for (i = 3; i <= 8; i++) { + cell = tolower($i) + if (cell !~ /browser deferred/) { row_ok = 0 } + } + if (row_ok) { ok = 1 } + } + END { exit !(rows == 1 && ok == 1) } +' "$CAPS"; then + pass "visual companion row defers browser for all hosts" else - bad "capability matrix missing Visual companion output row" + bad "visual companion row missing per-host browser deferral" fi has "$AGENTS" "tests/brainstorm-visual-companion.sh" && pass "AGENTS documents visual companion test" || bad "AGENTS missing visual companion test" +has "$BRAIN" "PII" && pass "brainstorming skill forbids PII" || bad "brainstorming skill missing PII guidance" has "$CAPS" "Visual companion" && pass "capability matrix includes visual companion" || bad "capability matrix missing visual companion" has "$SKILL_COVERAGE" "visual companion" && pass "skill coverage notes visual companion" || bad "skill coverage missing visual companion note" From 57ead8368182d03ce3910c2f9bd29c19fe4fe28c Mon Sep 17 00:00:00 2001 From: Jon Langevin Date: Tue, 30 Jun 2026 07:24:28 -0400 Subject: [PATCH 15/16] Polish visual companion guidance --- docs/plans/2026-06-30-brainstorm-visualization-design.md | 2 +- skills/brainstorming/SKILL.md | 2 +- skills/brainstorming/visual-companion.md | 2 +- tests/brainstorm-visual-companion.sh | 2 ++ 4 files changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/plans/2026-06-30-brainstorm-visualization-design.md b/docs/plans/2026-06-30-brainstorm-visualization-design.md index b717893..76a4ba3 100644 --- a/docs/plans/2026-06-30-brainstorm-visualization-design.md +++ b/docs/plans/2026-06-30-brainstorm-visualization-design.md @@ -127,7 +127,7 @@ | boundary | proof | |---|---| -| Skill behavior | Subagent pressure proof against `expected-behavior.md` → PASS for four paths. | +| Skill behavior | Subagent pressure proof against `expected-behavior.md` → PASS for five paths. | | Skill behavior markers | `bash tests/brainstorm-visual-companion.sh` → asserts just-in-time/not-upfront/per-question/text fallback/lazy-load/guide/fixture/coverage contract. | | Skill text ↔ content rules | `bash tests/skill-content-grep.sh` → host-neutral token guard passes. | | Skill refs ↔ repo files | `bash tests/skill-cross-refs.sh` → references resolve. | diff --git a/skills/brainstorming/SKILL.md b/skills/brainstorming/SKILL.md index c0b1377..8e86852 100644 --- a/skills/brainstorming/SKILL.md +++ b/skills/brainstorming/SKILL.md @@ -27,7 +27,7 @@ You MUST create a task for each of these items and complete them in order: 1. **Explore project context** — check files, docs, recent commits 2. **Apply project design guidance** — invoke `autodev:project-design-guidance`; read existing guidance or run its Q&A if no durable guidance exists -3. **Offer the visual companion just-in-time** — never upfront; only when a question would genuinely be clearer shown than described +3. **Watch for visual companion opportunities** — offer just-in-time, never upfront, only when a question would genuinely be clearer shown than described 4. **Ask clarifying questions** — adaptive batching: group related questions to reduce round-trips; use targeted singles for follow-ups 5. **Propose 2-3 approaches** — with trade-offs and your recommendation 6. **List load-bearing assumptions explicitly** — every design rests on assumptions ("upstream API is idempotent", "single-tenant", "user has admin"); write them down so the adversarial reviewer can attack them diff --git a/skills/brainstorming/visual-companion.md b/skills/brainstorming/visual-companion.md index 1adc26b..4ee5460 100644 --- a/skills/brainstorming/visual-companion.md +++ b/skills/brainstorming/visual-companion.md @@ -2,7 +2,7 @@ ## Lazy-load rule -Load this guide only after the user accepts the Visual Companion or when composing a visual artifact. `SKILL.md remains authoritative`; this file provides examples, accessibility guidance, and fallback details. +Load this guide only after the user accepts the Visual Companion, or when the user directly asks for a visual and you are composing that visual artifact. `SKILL.md remains authoritative`; this file provides examples, accessibility guidance, and fallback details. ## When to use visuals diff --git a/tests/brainstorm-visual-companion.sh b/tests/brainstorm-visual-companion.sh index d0299fa..b5645b7 100644 --- a/tests/brainstorm-visual-companion.sh +++ b/tests/brainstorm-visual-companion.sh @@ -22,6 +22,8 @@ has "$BRAIN" "text remains the source of truth" && pass "text source-of-truth ru has "$BRAIN" "counts as one question batch" && pass "visual offer batch-budget rule present" || bad "visual offer batch-budget rule missing" has "$BRAIN" "lazy-load" && pass "guide lazy-load rule present" || bad "guide lazy-load rule missing" has "$BRAIN" "visual-companion.md" && pass "guide reference present" || bad "guide reference missing" +has "$BRAIN" "do not re-offer unless the user raises visuals" && pass "decline no-reoffer rule present" || bad "decline no-reoffer rule missing" +hasE "$BRAIN" 'stale.*retire or update|retire or update.*stale' && pass "stale visual retire/update rule present" || bad "stale visual retire/update rule missing" [ -f "$GUIDE" ] && pass "visual companion guide exists" || bad "visual companion guide missing" if [ -f "$GUIDE" ]; then From 5cf388799deffe949fc30308c1e86cecf2adb1d4 Mon Sep 17 00:00:00 2001 From: Jon Langevin Date: Tue, 30 Jun 2026 07:25:48 -0400 Subject: [PATCH 16/16] Complete brainstorm visualization scope --- docs/plans/2026-06-30-brainstorm-visualization.md | 2 +- docs/plans/2026-06-30-brainstorm-visualization.md.scope-lock | 1 - 2 files changed, 1 insertion(+), 2 deletions(-) delete mode 100644 docs/plans/2026-06-30-brainstorm-visualization.md.scope-lock diff --git a/docs/plans/2026-06-30-brainstorm-visualization.md b/docs/plans/2026-06-30-brainstorm-visualization.md index 77b0fb8..0f1bbfd 100644 --- a/docs/plans/2026-06-30-brainstorm-visualization.md +++ b/docs/plans/2026-06-30-brainstorm-visualization.md @@ -29,7 +29,7 @@ |------|-------|-------|--------| | 1 | Add brainstorm visual companion guidance | Task 1, Task 2, Task 3 | issue-78-brainstorm-visualization | -**Status:** Locked 2026-06-30T10:37:25Z +**Status:** Complete 2026-06-30T11:25:38Z --- diff --git a/docs/plans/2026-06-30-brainstorm-visualization.md.scope-lock b/docs/plans/2026-06-30-brainstorm-visualization.md.scope-lock deleted file mode 100644 index c5e02d3..0000000 --- a/docs/plans/2026-06-30-brainstorm-visualization.md.scope-lock +++ /dev/null @@ -1 +0,0 @@ -889fc3ebf145472d192fcd78939c5edf14b962cac3a05717110f90cda3893915