Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 64 additions & 0 deletions docs/plans/2026-10-20-workflow-assistant-smoke-design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# Workflow Assistant release smoke compatibility

## Evidence and goal

The official v0.87.0 Darwin arm64 asset matches its published checksum and
GitHub digest, embeds clean source `4b2c11cc` and MCP SDK v1.3.0, and passes the
five-version wire matrix. The existing package smoke completes its fixture
checks but rejects stderr: Sonic 1.15.2 prints its exact public Go 1.27
`encoding/json` fallback warning at startup. The helper currently allows only
the exact legacy plugin-deprecation notice.

Recognize and report that known diagnostic without weakening unknown-stderr,
release-version, input-privacy or acceptance boundaries. This is a separate
package-maintenance follow-up; the completed two-PR/six-task MCP plan is unchanged.

## Chosen behavior

- Classify complete stderr lines by exact equality with the existing deprecation
notice or the observed Sonic fallback notice. Reject every other line,
including prefix/suffix mutations, injected extra lines and blank lines.
- Return symbolic diagnostic status; never include raw stderr in smoke reports.
Preserve existing empty/deprecation-only statuses and distinguish Sonic-only
and combined known diagnostics.
- Keep the strict release-version guard and mismatch handling unchanged.
Source pseudo-versions and snapshots remain unknown.
- Publish package patch 0.1.1 with the same three skills and direct MCP manifest.
Document verified release pairings and the nonfatal diagnostic explicitly.
- Retain runtime, desktop, binary-health and input-equivalence unknowns.

## Rejected alternatives

Blanket stderr suppression can hide sensitive or unexpected output. Relabeling a
source binary would fabricate release evidence. Upgrading Sonic changes core
runtime behavior and is unnecessary for this bounded maintainer-client fix.
The published engine tag, release assets, workflows and permissions are unchanged.

## Verification

Use RED/GREEN tests for exact known lines and adversarial variants. Run standalone
race/vet checks, then the actual unchanged v0.86.2 and official v0.87.0 assets.
Check wrong-version/source-version failures, safe reports and preserved unknowns.
Record the smoke client's source separately from the official binary's source.
Independent spec/quality review, public-data audit and ordinary draft-PR CI follow.

## Design backport — 2026-10-20

The genuine-child negative test disproved the assumption that existing cleanup
already rejected nonzero exit: `cmd.Wait()` errors were discarded. A child that
emitted the known warning and exited 2 returned no cleanup error (observed RED).
Retain its wait error and report bounded `check_failed` after reaping; protocol
and fixture failures keep their earlier precedence. Forced termination also
fails while cleanup still reaps the child and reader. Independent review found
this necessary to enforce the chosen successful-shutdown boundary within Task 1's
same helper/test files. The one-PR/one-task scope manifest and lock are unchanged.
Spec review also identified a child that handles forced SIGTERM with exit 0;
its genuine-child regression failed before correction. Record required signal
escalation as `check_failed` regardless of final exit status, while retaining
bounded cleanup and timeout precedence.
Quality review disproved another existing assumption: the bounded stderr read
did not reject overflow, so an exact-known prefix of 1 MiB + 1 bytes hid an
unknown suffix. A genuine-child canary test reproduced RED. Reject captured
stderr above 1 MiB as bounded `unexpected_stderr` before classification, retaining
earlier cleanup failures. This enforces Task 1's injected-output rejection in the
same files; the manifest and lock remain unchanged.
49 changes: 49 additions & 0 deletions docs/plans/2026-10-20-workflow-assistant-smoke.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Workflow Assistant release smoke plan

**Goal:** Classify the verified non-sensitive Sonic startup diagnostic while
preserving strict smoke-client guards and truthful release evidence.

**Design:** `2026-10-20-workflow-assistant-smoke-design.md`.

## Scope Manifest

**PR Count:** 1
**Tasks:** 1
**Estimated Lines of Change:** ~150

**Out of scope:**
- Engine/SDK/dependency, transport, skill, workflow, permission or deployment changes.
- Retagging, restarting or republishing the v0.87.0 engine release.
- Blanket stderr suppression, source-version relabeling or expanded acceptance claims.
- Changes to completed MCP plans, unrelated locks or installed plugins/sessions.

**PR Grouping:**

| PR # | Title | Tasks | Branch |
|------|-------|-------|--------|
| 1 | Handle known release startup diagnostic in Workflow Assistant smoke | Task 1 | fix/workflow-assistant-release-smoke |

**Status:** Locked

### Task 1: Correct the bounded smoke diagnostic classification

Files: package `tests/main.go`, `tests/smoke_test.go`, `plugin.json`, `README.md`,
and this design/plan. Add no engine or workflow changes.

1. Extract the existing exact-deprecation-only stderr classification without
changing behavior; confirm existing tests pass. Add adversarial tests and
observe the Sonic-positive case fail before support is added.
2. Add exact Sonic classification and explicit symbolic statuses. Reject unknown,
suffixed/prefixed, injected and malformed lines without echoing their content.
3. Bump package/client metadata to 0.1.1. Document both actually tested released
CLIs, diagnostic handling and unchanged host/runtime limits. Attribute the
existing Codex/Autodev host probe to historical package 0.1.0 evidence;
package 0.1.1 smoke does not establish fresh host activation.
4. Run standalone race/vet checks; run real released v0.86.2 and v0.87.0 smokes;
verify strict wrong-release/source-pseudo-version failures. Preserve the
existing actual v0.87.0 five-version wire evidence; runtime source is unchanged.
5. Complete independent spec/quality review, privacy/scope audit, clean package
export and ordinary draft-PR CI. Keep the original 0.1.0 export immutable.

Rollback: revert the package-only follow-up. No installed or engine release state
is changed by this task.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
5c9b8b5c0cbc4ffb64e61943b5a330020434f955d642b6b6d6cf3b87cefa85de
23 changes: 17 additions & 6 deletions example/workflow-agent-plugin/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,12 @@ with explicit coverage limits. No service, dashboard or orchestration platform.
## Prerequisites and local setup

Supply a trusted `wfctl` executable yourself. This package does not bundle,
download or update it. The tested release is
[v0.86.2](https://github.com/GoCodeAlone/workflow/releases/tag/v0.86.2).
download or update it. Package 0.1.1's maintainer smoke was tested with the Darwin
arm64 assets from [v0.86.2](https://github.com/GoCodeAlone/workflow/releases/tag/v0.86.2)
and [v0.87.0](https://github.com/GoCodeAlone/workflow/releases/tag/v0.87.0).
Choose the asset for your OS/architecture and verify its checksum against that
release's `checksums.txt` before using it. Follow the
[public CLI setup guide](https://github.com/GoCodeAlone/workflow/blob/v0.86.2/docs/mcp.md).
[public CLI setup guide](https://github.com/GoCodeAlone/workflow/blob/v0.87.0/docs/mcp.md).
`wfctl version` should report the selected version. Other releases are untested;
the skills discover tools dynamically and retain unknown version claims.

Expand Down Expand Up @@ -48,7 +49,7 @@ plugins; do not assume live inventory changes reload the server's type registry.

| Surface | Package intent and verification status |
|---|---|
| Local Codex supporting the portable format | Codex CLI 0.163.0 loaded this package and public Autodev 6.6.3 together in a disposable offline home: all three skills registered and the host discovered 33 tools/5 resources from wfctl v0.86.2. Model-driven selection and complete prompt behavior **not yet tested**. |
| Local Codex supporting the portable format | Historical package **0.1.0** probe: Codex CLI 0.163.0 loaded it and public Autodev 6.6.3 together in a disposable offline home; all three skills registered and the host discovered 33 tools/5 resources from wfctl v0.86.2. Package 0.1.1 has separate maintainer smoke evidence, with unchanged skills/MCP manifest; fresh host activation, model-driven selection and complete prompt behavior **not yet tested**. |
| ChatGPT desktop with local execution/plugin support | Same intended package, subject to account/workspace features. Complete desktop activation **not yet tested**. |
| Other Agent Plugins clients | Requires skills + stdio + both placeholders; no host certification claimed. |
| ChatGPT browser, remote/cloud executor | Cannot directly run this local executable. No remote endpoint, tunnel or hosted bridge included. |
Expand All @@ -69,7 +70,7 @@ SHA256 if available, the supplied source commit/path, observed server version,
tools/schemas used and separate check outcomes. Redaction/partial status and
equivalence to original input are distinct. No aggregate deploy-ready badge.

The real v0.86.2 interface differs from portions of its reference documentation:
The tested interfaces differ from portions of the v0.86.2 reference documentation:

- `get_module_schema` takes `module_type`; `get_step_schema` takes `step_type`.
- `compat_check` takes `yaml_content`. It checks the running module/step registry;
Expand Down Expand Up @@ -118,14 +119,24 @@ validator and ECMA-regex adapter. From this directory, run:
```sh
cd tests
GOWORK=off go test -race ./...
GOWORK=off go run . -wfctl /path/to/reviewed/wfctl -expect-version v0.86.2
GOWORK=off go run . -wfctl /path/to/reviewed/wfctl -expect-version v0.87.0
```

The smoke client uses only bundled public fixtures, launches the actual manifest command
with temporary plugin data/minimal environment, and emits safe summaries and
digests. It tests positive/negative configs and known upstream false positives.
It does not accept user YAML or read organization repositories/secrets. Missing
executable/version/tools or protocol failures return a bounded failure category.
Released-version matching remains strict; snapshots and source pseudo-versions
are rejected. The client recognizes only the exact legacy plugin-deprecation
notice and the exact Sonic Go 1.27 `encoding/json` fallback warning observed in
v0.87.0. It reports symbolic stderr statuses (`empty`, `known_deprecation_only`,
`known_sonic_fallback_only`, or `known_deprecation_and_sonic_fallback`) after
fixture checks and successful child shutdown. Changed, extra or malformed
diagnostics fail as `unexpected_stderr`; raw stderr is never included. Package
0.1.0's smoke rejected that new warning; 0.1.1 adds this bounded classification.
Stderr above the client's 1 MiB capture limit also fails, so truncation cannot
hide an unknown suffix behind recognized notices.
Fixture files use `.yaml.txt` because they include intentional failures and must
stay outside the repository's recursive runnable `.yaml` example discovery.
Unit-test doubles cover failures; only the real release run is MCP integration
Expand Down
2 changes: 1 addition & 1 deletion example/workflow-agent-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json",
"name": "workflow-assistant",
"version": "0.1.0",
"version": "0.1.1",
"description": "Configure and diagnose public Workflow apps with wfctl MCP and produce bounded readiness evidence.",
"author": {"name": "GoCodeAlone", "url": "https://github.com/GoCodeAlone"},
"homepage": "https://github.com/GoCodeAlone/workflow/tree/main/example/workflow-agent-plugin",
Expand Down
49 changes: 38 additions & 11 deletions example/workflow-agent-plugin/tests/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ var reasons = map[string]bool{
var requiredTools = []string{"list_module_types", "list_step_types", "list_trigger_types", "list_workflow_types", "get_module_schema", "get_config_skeleton", "validate_config", "inspect_config", "compat_check", "template_validate_config", "list_plugins"}

const deprecation = "[DEPRECATED] requires.plugins[].version and requires.plugins[].source in app.yaml are deprecated. Run 'wfctl migrate plugins' to migrate to wfctl.yaml + .wfctl-lock.yaml."
const sonicFallback = "WARNING: sonic/ast only supports (go1.17~1.26 and amd64 CPU) or (go1.20~1.26 and arm64 CPU), but your environment is not suitable and will fallback to encoding/json"
const maxStderrBytes = 1024 * 1024

func digest(data []byte) string { sum := sha256.Sum256(data); return hex.EncodeToString(sum[:]) }
func fileDigest(path string) (string, error) {
Expand Down Expand Up @@ -343,7 +345,8 @@ func (c *client) close() ([]byte, error) {
close(c.stopping)
_ = c.stdin.Close() // EOF first; broken stdin must not prevent reaping.
waited := make(chan struct{})
go func() { _ = c.cmd.Wait(); close(waited) }()
var waitErr error
go func() { waitErr = c.cmd.Wait(); close(waited) }()
wait := func() bool {
timer := time.NewTimer(3 * time.Second)
defer timer.Stop()
Expand All @@ -355,6 +358,7 @@ func (c *client) close() ([]byte, error) {
}
}
if !wait() {
c.closeErr = failure("check_failed")
_ = c.cmd.Process.Signal(syscall.SIGTERM)
if !wait() {
_ = c.cmd.Process.Kill()
Expand All @@ -363,6 +367,9 @@ func (c *client) close() ([]byte, error) {
}
}
}
if c.closeErr == nil && waitErr != nil {
c.closeErr = failure("check_failed")
}
_ = c.stdout.Close()
timer := time.NewTimer(time.Second)
defer timer.Stop()
Expand All @@ -372,9 +379,11 @@ func (c *client) close() ([]byte, error) {
c.closeErr = failure("timeout")
}
if _, err := c.stderr.Seek(0, io.SeekStart); err == nil {
c.closedStderr, err = io.ReadAll(io.LimitReader(c.stderr, 1024*1024+1))
c.closedStderr, err = io.ReadAll(io.LimitReader(c.stderr, maxStderrBytes+1))
if err != nil {
c.closeErr = failure("internal_error")
} else if len(c.closedStderr) > maxStderrBytes && c.closeErr == nil {
c.closeErr = failure("unexpected_stderr")
}
} else {
c.closeErr = failure("internal_error")
Expand Down Expand Up @@ -467,7 +476,7 @@ func runSmoke(ctx context.Context, opts options) (map[string]any, error) {
}
defer c.close()
s := session{client: c}
initialized := s.request("initialize", map[string]any{"protocolVersion": "2024-11-05", "capabilities": map[string]any{}, "clientInfo": map[string]any{"name": "workflow-package-smoke", "version": "0.1.0"}})
initialized := s.request("initialize", map[string]any{"protocolVersion": "2024-11-05", "capabilities": map[string]any{}, "clientInfo": map[string]any{"name": "workflow-package-smoke", "version": "0.1.1"}})
info := object(initialized["serverInfo"])
if s.err == nil {
s.err = checkVersion(info, opts.expected)
Expand Down Expand Up @@ -534,10 +543,9 @@ func runSmoke(ctx context.Context, opts options) (map[string]any, error) {
if closeErr != nil {
return nil, closeErr
}
for _, line := range strings.Split(strings.TrimSuffix(string(stderr), "\n"), "\n") {
if len(stderr) != 0 && line != deprecation {
return nil, failure("unexpected_stderr")
}
stderrStatus, err := classifyStderr(stderr)
if err != nil {
return nil, err
}
binaryHash, err := fileDigest(binary)
if err != nil {
Expand All @@ -551,10 +559,6 @@ func runSmoke(ctx context.Context, opts options) (map[string]any, error) {
for name, data := range fixtures {
fixtureHashes[name] = digest(data)
}
stderrStatus := "empty"
if len(stderr) != 0 {
stderrStatus = "known_deprecation_only"
}
return map[string]any{
"status": "passed", "scope": "bundled_public_fixtures_only", "captured_at": time.Now().UTC().Format(time.RFC3339Nano), "server": map[string]any{"name": info["name"], "version": info["version"]},
"release_source": "https://github.com/GoCodeAlone/workflow/tree/" + opts.expected, "binary_sha256": binaryHash, "tool_count": len(tools), "tool_inventory_sha256": digest(inventory), "fixture_sha256": fixtureHashes,
Expand All @@ -563,6 +567,29 @@ func runSmoke(ctx context.Context, opts options) (map[string]any, error) {
"runtime_acceptance": "not_checked", "desktop_activation": "not_checked", "plugin_binary_health": "not_checked", "original_input_equivalence": "not_checked",
}, nil
}
func classifyStderr(stderr []byte) (string, error) {
if len(stderr) == 0 {
return "empty", nil
}
var hasDeprecation, hasSonic bool
for _, line := range strings.Split(strings.TrimSuffix(string(stderr), "\n"), "\n") {
switch line {
case deprecation:
hasDeprecation = true
case sonicFallback:
hasSonic = true
default:
return "", failure("unexpected_stderr")
}
}
if hasSonic {
if hasDeprecation {
return "known_deprecation_and_sonic_fallback", nil
}
return "known_sonic_fallback_only", nil
}
return "known_deprecation_only", nil
}
func mainWith(ctx context.Context, args []string, out io.Writer, run func(context.Context, options) (map[string]any, error)) int {
flags := flag.NewFlagSet("workflow-package-smoke", flag.ContinueOnError)
flags.SetOutput(io.Discard)
Expand Down
Loading
Loading