Skip to content

[BUG] Harden work-package execution JSON and generated test repair #174

Description

@Joncallim

Bug Summary

Approved work-package execution can still fail after handoff when a specialist model returns malformed execution JSON or weak generated tests for a small local app. The July 9, 2026 tiny task tracker run reached execution, then failed across retries because the model produced an invalid test script, invalid JSON, and finally a test.js file that was not a focused node:test assertion.

Current Behaviour

Forge asks the execution model for one work_package_execution_json response and retries validation a small number of times. Some common bad outputs are not caught until after files are written to the sandbox and validation commands run. In the observed tiny tracker run, the final package attempt failed with Command failed: npm test and Generated test file is not a focused node:test assertion: test.js.

Expected Behaviour

Forge should repair predictable execution-output mistakes before the package burns all worker attempts. Malformed JSON wrappers, placeholder tests, console-only tests, and non-asserting npm test files should become generation-repair feedback so the model can return corrected execution JSON. A normal dependency-free tiny task tracker should complete with node:test-based regression coverage.

Reproduction Steps

  1. Create a local-only project with no existing app scaffold.
  2. Ask Forge to build a tiny task tracker web app with tests and a build check.
  3. Approve the Architect plan.
  4. Use a weak specialist execution model such as qwen2.5-coder:7b.
  5. Observe execution failing after model output contains malformed JSON or a non-focused test file such as test.js with no node:test assertion.

Impact

This blocks the core executable-workforce beta path after approval. Operators can approve a valid plan and still fail before getting a usable local app, even though the failure is often a predictable output-format or generated-test repair case.

Severity

High - major workflow broken

Acceptance Criteria

  • Execution JSON parsing accepts common salvageable wrappers, including one-line fenced JSON and JSON-encoded response strings.
  • Invalid JSON still produces clear repair feedback instead of an opaque package failure where possible.
  • Placeholder, console-only, or non-asserting generated JavaScript tests are rejected during generation validation before command validation.
  • The repair prompt tells the model to use dependency-free node --test, node:test, and node:assert/strict for tiny JavaScript apps.
  • The executor has a regression where a tiny task tracker flow recovers from invalid JSON, then a bad generated test, then succeeds with real node:test coverage.
  • Existing worker executor behavior remains covered by the unit suite.
  • Operators should prefer a stronger specialist execution model than zero-config qwen2.5-coder:7b for Frontend, Backend, and QA packages when available.

Environment

  • Date observed: July 9, 2026
  • Repository: Joncallim/Forge
  • Failing task: Tiny Task Tracker Web App
  • Specialist execution connector: Qwen 2.5 Coder via Ollama (qwen2.5-coder:7b)
  • Failure tail: Command failed: npm test / Generated test file is not a focused node:test assertion: test.js

Technical Notes

Likely files:

  • web/worker/work-package-executor.ts
  • web/__tests__/work-package-executor.test.ts

Keep this narrow. Do not redesign MCP admission, handoff scheduling, live MCP grants, commits, pull requests, or provider routing in this issue. Epic #172 should still handle the separate MCP admission contract work.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0Priority 0: urgent bug or vulnerabilitybugSomething isn't workingready-for-agent

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions