Skip to content
Merged

v3 #8

Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ Then orchestrate execution and review from the CLI:

```bash
productize tasks run <slug> # execute the task list via the daemon
productize tasks run <slug> --concurrent 3 # dependency-aware parallel worktrees
productize reviews fetch <slug> --pr 123 # pull PR review feedback
productize reviews fix <slug> # auto-remediate review issues
productize archive # file away completed work
Expand Down Expand Up @@ -241,7 +242,10 @@ access_mode = "full"
auto_commit = false

[tasks.run]
concurrent = 1
include_completed = false
output_format = "text"
verify_command = "make verify"
# Route certain task types to a specific agent/model:
task_runtime_rules = [
{ type = "frontend", ide = "codex", model = "gpt-5.5" },
Expand Down
28 changes: 26 additions & 2 deletions docs/cli-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,18 +211,42 @@ productize tasks run <slug> [flags]

The CLI resolves workspace defaults locally, validates task metadata, auto-starts
the daemon when needed, and starts the workflow through daemon transport.
Concurrency defaults to one. `--detach` controls only whether the client follows
the run; it does not change task scheduling or isolation.

| Flag | Default | Description |
| ---- | ------- | ----------- |
| `--name` | | Workflow slug, defaults to positional slug |
| `--concurrent` | `1` | Maximum dependency-ready tasks to execute concurrently; must be positive |
| `--include-completed` | `false` | Re-run completed tasks |
| `--format` | `text` | Output contract: `text`, `json`, or `raw-json` |
| `--skip-validation` | `false` | Skip task metadata preflight; use only when validation already ran elsewhere |
| `--force` | `false` | Continue after task metadata validation fails in non-interactive mode |
| `--attach` | `auto` | Attach mode: `auto`, `stream`, or `detach`; legacy `ui` maps to `stream` |
| `--ui` | `false` | Deprecated alias for `--stream` |
| `--attach` | `auto` | Attach mode: `auto`, `stream`, or `detach` |
| `--stream` | `false` | Force textual stream attach mode |
| `--detach` | `false` | Start the run without attaching a client |
| `--task-runtime` | | Per-task runtime override rule such as `type=...`, `id=...`, `ide=...`, `model=...` |
| `--verify-command` | | Shell command used to verify integrated parallel task changes |

Text mode preserves the human-readable run summary and watcher. JSON emits the
lean workflow event stream as JSONL, while raw JSON emits full canonical event
envelopes. A local parallel dry-run emits one schema-versioned plan object for
both machine-readable formats.

Task batch size remains fixed at one. Parallel runs require a verification
command unless Productize discovers a Make `verify` target, a Node `verify` or
`test` script, a Go module, or a Rust workspace. Productize executes that command
directly after every merged wave and once more before finalization, records its
exit code and logs, and rejects verifier changes to the integration checkout.

Parallel mode requires a clean Git-root checkout. Each ready task runs in a
Productize-owned worktree, successful branches merge deterministically, and the
starting branch advances only by safe fast-forward. Failed task worktrees are
retained; verified successful tasks are finalized so a rerun skips them.
Parallel task runs reject `--add-dir` because shared external directories are
not isolated by the repository worktrees.
`--dry-run --concurrent N` performs only local read-only planning and does not
start the daemon, create a run, worktree, branch, or repository file.

## `productize reviews`

Expand Down
11 changes: 10 additions & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,10 @@ retry_backoff_multiplier = 1.5
types = ["frontend", "backend", "docs", "test", "infra", "refactor", "chore", "bugfix"]

[tasks.run]
concurrent = 1
include_completed = false
output_format = "text"
verify_command = "make verify"

[exec]
output_format = "text"
Expand Down Expand Up @@ -91,7 +93,8 @@ marker remains intact across refreshes.
- `[tasks]` for the allowed task `type` list used by `create-tasks` and
`productize tasks validate`
- `[tasks.run]` for workflow-run defaults used by `productize tasks run`, such
as `include_completed`, `output_format`, and `task_runtime_rules`
as `concurrent`, `include_completed`, `output_format`, `task_runtime_rules`,
and `verify_command`
- `[fix_reviews]` for `concurrent`, `batch_size`, `include_resolved`, and
`output_format`
- `[fetch_reviews]` for `provider` and `nitpicks`
Expand All @@ -109,6 +112,12 @@ marker remains intact across refreshes.
- Both `~/.productize/config.toml` and `.productize/config.toml` are optional.
If both are absent, Productize keeps the built-in defaults.
- `.productize/tasks` remains the fixed workflow root in this version.
- `tasks.run.concurrent` must be greater than zero. The default is `1`, which
preserves ordered sequential execution; task batch size remains fixed at one.
- `tasks.run.verify_command` is optional. Parallel execution uses it to verify
integrated changes when Productize cannot discover a Make `verify` target,
Node `verify`/`test` script, Go module, or Rust workspace. The command runs
directly after each merged wave and before the final fast-forward.
- Unknown keys and invalid value types are rejected during config loading.
- Relative `add_dirs` are resolved against the owning config scope: the user
home directory for `~/.productize/config.toml` and the workspace root for
Expand Down
12 changes: 11 additions & 1 deletion docs/events.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Every line in `events.jsonl` is one `events.Event` object:
| `run_id` | `string` | Stable identifier for the workflow or exec run that emitted the event. |
| `seq` | `uint64` | Monotonic sequence number within a run. |
| `ts` | `RFC3339 timestamp` | Event timestamp in UTC. |
| `kind` | `string` | One of the 51 public event kinds below. |
| `kind` | `string` | One of the 52 public event kinds below. |
| `payload` | `object` | Kind-specific payload from `pkg/productize/events/kinds`. |

## Run Events
Expand Down Expand Up @@ -350,6 +350,16 @@ Payload type: `kinds.TaskMemoryUpdatedPayload`
- `mode`
- `bytes_written`

### `task.scheduler_updated`

Payload type: `kinds.TaskSchedulerUpdatedPayload`

Emitted after each durable parallel-task scheduler checkpoint. The payload
contains the manifest path, scheduler status, concurrency, starting and
integration Git state, dependency waves, per-task run/branch/worktree/commit
state, enforced verification command results and logs, finalization state, and
sorted next actions.

## Artifact Events

### `artifact.updated`
Expand Down
108 changes: 108 additions & 0 deletions docs/plans/2026-08-05-parallel-task-worktrees-design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
# Parallel Task Worktrees

## Goal

Allow `productize tasks run <slug> --concurrent N` to execute dependency-ready
PRD tasks concurrently without allowing multiple coding agents to mutate the
same checkout.

Sequential execution remains the default and preserves its current behavior.

## Architecture

Productize will treat task frontmatter dependencies as a directed acyclic
graph. Before a run starts, it validates missing references, self-dependencies,
and cycles. The scheduler selects a deterministic wave of pending tasks whose
dependencies have completed and caps active tasks at the requested concurrency.

For concurrent runs, the daemon creates a run integration branch and a
Productize-owned integration worktree. Each ready task receives its own branch
and worktree rooted at the current integration commit. Worktrees live under the
Productize home instead of inside the user's repository.

Each task executes through the existing preparation, agent, journal, and result
pipeline. Successful task changes are committed on the task branch. Productize
then merges successful branches into the integration branch in task-number
order. A new wave begins only after the preceding wave is integrated and
verified.

## Git Safety

Parallel mode requires a Git repository and a clean starting worktree. It never
stashes, resets, discards, or overwrites user changes.

Productize records the starting branch and commit. After final verification it
fast-forwards the starting checkout only when it is still clean, remains on the
same branch, and still points at the starting commit. Otherwise, it leaves the
verified integration branch intact and reports the exact merge command.

Merge conflicts are aborted in the Productize-owned integration worktree. The
task branch and diagnostic worktree are retained for inspection. No automatic
conflict resolution or force update is attempted.

## Workflow Memory

Parallel workers read the shared memory snapshot present at the beginning of a
wave but write only their per-task memory files. After task branches merge, the
coordinator rebuilds the shared memory deterministically from the integrated
per-task records before scheduling dependent tasks. This removes shared-file
write races while preserving durable workflow context.

## CLI and Configuration

`productize tasks run` gains `--concurrent`. Values greater than one enable the
worktree scheduler; one retains ordered execution. Task batch size remains one.

`[tasks.run].concurrent` provides the workspace default. `--detach` remains a
presentation choice and does not change isolation semantics.

Dry-run output includes the validated dependency graph, execution waves,
concurrency, branch names, worktree locations, merge order, and finalization
strategy, without creating Git refs, directories, runs, or daemon mutations.

## Run State and Recovery

The existing run journal and daemon read models will record scheduler state,
including task attempts, dependencies, waves, branches, worktrees, commits,
merge outcomes, verification outcomes, blocked tasks, and next actions.

Cancellation stops and drains owned agent processes and prevents new tasks from
starting. Worktrees containing failed or uncommitted changes are retained;
clean completed worktrees may be removed. Verified successful tasks are safely
finalized even when a sibling fails, so a normal rerun uses their completed task
metadata and does not duplicate completed work. Interrupted daemon processes
retain their checkpoint, branches, and worktrees for manual recovery.

Independent tasks may finish after another task fails, but dependents of a
failed or unmerged task remain blocked. The run reports partial progress rather
than pretending the workflow completed.

## Verification

Each isolated task continues to use the existing task verification contract.
The coordinator verifies every integrated wave and performs final verification
before advancing the starting branch.

Tests cover graph validation, deterministic waves, bounded concurrency, task
failure, merge conflict, cancellation, retained-resource recovery, dirty repositories,
changed starting branches, worktree cleanup and retention, dry-run purity,
workflow-memory folding, daemon transport, and race safety. Repository
acceptance requires `make verify`.

## Implementation Plan

1. Add graph parsing, validation, deterministic wave selection, and scheduler
state types around the existing task metadata.
2. Add a focused Git worktree lifecycle package with injectable command
execution and real-repository integration tests.
3. Extend task-run CLI/config/transport contracts with concurrency and stable
scheduler reporting.
4. Integrate parallel scheduling with the existing planner, executor, daemon,
journal, cancellation, and result paths without creating a second agent
execution stack.
5. Make parallel workflow memory task-local during a wave and rebuild shared
memory deterministically after integration.
6. Add safe merge, verification, final fast-forward, recovery, and cleanup
behavior.
7. Update runtime guidance and user documentation, then run the full
verification gate.
10 changes: 10 additions & 0 deletions docs/workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,10 +113,20 @@ productize tasks run <slug>

The daemon executes each task by driving your chosen agent over ACP. Useful flags:

- `--concurrent N` to run up to `N` dependency-ready tasks in parallel. The default is `1`; task batch size remains one.
- `--attach stream` to watch live, or `--detach` to run in the background (re-attach later with `productize runs watch <run-id>`).
- `--verify-command "make verify"` to specify the integration verification command when Productize cannot discover one.
- `--include-completed` to re-run tasks already marked done; `--skip-validation` or `--force` to bypass the metadata preflight.
- `--task-runtime type=frontend,ide=codex,model=gpt-5.5` to route a task type to a specific agent/model. Defaults also live in `[tasks.run].task_runtime_rules` in config.

With concurrency greater than one, Productize validates the task dependency
graph, requires a clean Git-root checkout, and gives every ready task its own
branch and Productize-owned worktree. It merges successful tasks in stable task
order, rebuilds shared workflow memory, and executes the verification command
after every wave and before a safe final fast-forward. Failed worktrees remain
available for inspection; verified partial successes are finalized so the next
run does not repeat them. Parallel `--dry-run` is local and read-only.

---

## 5. Review & remediate — `productize reviews`
Expand Down
18 changes: 17 additions & 1 deletion internal/api/client/reviews_exec_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -298,13 +298,29 @@ func TestClientReviewRequestsEncodeDaemonPathsAndBodies(t *testing.T) {
req.URL.EscapedPath(),
)
}
body, err := io.ReadAll(req.Body)
if err != nil {
t.Fatalf("read task run request body: %v", err)
}
var payload map[string]any
if err := json.Unmarshal(body, &payload); err != nil {
t.Fatalf("decode task run request body: %v", err)
}
runtimeOverrides, ok := payload["runtime_overrides"].(map[string]any)
if !ok || runtimeOverrides["concurrent"] != float64(3) {
t.Fatalf("runtime_overrides = %#v, want concurrent=3", payload["runtime_overrides"])
}
if runtimeOverrides["verification_command"] != "make verify" {
t.Fatalf("runtime_overrides = %#v, want verification_command", runtimeOverrides)
}
return jsonResponse(http.StatusCreated, `{"run":{"run_id":"task-run-1","mode":"task"}}`), nil
}),
},
}

run, err := client.StartTaskRun(context.Background(), " demo alpha/beta ", apicore.TaskRunRequest{
Workspace: "/tmp/workspace",
Workspace: "/tmp/workspace",
RuntimeOverrides: json.RawMessage(`{"concurrent":3,"verification_command":"make verify"}`),
})
if err != nil {
t.Fatalf("StartTaskRun() error = %v", err)
Expand Down
11 changes: 11 additions & 0 deletions internal/api/core/openapi_contract_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,17 @@ func TestOpenAPIContractKeepsWorkspaceContextAndProblemSemantics(t *testing.T) {
if schemaRequires(taskRunSchema, "workspace") {
t.Fatal("TaskRunRequest must not require workspace")
}
taskRunProperties := getMap(t, taskRunSchema, "properties")
taskRuntimeOverrides := getMap(t, taskRunProperties, "runtime_overrides")
taskRuntimeProperties := getMap(t, taskRuntimeOverrides, "properties")
concurrent := getMap(t, taskRuntimeProperties, "concurrent")
if concurrent["type"] != "integer" || concurrent["minimum"] != float64(1) {
t.Fatalf("TaskRunRequest concurrent schema = %#v, want positive integer", concurrent)
}
verificationCommand := getMap(t, taskRuntimeProperties, "verification_command")
if verificationCommand["type"] != "string" || verificationCommand["minLength"] != float64(1) {
t.Fatalf("TaskRunRequest verification command schema = %#v, want non-empty string", verificationCommand)
}
reviewRunSchema := getSchema(t, spec, "ReviewRunRequest")
if schemaRequires(reviewRunSchema, "workspace") {
t.Fatal("ReviewRunRequest must not require workspace")
Expand Down
7 changes: 7 additions & 0 deletions internal/cli/commands_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,13 @@ func TestNewTasksRunCommandDefaultsAttachModeToAuto(t *testing.T) {
t.Parallel()

cmd := newTasksRunCommandWithDefaults(nil, defaultCommandStateDefaults())
formatFlag := cmd.Flags().Lookup("format")
if formatFlag == nil {
t.Fatal("expected --format flag")
}
if formatFlag.DefValue != string(core.OutputFormatText) {
t.Fatalf("expected --format default %q, got %q", core.OutputFormatText, formatFlag.DefValue)
}
flag := cmd.Flags().Lookup("attach")
if flag == nil {
t.Fatal("expected --attach flag")
Expand Down
Loading