Skip to content

Terminate the script's process group when a SCRIPT_RUN stage is cancelled - #7320

Open
omlahore wants to merge 1 commit into
pipe-cd:masterfrom
omlahore:fix/scriptrun-cancel-process-group
Open

Terminate the script's process group when a SCRIPT_RUN stage is cancelled#7320
omlahore wants to merge 1 commit into
pipe-cd:masterfrom
omlahore:fix/scriptrun-cancel-process-group

Conversation

@omlahore

@omlahore omlahore commented Sep 4, 2026

Copy link
Copy Markdown

Related to #6734, which covers the same pattern in the v0 executors. This PR is the pipedv1 plugin only, to keep it to one concern.

What happens today

executeCommand in pkg/app/pipedv1/plugin/scriptrun/plugin.go runs the user script with exec.Command, taking no context at all, in a goroutine. The caller only selects on ctx.Done():

go func() { c <- executeCommand(opts.Run, opts.Env, request, lp) }()
select {
case result := <-c:
    return result
case <-ctx.Done():
    lp.Info("ScriptRun cancelled")
    return sdk.StageStatusFailure
}

So on cancel or timeout the stage reports cancelled and returns, and nothing ever signals the shell. The script keeps running, keeps mutating the cluster, and the goroutine stays parked in cmd.Run() for as long as it takes.

I reproduced it. A script that backgrounds a child, with both trapping SIGTERM:

exec.Command (current master):    grandchild 694602 alive after cancel = true
CommandContext + process group:   grandchild 694826 alive after cancel = false

The change

exec.CommandContext, the shell placed in its own process group with Setpgid, and a Cancel that signals the group rather than the child.

The process-group part is the bit that matters, and it is why CommandContext alone is not enough. Its default cancel signals only the direct child, and its WaitDelay fallback calls Process.Kill(), which is also only the child. Either way a grandchild that ignores SIGTERM outlives the stage. So Cancel sends SIGTERM to -pgid and a goroutine escalates to SIGKILL on the group after a grace period, stood down as soon as Wait returns.

The grace period is a named constant rather than an inline literal:

// commandTerminationGracePeriod is how long the script's process group gets
// after SIGTERM before it is SIGKILLed.
commandTerminationGracePeriod = 2 * time.Second

A long terraform apply may eventually want that configurable. Nothing in the stage config exposes it today, so I left it constant rather than inventing a schema field.

Portability

syscall.Setpgid and syscall.Kill are Unix-only, and I did not add a build tag, matching what the repo already does:

  • there are no //go:build files anywhere in the tree
  • pkg/lifecycle/binary.go:57 already calls c.cmd.Process.Signal(syscall.SIGTERM) unguarded
  • no workflow sets GOOS, and Makefile defaults to the host
  • this code path is already Unix-only in practice, since it hardcodes /bin/sh -l -c

Say the word if you would rather have //go:build unix on it anyway.

Tests

plugin_cancel_test.go:

  • TestExecuteCommandKillsProcessGroupOnCancel starts a script whose backgrounded grandchild traps SIGTERM, cancels the context, and asserts the grandchild is gone. It takes ~2s, which is the grace period, so it is genuinely exercising the SIGKILL escalation and not just the SIGTERM.
  • TestExecuteCommandSucceedsWithoutCancel pins the normal path: a command that exits on its own still returns success and does not wait out the grace period.

go build, go vet and go test ./... all pass in the plugin module.

One drive-by

Line 238 was lp.Errorf("failed to exec command: %w", err). StageLogPersister.Errorf does not wrap, so that rendered literally as %!w(*exec.ExitError=&{...}). I noticed it in my own test output and changed it to %v. Say the word if you would rather it were separate.

executeCommand ran /bin/sh with exec.Command and no context, in a
goroutine, while the caller only selected on ctx.Done(). On cancel or
timeout the stage reported CANCELLED and returned, but the shell and
everything it spawned kept running against the cluster.

Use exec.CommandContext, put the shell in its own process group with
Setpgid, and signal the group rather than the child. CommandContext's
default cancel signals only the direct child, and its WaitDelay fallback
calls Process.Kill() which is also only the child, so a grandchild that
ignores SIGTERM survives either way. Cancel now sends SIGTERM to the
group and escalates to SIGKILL after a 2s grace period.

Also corrects %w to %v in the exec failure log, which rendered as
%!w(*exec.ExitError=...) because StageLogPersister.Errorf does not wrap.

Signed-off-by: Om <omlahore47@gmail.com>
Copilot AI lite review requested due to automatic review settings September 4, 2026 11:13
@omlahore
omlahore requested review from a team as code owners September 4, 2026 11:13
@netlify

netlify Bot commented Sep 4, 2026

Copy link
Copy Markdown

Deploy Preview for pipecd-site canceled.

Name Link
🔨 Latest commit a79e677
🔍 Latest deploy log https://app.netlify.com/projects/pipecd-site/deploys/6a9aa7dbb0252c00085f7aeb

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The current cmd.Cancel implementation can return a spurious error on cancellation races (e.g., ESRCH), which should be handled to avoid incorrect error paths/logging during cancel.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR fixes cancellation behavior for the pipedv1 scriptrun plugin so that cancelling a SCRIPT_RUN (or rollback) stage terminates the entire shell process tree, preventing orphaned commands from continuing to mutate cluster state after the stage is cancelled.

Changes:

  • Switches executeCommand to exec.CommandContext and runs the shell as a separate process group (Setpgid) so the whole tree can be signaled.
  • Implements group-wide cancellation: SIGTERM to the process group, with SIGKILL escalation after a short grace period.
  • Adds tests to verify the process group is terminated on cancel and that the success path remains fast.
File summaries
File Description
pkg/app/pipedv1/plugin/scriptrun/plugin.go Adds context-aware execution and process-group termination logic for cancellation.
pkg/app/pipedv1/plugin/scriptrun/plugin_cancel_test.go Adds regression tests ensuring cancellation kills background descendants and normal execution stays unaffected.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +218 to +223
cmd.Cancel = func() error {
pgid := cmd.Process.Pid
lp.Infof("Cancelling script, sending SIGTERM to process group %d", pgid)
if err := syscall.Kill(-pgid, syscall.SIGTERM); err != nil {
return err
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants