fix(controller): stop starting the next stage after a cancel command - #6979
fix(controller): stop starting the next stage after a cancel command#6979Ahmad-Faraj wants to merge 4 commits into
Conversation
When a cancel arrives while a stage is finishing, the stage can still report success and the scheduler continues to the next stage. The next select then receives nil from the closed cancelledCh and falls through without cancelling or waiting for the just-started stage, leaving it running while rollback begins. Make a received cancel command terminate the deployment even when the stage reported success, and treat a nil receive as a cancellation. Signed-off-by: Ahmad Faraj <ahmedfrag4040@gmail.com>
There was a problem hiding this comment.
Pull request overview
This PR improves cancellation handling in the Piped controllers’ schedulers so that a user cancel reliably stops stage progression and avoids leaving a forward stage running while rollback begins.
Changes:
- Ensure a received cancel ends the deployment even if the current stage reports
STAGE_SUCCESS. - Treat receives from a closed
cancelledChas a cancellation signal by always cancelling/waiting for the current stage. - Apply the same scheduler behavior fix in both
pipedandpipedv1.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| pkg/app/pipedv1/controller/scheduler.go | Adjusts cancel-channel handling and cancellation precedence after stage completion. |
| pkg/app/piped/controller/scheduler.go | Mirrors the same cancel precedence and closed-channel handling changes for legacy piped. |
| // A received cancel command wins even if the stage finished successfully, | ||
| // otherwise the loop would continue to the next stage after the user cancelled. | ||
| if cancelCommand != nil { | ||
| deploymentStatus = model.DeploymentStatus_DEPLOYMENT_CANCELLED | ||
| statusReason = fmt.Sprintf("Cancelled by %s while executing stage %s", cancelCommander, ps.Id) | ||
| break | ||
| } |
| // A received cancel command wins even if the stage finished successfully, | ||
| // otherwise the loop would continue to the next stage after the user cancelled. | ||
| if cancelCommand != nil { | ||
| deploymentStatus = model.DeploymentStatus_DEPLOYMENT_CANCELLED | ||
| statusReason = fmt.Sprintf("Cancelled by %s while executing stage %s", cancelCommander, ps.Id) | ||
| break | ||
| } |
| case cmd := <-s.cancelledCh: | ||
| // A nil command means the channel was closed after delivering | ||
| // a cancel command, so treat it as a cancellation as well. | ||
| if cmd != nil { | ||
| cancelCommand = cmd | ||
| cancelCommander = cmd.Commander | ||
| handler.Cancel() | ||
| <-doneCh | ||
| } | ||
| handler.Cancel() | ||
| <-doneCh |
…age completes The select can pick doneCh while a cancel command is already buffered, which would let the next stage start before the cancel is observed. Drain the channel non-blocking after the stage completes. Signed-off-by: Ahmad Faraj <ahmedfrag4040@gmail.com>
|
The problem is we need to determine what we should do in case case cmd := <-s.cancelledCh:
if cmd != nil {
cancelCommand = cmd
cancelCommander = cmd.Commander
}
handler.Cancel()
<- doneCh
....
if s.cancelled && result == model.StageStatus_STAGE_SUCCESS {
// What should we do here ?
// Remain STAGE_SUCCESS or change to STAGE_CANCELLED
} |
|
I would keep STAGE_SUCCESS. The stage really did finish and executeStage has already committed that status by the time the loop sees the cancel. Rewriting it would misstate what ran on the target. A cancel that lands while the stage is still running already becomes STAGE_CANCELLED via determineStageStatus. So the stage status stays truthful and the cancel is honored at the deployment level with DEPLOYMENT_CANCELLED plus auto rollback when it is configured. |
|
@Ahmad-Faraj This case needs consideration, you can raise in next biweekly meeting |
|
Will raise it in the next biweekly meeting. |
|
@khanhtc1202 @mohammedfirdouss , PTAL |
|
This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. |
|
Still relevant, and the race in #6978 still reproduces. A cancel landing while a stage is finishing leaves the next stage started. My position has not changed, for anyone arriving at this cold. The finished stage keeps I can rebase if it has drifted. |
✅ Deploy Preview for pipecd-site canceled.
|
|
@armistcxy I tried raising this at three biweeklies and no maintainer was on the call any of them, so bringing it back here. My position hasn't changed. The finished stage keeps Is that the behaviour you want, or should the stage flip to |
Signed-off-by: Ahmad Faraj <ahmedfrag4040@gmail.com>
What this PR does:
A cancel that lands while a stage is finishing can be ignored, the stage still reports success and the scheduler moves on. The next iteration then receives nil from the closed cancelledCh and abandons the stage it just started while rollback begins. This makes a received cancel end the deployment even when the stage reported success, and treats a nil receive as a cancellation. Same fix in piped and pipedv1.
Why we need it:
A cancelled deployment should not keep executing stages, and rollback should not run in parallel with a forward stage.
Which issue(s) this PR fixes:
Fixes #6978
Does this PR introduce a user-facing change?: