Cancel the Code Engine job when a Fleets job times out

[Alberto Vilches (avilches)]

Summary

When a Fleets job goes past PROGRAM_TIMEOUT, stop_job_if_timeout writes STOPPED on the row but never asks Code Engine to cancel anything. The fleet keeps running, and since nothing else looks at a job that is already in a terminal status, nothing ever cleans it up. This adds the missing runner.stop() call before the status is written.

Details and comments

The timeout is the one path that decides to stop a job on its own, without Code Engine having reported a terminal state first. The other branches of update_job_status reach to_terminal because the task store already says the job succeeded, failed or was cancelled, so there is genuinely nothing left to cancel there. The timeout branch fires while the fleet is very likely still running, or while its state is unknown because the status poll returned nothing or raised.

If the cancel fails, the error is logged and the row still moves to STOPPED. Leaving it in a running status instead would keep it holding the user's concurrency slot for as long as Code Engine stays unreachable, which is the problem the timeout exists to bound.

Ray is not affected. Its timeout path has the same shape, but FreeResources deletes the RayCluster for any job in a terminal status, so the teardown happens regardless of whether stop() was called. Fleets has no equivalent task.

Worth knowing for reviewers: a follow-up will replace this single write with a two-phase transition, where a stop request moves the job to a new STOPPING status and a later poll confirms the real STOPPED. That work changes what this branch writes, not whether the cancel happens, so landing this first is still a strict improvement over today.

Closes https://github.ibm.com/IBM-Q-Software/qiskit-serverless/issues/1831

[Elena Peña Tapia (ElePT)]

Thanks for looking into this, also LGTM