Shutting down master won't shutdown the running ggp instances

[vangalamaheshh]

Hi @jbingham -

Canceling the pipeline using "stop job" feature on Dataflow UI, terminates the master. However, if there are any instances running (vms with name ggp*), they are not shut down, meaning, these instances are like zombie vms that user has to manually turn off.
This is clearly a bug.
Any thoughts on this?

Thank you,
Mahesh

[jbingham]

Correct. There's not currently an easy way to do it. You have to go in to Cloud Console and shutdown all of the VMs individually.

Ideal would be:

dockerflow shutdown my-job-id

[jbingham]

Just remembered how you would do this currently. When running dockerflow, set:

dockerflow --run-id=YOUR_UNIQUE_NAME ...

Then use:

gcloud alpha genomics pipelines list --where "client-id = YOUR_UNIQUE_NAME"

Parse the output to get the operationId and run

gcloud alpha genomics pipelines cancel OPERATION_ID

And about this being a bug... Dataflow has no way of knowing if the user code that it runs is launching a separate VM, or how it's tagging that VM. That's up to the Dataflow user. The way that Dockerflow makes it possible to cancel VMs launched by Dataflow is to add the run-id to the operations for the VMs.