This is copied from @trey-stafford's comment over in #106
There are still one or two points where we wait for argo to respond before returning a response to the user after submitting a recipe. A recipe submission should be just that - no work done, just a request that returns a reference ID that the user can then subsequently query for status). Consider a queue/worker approach?
I think this is a great idea. I actually implemented some of this on this branch: https://github.com/QGreenland-Net/ogdc-runner/tree/bug-166-background-task
On that branch, we generate an identifier for the workflow before submitting it to Argo. This identifier gets passed all the way down the call stack from the submit endpoint to the context manager that generates the argo workflow as a label. I also modified the status endpoint so that it could lookup workflows either by name, or by this identifier. The top level call to submit the workflow (submit_ogdc_recipe) in the /submit endpoint is run as a background task so there is (almost) no work done before exiting the function call.
If we think this is a good approach I can move forward with a PR
This is copied from @trey-stafford's comment over in #106
I think this is a great idea. I actually implemented some of this on this branch: https://github.com/QGreenland-Net/ogdc-runner/tree/bug-166-background-task
On that branch, we generate an identifier for the workflow before submitting it to Argo. This identifier gets passed all the way down the call stack from the submit endpoint to the context manager that generates the argo workflow as a label. I also modified the status endpoint so that it could lookup workflows either by name, or by this identifier. The top level call to submit the workflow (
submit_ogdc_recipe) in the/submitendpoint is run as a background task so there is (almost) no work done before exiting the function call.If we think this is a good approach I can move forward with a PR