Thinking about failure handling in orchestrated workflows

[ishita-0301]

I was reading through Orchestra's human-in-the-loop workflow model and it got me thinking about what happens when one worker or automation quietly stops making progress.

In multi-step pipelines, a task doesn't always fail loudly. Sometimes it just retries the same thing over and over or gets stuck in an unproductive state while the rest of the workflow waits.

[marcua]

Thank you for reaching out @ishita-0301! This issue is something we've had to consider at scale. Since Orchestra project/step data lives in the database, we've set up internal alerts for projects that have been in a certain state for too long. Just some simple SQL against our data warehouse and a Slack alert on our end, but I'm sure this can get more complex depending on your needs. What sorts of issues have you run into with in solving problems in the space?