Problem Statement
Two common patterns in a cyclic pipeline are to
- fall back to an earlier stage if a particular attempt fails, and
- trickle a list element by element out of a task and have a downstream task signal when it's ready for the next element in the list.
For example, one task might sample some configuration space for a target joint pose. After the pose is sampled, a path can be planned to that pose. In a dynamic environment you might need to then decompose the path into small segments, check if the segment is still valid, and then hand off just that segment to a control loop to execute. Once the segment is complete, the control loop requests from the "decompose path task" the next segment.
Proposed Solution
Falling back to an earlier stage if a particular attempt fails.
Approach 1: Retries in Expansion
If there is a maximum number of attempts, the subgraph that should be retried could be annotated in copperconfig.ron with a max_retry constant. Then, in the expansion, the copper loop could copy the subgraph max_retry number of times. There are some nuances to this; for example, if any of the repeated tasks have state, it will need to make a copy of their original state so that it can be propagated to the next task that corresponds to it in the repeated leg. Additionally, a "cancel" edge would probably need to be added to all the repeated tasks so that if a new sample is selected, repeat attempts do not continue to propagate through the pipeline. The task that fails may also want to output a "bias" value that is passed into the sampler. The bias value might indicate where not to search, given that the previous attempt failed. It would also probably be useful to update just dag to express repeats in a condensed fashion. Finally, for really high max_retry values it might result in longer compile times.
This approach feels theoretically pure — the graph is still internally expressed as a DAG, and there are no changes to the way the event loop is run — but it also feels a bit cumbersome. It also does not solve the case where someone wants infinite retries.
Approach 2: Retried with masked copperlist unwinding
Another approach would be to update the way the event loop is executed. Effectively, this would require some form of backtracking of copper lists in the event loop. If a task fails to complete, the event loop would need a way to unwind to an earlier state of the copper list.
Unwinding the whole copper list would be problematic. It would result in jitter in completely unrelated parts of the graph and mean no side effects could be performed while retry-able legs are occurring. A "mask" would need to be generated for the copper list that contains all tasks that have provenance from the subgraph marked as retry. Sink tasks that have provenance from any "retry leg" should not be executed until the retry subgraph completes successfully. There is a question of what the right behavior is when a new batch of inputs arrives at a retry component (either stop trying on the previous batch, or continue until success or max_retry attempts are made.)
Trickle a list element by element out of a task and have the receiving task signal when it's ready for the next element in the list.
Like in the case of retries, if the list has a maximum length it could just be expanded into a DAG. In this case, though, I think having a fixed upper bound will be less common.
I think the only solution here is to simply add a cycle by making a bridge to signal completion to the upstream node.
In the particular "decompose-and-collision-check" example, one work around approach might be to always issue a path segment that will take more than X ms to complete and then wait Y < X ms output the next segment. This feels brittle, though, and requires some way to predict the time to execute a path.
Problem Statement
Two common patterns in a cyclic pipeline are to
For example, one task might sample some configuration space for a target joint pose. After the pose is sampled, a path can be planned to that pose. In a dynamic environment you might need to then decompose the path into small segments, check if the segment is still valid, and then hand off just that segment to a control loop to execute. Once the segment is complete, the control loop requests from the "decompose path task" the next segment.
Proposed Solution
Falling back to an earlier stage if a particular attempt fails.
Approach 1: Retries in Expansion
If there is a maximum number of attempts, the subgraph that should be retried could be annotated in
copperconfig.ronwith amax_retryconstant. Then, in the expansion, the copper loop could copy the subgraph max_retry number of times. There are some nuances to this; for example, if any of the repeated tasks have state, it will need to make a copy of their original state so that it can be propagated to the next task that corresponds to it in the repeated leg. Additionally, a "cancel" edge would probably need to be added to all the repeated tasks so that if a new sample is selected, repeat attempts do not continue to propagate through the pipeline. The task that fails may also want to output a "bias" value that is passed into the sampler. The bias value might indicate where not to search, given that the previous attempt failed. It would also probably be useful to updatejust dagto express repeats in a condensed fashion. Finally, for really highmax_retryvalues it might result in longer compile times.This approach feels theoretically pure — the graph is still internally expressed as a DAG, and there are no changes to the way the event loop is run — but it also feels a bit cumbersome. It also does not solve the case where someone wants infinite retries.
Approach 2: Retried with masked copperlist unwinding
Another approach would be to update the way the event loop is executed. Effectively, this would require some form of backtracking of copper lists in the event loop. If a task fails to complete, the event loop would need a way to unwind to an earlier state of the copper list.
Unwinding the whole copper list would be problematic. It would result in jitter in completely unrelated parts of the graph and mean no side effects could be performed while retry-able legs are occurring. A "mask" would need to be generated for the copper list that contains all tasks that have provenance from the subgraph marked as retry. Sink tasks that have provenance from any "retry leg" should not be executed until the retry subgraph completes successfully. There is a question of what the right behavior is when a new batch of inputs arrives at a retry component (either stop trying on the previous batch, or continue until success or max_retry attempts are made.)
Trickle a list element by element out of a task and have the receiving task signal when it's ready for the next element in the list.
Like in the case of retries, if the list has a maximum length it could just be expanded into a DAG. In this case, though, I think having a fixed upper bound will be less common.
I think the only solution here is to simply add a cycle by making a bridge to signal completion to the upstream node.
In the particular "decompose-and-collision-check" example, one work around approach might be to always issue a path segment that will take more than X ms to complete and then wait Y < X ms output the next segment. This feels brittle, though, and requires some way to predict the time to execute a path.