Currently, Pipeline is using multiple private attributes to track task-level state info (e.g., status, failed/processed files). We may want to centralize this tracking using an abstraction like TaskState data class to make the code easier to understand, maintain, and extend.
Currently,
Pipelineis using multiple private attributes to track task-level state info (e.g., status, failed/processed files). We may want to centralize this tracking using an abstraction likeTaskStatedata class to make the code easier to understand, maintain, and extend.