An issue to document conversations related to supporting batching via a fan-in (task takes N small independent files as input), fan-out (task output gets split into N output files) task.
Not included in scope: a fan-out/fan-in process which breaks big files up into many smaller inputs which can go through an arbitrarily long pipeline independently and then get reconstructed into a single large output file at the end.
An issue to document conversations related to supporting batching via a fan-in (task takes N small independent files as input), fan-out (task output gets split into N output files) task.
Not included in scope: a fan-out/fan-in process which breaks big files up into many smaller inputs which can go through an arbitrarily long pipeline independently and then get reconstructed into a single large output file at the end.