One common situation that researchers might encounter is the need to experiment with many different combinations of task parameters. Currently, they would need to manually generate various configs themselves and run each. It would be nice if, instead, they could provide TIgerFlow with a config that specifies the different parameters/values they want to play with, and TigerFlow would automatically determine the combinations and run them all.
Some relevant points/open question:
-
The current consensus seems to be that the best way to do this would have researchers define a config with the parameters they want for all tasks in their pipeline and then have a way to generate X configs based on those parameter combinations. Then, there would be a way to run all X configs at once and orchestration to manage all X pipelines.
-
The way all X pipelines should be orchestrated is up for debate. Some options include:
- Simply running the pipelines sequentially (ie. 4 workers on pipeline1 until all files processed, then move on to pipeline2)
- Distributing resources equally across as many pipelines as possible (ie. 1 worker on pipelines 1-4)
- Specifying Y number of max workers, and having these workers dynamically move between pipelines when possible.
All of these have benefits in different situations depending on if the researcher wants all files to finish in one pipeline first, or if they want all file1s to finish across pipelines first.
-
How to specify the combinations in the original config? It would nice for researchers to be able to specify specific combinations of parameters that they would like to try as well as all values they want to try and have TigerFlow generate a matrix of all combinations. However, let's say I want to try 5 different models with 3 different prompt -- I want a 5x3 matrix of combinations. But different models of different sizes require different resource allocations, so I need a way of specifying the different resources I want, but tie these to specific models.
-
We need a way to be able to stop a specific pipeline or to stop the whole suite. We also need a way to display how many/which pipelines are running and the status for each.
-
We also talked briefly about the possibility of having an "evaluator" which checks the intermediate outputs of tasks in the pipeline(s) and determines if the output is good enough for the remaining tasks in the pipeline to run.
One common situation that researchers might encounter is the need to experiment with many different combinations of task parameters. Currently, they would need to manually generate various configs themselves and run each. It would be nice if, instead, they could provide TIgerFlow with a config that specifies the different parameters/values they want to play with, and TigerFlow would automatically determine the combinations and run them all.
Some relevant points/open question:
The current consensus seems to be that the best way to do this would have researchers define a config with the parameters they want for all tasks in their pipeline and then have a way to generate X configs based on those parameter combinations. Then, there would be a way to run all X configs at once and orchestration to manage all X pipelines.
The way all X pipelines should be orchestrated is up for debate. Some options include:
All of these have benefits in different situations depending on if the researcher wants all files to finish in one pipeline first, or if they want all
file1s to finish across pipelines first.How to specify the combinations in the original config? It would nice for researchers to be able to specify specific combinations of parameters that they would like to try as well as all values they want to try and have TigerFlow generate a matrix of all combinations. However, let's say I want to try 5 different models with 3 different prompt -- I want a 5x3 matrix of combinations. But different models of different sizes require different resource allocations, so I need a way of specifying the different resources I want, but tie these to specific models.
We need a way to be able to stop a specific pipeline or to stop the whole suite. We also need a way to display how many/which pipelines are running and the status for each.
We also talked briefly about the possibility of having an "evaluator" which checks the intermediate outputs of tasks in the pipeline(s) and determines if the output is good enough for the remaining tasks in the pipeline to run.