#138 adds support for parallel execution of shell commands via the parallel workflow configuration. When parallelism is enabled, it is expected that commands use the $INPUT_FILE and $OUTPUT_DIR envvars to set input/output data. E.g.,:
ogr2ogr -f GPKG -t_srs EPSG:4326 "$OUTPUT_DIR"/"$(basename "$INPUT_FILE" .gpkg)_4326.gpkg" "$INPUT_FILE"
When parallelism is not enabled, the command is expected to have input data in /input_dir/ it is expected that output data be placed in /output_dir/. E.g.,:
gdal_calc.py --calc "A / 10.0" -A /input_dir/*.tif --outfile=/output_dir/sic_as_percents.tif
This is confusing and ideally users should be able to enable/disable parallelism without needing to change the syntax of their commands.
If it is not practical to unify the interface between parallel and non parallel recipes, then this issue can be resolved by separating the shell recipe into two: shell and parallel-shell. This would make it clear that they have unique requirements.
#138 adds support for parallel execution of shell commands via the
parallelworkflow configuration. When parallelism is enabled, it is expected that commands use the$INPUT_FILEand$OUTPUT_DIRenvvars to set input/output data. E.g.,:When parallelism is not enabled, the command is expected to have input data in
/input_dir/it is expected that output data be placed in/output_dir/. E.g.,:This is confusing and ideally users should be able to enable/disable parallelism without needing to change the syntax of their commands.
If it is not practical to unify the interface between parallel and non parallel recipes, then this issue can be resolved by separating the shell recipe into two:
shellandparallel-shell. This would make it clear that they have unique requirements.