convert to WDL: force co-localization of secondary_files

[mr-c]

See biowdl/tasks#291 for a demonstration for both individual files and arrays of files

[illusional]

Thanks @mr-c, Janis should already do this on translation to WDL:

janis-core/janis_core/translations/wdl.py

Line 2079 in 0b5b79f

commands.append(wdl.Task.Command(f"cp -f '~{{{sectag}}}' {dest}"))

(Because I saw the same problems haha)

[mr-c]

@illusional Huh, I'm not seeing that behaviour for either a single File input with a secondary file specifier, nor an array of Files with a secondary file specifier:

secondary_files.cwl

cwlVersion: v1.2
class: CommandLineTool

arguments:
  - {shellQuote: false, valueFrom: "ls | grep -v lsout"}
inputs:
  input:
    type: File
    secondaryFiles:
      - ^.tar
stdout: lsout
outputs:
  output:
    type: File
    outputBinding:
      glob: lsout
requirements:
  ShellCommandRequirement: {}

$ janisdk fromcwl secondary_files.cwl -o .

secondary_files_v0_1_0.py

from datetime import datetime
from typing import List, Optional, Dict, Any

from janis_core import *
from janis_core.types.common_data_types import GenericFileWithSecondaries, File

Secondary_Files_V0_1_0 = CommandToolBuilder(tool="secondary_files", base_command=None, inputs=[ToolInput(tag="inp", input_type=GenericFileWithSecondaries(secondaries=["^.tar"]), doc=InputDocumentation(doc=None))], outputs=[ToolOutput(tag="outp", output_type=File(), selector=WildcardSelector(wildcard="lsout"), doc=OutputDocumentation(doc=None))], container="ubuntu:latest", version="v0.1.0", arguments=[ToolArgument(value="ls | grep -v lsout", position=None, doc=InputDocumentation(doc=None), shell_quote=False)])


if __name__ == "__main__":
    # or "cwl"
    Secondary_Files_V0_1_0().translate("wdl")

$ janis translate secondary_files_v0_1_0.py wdl -o .
2021-08-29T11:13:34 [INFO]: The command tool ({'tool': CommandTool<secondary_files>, 'toolid': 'secondary_files'}).outp' used a star-bind (*) glob to find the output, but the return type was not an array. For WDL, the first element will be used, ie: 'glob("lsout")[0]'

secondary_files_v0_1_0.wdl

version development

task secondary_files {
  input {
    Int? runtime_cpu
    Int? runtime_memory
    Int? runtime_seconds
    Int? runtime_disks
    File inp
    File inp_tar
  }
  command <<<
    set -e
     \
      ls | grep -v lsout
  >>>
  runtime {
    cpu: select_first([runtime_cpu, 1])
    disks: "local-disk ~{select_first([runtime_disks, 20])} SSD"
    docker: "ubuntu@sha256:1e48201ccc2ab83afc435394b3bf70af0fa0055215c1e26a5da9b50a1ae367c9"
    duration: select_first([runtime_seconds, 86400])
    memory: "~{select_first([runtime_memory, 4])}G"
    preemptible: 2
  }
  output {
    File outp = glob("lsout")[0]
  }

---

array_secondary_files.cwl

cwlVersion: v1.2
class: CommandLineTool

arguments:
  - {shellQuote: false, valueFrom: "ls | grep -v lsout"}
inputs:
  input_list:
    type: File[]
    secondaryFiles:
      - ^.tar
stdout: lsout
outputs:
  output:
    type: File
    outputBinding:
      glob: lsout
requirements:
  ShellCommandRequirement: {}

$ janisdk fromcwl array_secondary_files.cwl -o .
$ janis translate array_secondary_files_v0_1_0.py wdl -o .
2021-08-29T11:09:55 [INFO]: The command tool ({'tool': CommandTool<array_secondary_files>, 'toolid': 'array_secondary_files'}).outp' used a star-bind (*) glob to find the output, but the return type was not an array. For WDL, the first element will be used, ie: 'glob("lsout")[0]'

version development

task array_secondary_files {
  input {
    Int? runtime_cpu
    Int? runtime_memory
    Int? runtime_seconds
    Int? runtime_disks
    Array[File] input_list
    Array[File] input_list_tar
  }
  command <<<
    set -e
     \
      ls | grep -v lsout
  >>>
  runtime {
    cpu: select_first([runtime_cpu, 1])
    disks: "local-disk ~{select_first([runtime_disks, 20])} SSD"
    docker: "ubuntu@sha256:1e48201ccc2ab83afc435394b3bf70af0fa0055215c1e26a5da9b50a1ae367c9"
    duration: select_first([runtime_seconds, 86400])
    memory: "~{select_first([runtime_memory, 4])}G"
    preemptible: 2
  }
  output {
    File outp = glob("lsout")[0]
  }
}