move to 1 sample/row sample sheet and test with ashlar

[RobJY]

• change input_check.nf:create_fastq_channel():meta.id back to sample name
• supported formats:
  • ome.tif
  • moved zarr support to separate ticket
• change sample sheet format
  • schema change
  • document all the places that a sample sheet format change has to be made because I forget
    • Just in assets/schema_input.json
  • So should ashlar use all ome.tif or zarr images in the input directory as input?
    • doing this as default behavior
    • need to only use allowed file types
• make input parameter for each style of sample sheet
  • one row per sample
    • update mcmicro.nf
    • update nextflow_schema.json
    • rename assets/schema_input.json
    • nextflow.config
    • lib/WorkflowMain.groovy
  • one row per sample per cycle
    • update mcmicro.nf to handle either input sample sheet
    • update nextflow_schema.json
    • create assets/schema_input_cycle.json
    • update lib/WorkflowMain.groovy
    • update nextflow.config
  • use the '--input' parameter and for it check for cycle column to set input type
    • my concern with this is that I don't know how we'd get automatic validation because we have to set a schema for each input parameter
    • UPDATE: I thought I could get this working by setting the input parameter --input_sample or --input_cycle to the path given for --input and then use the correct one of those as input to Channel.fromSamplesheet(), but it doesn't work. It looks like fromSamplesheet() only accepts an input parameter name and it only gets the value of that input parameter from session.params. I tried updating the value of session.params['input_cycle'] from within the workflow, but this didn't work. It still had the default value defined in nextflow.config.
    • we're not going to support this at least at the moment
• question: what are better input parameter names; currently:
  • input_sample
  • input_cycle
• question: do the workflow inputs need to be added to nextflow.config?
  • I don't see any changes for the current parameters
  • Yes, I had to add them to pass lint tests
• marker_sheet as input parameter
  • add to nextflow_schema.json
  • create schma file assets/schema_marker.json
  • create channel from parameter in mcmicro.nf
  • question: is there a marker sheet for the tonsil data?
    • there is, Jeremy will send it
  • question: are the only columns for a marker sheet: channel_number, cycle_number, marker_name, Filter, excitation_wavelength, emission_wavelength?
    • needs to be discussed with larger group
  • question: which columns are required?
    • cycle, channel and marker_name
• question: my code assumes input sheet order is fixed. Is this a valid assumption or should I be expecting the columns in any order?
  • I ran into this when making ashlar input and validating the marker sheet. It was easy to make it work for any column order, so I just added it.
• nf-validation:
  • add paramsHelp
    • already there, but was in main.nf, so I didn't notice
• we would like more complex validation that checks associated files/directories. Can we do that with nf-validation?
  • it doesn't look like it
  • create new local subworkflow modules for the following validation:
    • change to subworkflow
      • made subworkflow for marker sheet validation
      • add sample sheet validation to subworkflow
      • clean-up new validation
      • clean-up (remove) old validation
    • validation:
      • uniqueness of marker name in marker sheet
      • uniqueness of (cycle, channel) tuple in marker sheet
      • cycle and channel are 1-based so 0 should throw an exception
      • cycle and channel cannot have skips and must be in order
      • validate that every sample has all the same cycle number values, and that those values match with the marker sheet 1:1
    • python exceptions should trigger Nextflow error messages
      • low priority because we'll probably be moving these python functions to groovy (move subworkflow input validation from python to groovy #2)
• question: any additional input validation on:
  • input_sample?
  • input_cycle?
  • marker_sheet?
• ext.args for ashlar input parameters
  • handle ffp & dfp
  • moved to workflow parameters
• question: what should the workflow output?
  • currently just passing the last module output file as placeholder
  • be sure to add an output.md describing it (required by lint tests)
• tests
  • nf-test
    • basic end-to-end test
    • test md5sum of each module's output
  • pytest end-to-end workflow test with md5sum check on module output files
  • some lint tests are failing
    • files_exist
    • files_unchanged
    • merge_markers
    • multiqc_config
    • nextflow_config
      • asked on nf-core:mcmicro Slack channel. Waiting for response
    • readme
      • posted question on Slack linting channel about this
    • schema_description
• fix lint warnings
• bug fix
  • not handling 1 row per sample per cycle inputs correctly. All rows with the same sample name need to go to one run of ashlar.
  • same issue for 1 row per sample inputs
• add image correction support to 1 row per sample case
  • manual
  • basicpy

[RobJY]

Hi @jmuhlich, I've got a few questions about the marker sheets:

• is there a marker sheet for the tonsil data?
• I'm looking at the marker sheet for the exemplar data. Are there other allowed columns besides the ones in this sheet?
  • Columns in that file are: channel_number, cycle_number, marker_name, Filter, excitation_wavelength, emission_wavelength
• Which columns are required?
  • From https://mcmicro.org/io.html#markers it looks like only cycle and marker_name are required. Is this correct?

[RobJY]

Hmm, I get ilastik_pixelclassification to start, but it errors out with:

Command error:
  Unable to find image 'biocontainers/ilastik:1.4.0_cv1' locally
  1.4.0_cv1: Pulling from biocontainers/ilastik
  677076032cca: Pulling fs layer
  1caf95c15ee1: Pulling fs layer
  8c24ac808f6a: Pulling fs layer
  acf6bb56eb1c: Pulling fs layer
  acf6bb56eb1c: Waiting
  8c24ac808f6a: Verifying Checksum
  8c24ac808f6a: Download complete
  677076032cca: Verifying Checksum
  677076032cca: Download complete
  677076032cca: Pull complete
  acf6bb56eb1c: Verifying Checksum
  acf6bb56eb1c: Download complete
  1caf95c15ee1: Verifying Checksum
  1caf95c15ee1: Download complete
  1caf95c15ee1: Pull complete
  8c24ac808f6a: Pull complete
  acf6bb56eb1c: Pull complete
  Digest: sha256:1900d2ef1f53acbd4fa180f706cc2ea9d7fca25ff8e161201b97f2aa93fdd9e4
  Status: Downloaded newer image for biocontainers/ilastik:1.4.0_cv1
  Matplotlib created a temporary config/cache directory at /tmp/matplotlib-sdpzk4kf because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
  Traceback (most recent call last):
    File "/opt/ilastik-1.4.0-Linux/bin/ilastik", line 10, in <module>
      sys.exit(main())
    File "/opt/ilastik-1.4.0-Linux/lib/python3.7/site-packages/ilastik_scripts/ilastik_startup.py", line 146, in main
      main()
    File "/opt/ilastik-1.4.0-Linux/lib/python3.7/site-packages/ilastik/__main__.py", line 30, in main
      hShell = app.main(parsed_args, workflow_cmdline_args)
    File "/opt/ilastik-1.4.0-Linux/lib/python3.7/site-packages/ilastik/app.py", line 134, in main
      _init_logging(parsed_args)  # Initialize logging before anything else
    File "/opt/ilastik-1.4.0-Linux/lib/python3.7/site-packages/ilastik/app.py", line 293, in _init_logging
      default_config.init(process_name, default_config.OutputMode.BOTH, logfile_path)
    File "/opt/ilastik-1.4.0-Linux/lib/python3.7/site-packages/ilastik/ilastik_logging/default_config.py", line 233, in init
      os.makedirs(os.path.dirname(DEFAULT_LOGFILE_PATH), exist_ok=True)
    File "/opt/ilastik-1.4.0-Linux/lib/python3.7/os.py", line 213, in makedirs
      makedirs(head, exist_ok=exist_ok)
    File "/opt/ilastik-1.4.0-Linux/lib/python3.7/os.py", line 213, in makedirs
      makedirs(head, exist_ok=exist_ok)
    File "/opt/ilastik-1.4.0-Linux/lib/python3.7/os.py", line 223, in makedirs
      mkdir(name, mode)
  PermissionError: [Errno 13] Permission denied: '/.cache'

I've left it in the code, but commented it out for now.

I tried setting MPLCONFIGDIR in illastik/pixelclassification/main.nf just before the call to run_ilastik.sh, but that didn't help. I still get the same error.
Oh, I think I see. That might fix the issue with matplotlib, but the problem is that something else is trying to write to .config and it doesn't have permission. I'm not sure how to fix this.

[RobJY]

Workflow now running with the modules: ashlar, deepcell_mesmer & mcquant. It also performs input validation on the sample and marker sheets.

Needs clean-up and nf-tests, but added end-to-end test using pytest.

[RobJY]

Current commands to run workflow and tests:

nextflow run ~/github/mcmicro-nf-core/main.nf --input_cycle ~/github/mcmicro-nf-core/assets/samplesheet_1_row_sample_cycle.csv --outdir ./output --marker_sheet ~/github/mcmicro-nf-core/assets/markers_1.csv -profile test_work -params-file ~/github/mcmicro-nf-core/conf/workflow.json

From base repo dir:
nf-test test tests/main.nf.test --profile test_work
nf-core lint