samtools sort aux_parse error: "Incomplete aux field, samtools sort: truncated file. Aborting"

### Operating System

Other Linux (please specify below)

### Other Linux

Rocky Linux 9.2

### Workflow Version

v1.2.3

### Workflow Execution

Command line (Cluster)

### Other workflow execution

_No response_

### EPI2ME Version

_No response_

### CLI command run

nextflow run epi2me-labs/wf-alignment -r v1.2.3 -w /weka/users/jefaa92p/tmp/nextflow -profile singularity --fastq /weka/users/jefaa92p/tmp/A549_ddjp/fastq/re-download --out_dir /weka/users/jefaa92p/tmp/A549_ddjp/analysis/wf-alignment_2025-07-03_1115 --references /projects/health_sciences/oms/pathology/sunali_group/ngs/refs/human/grch38_no_alt/ --sample A549_ddjp

### Workflow Execution - CLI Execution Profile

singularity

### What happened?

Running wf-alignment via slurm script fails on the "samtools sort" step of pipeline:alignReads:

```
Error executing process > 'pipeline:alignReads (1)'

Caused by:
  Process `pipeline:alignReads (1)` terminated with an error exit status (1)


Command executed:

  cat seqs.fastq.gz     | minimap2 -t 3 -ax map-ont -y --cap-kalloc 100m --cap-sw-mem 50m         combined_refs.mmi -     | samtools sort --write-index -@ 0 -o reads.bam -         -o aligned.sorted.bam##idx##aligned.sorted.bam.bai

Command exit status:
  1

Command output:
  (empty)

Command error:
  INFO:    Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred
  INFO:    Environment variable SINGULARITYENV_NXF_TASK_WORKDIR is set, but APPTAINERENV_NXF_TASK_WORKDIR is preferred
  INFO:    Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred
  [M::main::7.231*0.99] loaded/built the index for 195 target sequence(s)
  [M::mm_mapopt_update::8.843*0.99] mid_occ = 694
  [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 195
  [M::mm_idx_stat::9.598*0.99] distinct minimizers: 100167746 (38.80% are singletons); average occurrences: 5.519; average spacing: 5.607; total length: 3099922541
  [E::aux_parse] invalid aux tag id
  [W::sam_read1_sam] Parse error at line 198
  samtools sort: truncated file. Aborting
```


1. Input is fastq files downloaded from https://ddbj.nig.ac.jp/search/entry/sra-run/DRR171434
2. The workflow runs successfully on both the demo data and my own fastq's - it only fails with the fastq's from above.
3. The fastq checksums are good, pass validation with biopet-validatefastq and fastq_utils, and are aligned without issue by using nf-core/nanoseq.
4. seqs.fastq.gz is present in the work dir, readable (zcat) and valid according to the above validation tools.
5. Running minimap2 standalone with seqs.fastq.gz file and combined_refs.mmi  generated by wf-alignment works, so the concatentated fastqs and the .mmi appears valid.
6. Possibly related to this https://github.com/samtools/samtools/issues/1472?
7. There are two "-o" passed to samtools - is that expected?

Given I believe the offending bam is held in memory, I don't see a way of correcting an issue with the offending bam to allow the workflow to resume? 

Or maybe there actually is an issue with the fastq's or intermediate files and the samtools error is a red herring?

Any feedback very much appreciated!

Aaron.



### Relevant log output

```shell
Jul-03 11:15:38.849 [main] DEBUG nextflow.cli.Launcher - $> nextflow run epi2me-labs/wf-alignment -r v1.2.3 -w /weka/users/jefaa92p/tmp/nextflow -profile singularity --fastq /weka/users/jefaa92p/tmp/A549_ddjp/fastq/re-download --out_dir /weka/users/jefaa92p/tmp/A549_ddjp/analysis/wf-alignment_2025-07-03_1115 --references /projects/health_sciences/oms/pathology/sunali_group/ngs/refs/human/grch38_no_alt/ --sample A549_ddjp
Jul-03 11:15:38.967 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W  ~  version 24.04.3
Jul-03 11:15:38.988 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/home/jefaa92p/.nextflow/plugins; core-plugins: nf-amazon@2.5.2,nf-azure@1.6.0,nf-cloudcache@0.4.1,nf-codecommit@0.2.0,nf-console@1.1.3,nf-ga4gh@1.3.0,nf-google@1.13.2-patch1,nf-tower@1.9.1,nf-wave@1.4.2
Jul-03 11:15:39.002 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Jul-03 11:15:39.003 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Jul-03 11:15:39.005 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.10.0 in 'deployment' mode
Jul-03 11:15:39.020 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Jul-03 11:15:39.043 [main] DEBUG nextflow.scm.ProviderConfig - Using SCM config path: /home/jefaa92p/.nextflow/scm
Jul-03 11:15:39.676 [main] DEBUG nextflow.scm.AssetManager - Git config: /home/jefaa92p/.nextflow/assets/epi2me-labs/wf-alignment/.git/config; branch: master; remote: origin; url: https://github.com/epi2me-labs/wf-alignment.git
Jul-03 11:15:39.723 [main] DEBUG nextflow.scm.RepositoryFactory - Found Git repository result: [RepositoryFactory]
Jul-03 11:15:39.735 [main] DEBUG nextflow.scm.AssetManager - Git config: /home/jefaa92p/.nextflow/assets/epi2me-labs/wf-alignment/.git/config; branch: master; remote: origin; url: https://github.com/epi2me-labs/wf-alignment.git
Jul-03 11:15:40.947 [main] DEBUG nextflow.config.ConfigBuilder - Found config home: /home/jefaa92p/.nextflow/config
Jul-03 11:15:40.948 [main] DEBUG nextflow.config.ConfigBuilder - Found config base: /home/jefaa92p/.nextflow/assets/epi2me-labs/wf-alignment/nextflow.config
Jul-03 11:15:40.950 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /home/jefaa92p/.nextflow/config
Jul-03 11:15:40.951 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /home/jefaa92p/.nextflow/assets/epi2me-labs/wf-alignment/nextflow.config
Jul-03 11:15:40.958 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /home/jefaa92p/.nextflow/secrets/store.json
Jul-03 11:15:40.960 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@149c39b] - activable => nextflow.secret.LocalSecretsProvider@149c39b
Jul-03 11:15:40.971 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `singularity`
Jul-03 11:15:41.024 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `singularity`
Jul-03 11:15:41.116 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [standard, conda, singularity, awsbatch, local]
Jul-03 11:15:41.143 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from script declaration
Jul-03 11:15:41.144 [main] DEBUG nextflow.cli.CmdRun - Launching `https://github.com/epi2me-labs/wf-alignment` [desperate_gates] DSL2 - revision: 1bb08961e2 [v1.2.3]
Jul-03 11:15:41.145 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Jul-03 11:15:41.145 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Jul-03 11:15:41.204 [main] DEBUG nextflow.Session - Session UUID: 57504fa6-ebe9-421a-b755-9d4d57de8535
Jul-03 11:15:41.205 [main] DEBUG nextflow.Session - Run name: desperate_gates
Jul-03 11:15:41.205 [main] DEBUG nextflow.Session - Executor pool size: 24
Jul-03 11:15:41.212 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Jul-03 11:15:41.217 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=72; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jul-03 11:15:41.260 [main] DEBUG nextflow.cli.CmdRun - 
  Version: 24.04.3 build 5916
  Created: 09-07-2024 19:35 UTC (10-07-2024 07:35 NZDT)
  System: Linux 5.14.0-284.30.1.el9_2.x86_64
  Runtime: Groovy 4.0.21 on OpenJDK 64-Bit Server VM 17.0.11+9
  Encoding: UTF-8 (UTF-8)
  Process: 1625210@rtis-hpc-r04.uod.otago.ac.nz [192.168.55.164]
  CPUs: 24 - Mem: 125.4 GB (125 GB) - Swap: 0 (0)
Jul-03 11:15:41.285 [main] DEBUG nextflow.Session - Work-dir: /weka/users/jefaa92p/tmp/nextflow [UNKNOWN (0x18031977)]
Jul-03 11:15:41.325 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Jul-03 11:15:41.339 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Jul-03 11:15:41.398 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Jul-03 11:15:41.411 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 25; maxThreads: 1000
Jul-03 11:15:41.516 [main] DEBUG nextflow.Session - Session start
Jul-03 11:15:41.518 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow started -- trace file: /weka/users/jefaa92p/tmp/A549_ddjp/analysis/wf-alignment_2025-07-03_1115/execution/trace.txt
Jul-03 11:15:41.526 [main] DEBUG nextflow.Session - Using default localLib path: /home/jefaa92p/.nextflow/assets/epi2me-labs/wf-alignment/lib
Jul-03 11:15:41.531 [main] DEBUG nextflow.Session - Adding to the classpath library: /home/jefaa92p/.nextflow/assets/epi2me-labs/wf-alignment/lib
Jul-03 11:15:41.532 [main] DEBUG nextflow.Session - Adding to the classpath library: /home/jefaa92p/.nextflow/assets/epi2me-labs/wf-alignment/lib/nfcore_external_java_deps.jar
Jul-03 11:15:43.133 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Jul-03 11:15:44.693 [main] INFO  nextflow.Nextflow - 
[0;92m||||||||||   [0m[2m_____ ____ ___ ____  __  __ _____
[0;92m||||||||||  [0m[2m| ____|  _ \_ _|___ \|  \/  | ____|
[0;33m|||||       [0m[2m|  _| | |_) | |  __) | |\/| |  _|
[0;33m|||||       [0m[2m| |___|  __/| | / __/| |  | | |__
[0;94m||||||||||  [0m[2m|_____|_|  |___|_____|_|  |_|_____|
[0;94m||||||||||  [0m[1mwf-alignment v1.2.3-g1bb0896[0m
[2m--------------------------------------------------------------------------------[0m
[1mCore Nextflow options[0m
  [0;34mrevision       : [0;32mv1.2.3[0m
  [0;34mrunName        : [0;32mdesperate_gates[0m
  [0;34mcontainerEngine: [0;32msingularity[0m
  [0;34mcontainer      : [0;32m[withLabel:wfalignment:ontresearch/wf-alignment:shafd8c4500ccf8b133bef07111626b9e7d876c430d, withLabel:wf_common:ontresearch/wf-common:sha1c69fd30053aad5d516e9567b3944384325a0fee][0m
  [0;34mlaunchDir      : [0;32m/weka/users/jefaa92p/tmp/A549_ddjp[0m
  [0;34mworkDir        : [0;32m/weka/users/jefaa92p/tmp/nextflow[0m
  [0;34mprojectDir     : [0;32m/home/jefaa92p/.nextflow/assets/epi2me-labs/wf-alignment[0m
  [0;34muserName       : [0;32mjefaa92p[0m
  [0;34mprofile        : [0;32msingularity[0m
  [0;34mconfigFiles    : [0;32m/home/jefaa92p/.nextflow/config, /home/jefaa92p/.nextflow/assets/epi2me-labs/wf-alignment/nextflow.config[0m

[1mInput Options[0m
  [0;34mfastq          : [0;32m/weka/users/jefaa92p/tmp/A549_ddjp/fastq/re-download[0m
  [0;34mreferences     : [0;32m/projects/health_sciences/oms/pathology/sunali_group/ngs/refs/human/grch38_no_alt/[0m

[1mSample Options[0m
  [0;34msample         : [0;32mA549_ddjp[0m

[1mOutput Options[0m
  [0;34mout_dir        : [0;32m/weka/users/jefaa92p/tmp/A549_ddjp/analysis/wf-alignment_2025-07-03_1115[0m

!! Only displaying parameters that differ from the pipeline defaults !!
[2m--------------------------------------------------------------------------------[0m
If you use epi2me-labs/wf-alignment for your analysis please cite:

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x


[2m--------------------------------------------------------------------------------[0m
This is epi2me-labs/wf-alignment v1.2.3-g1bb0896.
[2m--------------------------------------------------------------------------------[0m
Jul-03 11:15:45.976 [main] INFO  nextflow.Nextflow - Searching input for [.fastq, .fastq.gz, .fq, .fq.gz] files.
Jul-03 11:15:46.081 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wf_common` matches labels `ingress,wf_common` for process with name fastcat
Jul-03 11:15:46.090 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-03 11:15:46.090 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-03 11:15:46.098 [main] DEBUG nextflow.executor.Executor - [warm up] executor > slurm
Jul-03 11:15:46.104 [main] DEBUG n.processor.TaskPollingMonitor - Creating task monitor for executor 'slurm' > capacity: 100; pollInterval: 5s; dumpInterval: 5m 
Jul-03 11:15:46.106 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: slurm)
Jul-03 11:15:46.108 [main] DEBUG n.executor.AbstractGridExecutor - Creating executor 'slurm' > queue-stat-interval: 1m
Jul-03 11:15:46.153 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wf_common` matches labels `ingress,wf_common` for process with name move_or_compress_fq_file
Jul-03 11:15:46.154 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-03 11:15:46.154 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-03 11:15:46.180 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfalignment` matches labels `wfalignment` for process with name pipeline:getParams
Jul-03 11:15:46.182 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-03 11:15:46.182 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-03 11:15:46.186 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfalignment` matches labels `wfalignment` for process with name pipeline:getVersions
Jul-03 11:15:46.187 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-03 11:15:46.187 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-03 11:15:46.234 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfalignment` matches labels `wfalignment` for process with name pipeline:process_references:fx2tab
Jul-03 11:15:46.235 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-03 11:15:46.236 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-03 11:15:46.240 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfalignment` matches labels `wfalignment` for process with name pipeline:process_references:combine
Jul-03 11:15:46.242 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-03 11:15:46.242 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-03 11:15:46.265 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfalignment` matches labels `wfalignment` for process with name pipeline:makeMMIndex
Jul-03 11:15:46.266 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-03 11:15:46.266 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-03 11:15:46.275 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfalignment` matches labels `wfalignment` for process with name pipeline:alignReads
Jul-03 11:15:46.276 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-03 11:15:46.276 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-03 11:15:46.284 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wf_common` matches labels `ingress,wf_common` for process with name pipeline:bamstats
Jul-03 11:15:46.286 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-03 11:15:46.286 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-03 11:15:46.318 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfalignment` matches labels `wfalignment` for process with name pipeline:addStepsColumn
Jul-03 11:15:46.319 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-03 11:15:46.319 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-03 11:15:46.326 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfalignment` matches labels `wfalignment` for process with name pipeline:readDepthPerRef
Jul-03 11:15:46.327 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-03 11:15:46.327 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-03 11:15:46.337 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfalignment` matches labels `wfalignment` for process with name pipeline:collectFilesInDir
Jul-03 11:15:46.337 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-03 11:15:46.338 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-03 11:15:46.347 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wf_common` matches labels `wf_common` for process with name pipeline:makeReport
Jul-03 11:15:46.347 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:makeReport` matches process pipeline:makeReport
Jul-03 11:15:46.348 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-03 11:15:46.349 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-03 11:15:46.363 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfalignment` matches labels `wfalignment` for process with name publish
Jul-03 11:15:46.364 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-03 11:15:46.364 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-03 11:15:46.366 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: pipeline:readDepthPerRef, move_or_compress_fq_file, pipeline:getParams, pipeline:collectFilesInDir, sortBam, pipeline:makeMMIndex, validate_sample_sheet, pipeline:makeReport, checkBamHeaders, mergeBams, alignReads, makeReport, getVersions, validateIndex, configure_igv, combine, addStepsColumn, readDepthPerRef, pipeline:alignReads, makeMMIndex, pipeline:process_references:combine, pipeline:bamstats, bamstats, pipeline:getVersions, split_fq_file, pipeline:addStepsColumn, fx2tab, pipeline:process_references:fx2tab, fastcat, publish, collectFilesInDir, samtools_index, checkReferences, catSortBams, getParams
Jul-03 11:15:46.368 [main] DEBUG nextflow.Session - Igniting dataflow network (20)
Jul-03 11:15:46.368 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > fastcat
Jul-03 11:15:46.368 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > move_or_compress_fq_file
Jul-03 11:15:46.369 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:getParams
Jul-03 11:15:46.369 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:getVersions
Jul-03 11:15:46.369 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:process_references:fx2tab
Jul-03 11:15:46.369 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:process_references:combine
Jul-03 11:15:46.369 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:makeMMIndex
Jul-03 11:15:46.369 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:alignReads
Jul-03 11:15:46.369 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:bamstats
Jul-03 11:15:46.369 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:addStepsColumn
Jul-03 11:15:46.369 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:readDepthPerRef
Jul-03 11:15:46.369 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:collectFilesInDir
Jul-03 11:15:46.369 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:makeReport
Jul-03 11:15:46.370 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > publish
Jul-03 11:15:46.370 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
  Script_0d574bf69965aed2: /home/jefaa92p/.nextflow/assets/epi2me-labs/wf-alignment/./lib/ingress.nf
  Script_0f8de0e416577fc9: /home/jefaa92p/.nextflow/assets/epi2me-labs/wf-alignment/main.nf
  Script_39b3afe0f2e5f561: /home/jefaa92p/.nextflow/assets/epi2me-labs/wf-alignment/./lib/common.nf
  Script_2a8147577bb1645f: /home/jefaa92p/.nextflow/assets/epi2me-labs/wf-alignment/./subworkflows/process_references.nf
Jul-03 11:15:46.370 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination 
Jul-03 11:15:46.370 [main] DEBUG nextflow.Session - Session await
Jul-03 11:15:46.412 [Actor Thread 9] DEBUG nextflow.container.SingularityCache - Singularity found local store for image=docker://ontresearch/wf-alignment:shafd8c4500ccf8b133bef07111626b9e7d876c430d; path=/weka/users/jefaa92p/tmp/nextflow/singularity/ontresearch-wf-alignment-shafd8c4500ccf8b133bef07111626b9e7d876c430d.img
Jul-03 11:15:46.414 [Actor Thread 9] DEBUG nextflow.container.SingularityCache - Singularity found local store for image=docker://ontresearch/wf-common:sha1c69fd30053aad5d516e9567b3944384325a0fee; path=/weka/users/jefaa92p/tmp/nextflow/singularity/ontresearch-wf-common-sha1c69fd30053aad5d516e9567b3944384325a0fee.img
Jul-03 11:15:46.548 [Task submitter] DEBUG nextflow.executor.GridTaskHandler - [SLURM] submitted process pipeline:getVersions > jobId: 2402341; workDir: /weka/users/jefaa92p/tmp/nextflow/98/44992769dd935aa62ff62cbb62cd13
Jul-03 11:15:46.549 [Task submitter] INFO  nextflow.Session - [98/449927] Submitted process > pipeline:getVersions
Jul-03 11:15:46.602 [Task submitter] DEBUG nextflow.executor.GridTaskHandler - [SLURM] submitted process pipeline:process_references:combine > jobId: 2402342; workDir: /weka/users/jefaa92p/tmp/nextflow/cc/e2ec6e1cb844cd005598a24b459450
Jul-03 11:15:46.603 [Task submitter] INFO  nextflow.Session - [cc/e2ec6e] Submitted process > pipeline:process_references:combine
Jul-03 11:15:46.648 [Task submitter] DEBUG nextflow.executor.GridTaskHandler - [SLURM] submitted process pipeline:process_references:fx2tab (1) > jobId: 2402343; workDir: /weka/users/jefaa92p/tmp/nextflow/e8/a246c743e542a980621f6b69d07923
Jul-03 11:15:46.649 [Task submitter] INFO  nextflow.Session - [e8/a246c7] Submitted process > pipeline:process_references:fx2tab (1)
Jul-03 11:15:46.690 [Task submitter] DEBUG nextflow.executor.GridTaskHandler - [SLURM] submitted process fastcat (1) > jobId: 2402344; workDir: /weka/users/jefaa92p/tmp/nextflow/75/2a2d3baa4728b2b4ab60a2e37eb0c8
Jul-03 11:15:46.690 [Task submitter] INFO  nextflow.Session - [75/2a2d3b] Submitted process > fastcat (1)
Jul-03 11:15:46.732 [Task submitter] DEBUG nextflow.executor.GridTaskHandler - [SLURM] submitted process pipeline:getParams > jobId: 2402345; workDir: /weka/users/jefaa92p/tmp/nextflow/f1/c0e1526c9f48c39fce6d98f52c3e27
Jul-03 11:15:46.732 [Task submitter] INFO  nextflow.Session - [f1/c0e152] Submitted process > pipeline:getParams
Jul-03 11:15:51.134 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 2402345; id: 3; name: pipeline:getParams; status: COMPLETED; exit: 0; error: -; workDir: /weka/users/jefaa92p/tmp/nextflow/f1/c0e1526c9f48c39fce6d98f52c3e27 started: 1751498151132; exited: 2025-07-02T23:15:49.539797386Z; ]
Jul-03 11:15:51.134 [Task monitor] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'TaskFinalizer' minSize=10; maxSize=72; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jul-03 11:15:51.151 [TaskFinalizer-1] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'PublishDir' minSize=10; maxSize=72; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jul-03 11:16:01.114 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 2402343; id: 4; name: pipeline:process_references:fx2tab (1); status: COMPLETED; exit: 0; error: -; workDir: /weka/users/jefaa92p/tmp/nextflow/e8/a246c743e542a980621f6b69d07923 started: 1751498151129; exited: 2025-07-02T23:15:59.999764843Z; ]
Jul-03 11:16:01.144 [Actor Thread 13] DEBUG nextflow.sort.BigSort - Sort completed -- entries: 1; slices: 1; internal sort time: 0.0 s; external sort time: 0.008 s; total time: 0.008 s
Jul-03 11:16:01.152 [Actor Thread 13] DEBUG nextflow.file.FileCollector - Saved collect-files list to: /weka/users/jefaa92p/tmp/nextflow/collect-file/c8645cf3cbf1144f6196f20405fcd59f
Jul-03 11:16:01.156 [Actor Thread 13] DEBUG nextflow.file.FileCollector - Deleting file collector temp dir: /tmp/nxf-16551644577675052473
Jul-03 11:16:01.215 [Task submitter] DEBUG nextflow.executor.GridTaskHandler - [SLURM] submitted process pipeline:addStepsColumn > jobId: 2402346; workDir: /weka/users/jefaa92p/tmp/nextflow/4b/989bb2cf6e6207f124bcdc6024f717
Jul-03 11:16:01.216 [Task submitter] INFO  nextflow.Session - [4b/989bb2] Submitted process > pipeline:addStepsColumn
Jul-03 11:16:06.114 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 2402341; id: 1; name: pipeline:getVersions; status: COMPLETED; exit: 0; error: -; workDir: /weka/users/jefaa92p/tmp/nextflow/98/44992769dd935aa62ff62cbb62cd13 started: 1751498151112; exited: 2025-07-02T23:16:02.522731172Z; ]
Jul-03 11:16:06.115 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 2402342; id: 2; name: pipeline:process_references:combine; status: COMPLETED; exit: 0; error: -; workDir: /weka/users/jefaa92p/tmp/nextflow/cc/e2ec6e1cb844cd005598a24b459450 started: 1751498151127; exited: 2025-07-02T23:16:05.520139657Z; ]
Jul-03 11:16:06.117 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 2402346; id: 6; name: pipeline:addStepsColumn; status: COMPLETED; exit: 0; error: -; workDir: /weka/users/jefaa92p/tmp/nextflow/4b/989bb2cf6e6207f124bcdc6024f717 started: 1751498166116; exited: 2025-07-02T23:16:05.439593144Z; ]
Jul-03 11:16:06.172 [Task submitter] DEBUG nextflow.executor.GridTaskHandler - [SLURM] submitted process pipeline:makeMMIndex > jobId: 2402347; workDir: /weka/users/jefaa92p/tmp/nextflow/1e/e7921b8dcc6e55fac3cd452ca2757f
Jul-03 11:16:06.172 [Task submitter] INFO  nextflow.Session - [1e/e7921b] Submitted process > pipeline:makeMMIndex
Jul-03 11:17:01.115 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 2402347; id: 7; name: pipeline:makeMMIndex; status: COMPLETED; exit: 0; error: -; workDir: /weka/users/jefaa92p/tmp/nextflow/1e/e7921b8dcc6e55fac3cd452ca2757f started: 1751498171113; exited: 2025-07-02T23:17:00.313812819Z; ]
Jul-03 11:20:51.119 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor slurm > tasks to be completed: 1 -- submitted tasks are shown below
~> TaskHandler[jobId: 2402344; id: 5; name: fastcat (1); status: RUNNING; exit: -; error: -; workDir: /weka/users/jefaa92p/tmp/nextflow/75/2a2d3baa4728b2b4ab60a2e37eb0c8 started: 1751498151131; exited: -; ]
Jul-03 11:25:51.134 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor slurm > tasks to be completed: 1 -- submitted tasks are shown below
~> TaskHandler[jobId: 2402344; id: 5; name: fastcat (1); status: RUNNING; exit: -; error: -; workDir: /weka/users/jefaa92p/tmp/nextflow/75/2a2d3baa4728b2b4ab60a2e37eb0c8 started: 1751498151131; exited: -; ]
Jul-03 11:30:21.147 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 2402344; id: 5; name: fastcat (1); status: COMPLETED; exit: 0; error: -; workDir: /weka/users/jefaa92p/tmp/nextflow/75/2a2d3baa4728b2b4ab60a2e37eb0c8 started: 1751498151131; exited: 2025-07-02T23:30:18.570583302Z; ]
Jul-03 11:30:21.213 [Task submitter] DEBUG nextflow.executor.GridTaskHandler - [SLURM] submitted process pipeline:alignReads (1) > jobId: 2402376; workDir: /weka/users/jefaa92p/tmp/nextflow/ca/d9cffa340d56d8d2795e89611a298a
Jul-03 11:30:21.213 [Task submitter] INFO  nextflow.Session - [ca/d9cffa] Submitted process > pipeline:alignReads (1)
Jul-03 11:30:51.147 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor slurm > tasks to be completed: 1 -- submitted tasks are shown below
~> TaskHandler[jobId: 2402376; id: 8; name: pipeline:alignReads (1); status: RUNNING; exit: -; error: -; workDir: /weka/users/jefaa92p/tmp/nextflow/ca/d9cffa340d56d8d2795e89611a298a started: 1751499026146; exited: -; ]
Jul-03 11:34:01.156 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 2402376; id: 8; name: pipeline:alignReads (1); status: COMPLETED; exit: 1; error: -; workDir: /weka/users/jefaa92p/tmp/nextflow/ca/d9cffa340d56d8d2795e89611a298a started: 1751499026146; exited: 2025-07-02T23:33:58.199765527Z; ]
Jul-03 11:34:01.160 [TaskFinalizer-8] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=pipeline:alignReads (1); work-dir=/weka/users/jefaa92p/tmp/nextflow/ca/d9cffa340d56d8d2795e89611a298a
  error [nextflow.exception.ProcessFailedException]: Process `pipeline:alignReads (1)` terminated with an error exit status (1)
Jul-03 11:34:01.190 [TaskFinalizer-8] ERROR nextflow.processor.TaskProcessor - Error executing process > 'pipeline:alignReads (1)'

Caused by:
  Process `pipeline:alignReads (1)` terminated with an error exit status (1)


Command executed:

  cat seqs.fastq.gz     | minimap2 -t 3 -ax map-ont -y --cap-kalloc 100m --cap-sw-mem 50m         combined_refs.mmi -     | samtools sort --write-index -@ 0 -o reads.bam -         -o aligned.sorted.bam##idx##aligned.sorted.bam.bai

Command exit status:
  1

Command output:
  (empty)

Command error:
  INFO:    Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred
  INFO:    Environment variable SINGULARITYENV_NXF_TASK_WORKDIR is set, but APPTAINERENV_NXF_TASK_WORKDIR is preferred
  INFO:    Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred
  [M::main::7.231*0.99] loaded/built the index for 195 target sequence(s)
  [M::mm_mapopt_update::8.843*0.99] mid_occ = 694
  [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 195
  [M::mm_idx_stat::9.598*0.99] distinct minimizers: 100167746 (38.80% are singletons); average occurrences: 5.519; average spacing: 5.607; total length: 3099922541
  [E::aux_parse] invalid aux tag id
  [W::sam_read1_sam] Parse error at line 198
  samtools sort: truncated file. Aborting

Work dir:
  /weka/users/jefaa92p/tmp/nextflow/ca/d9cffa340d56d8d2795e89611a298a

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
Jul-03 11:34:01.194 [TaskFinalizer-8] DEBUG nextflow.Session - Session aborted -- Cause: Process `pipeline:alignReads (1)` terminated with an error exit status (1)
Jul-03 11:34:01.194 [main] DEBUG nextflow.Session - Session await > all processes finished
Jul-03 11:34:02.168 [main] DEBUG nextflow.Session - Session await > all barriers passed
Jul-03 11:34:02.169 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: slurm) - terminating tasks monitor poll loop
Jul-03 11:34:02.483 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=7; failedCount=1; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=1h 2m 3s; failedDuration=14m 20s; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=5; peakCpus=8; peakMemory=33 GB; ]
Jul-03 11:34:02.483 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Jul-03 11:34:02.485 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
Jul-03 11:34:02.816 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Jul-03 11:34:02.857 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Jul-03 11:34:02.872 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye
```

### Application activity log entry

```shell

```

### Were you able to successfully run the latest version of the workflow with the demo data?

yes

### Other demo data information

```shell

```

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

samtools sort aux_parse error: "Incomplete aux field, samtools sort: truncated file. Aborting" #44

Operating System

Other Linux

Workflow Version

Workflow Execution

Other workflow execution

EPI2ME Version

CLI command run

Workflow Execution - CLI Execution Profile

What happened?

Relevant log output

Application activity log entry

Were you able to successfully run the latest version of the workflow with the demo data?

Other demo data information

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Uh oh!

samtools sort aux_parse error: "Incomplete aux field, samtools sort: truncated file. Aborting" #44

Description

Operating System

Other Linux

Workflow Version

Workflow Execution

Other workflow execution

EPI2ME Version

CLI command run

Workflow Execution - CLI Execution Profile

What happened?

Relevant log output

Application activity log entry

Were you able to successfully run the latest version of the workflow with the demo data?

Other demo data information

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions