GCTA_MAKE_GRM process failure with ConcurrentModificationException

[mckeowr1]

GCTA_MAKE_GRM process failure with ConcurrentModificationException

2026-05-13

Bug Report: Untitled Bug

Date: 2026-05-13 Nextflow version: Executor: RF

---

Description

GCTA_MAKE_GRM process failure. Threw
java.util.ConcurrentModificationException error.

Similar to #148, #141

Command Used

nextflow run main.nf   -profile rockfish   --strainfile data/sims_panelsize_ce-cb-ct_nqtl5_h2-0.8_50reps/strains_panelsize_ce.tsv   --nqtl data/sims_panelsize_ce-cb-ct_nqtl5_h2-0.8_50reps/nqtl.csv   --h2 data/sims_panelsize_ce-cb-ct_nqtl5_h2-0.8_50reps/h2.csv   --effect data/sims_panelsize_ce-cb-ct_nqtl5_h2-0.8_50reps/effect_sizes.csv   --reps 50   --cv_maf 0.05   --cv_ld 0.8   --output_dir /vast/eande106/projects/Ryan/simulation_pipeline/nemascan-sims-nf/Sims_panelsize_ce_nqtl5_h2-0.8_50reps   -work-dir /scratch4/eande106/Ryan_tmp/nf-work-panelsize

Error Output

    Pipeline execution summary
    ---------------------------
    Completed at: 2026-05-12T19:21:35.437327380-04:00
    Duration    : 3h 12m 8s
    Success     : false
    workDir     : /scratch4/eande106/Ryan_tmp/nf-work-panelsize
    exit status : null
    Error report: Error executing process > 'GCTA_MAKE_GRM (5 14 0.8 gamma loco ce.n100.r34_0.05)'

Caused by:
  java.util.ConcurrentModificationException


    Git info: https://github.com/AndersenLab/nemascan-sims-nf - prodrun-output-validation [63cdacbfb2ea5a9153d79c82ea9cfbc75e3afd4e]

    { Parameters }
    ---------------------------
    Strainfile                              = data/sims_panelsize_ce-cb-ct_nqtl5_h2-0.8_50reps/strains_panelsize_ce.tsv
    Causal variant MAF                      = (per-group ms_maf)
    Causal variant LD threshold             = null
    Number of simulated QTLs                = null
    Phenotype Heritability File             = null
    Number of simulation replicates         = 50
    Effect Size Range File                  = null
    Marker Genomic Range File               = null
    Significance Thresholds                 = BF, EIGEN
    Threshold for grouping QTL              = 1000
    Number of SNVs to define CI             = 150
    Relatedness Matrix Cutoff               = 0.05
    Mitochondrial chromosome name           = null
    Simulate QTLs in specific regions       = null
    Result Directory                        = /vast/eande106/projects/Ryan/simulation_pipeline/nemascan-sims-nf/Sims_panelsize_ce_nqtl5_h2-0.8_50reps

[mckeowr1]

RCA: GCTA_MAKE_GRM ConcurrentModificationException

Date: 2026-05-13

NF version: 24.10.1 build 5930

Branch: prodrun-output-validation

Stack Trace

java.util.ConcurrentModificationException: null
    at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1013)
    at java.base/java.util.ArrayList$Itr.next(ArrayList.java:967)
    at nextflow.executor.BashWrapperBuilder.createContainerBuilder(BashWrapperBuilder.groovy:706)
    at nextflow.executor.BashWrapperBuilder.makeBinding(BashWrapperBuilder.groovy:286)
    at nextflow.executor.BashWrapperBuilder.buildNew0(BashWrapperBuilder.groovy:410)
    at nextflow.executor.BashWrapperBuilder.build(BashWrapperBuilder.groovy:425)
    at nextflow.executor.GridTaskHandler.prepareLauncher(GridTaskHandler.groovy:107)
    at nextflow.processor.TaskArrayCollector.createTaskArray(TaskArrayCollector.groovy:141)
    at nextflow.processor.TaskArrayCollector.collect(TaskArrayCollector.groovy:103)
    at nextflow.processor.TaskProcessor.submitTask(TaskProcessor.groovy:2339)
    at nextflow.processor.TaskProcessor.invokeTask(TaskProcessor.groovy:655)
    at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor$1.run(...)

Direct Evidence from the Log

The .nextflow.log shows sibling inbred/loco tasks for the same replicate assigned consecutive indices of the same SLURM array job:

jobId: 24082819_0  GCTA_MAKE_GRM (5 13 0.8 gamma loco  ce.n100.r49_0.05)
jobId: 24082819_1  GCTA_MAKE_GRM (5 13 0.8 gamma inbred ce.n100.r49_0.05)
jobId: 24082819_2  GCTA_MAKE_GRM (5 20 0.8 gamma inbred ce.n100.r49_0.05)
jobId: 24082819_3  GCTA_MAKE_GRM (5 20 0.8 gamma loco  ce.n100.r49_0.05)

Inbred and loco siblings for the same (nqtl, rep, group) combination land in the same SLURM array batch. With array = 100, up to 50 inbred+loco pairs can be co-batched in one submission.

Why the #154 Fix Was Insufficient

Issue #154 replaced the wrap-combine-unwrap anti-pattern (.map{[it]}.combine(ch_mode).map{it[0]}) with merge + multiMap.

The tuple() call inside multiMap creates an independent Groovy ArrayList for each mode emission, so inbred and loco tasks receive different ArrayList objects at the Nextflow DSL channel level.

This sould have fixed the CME, but it appears the race is not occurring in the channel tuple. It occurs in BashWrapperBuilder.createContainerBuilder when iterating input file paths to build Singularity --bind mount arguments.

Its possible that the same ArrayList instance is shared between inbred and loco siblings at the BashWrapperBuilder level.

[mckeowr1]

Fix Attempt 1: SLURM-Array FileHolder Race

Branch: fix/perform-gwa-concurrent-mod-exception

Problem

Under SLURM array execution at high concurrency, sibling GCTA_MAKE_GRM and GCTA_PERFORM_GWA TaskRuns staged the same upstream files concurrently and raced on shared FileHolder references. Symptoms: NoSuchFileException, truncated stage-ins, and symlink collisions in work/stage-* directories.

Root cause: .combine(ch_mode) / .combine(ch_type) fanouts emit tuples that share the same Path references across sibling arms. NF caches FileHolder instances keyed by Path identity, so sibling tasks race when materializing the inner List<FileHolder> for input staging.

Approach

Fix two cascading fanouts at the GCTA stage with the same two-part pattern:

1. Race fix — flatMap + freshFiles. Replace .combine(ch_arm) with .flatMap { ... } that rewraps each path with file(p.toString()) per arm. The string round-trip forces FileHelper.asPath() to construct a fresh Path per arm, breaking FileHolder-cache aliasing across siblings.
2. Named aliases per arm. Each fanout arm becomes its own include { PROC as ALIAS } instance — distinct work-dir subtrees, distinct rows in nextflow log, and distinct DataflowBroadcast output channels for downstream consumers.

GCTA_MAKE_GRM (inbred / loco fanout)

• main.nf:22-27 — replaced single include { GCTA_MAKE_GRM } with two aliases: GCTA_MAKE_GRM_INBRED_FASTGWA, GCTA_MAKE_GRM_LOCO_MLMA.
• main.nf:650-718 — rewrote PLINK_UPDATE_BY_H2.out.out consumer:
  • .flatMap { ... } fans out inbred/loco arms with freshFiles() called separately per arm.
  • .branch { inbred / loco } routes by mode.
  • Per-arm .multiMap splits into params / plink / pheno_in (renamed from pheno to disambiguate from upstream .out.pheno).
  • Two invocations, one per alias.
  • ch_versions mixes both aliases' .versions.

GCTA_PERFORM_GWA (pca / nopca fanout × inbred / loco)

• main.nf:28-37 — four aliases:
  • GCTA_PERFORM_GWA_INBRED_PCA
  • GCTA_PERFORM_GWA_INBRED_NOPCA
  • GCTA_PERFORM_GWA_LOCO_PCA
  • GCTA_PERFORM_GWA_LOCO_NOPCA
• main.nf:729-878 — rewrote the GRM-output → GWA-input chain:
  • Per-alias .merge() of params + grm + plink + pheno keeps within-task alignment per GRM alias.
  • .mix() collapses the two GRM-alias streams.
  • .combine(ch_species_for_gwa, by: 0) attaches species once on the mixed stream (avoids forking ch_species_for_gwa four ways).
  • .flatMap { ... } fans out pca/nopca arms with freshFiles() called separately per arm.
  • .branch routes the 4 (mode, type) combinations to named sub-channels.
  • Four per-arm .multiMap blocks split into params / grm / plink / pheno_in.
  • Four alias invocations.
  • ch_versions mixes all four .versions.

Re-mix at the next consumer

• main.nf:875-915:
  • Per-alias 5-way merge (params + grm + plink + pheno + gwa) preserves within-alias task alignment.
  • .mix() across the 4 aliases collapses to a single 24-element tuple stream.
  • .multiMap breaks back out into ch_perform_gwa.{params, grm, plink, pheno, gwa} — sub-channels aligned by construction.
  • Existing .tap{} fork chain (ch_gwa_params_for_db / _analysis / _legacy, ch_gwa_gwa_for_db / _legacy, ch_gwa_pheno_for_analysis / _legacy) preserved verbatim against the new ch_perform_gwa.*.

Why merge → mix → multiMap (and not mix-per-output)

A naive "mix each output separately, then merge" approach would break per-task alignment. Within a single alias, .params[i] and .pheno[i] emit in the same task-completion order, so they pair correctly. But across two alias instances running in parallel, task-completion order is non-deterministic — .mix() interleaves arrivals, so a mixed .params stream and a mixed .pheno stream can be out of order relative to each other. Downstream .merge() would then mis-pair params[group=X] with pheno[group=Y].

The fix: merge within each alias first (alignment guaranteed by emission order on a single process), then mix across aliases (no merge across), then multiMap the mixed stream (sub-channels aligned by construction because they come from the same source tuple).

[mckeowr1]

Follow up Bug Report: ConcurrentModificationException in Nextflow's TaskArrayCollector

Date: 2026-05-18

Branch: fix/perform-gwa-concurrent-mod-exception

Summary

Full re-run of the inital command to test the fix in fix/perform-gwa-concurrent-mod-exception validation aborted on 2026-05-18 at 21:55:02 with a ConcurrentModificationException (CME) during TaskArrayCollector.createTaskArray() for GCTA_MAKE_GRM_LOCO (5 41 0.8 gamma loco ce.n200.r29_0.05).

The race is inside Nextflow itself, not in the pipeline. The session singleton ContainerConfig holds an ArrayList<String> under its envWhitelist key. TaskArrayRun.getContainerConfig() mutates that list on every array flush. BashWrapperBuilder.createContainerBuilder line 706 iterates the same list. Two threads, one reader and one mutator, race on the iterator's modCount snapshot. The pipeline-level branch work shipped on fix/perform-gwa-concurrent-mod-exception (freshFiles() helper + six-alias GCTA fanout) cannot reach this state and is unrelated to this CME.

The original rca.md framed the race as concurrent mutation of a per-task
FileHolder input list at the same line 706. That framing is wrong —
verified directly against the on-disk Nextflow 24.10.1 source. Line 706
iterates envWhitelist, not file holders.

Potential Mechanism

TaskArrayRun.getContainerConfig() calls .add() on a list it received from a session-wide singleton ContainerConfig via getEnvWhitelist() (which returns by reference, no defensive copy).

BashWrapperBuilder.createContainerBuilder at line 706 iterates that same list when building each task's container wrapper. Under sustained array-flush concurrency — multiple TaskArrayCollectors entering createTaskArray in tight wall-clock windows — one thread's ArrayList.iterator() snapshot of modCount becomes stale when another thread's add() bumps it. The next iter.next() calls checkForComodification(), sees the bumped modCount, and throws ConcurrentModificationException. Because the throw fires inside TaskProcessor.submitTask
TaskArrayCollector.collect (before any TaskHandler is registered), errorStrategy = 'ignore' cannot catch it; the session goes straight to Session.aborted.

Scale dependence

This race is more likely to manifest at larger scales because sustained concurrency increases the probability of interleaving a mutating add() between an iterator's snapshot and its exhaustion. The original report came from a Tier 3 run with 100+ concurrent array flushes, but the mechanism could manifest at smaller scales with enough bad luck.

Fix Attempt

Mitigation: a six-line addition to conf/rockfish.config substituting a java.util.concurrent.CopyOnWriteArrayList for the racy ArrayList.

conf/rockfish.config CoWAL substitution

Six lines added inside the existing singularity { } block:

singularity {
    enabled = true
    autoMounts = true
    runOptions = "--bind ${projectDir} --bind ${params.output_dir}"
    cacheDir = "${params.baseDir}/singularity"
    // Issue #175 mitigation: pre-populate envWhitelist with a thread-safe CoWAL
    // so TaskArrayRun.getContainerConfig mutations and BashWrapperBuilder line
    // 706 iteration cannot race. Sentinel keeps the list truthy so the Elvis
    // swap at TaskArrayRun.groovy:38 does not replace our CoWAL with a fresh
    // ArrayList. Remove once an upstream NF fix lands.
    envWhitelist = new java.util.concurrent.CopyOnWriteArrayList<String>(['__NXF_ENVWL_FIX_SENTINEL__'])
}

Why this could work

CopyOnWriteArrayList (CoWAL) is designed for concurrent mutation and iteration. Its add() creates a new copy of the underlying array, so iterators see a consistent snapshot without throwing ConcurrentModificationException. By substituting the session singleton's envWhitelist with a CoWAL, we can break the mutator. TaskArrayRun.getContainerConfig() will still add the array index name to the CoWAL, but that won't affect any iterators in BashWrapperBuilder.createContainerBuilder. The CME should be eliminated, allowing the session to complete even under high concurrency.

Debug Notes

Stack trace from .nextflow.log

java.util.ConcurrentModificationException: null
    at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1013)
    at java.util.ArrayList$Itr.next(ArrayList.java:967)
    at nextflow.executor.BashWrapperBuilder.createContainerBuilder(BashWrapperBuilder.groovy:706)
    at nextflow.executor.BashWrapperBuilder.makeBinding(BashWrapperBuilder.groovy:286)
    at nextflow.executor.BashWrapperBuilder.buildNew0(BashWrapperBuilder.groovy:410)
    at nextflow.executor.BashWrapperBuilder.buildNew0(BashWrapperBuilder.groovy:172)
    at nextflow.executor.BashWrapperBuilder.build(BashWrapperBuilder.groovy:425)
    at nextflow.executor.GridTaskHandler.prepareLauncher(GridTaskHandler.groovy:107)
    at nextflow.processor.TaskArrayCollector.createTaskArray(TaskArrayCollector.groovy:141)
    at nextflow.processor.TaskArrayCollector.collect(TaskArrayCollector.groovy:103)
    at nextflow.processor.TaskProcessor.submitTask(TaskProcessor.groovy:2339)

Reconstruct the actor-thread interleaving around the abort

LOG=/vast/eande106/projects/Ryan/simulation_pipeline/nemascan-sims-nf/.nextflow.log
grep -n "ConcurrentModificationException" ${LOG}

Output:

88638:  error [java.util.ConcurrentModificationException]: java.util.ConcurrentModificationException
88646:  java.util.ConcurrentModificationException
88649:java.util.ConcurrentModificationException: null
88675:May-18 21:55:02.637 [Actor Thread 278] DEBUG nextflow.Session - Session aborted -- Cause: java.util.ConcurrentModificationException
91090:  java.util.ConcurrentModificationException

The CME is logged at line 88638, then again at 88646 and 88649. The session abort is logged at 88675, about 3 seconds after the first CME log. The final CME log at 91090 is about 2 minutes after the abort log, likely a delayed flush of the same exception from the same thread.

sed -n '88625,88700p' ${LOG}

21:55:01.971 [Task monitor] Task completed > PLINK_UPDATE_BY_H2 (5 45 0.8 gamma ce.n200.r36_0.05)
21:55:02.041 [Task monitor] Task completed > PLINK_UPDATE_BY_H2 (5 24 0.8 gamma ce.n300.r21_0.05)
21:55:02.046 [Task monitor] Task completed > PLINK_UPDATE_BY_H2 (5 37 0.8 gamma ce.n200.r36_0.05)
21:55:02.051 [Task monitor] Task completed > PLINK_UPDATE_BY_H2 (5 35 0.8 gamma ce.n200.r36_0.05)
21:55:02.139 [Task monitor] Task completed > PLINK_UPDATE_BY_H2 (5 36 0.8 gamma ce.n200.r36_0.05)
... (21 PLINK_UPDATE_BY_H2 completions between 21:55:01.971 and 21:55:02.736)
21:55:02.338 [Actor Thread 278] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=GCTA_MAKE_GRM_LOCO (5 41 0.8 gamma loco ce.n200.r29_0.05)
  error [java.util.ConcurrentModificationException]
21:55:02.440 [Actor Thread 278] ERROR <stack trace>
21:55:02.536 [Task submitter] Submitted process > GCTA_MAKE_GRM_LOCO (5 25 0.8 gamma loco ce.n500.r13_0.05)
21:55:02.637 [Task submitter] Submitted process > GCTA_MAKE_GRM_LOCO (5 43 0.8 gamma loco ce.n500.r13_0.05)
21:55:02.637 [Task submitter] Submitted process > GCTA_MAKE_GRM_LOCO (5 28 0.8 gamma loco ce.n500.r13_0.05)
21:55:02.637 [Task submitter] Submitted process > GCTA_MAKE_GRM_LOCO (5 27 0.8 gamma loco ce.n500.r13_0.05)
21:55:02.637 [Actor Thread 278] DEBUG nextflow.Session - Session aborted -- Cause: ConcurrentModificationException
21:55:02.637 [Task submitter] Submitted process > GCTA_MAKE_GRM_LOCO (5 49 0.8 gamma loco ce.n200.r15_0.05)
... (continued submissions for ce.n300.r10, ce.n500.r32)

1. [Actor Thread 278] is inside TaskProcessor.submitTask for the
   failing GCTA_MAKE_GRM_LOCO task. The stack frames pin the failure
   to TaskArrayCollector.collect → createTaskArray →
   prepareLauncher → BashWrapperBuilder line 706. That task's
   collect() triggered a flush — its collector had reached its
   100-task threshold.
2. The 21 PLINK_UPDATE_BY_H2 completions between 21:55:01.971 and
   21:55:02.736 are the upstream events feeding new tasks into both
   GCTA_MAKE_GRM_INBRED and GCTA_MAKE_GRM_LOCO collectors in the
   same millisecond window. Several collectors fill and flush
   concurrently — direct evidence of the mutator/reader concurrency
   topology the mechanism requires.
3. The submission burst at 21:55:02.536–21:55:02.738 mixes panels
   ce.n500.r13, ce.n200.r15, ce.n300.r10, ce.n500.r32 into a
   single array job. This is "cross-panel batching" — a symptom of
   high concurrency at Tier 3 scale, but incidental to the race.
   One createTaskArray call bumps modCount once regardless of how
   many panels are in the batch.

Finding. Five distinct gpars actor threads logged
Creating task array run within seconds of each other. Steady-state
large scale run has multiple actor threads inside createTaskArray concurrently
— exactly the topology required for the envWhitelist race.

Source code references

Pull the v24.10.1 source files at issue

mkdir -p /tmp/nf-src-check && cd /tmp/nf-src-check
BASE="https://raw.githubusercontent.com/nextflow-io/nextflow/v24.10.1/modules/nextflow/src/main/groovy/nextflow"
curl -sSL -o BashWrapperBuilder.groovy ${BASE}/executor/BashWrapperBuilder.groovy
curl -sSL -o TaskArrayRun.groovy        ${BASE}/processor/TaskArrayRun.groovy
curl -sSL -o TaskArrayCollector.groovy  ${BASE}/processor/TaskArrayCollector.groovy
curl -sSL -o ContainerConfig.groovy     ${BASE}/container/ContainerConfig.groovy
curl -sSL -o Session.groovy             ${BASE}/Session.groovy

Verify what line 706 of BashWrapperBuilder iterates

sed -n '690,720p' /tmp/nf-src-check/BashWrapperBuilder.groovy

706         for( String var : containerConfig.getEnvWhitelist() ) {
707             builder.addEnv(var)
708         }

Line 706 iterates containerConfig.getEnvWhitelist(). The
rca.md claim that line 706 iterates an inner List<FileHolder> is
wrong.

Verify the mutator side

cat /tmp/nf-src-check/TaskArrayRun.groovy

Relevant excerpt:

@Slf4j
@CompileStatic
class TaskArrayRun extends TaskRun {
    ...
    @Override
    ContainerConfig getContainerConfig() {
        final config = super.getContainerConfig()
        final envWhitelist = config.getEnvWhitelist() ?: []
        final executor = (TaskArrayExecutor)processor.getExecutor()
        envWhitelist << executor.getArrayIndexName()
        config.put('envWhitelist', envWhitelist)
        return config
    }
    ...
}

Finding. envWhitelist << executor.getArrayIndexName() is Groovy shorthand for envWhitelist.add(...). Whether this mutates a shared singleton or a local copy depends entirely on what config.getEnvWhitelist() returns

Verify ContainerConfig.getEnvWhitelist() does not copy

grep -n -A 15 "getEnvWhitelist" /tmp/nf-src-check/ContainerConfig.groovy

Output:

95:    List<String> getEnvWhitelist() {
96:        def result = get('envWhitelist')
97-        if( !result )
98-            return Collections.emptyList()
99-        if( result instanceof CharSequence )
100-           return result.tokenize(',').collect { it.trim() }
101-       if( result instanceof List )
102-           return result                  // ← returns the underlying List by reference
103-       throw new IllegalArgumentException("Not a valid `envWhitelist` argument")
104-   }

Finding. When the underlying storage is already a List, line 104 returns it by reference. No defensive copy. So the envWhitelist that TaskArrayRun.getContainerConfig() mutates is the same list BashWrapperBuilder reads.

Verify Session.getContainerConfig() is a singleton

grep -n -B 2 -A 20 "getContainerConfig" /tmp/nf-src-check/Session.groovy

Relevant excerpt:

1252-     */
1253-    @Memoized
1254:    ContainerConfig getContainerConfig(String engine) {
...
1285:    ContainerConfig getContainerConfig() {
1286:        return getContainerConfig(null)
1287-    }

Finding. @Memoized caches the return value per engine key, so every call to session.getContainerConfig('singularity') returns the same ContainerConfig instance. That instance's underlying map holds the envWhitelist list that line 102 of ContainerConfig.groovy returns by reference. Therefore, the list is session-wide singleton state.