Head-node JVM `OutOfMemoryError` at 50-rep scale

[mckeowr1]

Details

• Date: 2026-05-21
• Executor: RF
• Branch: fix/175-stacked-cme-mitigations

Overview

Several task different threads are failing with different erorr messages.

For example, the GCTA_PERFORM_GWA_LOCO_NOPCA task is failing with an Cannot obtain the semaphore to fork operator's body

May-21 07:26:25.270 [Actor Thread 100] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=GCTA_PERFORM_GWA_LOCO_NOPCA; work-dir=null
  error [java.lang.IllegalStateException]: Cannot obtain the semaphore to fork operator's body.

This was preceeded in the log by a java.lang.OutOfMemoryError: Java heap space error

May-21 07:24:21.241 [Actor Thread 630] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=GCTA_PERFORM_GWA_LOCO_NOPCA (6828); work-dir=null
  error [java.lang.OutOfMemoryError]: Java heap space
May-21 07:24:37.041 [Actor Thread 630] ERROR nextflow.processor.TaskProcessor - Execution aborted due to an unexpected error
java.lang.OutOfMemoryError: Java heap space

Other actor threads reported the java.lang.OutOfMemoryError: Java heap space error

May-21 07:22:57.349 [Actor Thread 73] ERROR nextflow.util.CustomThreadPool - Unexpected thread exception
org.codehaus.groovy.runtime.InvokerInvocationException: java.lang.OutOfMemoryError: Java heap space
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:348)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
        at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
        at org.codehaus.groovy.runtime.InvokerHelper.invokePojoMethod(InvokerHelper.java:633)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:624)
        at groovyx.gpars.actor.Actor.callDynamic(Actor.java:369)
        at groovyx.gpars.actor.Actor.handleTermination(Actor.java:320)
        at groovyx.gpars.actor.AbstractLoopingActor.terminate(AbstractLoopingActor.java:204)
        at groovyx.gpars.dataflow.operator.DataflowProcessor.terminate(DataflowProcessor.java:147)
        at groovyx.gpars.dataflow.operator.DataflowProcessor.reportError(DataflowProcessor.java:294)
        at groovyx.gpars.dataflow.operator.DataflowProcessorActor.reportException(DataflowProcessorActor.java:125)
        at groovyx.gpars.dataflow.operator.DataflowProcessorActor.onException(DataflowProcessorActor.java:72)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
        at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
        at org.codehaus.groovy.runtime.InvokerHelper.invokePojoMethod(InvokerHelper.java:633)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:624)
        at groovyx.gpars.actor.Actor.callDynamic(Actor.java:368)
        at groovyx.gpars.actor.Actor.handleException(Actor.java:339)
        at groovyx.gpars.actor.AbstractLoopingActor$1.registerError(AbstractLoopingActor.java:63)
        at groovyx.gpars.util.AsyncMessagingCore.run(AsyncMessagingCore.java:140)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.OutOfMemoryError: Java heap space

From a quick visual check of the log, it looks like the actor threads throwing this error are reproted from the [Task submitter] and several different actor threads.

Check scratch space allocation

+----------------------------------------------------------------------------------------------+
|                GPFS Usage for Group eande106: as of Thu May 21 09:00:15 2026                 |
+------------+------------+------------+----------+---------------+----------------+-----------+
|     FS     |    Used    |   Quota    |  Used %  |     Files     |  Files Quota   |  Files %  |
+------------+------------+------------+----------+---------------+----------------+-----------+
|    data    |  2.22 TB   |  10.00 TB  |  22.00%  |   2,857,658   |   4,096,000    |   69.00%  |
|  scratch4  |  16.34 TB  |  20.00 TB  |  81.00%  |   35,957,832  |   40,960,000   |   87.00%  |
+------------+------------+------------+----------+---------------+----------------+-----------+

+---------------------------------------------------------------------------------+
|          Vast Usage for Group eande106: as of Thu May 21 09:02:21 2026          |
+-----------------------+-----------------------+----------------+----------------+
|          Used         |         Quota         |     Used %     |       DR       |
+-----------------------+-----------------------+----------------+----------------+
|       129.92 TB       |       200.00 TB       |     64.96%     |     1.38:1     |
+-----------------------+-----------------------+----------------+----------------+

We are using a large proportion of the scratch space, but it is not full.

Error Message

[81/e38fa6] process > LOCAL_GET_CONTIG_INFO (ce.n100.r01)                                                    [100%] 1 of 1, cached: 1 ✔                                                                                 
[79/4200f5] process > BCFTOOLS_EXTRACT_STRAINS (ce.n500.r50)                                                 [100%] 250 of 250, cached: 250 ✔                                                                           
[25/7f37af] process > BCFTOOLS_RENAME_CHROMS (ce.n500.r50)                                                   [100%] 250 of 250, cached: 250 ✔                                                                           
[0e/f4335e] process > PLINK_RECODE_MS_VCF (ce.n500.r50_0.05)                                                 [100%] 250 of 250, cached: 250 ✔                                                                           
[c2/8822a1] process > PLINK_RECODE_CV_VCF (ce.n500.r50_0.05)                                                 [100%] 250 of 250, cached: 250 ✔                                                                           
[81/b6c244] process > BCFTOOLS_CREATE_GENOTYPE_MATRIX (ce.n500.r50_0.05)                                     [100%] 250 of 250, cached: 250 ✔                                                                           
[5e/8209a6] process > R_FIND_GENOTYPE_MATRIX_EIGEN (6 ce.n500.r50_0.05)                                      [100%] 1500 of 1500, cached: 1500 ✔                                                                        
[4b/05ca31] process > LOCAL_COMPILE_EIGENS (ce.n500.r50 0.05)                                                [100%] 250 of 250, cached: 250 ✔                                                                           
[77/1585f9] process > PYTHON_SIMULATE_EFFECTS_GLOBAL (5 50 gammace.n500.r50_0.05)                            [100%] 12500 of 12500, cached: 12500 ✔                                                                     
[36/ee6526] process > GCTA_SIMULATE_PHENOTYPES (c_elegans_ce.n500.r34_0.05_5_gamma_0.8_49)                   [100%] 12500 of 12500, cached: 11631 ✔                                                                     
[cc/50aaf9] process > PLINK_UPDATE_BY_H2 (5 22 0.8 gamma ce.n500.r23_0.05)                                   [100%] 10997 of 10997, cached: 8691, failed: 6, retries: 6                                                 
[bd/943aae] process > GCTA_MAKE_GRM_INBRED (5 44 0.8 gamma inbred ce.n400.r42_0.05)                          [100%] 8511 of 8511, cached: 5511                                                                          
[bf/f61625] process > GCTA_MAKE_GRM_LOCO (5 48 0.8 gamma loco ce.n400.r40_0.05)                              [100%] 8297 of 8297, cached: 5797                                                                          
[45/be98fe] process > GCTA_PERFORM_GWA_INBRED_PCA (c_elegans_ce.n300.r46_0.05_5_gamma_0.8_12_inbred_pca)     [100%] 6059 of 6059, cached: 3044, failed: 20, retries: 15                                                 
[46/fc0071] process > GCTA_PERFORM_GWA_INBRED_NOPCA (c_elegans_ce.n300.r43_0.05_5_gamma_0.8_15_inbred_nopca) [100%] 6160 of 6160, cached: 3127, failed: 44, retries: 33                                                 
[e7/010588] process > GCTA_PERFORM_GWA_LOCO_PCA (c_elegans_ce.n400.r17_0.05_5_gamma_0.8_1_loco_pca)          [100%] 6661 of 6661, cached: 3655, failed: 8, retries: 6                                                   
[34/4ccceb] process > GCTA_PERFORM_GWA_LOCO_NOPCA (c_elegans_ce.n400.r11_0.05_5_gamma_0.8_50_loco_nopca)     [ 92%] 6333 of 6838, cached: 3324, failed: 11, retries: 9                                                  
[82/24db94] process > DB_MIGRATION_WRITE_MARKER_SET (ce.n500.r50_0.05)                                       [100%] 250 of 250, cached: 250 ✔                                                                           
[6f/aecb7c] process > DB_MIGRATION_WRITE_GENOTYPE_MATRIX (ce.n500.r50_0.05)                                  [100%] 250 of 250, cached: 250 ✔                                                                           
[1b/a2bfd6] process > DB_MIGRATION_WRITE_TRAIT_DATA (ce.n500.r34_5_14_0.8)                                   [100%] 12500 of 12500, cached: 6741 ✔                                                                      
[36/f84ccb] process > DB_MIGRATION_WRITE_GWA_TO_DB (c_elegans_ce.n300.r33_0.05_5_gamma_0.8_23_inbred_nopca)  [100%] 23447 of 23447, cached: 7447                                                                        
[-        ] process > DB_MIGRATION_AGGREGATE_METADATA                                                        -                                                                                                          
[-        ] process > VALIDATE_REPLICATION_COMPLETE                                                          -                                                                                                          
ERROR ~ Execution aborted due to an unexpected error

 -- Check '.nextflow.log' file for details
ERROR ~ Execution aborted due to an unexpected error

 -- Check '.nextflow.log' file for details
ERROR ~ Execution aborted due to an unexpected error

 -- Check '.nextflow.log' file for details
ERROR ~ Unexpected thread exception

 -- Check '.nextflow.log' file for details
ERROR ~ Execution aborted due to an unexpected error

 -- Check '.nextflow.log' file for details

.nextflow.log

tail -300 .nextflow.log

May-21 06:00:28.767 [Task submitter] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=GCTA_PERFORM_GWA_LOCO_NOPCA (5801); work-dir=/scratch4/eande106/Ryan_tmp/nf-work-175-stacked-smoke_full/3f/075fda8d8f5441dab9dc9450ac2fc6
  error [java.lang.OutOfMemoryError]: Java heap space
May-21 06:01:09.352 [Task submitter] ERROR nextflow.processor.TaskProcessor - Execution aborted due to an unexpected error
java.lang.OutOfMemoryError: Java heap space
        at java.base/java.io.BufferedReader.<init>(BufferedReader.java:106)
        at java.base/java.io.BufferedReader.<init>(BufferedReader.java:117)
        at org.codehaus.groovy.runtime.IOGroovyMethods.iterator(IOGroovyMethods.java:943)
        at org.codehaus.groovy.runtime.StringGroovyMethods$LineIterable.iterator(StringGroovyMethods.java:614)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.toList(DefaultGroovyMethods.java:3388)
        at org.codehaus.groovy.runtime.StringGroovyMethods.readLines(StringGroovyMethods.java:2256)
        at nextflow.util.MustacheTemplateEngine.replace0(MustacheTemplateEngine.groovy:90)
        at nextflow.util.MustacheTemplateEngine.render(MustacheTemplateEngine.groovy:63)
        at nextflow.executor.BashWrapperBuilder.buildNew0(BashWrapperBuilder.groovy:411)
        at nextflow.executor.BashWrapperBuilder.buildNew0(BashWrapperBuilder.groovy:172)
        at nextflow.executor.BashWrapperBuilder.build(BashWrapperBuilder.groovy:425)
        at nextflow.executor.GridTaskHandler.prepareLauncher(GridTaskHandler.groovy:107)
        at nextflow.processor.TaskPollingMonitor.submit(TaskPollingMonitor.groovy:208)
        at nextflow.processor.TaskPollingMonitor.submitPendingTasks(TaskPollingMonitor.groovy:601)
        at nextflow.processor.TaskPollingMonitor.submitLoop(TaskPollingMonitor.groovy:415)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
        at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
        at groovy.lang.MetaClassImpl.invokeMethodClosure(MetaClassImpl.java:1017)
        at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1207)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
        at groovy.lang.Closure.call(Closure.java:433)
        at groovy.lang.Closure.call(Closure.java:412)
        at groovy.lang.Closure.run(Closure.java:505)
        at java.base/java.lang.Thread.run(Thread.java:840)
May-21 06:02:44.079 [Task submitter] DEBUG nextflow.Session - Session aborted -- Cause: Java heap space
May-21 06:16:39.877 [Actor Thread 626] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=GCTA_MAKE_GRM_LOCO (5 32 0.8 gamma loco ce.n400.r04_0.05); work-dir=/scratch4/eande106/Ryan_tmp/nf-work-175-stacked-smoke_full/6a/5aa62f1157f51d0bc630a84b7a1df2
  error [java.lang.OutOfMemoryError]: Java heap space
May-21 06:16:46.379 [Actor Thread 626] ERROR nextflow.processor.TaskProcessor - Execution aborted due to an unexpected error
java.lang.OutOfMemoryError: Java heap space
        at java.base/java.util.Arrays.copyOf(Arrays.java:3537)
        at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:228)
        at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:582)
        at java.base/java.lang.StringBuilder.append(StringBuilder.java:179)
        at java.base/java.lang.StringBuilder.append(StringBuilder.java:173)
        at org.codehaus.groovy.runtime.StringGroovyMethods.leftShift(StringGroovyMethods.java:1863)
        at nextflow.container.SingularityBuilder.makeEnv(SingularityBuilder.groovy:207)
        at java.base/java.lang.invoke.LambdaForm$DMH/0x00001554f50b2000.invokeVirtual(LambdaForm$DMH)
        at java.base/java.lang.invoke.LambdaForm$MH/0x00001554f52eac00.invoke(LambdaForm$MH)
        at java.base/java.lang.invoke.LambdaForm$MH/0x00001554f515e800.guardWithCatch(LambdaForm$MH)
        at java.base/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base/java.lang.invoke.LambdaForm$MH/0x00001554f515f800.guard(LambdaForm$MH)
        at java.base/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base/java.lang.invoke.LambdaForm$MH/0x00001554f515f800.guard(LambdaForm$MH)
        at java.base/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base/java.lang.invoke.LambdaForm$MH/0x00001554f515f800.guard(LambdaForm$MH)
        at java.base/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
        at nextflow.container.ContainerBuilder.appendEnv(ContainerBuilder.groovy:224)
        at nextflow.container.SingularityBuilder.appendEnv(SingularityBuilder.groovy:164)
        at nextflow.container.SingularityBuilder.build(SingularityBuilder.groovy:116)
        at java.base/java.lang.invoke.LambdaForm$DMH/0x00001554f50b3400.invokeVirtual(LambdaForm$DMH)
        at java.base/java.lang.invoke.LambdaForm$MH/0x00001554f5670c00.invoke(LambdaForm$MH)
        at java.base/java.lang.invoke.LambdaForm$MH/0x00001554f5105800.guardWithCatch(LambdaForm$MH)
        at java.base/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base/java.lang.invoke.LambdaForm$MH/0x00001554f5106400.guard(LambdaForm$MH)
        at java.base/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base/java.lang.invoke.LambdaForm$MH/0x00001554f5106400.guard(LambdaForm$MH)
        at java.base/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base/java.lang.invoke.LambdaForm$MH/0x00001554f5106400.guard(LambdaForm$MH)
        at java.base/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
        at nextflow.container.ContainerBuilder.build(ContainerBuilder.groovy:179)
        at nextflow.executor.BashWrapperBuilder.createContainerBuilder(BashWrapperBuilder.groovy:745)
May-21 06:21:04.466 [Actor Thread 754] INFO  nextflow.processor.TaskProcessor - [4f/4231c0] Cached process > GCTA_PERFORM_GWA_INBRED_NOPCA (c_elegans_ce.n300.r43_0.05_5_gamma_0.8_11_inbred_nopca)
May-21 06:26:57.980 [Actor Thread 73] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=GCTA_MAKE_GRM_LOCO; work-dir=null
  error [java.lang.IllegalStateException]: Cannot obtain the semaphore to fork operator's body.
May-21 06:43:39.070 [Actor Thread 972] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=DB_MIGRATION_WRITE_GWA_TO_DB (c_elegans_ce.n300.r11_0.05_5_gamma_0.8_27_inbred_nopca); work-dir=/scratch4/eande106/Ryan_tmp/nf-work-175-stacked-smoke_full/63/c43b3b6d147d0f6c4f408fa8de2c2d
  error [java.lang.OutOfMemoryError]: Java heap space
May-21 06:44:12.656 [Actor Thread 972] ERROR nextflow.processor.TaskProcessor - Execution aborted due to an unexpected error
java.lang.OutOfMemoryError: Java heap space
May-21 06:49:53.180 [Actor Thread 120] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=DB_MIGRATION_WRITE_GWA_TO_DB; work-dir=null
  error [java.lang.IllegalStateException]: Cannot obtain the semaphore to fork operator's body.
May-21 07:09:24.560 [Thread-1] DEBUG nextflow.util.SimpleAgent - Unexpected error while polling agent event | Java heap space
May-21 07:13:20.070 [Actor Thread 875] WARN  nextflow.processor.TaskProcessor - [PLINK_UPDATE_BY_H2 (11046)] Unable to resume cached task -- See log file for details
java.lang.OutOfMemoryError: Java heap space
May-21 07:22:57.349 [Actor Thread 73] ERROR nextflow.util.CustomThreadPool - Unexpected thread exception
org.codehaus.groovy.runtime.InvokerInvocationException: java.lang.OutOfMemoryError: Java heap space
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:348)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
        at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
        at org.codehaus.groovy.runtime.InvokerHelper.invokePojoMethod(InvokerHelper.java:633)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:624)
        at groovyx.gpars.actor.Actor.callDynamic(Actor.java:369)
        at groovyx.gpars.actor.Actor.handleTermination(Actor.java:320)
        at groovyx.gpars.actor.AbstractLoopingActor.terminate(AbstractLoopingActor.java:204)
        at groovyx.gpars.dataflow.operator.DataflowProcessor.terminate(DataflowProcessor.java:147)
        at groovyx.gpars.dataflow.operator.DataflowProcessor.reportError(DataflowProcessor.java:294)
        at groovyx.gpars.dataflow.operator.DataflowProcessorActor.reportException(DataflowProcessorActor.java:125)
        at groovyx.gpars.dataflow.operator.DataflowProcessorActor.onException(DataflowProcessorActor.java:72)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
        at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
        at org.codehaus.groovy.runtime.InvokerHelper.invokePojoMethod(InvokerHelper.java:633)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:624)
        at groovyx.gpars.actor.Actor.callDynamic(Actor.java:368)
        at groovyx.gpars.actor.Actor.handleException(Actor.java:339)
        at groovyx.gpars.actor.AbstractLoopingActor$1.registerError(AbstractLoopingActor.java:63)
        at groovyx.gpars.util.AsyncMessagingCore.run(AsyncMessagingCore.java:140)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.OutOfMemoryError: Java heap space
May-21 07:24:21.241 [Actor Thread 630] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=GCTA_PERFORM_GWA_LOCO_NOPCA (6828); work-dir=null
  error [java.lang.OutOfMemoryError]: Java heap space
May-21 07:24:37.041 [Actor Thread 630] ERROR nextflow.processor.TaskProcessor - Execution aborted due to an unexpected error
java.lang.OutOfMemoryError: Java heap space
May-21 07:26:25.270 [Actor Thread 100] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=GCTA_PERFORM_GWA_LOCO_NOPCA; work-dir=null
  error [java.lang.IllegalStateException]: Cannot obtain the semaphore to fork operator's body.
May-21 07:30:05.055 [Actor Thread 1021] INFO  nextflow.processor.TaskProcessor - [bd/943aae] Cached process > GCTA_MAKE_GRM_INBRED (5 44 0.8 gamma inbred ce.n400.r42_0.05)
May-21 07:30:08.154 [Actor Thread 998] INFO  nextflow.processor.TaskProcessor - [35/424cb8] Cached process > GCTA_PERFORM_GWA_INBRED_NOPCA (c_elegans_ce.n300.r43_0.05_5_gamma_0.8_13_inbred_nopca)
May-21 07:30:08.154 [Actor Thread 1092] INFO  nextflow.processor.TaskProcessor - [e7/010588] Cached process > GCTA_PERFORM_GWA_LOCO_PCA (c_elegans_ce.n400.r17_0.05_5_gamma_0.8_1_loco_pca)
May-21 07:30:08.154 [Actor Thread 1002] INFO  nextflow.processor.TaskProcessor - [63/1ee362] Cached process > GCTA_PERFORM_GWA_INBRED_PCA (c_elegans_ce.n300.r46_0.05_5_gamma_0.8_9_inbred_pca)
May-21 07:30:08.154 [Actor Thread 1081] INFO  nextflow.processor.TaskProcessor - [8a/0df892] Cached process > GCTA_PERFORM_GWA_LOCO_PCA (c_elegans_ce.n400.r17_0.05_5_gamma_0.8_2_loco_pca)
May-21 07:30:11.259 [Actor Thread 899] INFO  nextflow.processor.TaskProcessor - [ae/bf4e8e] Cached process > PLINK_UPDATE_BY_H2 (5 20 0.8 gamma ce.n500.r23_0.05)
May-21 08:22:56.986 [Actor Thread 754] INFO  nextflow.processor.TaskProcessor - [46/fc0071] Cached process > GCTA_PERFORM_GWA_INBRED_NOPCA (c_elegans_ce.n300.r43_0.05_5_gamma_0.8_15_inbred_nopca)
May-21 08:29:10.338 [Actor Thread 100] DEBUG n.processor.TaskArrayCollector - Creating task array run >> /scratch4/eande106/Ryan_tmp/nf-work-175-stacked-smoke_full/54/6a8e030952f7422269b3c30d5c14c5
array=( /scratch4/eande106/Ryan_tmp/nf-work-175-stacked-smoke_full/d0/88951e9660a8e6929f2afeb7ad32da /scratch4/eande106/Ryan_tmp/nf-work-175-stacked-smoke_full/5b/182226fed468f97d14da5f98f132b2 /scratch4/eande106/Ryan_tmp/nf-work-175-stacked-smoke_full/84/c5c591714d7fca2cc28bcc299931a6 /scratch4/eande106/Ryan_tmp/nf-work-175-stacked-smoke_full/6a/f2f1dd3c565d4cc35b3c0306ff2078 )
export nxf_array_task_dir=${array[SLURM_ARRAY_TASK_ID]}
bash $nxf_array_task_dir/.command.run 2>&1 > $nxf_array_task_dir/.command.log

May-21 08:51:38.347 [Actor Thread 1104] INFO  nextflow.processor.TaskProcessor - [45/be98fe] Cached process > GCTA_PERFORM_GWA_INBRED_PCA (c_elegans_ce.n300.r46_0.05_5_gamma_0.8_12_inbred_pca)
May-21 08:58:59.255 [Actor Thread 899] INFO  nextflow.processor.TaskProcessor - [cc/50aaf9] Cached process > PLINK_UPDATE_BY_H2 (5 22 0.8 gamma ce.n500.r23_0.05)
May-21 09:03:26.951 [Actor Thread 998] INFO  nextflow.processor.TaskProcessor - [b5/3dda1f] Cached process > GCTA_PERFORM_GWA_INBRED_NOPCA (c_elegans_ce.n300.r43_0.05_5_gamma_0.8_14_inbred_nopca)
May-21 09:03:26.951 [Actor Thread 1092] INFO  nextflow.processor.TaskProcessor - [43/e87e53] Cached process > GCTA_PERFORM_GWA_LOCO_PCA (c_elegans_ce.n400.r17_0.05_5_gamma_0.8_3_loco_pca)
May-21 09:03:26.951 [Actor Thread 1002] INFO  nextflow.processor.TaskProcessor - [6b/18097a] Cached process > GCTA_PERFORM_GWA_INBRED_PCA (c_elegans_ce.n300.r46_0.05_5_gamma_0.8_15_inbred_pca)
May-21 09:07:16.881 [Actor Thread 1081] INFO  nextflow.processor.TaskProcessor - [46/6e3bfd] Cached process > GCTA_PERFORM_GWA_LOCO_PCA (c_elegans_ce.n400.r17_0.05_5_gamma_0.8_4_loco_pca)

[mckeowr1]

Context

The 50-rep -resume run on Rockfish run on fix/175-stacked-cme-mitigations aborted with java.lang.OutOfMemoryError: Java heap space in the Nextflow orchestrator (head-node) after encountering another CME error. After the first OOM, everything cascades (Cannot obtain the semaphore to fork operator's body, Unexpected thread exception).

This was a test run resuming the pipeline after a CME error was encounted when testing CME fixes on fix/175-stacked-cme-mitigations

Potential causes

Undersized orchestrator heap on the login node (+ -resume re-materialization).

The login node executing the pipeline might have run into memory restrictions. A 200K-task -resume run (which re-materializes the whole cached task graph + all channel emissions) could have run out of memory.

envWhitelist bloat

The addition to the RockFish config in the fix for #175 could be causing bloat in the envWhitelist

Investigation notes

Env set up:

WORK=/scratch4/eande106/Ryan_tmp/nf-work-175-stacked-smoke_full
OUT=/scratch4/eande106/Ryan_tmp/175-stacked-smoke_full
LAUNCH_DIR=$(pwd)

First OOM in the log

grep -nH "OutOfMemoryError" .nextflow.log* | head

.nextflow.log:176785:  error [java.lang.OutOfMemoryError]: Java heap space
.nextflow.log:176787:java.lang.OutOfMemoryError: Java heap space
.nextflow.log:176822:  error [java.lang.OutOfMemoryError]: Java heap space
.nextflow.log:176824:java.lang.OutOfMemoryError: Java heap space
.nextflow.log:176863:  error [java.lang.OutOfMemoryError]: Java heap space
.nextflow.log:176865:java.lang.OutOfMemoryError: Java heap space
.nextflow.log:176871:java.lang.OutOfMemoryError: Java heap space
.nextflow.log:176873:org.codehaus.groovy.runtime.InvokerInvocationException: java.lang.OutOfMemoryError: Java heap space
.nextflow.log:176906:Caused by: java.lang.OutOfMemoryError: Java heap space
.nextflow.log:176909:  error [java.lang.OutOfMemoryError]: Java heap space

Now check the context of the first OOM

grep -n -B3 -A30 "OutOfMemoryError" .nextflow.log* | head -70

.nextflow.log-176782-May-21 04:47:53.374 [Actor Thread 875] INFO  nextflow.processor.TaskProcessor - [6b/42a982] Cached process > PLINK_UPDATE_BY_H2 (5 18 0.8 gamma ce.n500.r23_0.05)
.nextflow.log-176783-May-21 06:00:28.767 [Task submitter] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
.nextflow.log-176784-  task: name=GCTA_PERFORM_GWA_LOCO_NOPCA (5801); work-dir=/scratch4/eande106/Ryan_tmp/nf-work-175-stacked-smoke_full/3f/075fda8d8f5441dab9dc9450ac2fc6
.nextflow.log:176785:  error [java.lang.OutOfMemoryError]: Java heap space
.nextflow.log-176786-May-21 06:01:09.352 [Task submitter] ERROR nextflow.processor.TaskProcessor - Execution aborted due to an unexpected error
.nextflow.log:176787:java.lang.OutOfMemoryError: Java heap space
.nextflow.log-176788-   at java.base/java.io.BufferedReader.<init>(BufferedReader.java:106)
.nextflow.log-176789-   at java.base/java.io.BufferedReader.<init>(BufferedReader.java:117)
.nextflow.log-176790-   at org.codehaus.groovy.runtime.IOGroovyMethods.iterator(IOGroovyMethods.java:943)
.nextflow.log-176791-   at org.codehaus.groovy.runtime.StringGroovyMethods$LineIterable.iterator(StringGroovyMethods.java:614)
.nextflow.log-176792-   at org.codehaus.groovy.runtime.DefaultGroovyMethods.toList(DefaultGroovyMethods.java:3388)
.nextflow.log-176793-   at org.codehaus.groovy.runtime.StringGroovyMethods.readLines(StringGroovyMethods.java:2256)
.nextflow.log-176794-   at nextflow.util.MustacheTemplateEngine.replace0(MustacheTemplateEngine.groovy:90)
.nextflow.log-176795-   at nextflow.util.MustacheTemplateEngine.render(MustacheTemplateEngine.groovy:63)
.nextflow.log-176796-   at nextflow.executor.BashWrapperBuilder.buildNew0(BashWrapperBuilder.groovy:411)
.nextflow.log-176797-   at nextflow.executor.BashWrapperBuilder.buildNew0(BashWrapperBuilder.groovy:172)
.nextflow.log-176798-   at nextflow.executor.BashWrapperBuilder.build(BashWrapperBuilder.groovy:425)
.nextflow.log-176799-   at nextflow.executor.GridTaskHandler.prepareLauncher(GridTaskHandler.groovy:107)
.nextflow.log-176800-   at nextflow.processor.TaskPollingMonitor.submit(TaskPollingMonitor.groovy:208)
.nextflow.log-176801-   at nextflow.processor.TaskPollingMonitor.submitPendingTasks(TaskPollingMonitor.groovy:601)
.nextflow.log-176802-   at nextflow.processor.TaskPollingMonitor.submitLoop(TaskPollingMonitor.groovy:415)
.nextflow.log-176803-   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
.nextflow.log-176804-   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
.nextflow.log-176805-   at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
.nextflow.log-176806-   at java.base/java.lang.reflect.Method.invoke(Method.java:569)
.nextflow.log-176807-   at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
.nextflow.log-176808-   at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
.nextflow.log-176809-   at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
.nextflow.log-176810-   at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
.nextflow.log-176811-   at groovy.lang.MetaClassImpl.invokeMethodClosure(MetaClassImpl.java:1017)
.nextflow.log-176812-   at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1207)
.nextflow.log-176813-   at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
.nextflow.log-176814-   at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
.nextflow.log-176815-   at groovy.lang.Closure.call(Closure.java:433)
.nextflow.log-176816-   at groovy.lang.Closure.call(Closure.java:412)
.nextflow.log-176817-   at groovy.lang.Closure.run(Closure.java:505)
--
.nextflow.log-176819-May-21 06:02:44.079 [Task submitter] DEBUG nextflow.Session - Session aborted -- Cause: Java heap space
.nextflow.log-176820-May-21 06:16:39.877 [Actor Thread 626] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
.nextflow.log-176821-  task: name=GCTA_MAKE_GRM_LOCO (5 32 0.8 gamma loco ce.n400.r04_0.05); work-dir=/scratch4/eande106/Ryan_tmp/nf-work-175-stacked-smoke_full/6a/5aa62f1157f51d0bc630a84b7a1df2
.nextflow.log:176822:  error [java.lang.OutOfMemoryError]: Java heap space
.nextflow.log-176823-May-21 06:16:46.379 [Actor Thread 626] ERROR nextflow.processor.TaskProcessor - Execution aborted due to an unexpected error
.nextflow.log:176824:java.lang.OutOfMemoryError: Java heap space
.nextflow.log-176825-   at java.base/java.util.Arrays.copyOf(Arrays.java:3537)
.nextflow.log-176826-   at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:228)
.nextflow.log-176827-   at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:582)
.nextflow.log-176828-   at java.base/java.lang.StringBuilder.append(StringBuilder.java:179)
.nextflow.log-176829-   at java.base/java.lang.StringBuilder.append(StringBuilder.java:173)
.nextflow.log-176830-   at org.codehaus.groovy.runtime.StringGroovyMethods.leftShift(StringGroovyMethods.java:1863)
.nextflow.log-176831-   at nextflow.container.SingularityBuilder.makeEnv(SingularityBuilder.groovy:207)
.nextflow.log-176832-   at java.base/java.lang.invoke.LambdaForm$DMH/0x00001554f50b2000.invokeVirtual(LambdaForm$DMH)
.nextflow.log-176833-   at java.base/java.lang.invoke.LambdaForm$MH/0x00001554f52eac00.invoke(LambdaForm$MH)
.nextflow.log-176834-   at java.base/java.lang.invoke.LambdaForm$MH/0x00001554f515e800.guardWithCatch(LambdaForm$MH)
.nextflow.log-176835-   at java.base/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
.nextflow.log-176836-   at java.base/java.lang.invoke.LambdaForm$MH/0x00001554f515f800.guard(LambdaForm$MH)
.nextflow.log-176837-   at java.base/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
.nextflow.log-176838-   at java.base/java.lang.invoke.LambdaForm$MH/0x00001554f515f800.guard(LambdaForm$MH)
.nextflow.log-176839-   at java.base/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
.nextflow.log-176840-   at java.base/java.lang.invoke.LambdaForm$MH/0x00001554f515f800.guard(LambdaForm$MH)
.nextflow.log-176841-   at java.base/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
.nextflow.log-176842-   at nextflow.container.ContainerBuilder.appendEnv(ContainerBuilder.groovy:224)
.nextflow.log-176843-   at nextflow.container.SingularityBuilder.appendEnv(SingularityBuilder.groovy:164)
.nextflow.log-176844-   at nextflow.container.SingularityBuilder.build(SingularityBuilder.groovy:116)
.nextflow.log-176845-   at java.base/java.lang.invoke.LambdaForm$DMH/0x00001554f50b3400.invokeVirtual(LambdaForm$DMH)
.nextflow.log-176846-   at java.base/java.lang.invoke.LambdaForm$MH/0x00001554f5670c00.invoke(LambdaForm$MH)
.nextflow.log-176847-   at java.base/java.lang.invoke.LambdaForm$MH/0x00001554f5105800.guardWithCatch(LambdaForm$MH)
.nextflow.log-176848-   at java.base/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
.nextflow.log-176849-   at java.base/java.lang.invoke.LambdaForm$MH/0x00001554f5106400.guard(LambdaForm$MH)
.nextflow.log-176850-   at java.base/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
.nextflow.log-176851-   at java.base/java.lang.invoke.LambdaForm$MH/0x00001554f5106400.guard(LambdaForm$MH)

Look for GC-signature classifier in the log

grep -ciE "GC overhead limit exceeded" .nextflow.log

0

grep -ciE "Full GC|Allocation Failure|Pause Full" .nextflow.log

0

JVM max heap on login node

Check what they JVM heap size is set to by default

echo "NXF_OPTS=$NXF_OPTS"; echo "JAVA_TOOL_OPTIONS=$JAVA_TOOL_OPTIONS"

NXF_OPTS=
JAVA_TOOL_OPTIONS=

java -XX:+PrintFlagsFinal -version 2>/dev/null \
| grep -E "MaxHeapSize|UseContainerSupport|MaxRAMPercentage"

 size_t MaxHeapSize                              = 1342177280                                {product} {ergonomic}
   double MaxRAMPercentage                         = 25.000000                                 {product} {default}
   size_t SoftMaxHeapSize                          = 1342177280                             {manageable} {ergonomic}
     bool UseContainerSupport                      = true                                      {product} {default}

The heap size is ~1.25GB which is small for for this level of of parallelism. A 200K-task -resume run (which re-materializes the whole cached task graph + all channel emissions) under a small default heap simply runs out.

Its set to be 25% of the login-node memory limits. Because the login-node cap per-user RAM, its likle that or pipeline orchestrator running on the login node is running out of memory.

Check for envWhitelist bloat

If the CoWAL grew, late-submitted .command.run files have a larger env section than early ones.

P=GCTA_PERFORM_GWA_LOCO_NOPCA            # one containerized process, held constant
  while read -r t s f; do
    grep -aq "$P" "$f" || continue
    env=$(grep -aoE 'SINGULARITYENV_[^ ]+|--env [^ ]+' "$f" | wc -c)
    sent=$(grep -aoc '__NXF_ENVWL_FIX_SENTINEL__' "$f")
    printf '%s\t%s\t%s\n' "$t" "$env" "$sent"
  done < /tmp/cmdrun_by_time.txt | sort -n | sed -n '1p;$p'

Output

1779252817.2137203610   16631   1
1779366547.1425550530   33977   1

For the same containerized process (GCTA_PERFORM_GWA_LOCO_NOPCA), the env section roughly doubled (16.6 KB → 34.0 KB) for an identical process. A healthy run
renders a near-constant env per process — rep-to-rep differences are a handful of bytes (paths,values). A +17 KB swing for the same process is not normal task variation; env bloat is real.

So the most likely source: the CoWAL+sentinel mitigation turned a per-array-fresh whitelist into an ever-growing one; the rendered per-task env (and the retained whitelist) grow over the run, and on the undersized ~1.25 GB login-node heap the 50-rep run accumulates enough to OOM.