Hang when using ox and scopt --help

[jackkoenig]

Consider the following small example of a minimal OxApp with a minimal CLI

//> using scala 3.7.3
//> using dep com.softwaremill.ox::core:1.0.0
//> using dep com.github.scopt::scopt:4.1.0

import ox.*
import scopt.OParser

case class Config(
  input: String = "",
)

object MinimalApp extends OxApp {

  def run(args: Vector[String])(using Ox): ExitCode = {
    val builder = OParser.builder[Config]
    val parser = {
      import builder.*
      OParser.sequence(
        programName("minimal-app"),
        head("minimal-app", "1.0"),
        arg[String]("<input>")
          .required()
          .action((x, c) => c.copy(input = x))
          .text("input file path (required)"),
        help("help").text("prints this usage text")
      )
    }

    OParser.parse(parser, args.toSeq, Config()) match {
      case Some(config) =>
        println(s"Running with config: $config")
        ExitCode.Success

      case None =>
        // Command line arguments parsing failed
        ExitCode.Failure(2)
    }
  }
}

This just parses the arguments and returns Success if it can or Failure if it can't.

If you run this with scala-cli, you'll see that it works fine with correct or incorrect arguments. However, if you pass --help, the process hangs deep in ox. It will ignore SIGINT, you have to SIGKILL it to terminate the process.

Using jstack the trace of where it is stuck is:

2025-10-01 19:57:30
Full thread dump OpenJDK 64-Bit Server VM (25+37-jvmci-b01 mixed mode, sharing):

Threads class SMR info:
_java_thread_list=0x00007f80dc001d30, length=16, elements={
0x00007f815002c080, 0x00007f815020ccb0, 0x00007f815020db40, 0x00007f815020eeb0,
0x00007f8150210190, 0x00007f81502113d0, 0x00007f8150213230, 0x00007f8150214c20,
0x00007f8150261110, 0x00007f81502636d0, 0x00007f815047e570, 0x00007f8150480420,
0x00007f60880035a0, 0x00007f608806f070, 0x00007f607c004730, 0x00007f80dc000eb0
}

"main" #3 [2834181] prio=5 os_prio=0 cpu=187.34ms elapsed=21.38s tid=0x00007f815002c080 nid=2834181 waiting on condition  [0x00007f815a585000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@25/Native Method)
	- parking to wait for  <0x00000001001bdc78> (a java.util.concurrent.CompletableFuture$Signaller)
	at java.util.concurrent.locks.LockSupport.park(java.base@25/LockSupport.java:223)
	at java.util.concurrent.CompletableFuture$Signaller.block(java.base@25/CompletableFuture.java:1885)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@25/ForkJoinPool.java:4364)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@25/ForkJoinPool.java:4310)
	at java.util.concurrent.CompletableFuture.waitingGet(java.base@25/CompletableFuture.java:1919)
	at java.util.concurrent.CompletableFuture.get(java.base@25/CompletableFuture.java:2093)
	at ox.ForkUsingResult.f$proxy1$1(fork.scala:220)
	at ox.ForkUsingResult.join(fork.scala:220)
	at ox.ForkUsingResult.join$(fork.scala:219)
	at ox.fork$package$$anon$7.join(fork.scala:187)
	at ox.UnsupervisedFork.joinEither(fork.scala:270)
	at ox.UnsupervisedFork.joinEither$(fork.scala:262)
	at ox.fork$package$$anon$7.joinEither(fork.scala:187)
	at ox.OxApp.main$$anonfun$2(OxApp.scala:54)
	at ox.OxApp$$Lambda/0x0000000012048210.applyVoid(Unknown Source)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
	at ox.unsupervised$package$.runWithCurrentScopeSet$1(unsupervised.scala:56)
	at ox.unsupervised$package$.scopedWithCapability(unsupervised.scala:69)
	at ox.unsupervised$package$.unsupervised(unsupervised.scala:25)
	at ox.unsupervised$package$.unsupervised(unsupervised.scala:22)
	at ox.OxApp.main(OxApp.scala:37)
	at ox.OxApp.main$(OxApp.scala:31)
	at MinimalApp$.main(bug.scala:12)
	at MinimalApp.main(bug.scala)

"Reference Handler" #17 [2834195] daemon prio=10 os_prio=0 cpu=0.13ms elapsed=21.37s tid=0x00007f815020ccb0 nid=2834195 waiting on condition  [0x00007f80fefce000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.ref.Reference.waitForReferencePendingList(java.base@25/Native Method)
	at java.lang.ref.Reference.processPendingReferences(java.base@25/Reference.java:246)
	at java.lang.ref.Reference$ReferenceHandler.run(java.base@25/Reference.java:208)

"Finalizer" #18 [2834196] daemon prio=8 os_prio=0 cpu=0.10ms elapsed=21.37s tid=0x00007f815020db40 nid=2834196 in Object.wait()  [0x00007f80fedcc000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait0(java.base@25/Native Method)
	- waiting on <0x0000000101002368> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.Object.wait(java.base@25/Object.java:389)
	at java.lang.Object.wait(java.base@25/Object.java:351)
	at java.lang.ref.ReferenceQueue.remove0(java.base@25/ReferenceQueue.java:137)
	at java.lang.ref.ReferenceQueue.remove(java.base@25/ReferenceQueue.java:215)
	- locked <0x0000000101002368> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.Finalizer$FinalizerThread.run(java.base@25/Finalizer.java:165)

"Signal Dispatcher" #19 [2834197] daemon prio=9 os_prio=0 cpu=0.40ms elapsed=21.37s tid=0x00007f815020eeb0 nid=2834197 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Service Thread" #20 [2834198] daemon prio=9 os_prio=0 cpu=0.40ms elapsed=21.37s tid=0x00007f8150210190 nid=2834198 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Monitor Deflation Thread" #21 [2834199] daemon prio=9 os_prio=0 cpu=0.79ms elapsed=21.37s tid=0x00007f81502113d0 nid=2834199 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"JVMCI-native CompilerThread0" #22 [2834200] daemon prio=9 os_prio=0 cpu=72.41ms elapsed=21.37s tid=0x00007f8150213230 nid=2834200 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"C1 CompilerThread0" #33 [2834201] daemon prio=9 os_prio=0 cpu=49.05ms elapsed=21.37s tid=0x00007f8150214c20 nid=2834201 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"Notification Thread" #40 [2834233] daemon prio=9 os_prio=0 cpu=0.06ms elapsed=21.34s tid=0x00007f8150261110 nid=2834233 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Common-Cleaner" #41 [2834234] daemon prio=8 os_prio=0 cpu=0.15ms elapsed=21.34s tid=0x00007f81502636d0 nid=2834234 in Object.wait()  [0x00007f80ca840000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait0(java.base@25/Native Method)
	- waiting on <0x000000010113fc38> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.Object.wait(java.base@25/Object.java:389)
	at java.lang.ref.ReferenceQueue.remove0(java.base@25/ReferenceQueue.java:123)
	at java.lang.ref.ReferenceQueue.remove(java.base@25/ReferenceQueue.java:201)
	- locked <0x000000010113fc38> (a java.lang.ref.ReferenceQueue$Lock)
	at jdk.internal.ref.CleanerImpl.run(java.base@25/CleanerImpl.java:146)
	at java.lang.Thread.runWith(java.base@25/Thread.java:1487)
	at java.lang.Thread.run(java.base@25/Thread.java:1474)
	at jdk.internal.misc.InnocuousThread.run(java.base@25/InnocuousThread.java:148)

"VirtualThread-unblocker" #42 [2834248] daemon prio=5 os_prio=0 cpu=0.10ms elapsed=21.20s tid=0x00007f815047e570 nid=2834248 runnable  [0x00007f80c9f37000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.VirtualThread.takeVirtualThreadListToUnblock(java.base@25/Native Method)
	at java.lang.VirtualThread.unblockVirtualThreads(java.base@25/VirtualThread.java:1507)
	at java.lang.VirtualThread$$Lambda/0x000000001200e718.run(java.base@25/Unknown Source)
	at java.lang.Thread.runWith(java.base@25/Thread.java:1487)
	at java.lang.Thread.run(java.base@25/Thread.java:1474)
	at jdk.internal.misc.InnocuousThread.run(java.base@25/InnocuousThread.java:148)

"ForkJoinPool-1-worker-1" #44 [2834249] daemon prio=5 os_prio=0 cpu=76.58ms elapsed=21.20s tid=0x00007f8150480420 nid=2834249 waiting on condition  [0x00007f80c9e36000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@25/Native Method)
	- parking to wait for  <0x0000000100186350> (a java.util.concurrent.ForkJoinPool)
	at java.util.concurrent.ForkJoinPool.awaitWork(java.base@25/ForkJoinPool.java:2109)
	at java.util.concurrent.ForkJoinPool.deactivate(java.base@25/ForkJoinPool.java:2063)
	at java.util.concurrent.ForkJoinPool.runWorker(java.base@25/ForkJoinPool.java:2027)
	at java.util.concurrent.ForkJoinWorkerThread.run(java.base@25/ForkJoinWorkerThread.java:187)

"ForkJoinPool-1-worker-2" #47 [2834250] daemon prio=5 os_prio=0 cpu=5.44ms elapsed=21.19s tid=0x00007f60880035a0 nid=2834250 waiting on condition  [0x00007f80c9d35000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@25/Native Method)
	- parking to wait for  <0x0000000100186350> (a java.util.concurrent.ForkJoinPool)
	at java.util.concurrent.ForkJoinPool.awaitWork(java.base@25/ForkJoinPool.java:2109)
	at java.util.concurrent.ForkJoinPool.deactivate(java.base@25/ForkJoinPool.java:2063)
	at java.util.concurrent.ForkJoinPool.runWorker(java.base@25/ForkJoinPool.java:2027)
	at java.util.concurrent.ForkJoinWorkerThread.run(java.base@25/ForkJoinWorkerThread.java:187)

"ox-interrupt-hook" #45 [2834252] prio=5 os_prio=0 cpu=0.28ms elapsed=21.12s tid=0x00007f608806f070 nid=2834252 waiting on condition  [0x00007f80c9c34000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@25/Native Method)
	- parking to wait for  <0x000000010099a7f0> (a java.util.concurrent.CompletableFuture$Signaller)
	at java.util.concurrent.locks.LockSupport.park(java.base@25/LockSupport.java:223)
	at java.util.concurrent.CompletableFuture$Signaller.block(java.base@25/CompletableFuture.java:1885)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@25/ForkJoinPool.java:4364)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@25/ForkJoinPool.java:4310)
	at java.util.concurrent.CompletableFuture.waitingGet(java.base@25/CompletableFuture.java:1919)
	at java.util.concurrent.CompletableFuture.get(java.base@25/CompletableFuture.java:2093)
	at ox.fork$package$$anon$7.cancel(fork.scala:190)
	at ox.OxApp.$anonfun$2(OxApp.scala:50)
	at ox.OxApp$$Lambda/0x0000000012096f10.run(Unknown Source)
	at java.lang.Thread.runWith(java.base@25/Thread.java:1487)
	at java.lang.Thread.run(java.base@25/Thread.java:1474)

"ForkJoinPool-1-worker-3" #49 [2834253] daemon prio=5 os_prio=0 cpu=0.52ms elapsed=21.11s tid=0x00007f607c004730 nid=2834253 waiting on condition  [0x00007f80c9b33000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@25/Native Method)
	- parking to wait for  <0x0000000100186350> (a java.util.concurrent.ForkJoinPool)
	at java.util.concurrent.ForkJoinPool.awaitWork(java.base@25/ForkJoinPool.java:2109)
	at java.util.concurrent.ForkJoinPool.deactivate(java.base@25/ForkJoinPool.java:2063)
	at java.util.concurrent.ForkJoinPool.runWorker(java.base@25/ForkJoinPool.java:2027)
	at java.util.concurrent.ForkJoinWorkerThread.run(java.base@25/ForkJoinWorkerThread.java:187)

"Attach Listener" #50 [2834603] daemon prio=9 os_prio=0 cpu=0.89ms elapsed=2.52s tid=0x00007f80dc000eb0 nid=2834603 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"VM Thread" os_prio=0 cpu=1.48ms elapsed=21.38s tid=0x00007f8150202ee0 nid=2834194 runnable  

"VM Periodic Task Thread" os_prio=0 cpu=0.22ms elapsed=21.38s tid=0x00007f81501eed90 nid=2834193 waiting on condition  

"G1 Service" os_prio=0 cpu=0.41ms elapsed=21.38s tid=0x00007f81501dfae0 nid=2834186 runnable  

"G1 Refine#0" os_prio=0 cpu=0.05ms elapsed=21.38s tid=0x00007f81501de8f0 nid=2834185 runnable  

"G1 Conc#0" os_prio=0 cpu=0.03ms elapsed=21.39s tid=0x00007f815007c340 nid=2834184 runnable  

"G1 Main Marker" os_prio=0 cpu=0.04ms elapsed=21.39s tid=0x00007f815007b100 nid=2834183 runnable  

"GC Thread#0" os_prio=0 cpu=0.04ms elapsed=21.39s tid=0x00007f8150062f60 nid=2834182 runnable  

JNI global refs: 23, weak refs: 0

That's as far as I've gotten debugging, very puzzling.

[adamw]

Here's the full stack trace, with virtual threads, obtained using jcmd [pid] Thread.dump_to_file [file]. It seems the problem is a System.exit call:

91953
2025-10-02T10:20:58.751941Z
25+36-LTS

#3 "main" WAITING 2025-10-02T10:20:58.756490Z
    at java.base/jdk.internal.misc.Unsafe.park(Native Method)
    - parking to wait for <java.util.concurrent.CompletableFuture$Signaller@2f37aeaa>
    at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:223)
    at java.base/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1885)
    at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:4364)
    at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:4310)
    at java.base/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1919)
    at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2093)
    at ox.ForkUsingResult.f$proxy1$1(fork.scala:220)
    at ox.ForkUsingResult.join(fork.scala:220)
    at ox.ForkUsingResult.join$(fork.scala:219)
    at ox.fork$package$$anon$7.join(fork.scala:187)
    at ox.UnsupervisedFork.joinEither(fork.scala:270)
    at ox.UnsupervisedFork.joinEither$(fork.scala:262)
    at ox.fork$package$$anon$7.joinEither(fork.scala:187)
    at ox.OxApp.main$$anonfun$2(OxApp.scala:54)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
    at ox.unsupervised$package$.runWithCurrentScopeSet$1(unsupervised.scala:56)
    at ox.unsupervised$package$.scopedWithCapability(unsupervised.scala:69)
    at ox.unsupervised$package$.unsupervised(unsupervised.scala:25)
    at ox.unsupervised$package$.unsupervised(unsupervised.scala:22)
    at ox.OxApp.main(OxApp.scala:37)
    at ox.OxApp.main$(OxApp.scala:31)
    at MinimalApp$.main(x.scala:12)
    at MinimalApp.main(x.scala)

#14 "Reference Handler" RUNNABLE 2025-10-02T10:20:58.757265Z
    at java.base/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
    at java.base/java.lang.ref.Reference.processPendingReferences(Reference.java:246)
    at java.base/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:208)

#15 "Finalizer" WAITING 2025-10-02T10:20:58.757628Z
    at java.base/java.lang.Object.wait0(Native Method)
    - waiting on <java.lang.ref.ReferenceQueue$Lock@6ab3e608>
    at java.base/java.lang.Object.wait(Object.java:389)
    at java.base/java.lang.Object.wait(Object.java:351)
    at java.base/java.lang.ref.ReferenceQueue.remove0(ReferenceQueue.java:137)
    at java.base/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:215)
    - locked <java.lang.ref.ReferenceQueue$Lock@6ab3e608>
    at java.base/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:165)

#16 "Signal Dispatcher" RUNNABLE 2025-10-02T10:20:58.757802Z

#23 "Notification Thread" RUNNABLE 2025-10-02T10:20:58.757825Z

#24 "Common-Cleaner" TIMED_WAITING 2025-10-02T10:20:58.757863Z
    at java.base/java.lang.Object.wait0(Native Method)
    - waiting on <java.lang.ref.ReferenceQueue$Lock@2f2d15db>
    at java.base/java.lang.Object.wait(Object.java:389)
    at java.base/java.lang.ref.ReferenceQueue.remove0(ReferenceQueue.java:123)
    at java.base/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:201)
    - locked <java.lang.ref.ReferenceQueue$Lock@2f2d15db>
    at java.base/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:146)
    at java.base/java.lang.Thread.run(Thread.java:1474)
    at java.base/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:148)

#25 "VirtualThread-unblocker" RUNNABLE 2025-10-02T10:20:58.757986Z
    at java.base/java.lang.VirtualThread.takeVirtualThreadListToUnblock(Native Method)
    at java.base/java.lang.VirtualThread.unblockVirtualThreads(VirtualThread.java:1507)
    at java.base/java.lang.Thread.run(Thread.java:1474)
    at java.base/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:148)

#28 "ox-interrupt-hook" WAITING 2025-10-02T10:20:58.758066Z
    at java.base/jdk.internal.misc.Unsafe.park(Native Method)
    - parking to wait for <java.util.concurrent.CompletableFuture$Signaller@3e711a32>
    at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:223)
    at java.base/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1885)
    at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:4364)
    at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:4310)
    at java.base/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1919)
    at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2093)
    at ox.fork$package$$anon$7.cancel(fork.scala:190)
    at ox.OxApp.$anonfun$2(OxApp.scala:50)
    at java.base/java.lang.Thread.run(Thread.java:1474)

#33 "Attach Listener" RUNNABLE 2025-10-02T10:20:58.758210Z
    at java.base/jdk.internal.vm.ThreadSnapshot.create(Native Method)
    at java.base/jdk.internal.vm.ThreadSnapshot.of(ThreadSnapshot.java:60)
    at java.base/jdk.internal.vm.ThreadDumper.dumpThread(ThreadDumper.java:181)
    at java.base/jdk.internal.vm.ThreadDumper.lambda$dumpThreads$0(ThreadDumper.java:176)
    at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:197)
    at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:1024)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560)
    at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:315)
    at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734)
    at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:803)
    at java.base/jdk.internal.vm.ThreadDumper.dumpThreads(ThreadDumper.java:176)
    at java.base/jdk.internal.vm.ThreadDumper.dumpThreads(ThreadDumper.java:172)
    at java.base/jdk.internal.vm.ThreadDumper.dumpThreadsToFile(ThreadDumper.java:133)
    at java.base/jdk.internal.vm.ThreadDumper.dumpThreads(ThreadDumper.java:73)

#26 "" virtual WAITING 2025-10-02T10:20:58.758432Z
    at java.base/java.lang.VirtualThread.park(VirtualThread.java:738)
    - parking to wait for <java.util.concurrent.CountDownLatch$Sync@5c5a91f9>
    at java.base/java.lang.System$1.parkVirtualThread(System.java:2284)
    at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:221)
    at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:790)
    at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1139)
    at java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:230)
    at java.base/java.lang.VirtualThread.joinNanos(VirtualThread.java:1002)
    at java.base/java.lang.Thread.join(Thread.java:1870)
    at java.base/java.lang.Thread.join(Thread.java:1963)
    at ox.internal.ThreadHerd.interruptAllAndJoinUntilCompleted$$anonfun$2(ThreadHerd.scala:62)
    at java.base/java.util.concurrent.ConcurrentHashMap$KeySetView.forEach(ConcurrentHashMap.java:4726)
    at ox.internal.ThreadHerd.interruptAllAndJoinUntilCompleted(ThreadHerd.scala:60)
    at ox.unsupervised$package$.runWithCurrentScopeSet$1(unsupervised.scala:57)
    at ox.unsupervised$package$.scopedWithCapability(unsupervised.scala:69)
    at ox.fork$package$.forkCancellable$$anonfun$1(fork.scala:169)
    at ox.fork$package$.forkCancellable$$anonfun$adapted$1(fork.scala:185)
    at scala.Function0.apply$mcV$sp(Function0.scala:42)
    at ox.internal.ThreadHerd.$anonfun$1(ThreadHerd.scala:29)
    at java.base/java.lang.VirtualThread.run(VirtualThread.java:456)

#29 "" virtual WAITING 2025-10-02T10:20:58.758659Z
    at java.base/java.lang.VirtualThread.park(VirtualThread.java:738)
    - parking to wait for <java.util.concurrent.CountDownLatch$Sync@29acafdc>
    at java.base/java.lang.System$1.parkVirtualThread(System.java:2284)
    at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:221)
    at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:790)
    at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1139)
    at java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:230)
    at java.base/java.lang.VirtualThread.joinNanos(VirtualThread.java:1002)
    at java.base/java.lang.Thread.join(Thread.java:1870)
    at java.base/java.lang.Thread.join(Thread.java:1963)
    at ox.internal.ThreadHerd.interruptAllAndJoinUntilCompleted$$anonfun$2(ThreadHerd.scala:62)
    at java.base/java.util.concurrent.ConcurrentHashMap$KeySetView.forEach(ConcurrentHashMap.java:4726)
    at ox.internal.ThreadHerd.interruptAllAndJoinUntilCompleted(ThreadHerd.scala:60)
    at ox.unsupervised$package$.runWithCurrentScopeSet$1(unsupervised.scala:57)
    at ox.unsupervised$package$.scopedWithCapability(unsupervised.scala:69)
    at ox.supervised$package$.supervisedError(supervised.scala:52)
    at ox.supervised$package$.supervisedError(supervised.scala:46)
    at ox.supervised$package$.supervised(supervised.scala:34)
    at ox.OxApp.$anonfun$1(OxApp.scala:39)
    at ox.fork$package$.forkCancellable$$anonfun$1$$anonfun$1$$anonfun$1(fork.scala:174)
    at ox.fork$package$.$anonfun$adapted$1(fork.scala:178)
    at scala.Function0.apply$mcV$sp(Function0.scala:42)
    at ox.internal.ThreadHerd.$anonfun$1(ThreadHerd.scala:29)
    at java.base/java.lang.VirtualThread.run(VirtualThread.java:456)

#31 "" virtual WAITING 2025-10-02T10:20:58.759118Z
    at java.base/java.lang.Object.wait0(Native Method)
    - waiting on <java.lang.Thread@97e1986>
    at java.base/java.lang.Object.wait(Object.java:382)
    at java.base/java.lang.Thread.join(Thread.java:1887)
    - locked <java.lang.Thread@97e1986>
    at java.base/java.lang.Thread.join(Thread.java:1963)
    at java.base/java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:114)
    at java.base/java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:47)
    at java.base/java.lang.Shutdown.runHooks(Shutdown.java:130)
    at java.base/java.lang.Shutdown.exit(Shutdown.java:167)
    - locked <java.lang.Class@687b574f>
    at java.base/java.lang.Runtime.exit(Runtime.java:177)
    at java.base/java.lang.System.exit(System.java:1540)
    at scala.sys.package$.exit(package.scala:40)
    at scopt.platform$.exit(PlatformReadInstances.scala:60)
    at scopt.DefaultOEffectSetup.terminate(OEffect.scala:36)
    at scopt.ORunner$.runEffects$$anonfun$1(ORunner.scala:440)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
    at scala.collection.immutable.List.foreach(List.scala:334)
    at scopt.ORunner$.runEffects(ORunner.scala:441)
    at scopt.OParser$.parse(OParser.scala:291)
    at MinimalApp$.run(x.scala:29)
    at ox.OxApp.$anonfun$1$$anonfun$1(OxApp.scala:39)
    at ox.supervised$package$.$anonfun$2(supervised.scala:53)
    at ox.fork$package$.forkUserError$$anonfun$1(fork.scala:96)
    at ox.fork$package$.forkUserError$$anonfun$adapted$1(fork.scala:107)
    at scala.Function0.apply$mcV$sp(Function0.scala:42)
    at ox.internal.ThreadHerd.$anonfun$1(ThreadHerd.scala:29)
    at java.base/java.lang.VirtualThread.run(VirtualThread.java:456)

#27 "ForkJoinPool-1-worker-1" WAITING 2025-10-02T10:20:58.760238Z
    at java.base/jdk.internal.misc.Unsafe.park(Native Method)
    - parking to wait for <java.util.concurrent.ForkJoinPool@51df094f>
    at java.base/java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:2109)
    at java.base/java.util.concurrent.ForkJoinPool.deactivate(ForkJoinPool.java:2063)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2027)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:187)

#30 "ForkJoinPool-1-worker-2" TIMED_WAITING 2025-10-02T10:20:58.760322Z
    at java.base/jdk.internal.misc.Unsafe.park(Native Method)
    - parking to wait for <java.util.concurrent.ForkJoinPool@51df094f>
    at java.base/java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:2109)
    at java.base/java.util.concurrent.ForkJoinPool.deactivate(ForkJoinPool.java:2063)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2027)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:187)

#32 "ForkJoinPool-1-worker-3" WAITING 2025-10-02T10:20:58.760367Z
    at java.base/jdk.internal.misc.Unsafe.park(Native Method)
    - parking to wait for <java.util.concurrent.ForkJoinPool@51df094f>
    at java.base/java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:2109)
    at java.base/java.util.concurrent.ForkJoinPool.deactivate(ForkJoinPool.java:2063)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2027)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:187)

[adamw]

A simple reproduction is:

object App extends OxApp:
  def run(args: Vector[String])(using Ox): ExitCode =
    System.exit(1)
    ExitCode.Success

[adamw]

The problem is that System.exit hangs indefinitely, and we initiate an orderly shutdown (terminating all scopes) in OxApp.interruptThread (line 50). So this in turns waits for the thread which called exit...

I don't think we can detect that a System.exit was called, so the only thing that comes to mind is to add a timeout to the cleanup and just give up after ... 1s? Or not using System.exit ;)

@lbialy maybe you have better ideas :)

[jackkoenig]

Thank you for the quick analysis!

A simple reproduction is:

Doh, I tried to reduce further but missed the System.exit 🙈

Or not using System.exit ;)

No arguments here, I don't know how it would be possible but if something about the hang could point to System.exit being considered harmful that might help other people who hit this foot-gun (especially via libraries with hidden exit calls 🫠)

[jackkoenig]

I don't know how it would be possible but if something about the hang could point to System.exit being considered harmful

Maybe a function on the stacktrace called IfYouAreSeeingThisAndThingsAreHangingThereMayHaveBeenASystemExit 💁‍♀️

[adamw]

No arguments here, I don't know how it would be possible but if something about the hang could point to System.exit being considered harmful that might help other people who hit this foot-gun (especially via libraries with hidden exit calls 🫠)

I suppose one thing we can do is include a way to create thread dumps which include virtual threads. jstack does not include them as you could see, but jmd to-file does. Similarly, if you debug in IntelliJ you won't see them, but they are in the thread dump.

[jackkoenig]

A docs page on debugging techniques, especially around hangs would be really helpful and could cover this particular topic.

Anyway, I think this particular issue is resolved. Thank you and thanks for the great library!

[lbialy]

Hold my beer...

what if we used Thread.getAllStackTraces and scour them in search of System.exit or Runtime.exit in our shutdown hook? If we find such an entry, we could make a decision that we're not interrupting the main fiber because someone is already making that informed decision of calling System.exit after all resources have been cleaned up. It's all rather shitty but with our current semantics System.exit causes a deadlock which is probably worse. We could issue a warning to stderr that it's a bad idea to call System.exit in user code perhaps?

[lbialy]

Huh, there's no way to fetch stack traces of all virtual threads (Thread.getAllStackTraces and the related MX Bean do not cover VTs) so this idea won't work. @adamw We could, in theory, patch all Forks with an ability to provide information whether they are currently waiting for System.exit / Runtime.exit to finish, but
a) that breaks the current abstraction in which all actual Thread instances are held by the herds and Forks are not referencing them directly, I don't know how big of an impact that would be, I guess significant
b) we'd have to call that recursively from the shutdown hook thread to make the decision whether to interrupt the main thread; we can't just check for that before any call to interrupt on any thread as that's way to expensive so it'd be a special route for this particular case and we'd still be unable to clean up all other threads without breaking our guarantees.

It's all quite horrible.

[adamw]

You can visit all threads, but only through internal APIs. That was my first idea as well ;)

I'm not sure I understand the idea with patching Forks, what exactly would you add to forks and how that would enable us to detect problems?

Other possibility is to store child herds and traverse all herds recursively, although this could complicate the bookkeeping a bit - that would have to be a concurrent collection I guess

[adamw]

So far I added some docs. Let's keep this open, as the problem isn't solved, just documented :)

[lbialy]

The idea was that if you have a reference to the Thread instance that's executing the piece of code that computes the result of the Fork, you are able to call thread.getStackTrace and check if it has System.exit and/or Runtime.exit in there. Currently from what I can see you maintain a clear separation of threads and forks and forks don't have any reference to the threads that are running user's functions. The only thing you use is a reference to a CompletableFuture and, in case of CancellableFork, a reference to semaphore that, if released, triggers interruption of the Thread and subsequent cleanup.

[adamw]

Ah right, we'd need both full parent-child relations in the herds, and keeping threads in forks. So that we can navigate from the root, to all threads.