From 4be4ba99c0ab143f768e30a7a9b90826926687b5 Mon Sep 17 00:00:00 2001 From: Adam Warski Date: Sun, 28 Jun 2026 08:31:08 +0000 Subject: [PATCH 1/8] Add chapter: subprocesses and external streams (structured teardown) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds guidance for the case structured concurrency is easiest to get subtly wrong: driving a long-lived subprocess (or socket/SSE) whose output is read line-by-line. The reader is a fork whose return value is the result; the killer is teardown — a blocking native read ignores interruption, and Ox joins forks before running `releaseAfterScope` finalizers, so the resource must be destroyed in the scope body's `finally` (before the join), and a launcher subprocess needs its whole tree killed or an orphan keeps the pipe open. - New chapter 170-subprocesses-and-external-streams.md. - SKILL.md: two always-loaded design rules so the agent decides the owning scope and the teardown point UP FRONT (model readers as forks with return-value results; never return a live fork-owning handle; destroy blocking resources in the body finally, not releaseAfterScope), plus the index entry. Grounded in Ox's documented scope-teardown semantics (interruptAllAndJoin runs in the body finally; finalizers after). Co-Authored-By: Claude Opus 4.8 (1M context) --- .../170-subprocesses-and-external-streams.md | 125 ++++++++++++++++++ .../skills/direct-style-scala/SKILL.md | 20 +++ 2 files changed, 145 insertions(+) create mode 100644 direct-style-scala/skills/direct-style-scala/170-subprocesses-and-external-streams.md diff --git a/direct-style-scala/skills/direct-style-scala/170-subprocesses-and-external-streams.md b/direct-style-scala/skills/direct-style-scala/170-subprocesses-and-external-streams.md new file mode 100644 index 0000000..0ff9daf --- /dev/null +++ b/direct-style-scala/skills/direct-style-scala/170-subprocesses-and-external-streams.md @@ -0,0 +1,125 @@ +# Subprocesses and External Streams + +Driving a long-lived subprocess (or any blocking external connection — a raw +socket, an SSE stream) whose output you consume line by line, under structured +concurrency. The reader runs as a fork; its result is the fork's return value; +and teardown must account for a blocking native read not being interruptible. +This is where structured concurrency is easiest to get subtly wrong — getting +the ownership and teardown right *before* writing the code avoids a deadlock that +only shows up at shutdown. + +## Dependencies + +- `"com.softwaremill.ox" %% "core"` — `supervised`, `fork`, `forkDiscard`, + `releaseAfterScope` + +--- + +## Own the work with a scope; return a result, not a live handle + +Decide the owning scope first. A reader that consumes a process's output is +concurrent work, so it belongs to a `supervised` scope whose lifetime matches the +work: open the scope, spawn the process, fork the reader, drive it, and return +the computed value. The reader fork's **return value is the outcome** — read it +with `join()`. Don't publish the result through a shared `AtomicReference` that +another thread writes and the caller polls; there is one producer and one value. + +```scala +def runAndCollect(command: Seq[String]): Summary = + supervised: + val process = ProcessBuilder(command*).start() + try + val reader = fork: + scala.io.Source + .fromInputStream(process.getInputStream) + .getLines() + .foldLeft(Summary.empty)(_.add(_)) // the fork returns the Summary + reader.join() // its return value IS the result + finally process.destroyForcibly().discard // see the next section +``` + +The opposite shape — a constructor that spawns reader threads and returns an +object the caller drives later — is the pattern to avoid. Its lifetime no longer +matches any scope, so cancellation, error propagation, and cleanup all become +manual flags and `try`/`finally` ladders, exactly the bookkeeping structured +concurrency removes. + +> **Warning:** Never return an object that owns running forks or threads, to be +> driven by the caller afterwards. Keep the work inside the scope that owns it +> and return a plain value. If the caller must interleave with the work (send +> input, react to events), pass a *consumer* into the scope (`run(...)(use: +> Handle => T)`) rather than handing a live handle out of it. + +## Teardown: a blocking read is not interruptible — destroy before the join + +A fork blocked in a native read — `InputStream.read`, a blocking socket +`recv` — does **not** observe `Thread.interrupt`. Ox ends a scope by interrupting +its forks and then joining them, so interruption alone will not stop such a fork, +and the join blocks forever. The only thing that unblocks the read is closing the +underlying resource (destroy the process, close the socket), which makes the read +return EOF. + +That close must happen **before** the scope joins the fork. Ox joins forks in the +scope body's own `finally`, and runs `releaseAfterScope` finalizers only *after* +that join returns: + +```scala +// ox core, scopedWithCapability (simplified): +try f() // your scope body +finally herd.interruptAllAndJoinUntilCompleted() // forks are joined HERE +runFinalizers() // releaseAfterScope runs AFTER +``` + +So a kill registered with `releaseAfterScope` runs too late: the join has already +deadlocked on the blocked read. Put the destroy in your own body `finally` (as +above) — or bracket the resource so its teardown is sequenced before the scope +joins. On the normal path the process has already exited and the destroy is a +no-op; on cancellation or error it is what lets the scope complete. + +> **Required:** Close or destroy a blocking external resource in the scope +> BODY's `finally`, before the scope joins the reader fork. Do NOT rely on +> `releaseAfterScope` for it — that finalizer runs *after* the join and will +> deadlock on a fork stuck in a non-interruptible read. + +## Kill the whole process tree, not just the direct child + +If the process you spawned is a launcher or wrapper that forks the real worker +(`some-launcher run the-tool …`), destroying only your direct child orphans the +worker — and the orphan keeps the inherited stdout/stderr pipe write-ends open. +A POSIX pipe reports EOF only once *every* write-end is closed, so the reader +fork never unblocks and the join still hangs. Destroy descendants first, then the +root: + +```scala +val handle = process.toHandle +handle.descendants().forEach(_.destroyForcibly().discard) +handle.destroyForcibly().discard +``` + +> **Warning:** A forked grandchild inherits the pipe file descriptors. If it +> survives the kill, the reader never sees EOF — so terminate the descendants, +> not just the process you directly spawned. + +## Don't let an unread pipe stall the process + +A subprocess blocks once an OS pipe buffer (~64 KB) fills. If you read stdout but +ignore stderr, a chatty process wedges mid-run. Either let the child inherit the +parent's stderr, or drain stderr in its own fork for the resource's lifetime — +a `forkDiscard`, torn down by the same body-`finally` destroy: + +```scala +supervised: + val process = ProcessBuilder(command*).start() + try + forkDiscard: + try scala.io.Source.fromInputStream(process.getErrorStream).getLines().foreach(log.debug) + catch case NonFatal(e) => log.debug("stderr drain ended", e) + val reader = fork(consume(process.getInputStream)) + reader.join() + finally process.destroyForcibly().discard +``` + +The drain fork swallows `NonFatal` so a stray read error can't tear down the +scope, and logs rather than discards silently so a real failure stays +diagnosable. It needs no separate stop signal: the body-`finally` destroy EOFs +its read, and the scope then joins it. diff --git a/direct-style-scala/skills/direct-style-scala/SKILL.md b/direct-style-scala/skills/direct-style-scala/SKILL.md index ad8be96..ef2f008 100644 --- a/direct-style-scala/skills/direct-style-scala/SKILL.md +++ b/direct-style-scala/skills/direct-style-scala/SKILL.md @@ -101,6 +101,20 @@ def findUser(id: Id[User])(using DbTx): Either[Fail, User] = resource must be tied to that parent scope's lifetime. * keep constructors plain; use factories that take `(using Ox)` and return values that do not carry the capability. +* decide the owning scope BEFORE writing concurrent code. Model a stream reader + or worker as a fork in a `supervised` scope whose lifetime matches the work; + its result is the fork's **return value** (`fork{…}.join()`), not a value + published through a shared `AtomicReference`. NEVER return an object that owns + running forks/threads to be driven later — its lifetime escapes every scope, + making cancellation and cleanup manual again. Pass a consumer into the scope + instead of handing a live handle out. +* a fork blocked on a non-interruptible read (subprocess pipe, socket, blocking + recv) is NOT ended by scope cancellation — only closing the underlying resource + EOFs it. Close/destroy it in the scope body's `finally`, BEFORE the scope joins + the fork; NEVER rely on `releaseAfterScope`, which Ox runs after the join and so + deadlocks on the blocked read. For a launcher subprocess, destroy the whole + process tree. See [Subprocesses and External + Streams](170-subprocesses-and-external-streams.md). # Functional programming @@ -266,6 +280,12 @@ https://raw.githubusercontent.com/virtuslab/scala-skill/refs/heads/master/direct `mapStateful`), Ox primitive selection, channels for worker mailboxes and shutdown, actors for serialized mutable state. +- [Subprocesses and External Streams](170-subprocesses-and-external-streams.md) + — driving a subprocess / socket / SSE reader as a fork whose return value is + the result; why a blocking non-interruptible read needs the resource destroyed + in the scope body's `finally` (before the join) rather than via + `releaseAfterScope`; process-tree teardown; pipe back-pressure. + ## Error Handling - [Error Handling](200-error-handling.md) — `Fail` ADT, Ox `either` blocks with From 6ae4d8908b9c14eac802dbc2c455810fb4802eda Mon Sep 17 00:00:00 2001 From: Adam Warski Date: Thu, 23 Jul 2026 09:10:29 +0000 Subject: [PATCH 2/8] Update for Ox 1.0.6: abandonOnInterruptReads, interruptibility corrections MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ox 1.0.6 (#479) adds abandon-on-interrupt utilities and officially documents which JVM operations are (un)interruptible. Update the chapter accordingly: - correct the socket claim: blocking socket reads on virtual threads (which Ox forks run on) ARE interruptible since Java 21, destructively; the teardown deadlock is specific to classic java.io stream reads (subprocess pipes, files, stdin); - ground the releaseAfterScope-runs-after-the-join claim in the now documented behavior instead of a paraphrased-internals snippet; - add a section on abandonOnInterruptReads / abandonOnInterrupt for reads that can't be unblocked by closing (stdin, FileInputStream) and per-read cancellation — while keeping body-finally destroy as the primary subprocess teardown, since the wrapper doesn't terminate the child. Co-Authored-By: Claude Fable 5 --- .../170-subprocesses-and-external-streams.md | 96 +++++++++++++------ .../skills/direct-style-scala/SKILL.md | 20 ++-- 2 files changed, 81 insertions(+), 35 deletions(-) diff --git a/direct-style-scala/skills/direct-style-scala/170-subprocesses-and-external-streams.md b/direct-style-scala/skills/direct-style-scala/170-subprocesses-and-external-streams.md index 0ff9daf..31f5321 100644 --- a/direct-style-scala/skills/direct-style-scala/170-subprocesses-and-external-streams.md +++ b/direct-style-scala/skills/direct-style-scala/170-subprocesses-and-external-streams.md @@ -3,7 +3,7 @@ Driving a long-lived subprocess (or any blocking external connection — a raw socket, an SSE stream) whose output you consume line by line, under structured concurrency. The reader runs as a fork; its result is the fork's return value; -and teardown must account for a blocking native read not being interruptible. +and teardown must account for a subprocess pipe read not being interruptible. This is where structured concurrency is easiest to get subtly wrong — getting the ownership and teardown right *before* writing the code avoids a deadlock that only shows up at shutdown. @@ -11,7 +11,7 @@ only shows up at shutdown. ## Dependencies - `"com.softwaremill.ox" %% "core"` — `supervised`, `fork`, `forkDiscard`, - `releaseAfterScope` + `releaseAfterScope`, `abandonOnInterruptReads` --- @@ -50,31 +50,32 @@ concurrency removes. > input, react to events), pass a *consumer* into the scope (`run(...)(use: > Handle => T)`) rather than handing a live handle out of it. -## Teardown: a blocking read is not interruptible — destroy before the join - -A fork blocked in a native read — `InputStream.read`, a blocking socket -`recv` — does **not** observe `Thread.interrupt`. Ox ends a scope by interrupting -its forks and then joining them, so interruption alone will not stop such a fork, -and the join blocks forever. The only thing that unblocks the read is closing the -underlying resource (destroy the process, close the socket), which makes the read -return EOF. - -That close must happen **before** the scope joins the fork. Ox joins forks in the -scope body's own `finally`, and runs `releaseAfterScope` finalizers only *after* -that join returns: - -```scala -// ox core, scopedWithCapability (simplified): -try f() // your scope body -finally herd.interruptAllAndJoinUntilCompleted() // forks are joined HERE -runFinalizers() // releaseAfterScope runs AFTER -``` - -So a kill registered with `releaseAfterScope` runs too late: the join has already -deadlocked on the blocked read. Put the destroy in your own body `finally` (as -above) — or bracket the resource so its teardown is sequenced before the scope -joins. On the normal path the process has already exited and the destroy is a -no-op; on cancellation or error it is what lets the scope complete. +## Teardown: a pipe read is not interruptible — destroy before the join + +A fork blocked reading a classic `java.io` stream — a subprocess's stdout pipe, +a `FileInputStream`, stdin — does **not** observe `Thread.interrupt`. This is +intended, permanent JVM behavior (the request to change it was closed as "Won't +Fix"; see [which operations are +interruptible](https://ox.softwaremill.com/latest/structured-concurrency/interruptions.html) +in the Ox docs). Ox ends a scope by interrupting its forks and then joining +them, so interruption alone will not stop such a fork, and the join blocks +forever. The only thing that unblocks the read is closing the underlying +resource — destroying the process — which makes the read return EOF. + +Blocking *socket* reads are different: on virtual threads — which Ox forks run +on — they are interruptible since Java 21, though destructively (the interrupt +closes the socket). A socket-backed reader is therefore ended by scope +cancellation on its own; the deadlock below is specific to pipe and other +classic stream reads. + +That close must happen **before** the scope joins the fork. Ox releases +scope-managed resources (`useInScope`, `releaseAfterScope`) only *after* all +forks complete — so, as the Ox docs note, they won't unblock a fork stuck in +uninterruptible I/O: the join has already deadlocked by the time the finalizer +would run. Put the destroy in your own body `finally` (as above) — it is +sequenced before the scope joins the forks. On the normal path the process has +already exited and the destroy is a no-op; on cancellation or error it is what +lets the scope complete. > **Required:** Close or destroy a blocking external resource in the scope > BODY's `finally`, before the scope joins the reader fork. Do NOT rely on @@ -100,6 +101,47 @@ handle.destroyForcibly().discard > survives the kill, the reader never sees EOF — so terminate the descendants, > not just the process you directly spawned. +## Reads you can't unblock by closing: `abandonOnInterruptReads` + +Destroy-and-EOF works because a subprocess supports asynchronous close. Some +resources don't: stdin can't be meaningfully closed, and closing a +`FileInputStream` does not unblock a pending read. And sometimes a single read +should be cancellable (a `timeout` around one read) without tearing the +resource down. For these, Ox (since 1.0.6) provides `abandonOnInterruptReads`: +it wraps an `InputStream` so the actual reads run on a *detached* virtual +thread — unmanaged, never joined — while the calling fork awaits each chunk +interruptibly. On interruption the wait is abandoned: the fork proceeds with an +`InterruptedException`, the in-flight chunk is not lost (the next read returns +it), and with `closeOnAbandon = true` the underlying stream is additionally +closed. + +```scala +import ox.* +import scala.concurrent.duration.* + +// stdin can be neither interrupted nor usefully closed — wrap it once, +// process-wide (multiple wrappers would compete for input): +lazy val stdin = abandonOnInterruptReads(System.in) + +supervised: + val firstByte: Option[Int] = timeoutOption(1.second)(stdin.read()) +``` + +The trade-off: an abandoned read leaves the detached thread blocked until the +underlying read completes — for stdin, possibly for the application's lifetime. +That is a cheap virtual thread in exchange for interruptibility. For a +subprocess, the body-`finally` destroy remains the primary teardown: it +terminates the child *and* EOFs the pipe, leaving nothing behind. Wrapping the +process's output stream as well makes the reader fork immune to a mis-sequenced +teardown (scope cancellation can then always end it) — but it never replaces +destroying the process. + +For one-off uninterruptible calls (a JDBC `execute`, a DNS lookup) there is +`abandonOnInterrupt(op)(onAbandon)`, where the cleanup starts on abandonment — +e.g. `abandonOnInterrupt(statement.execute())(connection.close())`. For +resources supporting asynchronous close, the cleanup also unblocks the +abandoned operation, so nothing is leaked. + ## Don't let an unread pipe stall the process A subprocess blocks once an OS pipe buffer (~64 KB) fills. If you read stdout but diff --git a/direct-style-scala/skills/direct-style-scala/SKILL.md b/direct-style-scala/skills/direct-style-scala/SKILL.md index ef2f008..0846705 100644 --- a/direct-style-scala/skills/direct-style-scala/SKILL.md +++ b/direct-style-scala/skills/direct-style-scala/SKILL.md @@ -108,12 +108,15 @@ def findUser(id: Id[User])(using DbTx): Either[Fail, User] = running forks/threads to be driven later — its lifetime escapes every scope, making cancellation and cleanup manual again. Pass a consumer into the scope instead of handing a live handle out. -* a fork blocked on a non-interruptible read (subprocess pipe, socket, blocking - recv) is NOT ended by scope cancellation — only closing the underlying resource - EOFs it. Close/destroy it in the scope body's `finally`, BEFORE the scope joins - the fork; NEVER rely on `releaseAfterScope`, which Ox runs after the join and so - deadlocks on the blocked read. For a launcher subprocess, destroy the whole - process tree. See [Subprocesses and External +* a fork blocked in a classic `java.io` stream read (subprocess pipe, + `FileInputStream`, stdin) is NOT ended by scope cancellation — such reads + ignore interruption. Either close/destroy the resource in the scope body's + `finally`, BEFORE the scope joins the fork — NEVER via `releaseAfterScope`, + which Ox runs after the join and so deadlocks on the blocked read — or wrap + the stream with `abandonOnInterruptReads` so reads become interruptible. + (Socket reads on Ox's virtual-thread forks ARE interruptible: the interrupt + closes the socket.) For a launcher subprocess, destroy the whole process + tree. See [Subprocesses and External Streams](170-subprocesses-and-external-streams.md). # Functional programming @@ -282,9 +285,10 @@ https://raw.githubusercontent.com/virtuslab/scala-skill/refs/heads/master/direct - [Subprocesses and External Streams](170-subprocesses-and-external-streams.md) — driving a subprocess / socket / SSE reader as a fork whose return value is - the result; why a blocking non-interruptible read needs the resource destroyed + the result; why a non-interruptible pipe read needs the resource destroyed in the scope body's `finally` (before the join) rather than via - `releaseAfterScope`; process-tree teardown; pipe back-pressure. + `releaseAfterScope`; process-tree teardown; `abandonOnInterruptReads` for + reads that can't be unblocked by closing; pipe back-pressure. ## Error Handling From a666dadd030ecf1f3d9f7603958370c2f7a13ff3 Mon Sep 17 00:00:00 2001 From: Adam Warski Date: Thu, 23 Jul 2026 09:36:23 +0000 Subject: [PATCH 3/8] Address review: closing only unblocks async-closeable streams MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The teardown text (chapter + always-loaded SKILL.md bullet) presented close/destroy as applicable to all uninterruptible stream reads, but closing does not unblock a pending FileInputStream/stdin read — scope the destroy guidance to subprocesses and direct non-closeable streams to abandonOnInterruptReads. Also: closeOnAbandon permanently closes the wrapper (no next read); disambiguate "process output stream" to process.getInputStream; drop releaseAfterScope from the dependency list (the chapter only warns against it); mention abandonOnInterruptWrites for the write side; trim a comment duplicating prose. Co-Authored-By: Claude Fable 5 --- .../170-subprocesses-and-external-streams.md | 26 ++++++++++--------- .../skills/direct-style-scala/SKILL.md | 15 ++++++----- 2 files changed, 22 insertions(+), 19 deletions(-) diff --git a/direct-style-scala/skills/direct-style-scala/170-subprocesses-and-external-streams.md b/direct-style-scala/skills/direct-style-scala/170-subprocesses-and-external-streams.md index 31f5321..8c23b3a 100644 --- a/direct-style-scala/skills/direct-style-scala/170-subprocesses-and-external-streams.md +++ b/direct-style-scala/skills/direct-style-scala/170-subprocesses-and-external-streams.md @@ -11,7 +11,7 @@ only shows up at shutdown. ## Dependencies - `"com.softwaremill.ox" %% "core"` — `supervised`, `fork`, `forkDiscard`, - `releaseAfterScope`, `abandonOnInterruptReads` + `abandonOnInterruptReads` --- @@ -59,8 +59,10 @@ Fix"; see [which operations are interruptible](https://ox.softwaremill.com/latest/structured-concurrency/interruptions.html) in the Ox docs). Ox ends a scope by interrupting its forks and then joining them, so interruption alone will not stop such a fork, and the join blocks -forever. The only thing that unblocks the read is closing the underlying -resource — destroying the process — which makes the read return EOF. +forever. For a subprocess, what unblocks the read is destroying the process: +that closes the pipe's write-end, and the read returns EOF. Streams that don't +support an asynchronous close — stdin, `FileInputStream` — can't be unblocked +this way at all; wrap those with `abandonOnInterruptReads` (below). Blocking *socket* reads are different: on virtual threads — which Ox forks run on — they are interruptible since Java 21, though destructively (the interrupt @@ -111,16 +113,15 @@ resource down. For these, Ox (since 1.0.6) provides `abandonOnInterruptReads`: it wraps an `InputStream` so the actual reads run on a *detached* virtual thread — unmanaged, never joined — while the calling fork awaits each chunk interruptibly. On interruption the wait is abandoned: the fork proceeds with an -`InterruptedException`, the in-flight chunk is not lost (the next read returns -it), and with `closeOnAbandon = true` the underlying stream is additionally -closed. +`InterruptedException`, and the in-flight chunk is not lost — the next read +returns it. With `closeOnAbandon = true`, an interrupted read instead closes +the underlying stream, and the wrapper becomes permanently closed. ```scala import ox.* import scala.concurrent.duration.* -// stdin can be neither interrupted nor usefully closed — wrap it once, -// process-wide (multiple wrappers would compete for input): +// one process-wide wrapper: multiple wrappers over System.in would compete for input lazy val stdin = abandonOnInterruptReads(System.in) supervised: @@ -132,15 +133,16 @@ underlying read completes — for stdin, possibly for the application's lifetime That is a cheap virtual thread in exchange for interruptibility. For a subprocess, the body-`finally` destroy remains the primary teardown: it terminates the child *and* EOFs the pipe, leaving nothing behind. Wrapping the -process's output stream as well makes the reader fork immune to a mis-sequenced -teardown (scope cancellation can then always end it) — but it never replaces -destroying the process. +process's stdout (`process.getInputStream`) as well makes the reader fork +immune to a mis-sequenced teardown (scope cancellation can then always end +it) — but it never replaces destroying the process. For one-off uninterruptible calls (a JDBC `execute`, a DNS lookup) there is `abandonOnInterrupt(op)(onAbandon)`, where the cleanup starts on abandonment — e.g. `abandonOnInterrupt(statement.execute())(connection.close())`. For resources supporting asynchronous close, the cleanup also unblocks the -abandoned operation, so nothing is leaked. +abandoned operation, so nothing is leaked. `abandonOnInterruptWrites` covers +the write side — e.g. a write to the child's stdin blocked on a full pipe. ## Don't let an unread pipe stall the process diff --git a/direct-style-scala/skills/direct-style-scala/SKILL.md b/direct-style-scala/skills/direct-style-scala/SKILL.md index 0846705..3cbd65e 100644 --- a/direct-style-scala/skills/direct-style-scala/SKILL.md +++ b/direct-style-scala/skills/direct-style-scala/SKILL.md @@ -110,13 +110,14 @@ def findUser(id: Id[User])(using DbTx): Either[Fail, User] = instead of handing a live handle out. * a fork blocked in a classic `java.io` stream read (subprocess pipe, `FileInputStream`, stdin) is NOT ended by scope cancellation — such reads - ignore interruption. Either close/destroy the resource in the scope body's - `finally`, BEFORE the scope joins the fork — NEVER via `releaseAfterScope`, - which Ox runs after the join and so deadlocks on the blocked read — or wrap - the stream with `abandonOnInterruptReads` so reads become interruptible. - (Socket reads on Ox's virtual-thread forks ARE interruptible: the interrupt - closes the socket.) For a launcher subprocess, destroy the whole process - tree. See [Subprocesses and External + ignore interruption. For a subprocess, destroy it in the scope body's + `finally`, BEFORE the scope joins the fork (the pipe read then EOFs) — NEVER + via `releaseAfterScope`, which Ox runs after the join and so deadlocks on the + blocked read. For streams that closing can't unblock (stdin, + `FileInputStream`), wrap the stream with `abandonOnInterruptReads` so reads + become interruptible. (Socket reads on Ox's virtual-thread forks ARE + interruptible: the interrupt closes the socket.) For a launcher subprocess, + destroy the whole process tree. See [Subprocesses and External Streams](170-subprocesses-and-external-streams.md). # Functional programming From 691a9525f650c7b0bcc726fb9adddf710f800be8 Mon Sep 17 00:00:00 2001 From: Adam Warski Date: Thu, 23 Jul 2026 09:50:03 +0000 Subject: [PATCH 4/8] Cover remaining Ox 1.0.6 additions: computeIntensive/cede, resourceScope MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Chapter 150 gains a "CPU-intensive work" section (plus rows in both selection tables): virtual threads are never preempted, so long CPU-bound work must run via computeIntensive (or cooperate with cede()); scope context deliberately doesn't propagate into the compute pool. A matching decision-trigger bullet goes in SKILL.md's Performance section, since the skill otherwise pushes all work onto virtual threads with no starvation warning. - Chapter 100 gains a resourceScope section: scoped cleanup without claiming a concurrency capability, and `using ResourceScope` as the narrower alternative to `using Ox` for registration-only methods — reflected in the SKILL.md factories bullet and chapter 150's capability callout. Co-Authored-By: Claude Fable 5 --- .../100-resource-management.md | 32 +++++++++++- .../150-shared-state-across-threads.md | 52 +++++++++++++++++-- .../skills/direct-style-scala/SKILL.md | 18 +++++-- 3 files changed, 95 insertions(+), 7 deletions(-) diff --git a/direct-style-scala/skills/direct-style-scala/100-resource-management.md b/direct-style-scala/skills/direct-style-scala/100-resource-management.md index 09b7ef2..17d8e38 100644 --- a/direct-style-scala/skills/direct-style-scala/100-resource-management.md +++ b/direct-style-scala/skills/direct-style-scala/100-resource-management.md @@ -2,7 +2,8 @@ ## Dependencies -- `"com.softwaremill.ox" %% "core"` — `useInScope`, `useCloseableInScope` +- `"com.softwaremill.ox" %% "core"` — `useInScope`, `useCloseableInScope`, + `resourceScope` --- @@ -38,3 +39,32 @@ def create(using Ox): Dependencies = On scope termination (e.g., SIGTERM via `OxApp` — see [Background Processes](110-background-processes.md)), resources are released in reverse acquisition order: first the database pool, then the sttp backend. + +## Resources without concurrency: resourceScope + +When a method only needs scoped cleanup — no forks — don't open a `supervised` +scope just to get `useInScope`: that claims a concurrency capability the code +doesn't use. `resourceScope` (Ox ≥ 1.0.6) starts a dedicated, resource-only +scope, Ox's analogue of `scala.util.Using.Manager`: + +```scala +import ox.{resourceScope, useCloseableInScope} +import java.io.{FileReader, FileWriter} + +def process(): Unit = resourceScope: + val in = useCloseableInScope(FileReader("in.txt")) + val out = useCloseableInScope(FileWriter("out.txt")) + // both closed when the scope completes, out first + out.write(in.read()) +``` + +A resource scope can only be started where no concurrency scope is lexically +visible — verified at compile time, because a fork started inside a visible +resource scope could outlive it. Within a concurrency scope, register +resources directly instead; to use a resource scope e.g. in a fork's body, +extract it to a method that doesn't take `using Ox` — good practice in itself. + +Methods that only register resources should declare exactly that capability: +`using ResourceScope` instead of `using Ox`. Every concurrency scope is also a +`ResourceScope`, so such a method can be called from both — while its +signature no longer claims the ability to fork. diff --git a/direct-style-scala/skills/direct-style-scala/150-shared-state-across-threads.md b/direct-style-scala/skills/direct-style-scala/150-shared-state-across-threads.md index 0484c8b..f1fa2a5 100644 --- a/direct-style-scala/skills/direct-style-scala/150-shared-state-across-threads.md +++ b/direct-style-scala/skills/direct-style-scala/150-shared-state-across-threads.md @@ -3,7 +3,8 @@ ## Dependencies - `"com.softwaremill.ox" %% "core"` — `Flow`, `par`, `supervised`, `fork`, - `forkDiscard`, `forkUserDiscard`, channels, actors + `forkDiscard`, `forkUserDiscard`, channels, actors, `computeIntensive`, + `cede` --- @@ -26,6 +27,7 @@ when bridging a foreign API that Ox does not cover. | Mailbox or producer-consumer queue | `ox.channels.Channel[T]` | | Collection fan-out | `Flow.mapPar` or `Flow.mapParUnordered` | | Serialized access to mutable state | `Actor` | +| Long CPU-bound computation | `computeIntensive` (see below) | ## Flows @@ -287,6 +289,47 @@ stateRef.updateAndGet(state => process(state, item)).discard > **Warning:** Never use `AtomicReference` as a lifecycle, cancellation, or > shutdown flag. Model lifecycle with Ox scope structure or channel completion. +## CPU-intensive work + +Ox forks run on virtual threads, which are never preempted: a thread yields +only when it blocks. A long CPU-bound computation therefore monopolizes a +carrier thread — and since there are only as many carriers as CPU cores, a +handful of such computations can starve every other virtual thread in the +process, including latency-sensitive ones like HTTP handlers. Two remedies +(Ox ≥ 1.0.6), depending on the computation: + +- for short bursts in code you control, call `cede()` about once every + millisecond of computation (a call costs ~1µs): it yields the virtual thread + back to the scheduler and checks the interrupt flag, making the loop both + fair and cancellable; +- for long-running computations, or code that can't be instrumented with + yields (a third-party library), wrap the call in `computeIntensive`: the + computation runs on a pool of platform threads — which the OS preempts — + while the calling virtual thread blocks until the result is available. + +```scala +def expensive(): BigInt = (1 to 1_000_000).map(BigInt(_)).product + +supervised: + val f1 = fork(computeIntensive(expensive())) + val f2 = fork(computeIntensive(expensive())) + f1.join() + f2.join() +``` + +Because the caller blocks, `computeIntensive` stays structured — the +computation never outlives the enclosing scope — and composes with `fork`, +`par`, `race`, and `mapPar`. On cancellation the pool task is interrupted and +the caller waits for it to complete, so computations should still call +`checkInterrupt()` (or `cede()`) periodically where possible — a +non-cooperating computation delays the scope's shutdown until it finishes. + +> **Warning:** Scope context does not propagate into the computation: `fork` +> inside `computeIntensive` fails, `ForkLocal`s read their defaults, and +> thread-local integrations (MDC, OpenTelemetry context — see [OpenTelemetry +> Observability](510-opentelemetry-observability.md)) do not cross the +> boundary. Compute the value on the pool; do the forking, logging context, +> and tracing on the virtual-thread side. + ## Scope propagation Prefer local, focused scopes. If a method only needs concurrency to compute its @@ -312,8 +355,10 @@ def loadBoth(userId: UserId, accountId: AccountId): (User, Account) = ``` > **Important:** `(using Ox)` in a method signature means "I will start -> forks or register resources in your scope." If the method manages its own -> concurrency lifecycle, use a local `supervised` block instead. +> forks or register resources in your scope." If the method only registers +> resources and starts no forks, declare the narrower `(using ResourceScope)` +> instead (see [Resource Management](100-resource-management.md)); if it +> manages its own concurrency lifecycle, use a local `supervised` block. ## Choosing the right pattern @@ -326,3 +371,4 @@ Prefer the highest-level primitive that fits the shape of the problem: | **Channel** | Modeling an explicit protocol, mailbox, producer-consumer queue, or callback boundary. | | **Actor** | Multiple concurrent callers must access one mutable object serially. | | **AtomicReference** | A single shared value needs pure atomic updates; never use it for lifecycle flags. | +| **computeIntensive** | Long or non-instrumentable CPU-bound work that must not starve virtual threads. | diff --git a/direct-style-scala/skills/direct-style-scala/SKILL.md b/direct-style-scala/skills/direct-style-scala/SKILL.md index 3cbd65e..c6cd63e 100644 --- a/direct-style-scala/skills/direct-style-scala/SKILL.md +++ b/direct-style-scala/skills/direct-style-scala/SKILL.md @@ -85,6 +85,12 @@ def findUser(id: Id[User])(using DbTx): Either[Fail, User] = * NEVER materialize unbounded data into memory. Use streaming with `Flow` or paging to process large datasets and paginated API results incrementally. +* virtual threads are never preempted — a long CPU-bound computation (even + inside `mapPar`) starves every other virtual thread in the process. Run long + or non-instrumentable compute via `computeIntensive` (platform-thread pool; + the blocking caller keeps it structured); in CPU-bound loops you control, + call `cede()` about once per millisecond. See [Concurrency and Inter-Thread + Communication](150-shared-state-across-threads.md). # Direct-style Scala @@ -100,7 +106,10 @@ def findUser(id: Id[User])(using DbTx): Either[Fail, User] = job-level concurrency. Accept a parent `(using Ox)` only when a fork or resource must be tied to that parent scope's lifetime. * keep constructors plain; use factories that take `(using Ox)` and return - values that do not carry the capability. + values that do not carry the capability. If the factory only registers + resources and starts no forks, take the narrower `(using ResourceScope)`; + for scoped cleanup with no enclosing scope and no concurrency, use + `resourceScope` instead of `supervised`. * decide the owning scope BEFORE writing concurrent code. Model a stream reader or worker as a fork in a `supervised` scope whose lifetime matches the work; its result is the fork's **return value** (`fork{…}.join()`), not a value @@ -265,7 +274,9 @@ https://raw.githubusercontent.com/virtuslab/scala-skill/refs/heads/master/direct sbt/Scalafix boundary enforcement. - [Resource Management](100-resource-management.md) — `useInScope`, - `useCloseableInScope`, reverse-order release, scope-based cleanup. + `useCloseableInScope`, reverse-order release, scope-based cleanup; + `resourceScope` for cleanup without concurrency, `using ResourceScope` as + the narrower capability. - [Background Processes](110-background-processes.md) — `OxApp` entry point, `forkDiscard`/`forkUserDiscard` for daemon vs. user threads, @@ -282,7 +293,8 @@ https://raw.githubusercontent.com/virtuslab/scala-skill/refs/heads/master/direct - [Concurrency and Inter-Thread Communication](150-shared-state-across-threads.md) — Flows for declarative concurrent pipelines (`mapPar`, `merge`, `mapStateful`), Ox primitive selection, channels for worker mailboxes and - shutdown, actors for serialized mutable state. + shutdown, actors for serialized mutable state, `computeIntensive`/`cede` for + CPU-bound work on virtual threads. - [Subprocesses and External Streams](170-subprocesses-and-external-streams.md) — driving a subprocess / socket / SSE reader as a fork whose return value is From 00b7da5694225690ebc2997b4e7b1c0705f90b81 Mon Sep 17 00:00:00 2001 From: Adam Warski Date: Thu, 23 Jul 2026 10:14:01 +0000 Subject: [PATCH 5/8] Address second review round - SKILL.md: fix overstated starvation claim (one computation holds one carrier; a few suffice to starve), extend the parent-scope bullet with the ResourceScope option. - Chapter 100: Dependencies.create now declares `using ResourceScope`, matching the rule the chapter itself introduces; state the actual harm behind the resourceScope compile-time guard (use-after-release). - Chapter 150: "(by default)" on the carrier count, `import ox.*` in the computeIntensive example, note that a `timeout` around a non-cooperating computation overshoots. - Harmonize version phrasing on "(Ox >= 1.0.6)" across additions. Co-Authored-By: Claude Fable 5 --- .../direct-style-scala/100-resource-management.md | 9 ++++++--- .../150-shared-state-across-threads.md | 12 ++++++++---- .../170-subprocesses-and-external-streams.md | 2 +- .../skills/direct-style-scala/SKILL.md | 8 +++++--- 4 files changed, 20 insertions(+), 11 deletions(-) diff --git a/direct-style-scala/skills/direct-style-scala/100-resource-management.md b/direct-style-scala/skills/direct-style-scala/100-resource-management.md index 17d8e38..570cf93 100644 --- a/direct-style-scala/skills/direct-style-scala/100-resource-management.md +++ b/direct-style-scala/skills/direct-style-scala/100-resource-management.md @@ -17,10 +17,12 @@ when the scope ends. ## Application resource setup The `Dependencies.create` method acquires all resources within the application's -root scope: +root scope. It only registers resources — no forks — so it declares +`using ResourceScope` rather than `using Ox` (see below); the root `Ox` scope +satisfies it: ```scala -def create(using Ox): Dependencies = +def create(using ResourceScope): Dependencies = val config = Config.read.tap(Config.log) val otel = initializeOtel() val sttpBackend = useInScope( @@ -60,7 +62,8 @@ def process(): Unit = resourceScope: A resource scope can only be started where no concurrency scope is lexically visible — verified at compile time, because a fork started inside a visible -resource scope could outlive it. Within a concurrency scope, register +resource scope could outlive it, using resources after they have been +released. Within a concurrency scope, register resources directly instead; to use a resource scope e.g. in a fork's body, extract it to a method that doesn't take `using Ox` — good practice in itself. diff --git a/direct-style-scala/skills/direct-style-scala/150-shared-state-across-threads.md b/direct-style-scala/skills/direct-style-scala/150-shared-state-across-threads.md index f1fa2a5..c2d427b 100644 --- a/direct-style-scala/skills/direct-style-scala/150-shared-state-across-threads.md +++ b/direct-style-scala/skills/direct-style-scala/150-shared-state-across-threads.md @@ -293,9 +293,10 @@ stateRef.updateAndGet(state => process(state, item)).discard Ox forks run on virtual threads, which are never preempted: a thread yields only when it blocks. A long CPU-bound computation therefore monopolizes a -carrier thread — and since there are only as many carriers as CPU cores, a -handful of such computations can starve every other virtual thread in the -process, including latency-sensitive ones like HTTP handlers. Two remedies +carrier thread — and since there are only as many carriers as CPU cores (by +default), a handful of such computations can starve every other virtual +thread in the process, including latency-sensitive ones like HTTP handlers. +Two remedies (Ox ≥ 1.0.6), depending on the computation: - for short bursts in code you control, call `cede()` about once every @@ -308,6 +309,8 @@ process, including latency-sensitive ones like HTTP handlers. Two remedies while the calling virtual thread blocks until the result is available. ```scala +import ox.* + def expensive(): BigInt = (1 to 1_000_000).map(BigInt(_)).product supervised: @@ -321,7 +324,8 @@ computation never outlives the enclosing scope — and composes with `fork`, `par`, `race`, and `mapPar`. On cancellation the pool task is interrupted and the caller waits for it to complete, so computations should still call `checkInterrupt()` (or `cede()`) periodically where possible — a -non-cooperating computation delays the scope's shutdown until it finishes. +non-cooperating computation delays the scope's shutdown until it finishes, and +a `timeout` around it overshoots accordingly. > **Warning:** Scope context does not propagate into the computation: `fork` > inside `computeIntensive` fails, `ForkLocal`s read their defaults, and diff --git a/direct-style-scala/skills/direct-style-scala/170-subprocesses-and-external-streams.md b/direct-style-scala/skills/direct-style-scala/170-subprocesses-and-external-streams.md index 8c23b3a..70978d5 100644 --- a/direct-style-scala/skills/direct-style-scala/170-subprocesses-and-external-streams.md +++ b/direct-style-scala/skills/direct-style-scala/170-subprocesses-and-external-streams.md @@ -109,7 +109,7 @@ Destroy-and-EOF works because a subprocess supports asynchronous close. Some resources don't: stdin can't be meaningfully closed, and closing a `FileInputStream` does not unblock a pending read. And sometimes a single read should be cancellable (a `timeout` around one read) without tearing the -resource down. For these, Ox (since 1.0.6) provides `abandonOnInterruptReads`: +resource down. For these, use `abandonOnInterruptReads` (Ox ≥ 1.0.6): it wraps an `InputStream` so the actual reads run on a *detached* virtual thread — unmanaged, never joined — while the calling fork awaits each chunk interruptibly. On interruption the wait is abandoned: the fork proceeds with an diff --git a/direct-style-scala/skills/direct-style-scala/SKILL.md b/direct-style-scala/skills/direct-style-scala/SKILL.md index c6cd63e..e5379e4 100644 --- a/direct-style-scala/skills/direct-style-scala/SKILL.md +++ b/direct-style-scala/skills/direct-style-scala/SKILL.md @@ -85,8 +85,9 @@ def findUser(id: Id[User])(using DbTx): Either[Fail, User] = * NEVER materialize unbounded data into memory. Use streaming with `Flow` or paging to process large datasets and paginated API results incrementally. -* virtual threads are never preempted — a long CPU-bound computation (even - inside `mapPar`) starves every other virtual thread in the process. Run long +* virtual threads are never preempted — long CPU-bound computations (a few + suffice, e.g. a `mapPar` over such work) can starve every other virtual + thread in the process. Run long or non-instrumentable compute via `computeIntensive` (platform-thread pool; the blocking caller keeps it structured); in CPU-bound loops you control, call `cede()` about once per millisecond. See [Concurrency and Inter-Thread @@ -104,7 +105,8 @@ def findUser(id: Id[User])(using DbTx): Either[Fail, User] = pure atomic state or when bridging a foreign API that Ox does not cover. * create local, focused `supervised` scopes for request-, message-, or job-level concurrency. Accept a parent `(using Ox)` only when a fork or - resource must be tied to that parent scope's lifetime. + resource must be tied to that parent scope's lifetime (if it's only + resources, take `(using ResourceScope)`). * keep constructors plain; use factories that take `(using Ox)` and return values that do not carry the capability. If the factory only registers resources and starts no forks, take the narrower `(using ResourceScope)`; From c9a4fa73be672f3470511266457de6537a26abe7 Mon Sep 17 00:00:00 2001 From: Adam Warski Date: Thu, 23 Jul 2026 11:53:24 +0000 Subject: [PATCH 6/8] Deduplicate ResourceScope rule in SKILL.md; fix line wrap The resources-only -> (using ResourceScope) rule was stated in two adjacent bullets; keep the full statement in the factories bullet only. Co-Authored-By: Claude Fable 5 --- direct-style-scala/skills/direct-style-scala/SKILL.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/direct-style-scala/skills/direct-style-scala/SKILL.md b/direct-style-scala/skills/direct-style-scala/SKILL.md index e5379e4..f841e4a 100644 --- a/direct-style-scala/skills/direct-style-scala/SKILL.md +++ b/direct-style-scala/skills/direct-style-scala/SKILL.md @@ -87,10 +87,10 @@ def findUser(id: Id[User])(using DbTx): Either[Fail, User] = paging to process large datasets and paginated API results incrementally. * virtual threads are never preempted — long CPU-bound computations (a few suffice, e.g. a `mapPar` over such work) can starve every other virtual - thread in the process. Run long - or non-instrumentable compute via `computeIntensive` (platform-thread pool; - the blocking caller keeps it structured); in CPU-bound loops you control, - call `cede()` about once per millisecond. See [Concurrency and Inter-Thread + thread in the process. Run long or non-instrumentable compute via + `computeIntensive` (platform-thread pool; the blocking caller keeps it + structured); in CPU-bound loops you control, call `cede()` about once per + millisecond. See [Concurrency and Inter-Thread Communication](150-shared-state-across-threads.md). # Direct-style Scala @@ -105,8 +105,7 @@ def findUser(id: Id[User])(using DbTx): Either[Fail, User] = pure atomic state or when bridging a foreign API that Ox does not cover. * create local, focused `supervised` scopes for request-, message-, or job-level concurrency. Accept a parent `(using Ox)` only when a fork or - resource must be tied to that parent scope's lifetime (if it's only - resources, take `(using ResourceScope)`). + resource must be tied to that parent scope's lifetime. * keep constructors plain; use factories that take `(using Ox)` and return values that do not carry the capability. If the factory only registers resources and starts no forks, take the narrower `(using ResourceScope)`; From 2ce3f959872a9048d84baaf7e939956f0680f81f Mon Sep 17 00:00:00 2001 From: Adam Warski Date: Thu, 23 Jul 2026 11:58:40 +0000 Subject: [PATCH 7/8] Tighten chapter 170; drop redundant chapter link from Performance bullet MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Each fact is now stated once: the anti-pattern paragraph merged into its warning callout, the pipe-EOF mechanism lives only in the process-tree callout, the teardown-ordering paragraph no longer restates itself, and the intro no longer pre-summarizes the sections. No facts or callouts dropped. The SKILL.md Performance bullet loses its chapter link — the index below already carries it. Co-Authored-By: Claude Fable 5 --- .../170-subprocesses-and-external-streams.md | 93 ++++++++----------- .../skills/direct-style-scala/SKILL.md | 3 +- 2 files changed, 40 insertions(+), 56 deletions(-) diff --git a/direct-style-scala/skills/direct-style-scala/170-subprocesses-and-external-streams.md b/direct-style-scala/skills/direct-style-scala/170-subprocesses-and-external-streams.md index 70978d5..5668c86 100644 --- a/direct-style-scala/skills/direct-style-scala/170-subprocesses-and-external-streams.md +++ b/direct-style-scala/skills/direct-style-scala/170-subprocesses-and-external-streams.md @@ -2,11 +2,8 @@ Driving a long-lived subprocess (or any blocking external connection — a raw socket, an SSE stream) whose output you consume line by line, under structured -concurrency. The reader runs as a fork; its result is the fork's return value; -and teardown must account for a subprocess pipe read not being interruptible. -This is where structured concurrency is easiest to get subtly wrong — getting -the ownership and teardown right *before* writing the code avoids a deadlock that -only shows up at shutdown. +concurrency. Getting the ownership and teardown shape right *before* writing +the code avoids a deadlock that only shows up at shutdown. ## Dependencies @@ -18,11 +15,10 @@ only shows up at shutdown. ## Own the work with a scope; return a result, not a live handle Decide the owning scope first. A reader that consumes a process's output is -concurrent work, so it belongs to a `supervised` scope whose lifetime matches the -work: open the scope, spawn the process, fork the reader, drive it, and return -the computed value. The reader fork's **return value is the outcome** — read it -with `join()`. Don't publish the result through a shared `AtomicReference` that -another thread writes and the caller polls; there is one producer and one value. +concurrent work, so it belongs to a `supervised` scope whose lifetime matches +the work. The reader fork's **return value is the outcome** — read it with +`join()`; don't publish it through a shared `AtomicReference` that another +thread writes and the caller polls. There is one producer and one value. ```scala def runAndCollect(command: Seq[String]): Summary = @@ -38,24 +34,20 @@ def runAndCollect(command: Seq[String]): Summary = finally process.destroyForcibly().discard // see the next section ``` -The opposite shape — a constructor that spawns reader threads and returns an -object the caller drives later — is the pattern to avoid. Its lifetime no longer -matches any scope, so cancellation, error propagation, and cleanup all become -manual flags and `try`/`finally` ladders, exactly the bookkeeping structured -concurrency removes. - -> **Warning:** Never return an object that owns running forks or threads, to be -> driven by the caller afterwards. Keep the work inside the scope that owns it -> and return a plain value. If the caller must interleave with the work (send -> input, react to events), pass a *consumer* into the scope (`run(...)(use: -> Handle => T)`) rather than handing a live handle out of it. +> **Warning:** Never return an object that owns running forks or threads, to +> be driven by the caller afterwards — its lifetime then matches no scope, so +> cancellation, error propagation, and cleanup become manual flags and +> `try`/`finally` ladders, exactly the bookkeeping structured concurrency +> removes. Keep the work inside the scope and return a plain value; if the +> caller must interleave with the work (send input, react to events), pass a +> *consumer* into the scope (`run(...)(use: Handle => T)`) rather than handing +> a live handle out of it. ## Teardown: a pipe read is not interruptible — destroy before the join -A fork blocked reading a classic `java.io` stream — a subprocess's stdout pipe, -a `FileInputStream`, stdin — does **not** observe `Thread.interrupt`. This is -intended, permanent JVM behavior (the request to change it was closed as "Won't -Fix"; see [which operations are +A fork blocked reading a classic `java.io` stream — a subprocess's stdout +pipe, a `FileInputStream`, stdin — does **not** observe `Thread.interrupt`, +permanently and by design (see [which operations are interruptible](https://ox.softwaremill.com/latest/structured-concurrency/interruptions.html) in the Ox docs). Ox ends a scope by interrupting its forks and then joining them, so interruption alone will not stop such a fork, and the join blocks @@ -65,19 +57,16 @@ support an asynchronous close — stdin, `FileInputStream` — can't be unblocke this way at all; wrap those with `abandonOnInterruptReads` (below). Blocking *socket* reads are different: on virtual threads — which Ox forks run -on — they are interruptible since Java 21, though destructively (the interrupt -closes the socket). A socket-backed reader is therefore ended by scope -cancellation on its own; the deadlock below is specific to pipe and other -classic stream reads. - -That close must happen **before** the scope joins the fork. Ox releases -scope-managed resources (`useInScope`, `releaseAfterScope`) only *after* all -forks complete — so, as the Ox docs note, they won't unblock a fork stuck in -uninterruptible I/O: the join has already deadlocked by the time the finalizer -would run. Put the destroy in your own body `finally` (as above) — it is -sequenced before the scope joins the forks. On the normal path the process has -already exited and the destroy is a no-op; on cancellation or error it is what -lets the scope complete. +on — interruption works since Java 21, destructively closing the socket. A +socket-backed reader is thus ended by scope cancellation on its own; the +deadlock here is specific to pipe and other classic stream reads. + +The destroy must be sequenced **before** the scope joins the fork — which is +why it sits in the scope body's own `finally` above. Ox releases scope-managed +resources (`useInScope`, `releaseAfterScope`) only *after* all forks complete, +so a kill registered there runs when the join has already deadlocked. On the +normal path the process has already exited and the destroy is a no-op; on +cancellation or error it is what lets the scope complete. > **Required:** Close or destroy a blocking external resource in the scope > BODY's `finally`, before the scope joins the reader fork. Do NOT rely on @@ -88,10 +77,7 @@ lets the scope complete. If the process you spawned is a launcher or wrapper that forks the real worker (`some-launcher run the-tool …`), destroying only your direct child orphans the -worker — and the orphan keeps the inherited stdout/stderr pipe write-ends open. -A POSIX pipe reports EOF only once *every* write-end is closed, so the reader -fork never unblocks and the join still hangs. Destroy descendants first, then the -root: +worker. Destroy descendants first, then the root: ```scala val handle = process.toHandle @@ -99,9 +85,10 @@ handle.descendants().forEach(_.destroyForcibly().discard) handle.destroyForcibly().discard ``` -> **Warning:** A forked grandchild inherits the pipe file descriptors. If it -> survives the kill, the reader never sees EOF — so terminate the descendants, -> not just the process you directly spawned. +> **Warning:** A forked grandchild inherits the pipe file descriptors, and a +> POSIX pipe reports EOF only once *every* write-end is closed. An orphan that +> survives the kill keeps the reader blocked forever — terminate the +> descendants, not just the process you directly spawned. ## Reads you can't unblock by closing: `abandonOnInterruptReads` @@ -131,11 +118,10 @@ supervised: The trade-off: an abandoned read leaves the detached thread blocked until the underlying read completes — for stdin, possibly for the application's lifetime. That is a cheap virtual thread in exchange for interruptibility. For a -subprocess, the body-`finally` destroy remains the primary teardown: it -terminates the child *and* EOFs the pipe, leaving nothing behind. Wrapping the -process's stdout (`process.getInputStream`) as well makes the reader fork -immune to a mis-sequenced teardown (scope cancellation can then always end -it) — but it never replaces destroying the process. +subprocess, keep the body-`finally` destroy as the primary teardown — it +terminates the child *and* EOFs the pipe, leaving nothing behind; wrapping the +process's stdout (`process.getInputStream`) additionally protects the reader +from a mis-sequenced teardown, but does not replace the destroy. For one-off uninterruptible calls (a JDBC `execute`, a DNS lookup) there is `abandonOnInterrupt(op)(onAbandon)`, where the cleanup starts on abandonment — @@ -163,7 +149,6 @@ supervised: finally process.destroyForcibly().discard ``` -The drain fork swallows `NonFatal` so a stray read error can't tear down the -scope, and logs rather than discards silently so a real failure stays -diagnosable. It needs no separate stop signal: the body-`finally` destroy EOFs -its read, and the scope then joins it. +The drain catches `NonFatal` — logged, so a real failure stays diagnosable — +because a stray read error must not tear down the scope. It needs no stop +signal: the body-`finally` destroy EOFs its read, and the scope then joins it. diff --git a/direct-style-scala/skills/direct-style-scala/SKILL.md b/direct-style-scala/skills/direct-style-scala/SKILL.md index f841e4a..a690203 100644 --- a/direct-style-scala/skills/direct-style-scala/SKILL.md +++ b/direct-style-scala/skills/direct-style-scala/SKILL.md @@ -90,8 +90,7 @@ def findUser(id: Id[User])(using DbTx): Either[Fail, User] = thread in the process. Run long or non-instrumentable compute via `computeIntensive` (platform-thread pool; the blocking caller keeps it structured); in CPU-bound loops you control, call `cede()` about once per - millisecond. See [Concurrency and Inter-Thread - Communication](150-shared-state-across-threads.md). + millisecond. # Direct-style Scala From 44cfdc67b3bace4d36a55ac03f3d9093ab17fb60 Mon Sep 17 00:00:00 2001 From: Adam Warski Date: Thu, 23 Jul 2026 12:01:01 +0000 Subject: [PATCH 8/8] Shrink the stream-teardown bullet to a trigger; the how lives in ch. 170 The bullet had grown into a restatement of the chapter's full teardown rules. Keep only the hazard (uninterruptible reads, shutdown deadlock) and the instruction to read the chapter before writing such code. Co-Authored-By: Claude Fable 5 --- .../skills/direct-style-scala/SKILL.md | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) diff --git a/direct-style-scala/skills/direct-style-scala/SKILL.md b/direct-style-scala/skills/direct-style-scala/SKILL.md index a690203..96b3614 100644 --- a/direct-style-scala/skills/direct-style-scala/SKILL.md +++ b/direct-style-scala/skills/direct-style-scala/SKILL.md @@ -117,16 +117,11 @@ def findUser(id: Id[User])(using DbTx): Either[Fail, User] = running forks/threads to be driven later — its lifetime escapes every scope, making cancellation and cleanup manual again. Pass a consumer into the scope instead of handing a live handle out. -* a fork blocked in a classic `java.io` stream read (subprocess pipe, - `FileInputStream`, stdin) is NOT ended by scope cancellation — such reads - ignore interruption. For a subprocess, destroy it in the scope body's - `finally`, BEFORE the scope joins the fork (the pipe read then EOFs) — NEVER - via `releaseAfterScope`, which Ox runs after the join and so deadlocks on the - blocked read. For streams that closing can't unblock (stdin, - `FileInputStream`), wrap the stream with `abandonOnInterruptReads` so reads - become interruptible. (Socket reads on Ox's virtual-thread forks ARE - interruptible: the interrupt closes the socket.) For a launcher subprocess, - destroy the whole process tree. See [Subprocesses and External +* a fork blocked reading a subprocess pipe, stdin, a file, or any other + classic `java.io` stream is NOT ended by scope cancellation — without the + right teardown shape, shutdown deadlocks. BEFORE writing code that drives a + subprocess or reads a blocking external stream (socket, SSE), read + [Subprocesses and External Streams](170-subprocesses-and-external-streams.md). # Functional programming