Skip to content

Fleet agent: don't let a dropped connection kill the launcher - #169

Open
rudysev wants to merge 1 commit into
starbrightlab:mainfrom
rudysev:fix/fleet-http-connection-crash
Open

Fleet agent: don't let a dropped connection kill the launcher#169
rudysev wants to merge 1 commit into
starbrightlab:mainfrom
rudysev:fix/fleet-http-connection-crash

Conversation

@rudysev

@rudysev rudysev commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

What

Loading the phone remote page crashed the launcher. FleetHttpServer ran each connection directly as a pool task:

pool.execute { serve(socket) }

Browsers preconnect speculatively and abandon connections when a tab is backgrounded, so readHead() blocked until the 30s soTimeout and threw SocketTimeoutException. A checked exception escaping a Runnable gets wrapped by ThreadPoolExecutor.runWorker into java.lang.Error, which reaches the thread's uncaught handler:

FATAL EXCEPTION: pool-2-thread-2
java.lang.Error: java.net.SocketTimeoutException: Read timed out
    at com.immortal.launcher.FleetHttpServer.readHead(FleetHttpServer.kt:239)
    at com.immortal.launcher.FleetHttpServer.serve(FleetHttpServer.kt:201)

CrashGuard then recovered the process, so turning on "remote — control from phone" and opening the page from a phone browser bounced the launcher. Every other failure path in the file was already inside runCatching; the header read was the one hole.

Changes

  • serveSafely() wraps serve() — timeouts and resets log at debug, anything else warns, and the socket is always closed. A dead connection can no longer take the process down.
  • Split the read timeout: 10s while waiting for a request line, 30s once a real request is in flight (uploads keep the long budget). An abandoned connection previously pinned a pool thread for the full 30s.
  • Pool 4 → 8: with Connection: close, every asset and poll costs a fresh connection and browsers fan out ~6 in parallel, so the remote page could queue behind itself.

Testing

FleetHttpServerSocketTest drives a real loopback server: reset connections stay off the uncaught handler and the pool keeps serving, plus coverage for preflight, a malformed request, a throwing handler, and stop(). Full suite passes (298 tests).

The unit-test change to app/build.gradle.kts (unitTests.isReturnDefaultValues) is needed because the off-device android.util.Log stub throws "not mocked", which put every logging path out of reach of a JVM test. No test regressions.

Verified on a Portal 10" gen2 (Android 10) with the dev build: held idle connections open past the header timeout, and the same exception now surfaces as

D ImmortalFleet: idle connection timed out: Read timed out

with the PID unchanged and the next request answered normally. The phone remote is usable end to end.

🤖 Generated with Claude Code

A phone browser opening the remote page crashed the launcher on a Portal 10"
gen2 (Android 10). Browsers preconnect speculatively and abandon connections
when a tab is backgrounded, so FleetHttpServer.readHead() would block until the
30s soTimeout and throw SocketTimeoutException. serve() ran directly as a pool
task, and a checked exception escaping a Runnable is wrapped by
ThreadPoolExecutor.runWorker into java.lang.Error — reaching the thread's
uncaught handler:

    FATAL EXCEPTION: pool-2-thread-2
    java.lang.Error: java.net.SocketTimeoutException: Read timed out
        at com.immortal.launcher.FleetHttpServer.readHead(FleetHttpServer.kt:239)

CrashGuard then recovered the process, so enabling "remote control from phone"
and loading the page bounced the launcher. Every other failure path in the file
was already inside runCatching; the header read was the one hole.

- serveSafely() wraps serve(): timeouts and resets log at debug, anything else
  warns, and the socket is always closed.
- Split the read timeout — 10s waiting for a request line, 30s once a request is
  in flight (uploads keep the long budget). An abandoned connection previously
  pinned a pool thread for the full 30s.
- Pool 4 -> 8. With `Connection: close` every asset and poll costs a fresh
  connection and browsers fan out ~6 in parallel, so the page queued behind
  itself.

FleetHttpServerSocketTest drives a real loopback server: reset connections stay
off the uncaught handler and the pool keeps serving, plus preflight, malformed
request, throwing handler, and stop(). Unit tests need
`unitTests.isReturnDefaultValues` because the off-device android.util.Log stub
throws "not mocked", which put every logging path out of reach of a JVM test.

Verified on a Portal 10" gen2 (Android 10): held idle connections past the
header timeout, same exception now logs as
`D ImmortalFleet: idle connection timed out: Read timed out`, PID unchanged, and
the next request answered normally.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant