Skip to content

fix(llminternal): classify a dropped live connection by type, not error text - #1603

Open
harshitwandhare wants to merge 1 commit into
google:mainfrom
harshitwandhare:fix/llminternal-resumable-socket-error
Open

harshitwandhare wants to merge 1 commit into
google:mainfrom
harshitwandhare:fix/llminternal-resumable-socket-error

Conversation

@harshitwandhare

@harshitwandhare harshitwandhare commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1602.

What was broken

RunLive runs a reader goroutine and a sender goroutine against one live connection. Both report failures into the same unbuffered errChan and the flow's select consumes exactly one, so whichever arrives first decides whether the session resumes or dies. isResumable made that call by matching substrings of the error text, and the two goroutines do not produce the same text for the same event.

Measured with a probe that kills the peer and then calls each path directly:

platform sender error on a dropped connection old isResumable
Linux write tcp ...: write: broken pipe true
Windows write tcp ...: wsasend: An established connection was aborted by the software in your host machine. false

The reader's error is the websocket layer's close (close 1006 (abnormal closure): unexpected EOF) and is the same on both, so it matched EOF and resumed. On Windows the sender's error matched nothing in the list, so an identical connection loss was pushed to the caller as fatal and the session stopped after one connection. Which goroutine noticed first decided whether a Windows live session survived a transient drop.

errors.Is against the POSIX constants does not close the gap. Go does not map WSA error numbers onto syscall.ECONNABORTED / ECONNRESET, verified directly:

errors.Is(senderErr, ECONNABORTED) = false
errors.Is(senderErr, ECONNRESET)   = false
errors.Is(senderErr, EPIPE)        = false
UNWRAP *net.OpError      "write tcp ...: wsasend: An established connection was aborted ..."
UNWRAP *os.SyscallError  "wsasend: An established connection was aborted ..."
UNWRAP syscall.Errno     "An established connection was aborted ..."
   --> syscall.Errno = 10053 (0x2745)

What the fix does

Classifies the transport failure by type instead of by text:

var opErr *net.OpError
if errors.As(err, &opErr) {
	return true
}

Any *net.OpError reaching this path is a failed read or write on an already-established live socket, because the dial has its own error path higher up that returns before this loop. The substring checks stay for the websocket-level cases (1006, 1008, GoAway), which are not *net.OpError, so no existing behaviour changes.

isResumable moves from a closure inside RunLive to a package-level function, which is what makes it testable. Nothing else about it changed: the io.EOF check and the substring list are byte-identical to before.

How it surfaced

TestRunLiveNoGoroutineLeak/realtime_sender_error_after_connection_loss_does_not_leak fails intermittently on Windows:

base_flow_live_test.go:318: never received a model turn for the retried realtime send
base_flow_live_test.go:392: connection count = 1, want 2

connection count = 1, want 2 is the tell: the flow never opened the second connection. Despite the test's name this is not a leak.

Being straight about that repro: I saw it once in roughly 550 runs of the test binary, and only under heavy CPU load, which is what makes the sender win the race often enough to observe. I did not try to turn that into a regression test, because a 1-in-550 race would be a flake in CI rather than a guard. The new unit test pins the mechanism instead, which is deterministic.

Testing Plan

Failing first, and on the real defect rather than on a missing symbol. Since isResumable did not exist as a callable symbol before, running the new test against unmodified main only produces undefined: isResumable, which proves nothing. So the extraction was applied on its own, without the *net.OpError check, and the new tests were run against that:

--- FAIL: TestIsResumable/sender:_windows_WSAECONNABORTED
--- FAIL: TestIsResumable/sender:_windows_WSAECONNRESET
--- FAIL: TestIsResumable/reader:_windows_WSAECONNRESET_on_recv
--- FAIL: TestIsResumable/sender:_wrapped_socket_error_still_resumable
--- PASS: TestIsResumable/sender:_linux_EPIPE
--- PASS: TestIsResumable/sender:_linux_ECONNRESET
--- PASS: TestIsResumable/protocol_error_1002_is_fatal
--- FAIL: TestIsResumableAgreesAcrossReaderAndSender/windows
    same connection loss classified differently: reader=true sender=false;
    whether the session resumes would depend on which goroutine reported first
--- PASS: TestIsResumableAgreesAcrossReaderAndSender/linux

With the *net.OpError check restored, all 16 subtests pass.

The test synthesizes the error shapes (*net.OpError wrapping *os.SyscallError wrapping an explicit errno) rather than relying on the host's own spelling, so a Linux CI runner exercises the Windows cases. Confirmed: the same 16 subtests pass on linux/amd64 (WSL2) as well as windows/amd64.

Gates run on this branch, windows/amd64, Go 1.26.6:

  • go test ./internal/llminternal/ -count=1 ok

  • go test -race ./internal/llminternal/ -count=1 ok, and 20 further runs as separate processes, 0 failures

  • go vet ./internal/llminternal/ clean

  • go test -race -mod=readonly -count=1 -shuffle=on work: 5 packages fail, and the same 5 fail on pristine main at f7e16e0. Failure sets extracted and diffed rather than eyeballed, and they are identical: cmd/adkgo/internal/deploy/agentengine, cmd/adkgo/internal/deploy/cloudrun, internal/configurable/conformance/replayplugin, internal/telemetry/functionaltest, runner. All are Windows-only and none is internal/llminternal

  • golangci-lint run: 0 issues, exit 0. It flagged one real gofumpt violation in the new test file first, which is fixed.

    Worth recording for anyone else reproducing locally: with a stale GOLANGCI_LINT_CACHE this run instead reports 6 spurious SA5011 findings in unrelated test files, and the files it names change between runs on an unmodified tree. That is exactly the failure mode the skip-cache: true comment in .github/workflows/go.yml describes, and pointing GOLANGCI_LINT_CACHE at a fresh directory gives 0 issues. I had it backwards at first and am noting it in case it saves someone the same detour.

One thing worth flagging separately, since it cost me time and is not caused by this change: -count=N with N > 1 is not usable on this package. TestLoggingSpanIDPropagation fails on every repeat after the first because it depends on process-global logger state, 19 failures out of 20 on pristine main with no changes applied. Repeated runs here have to be separate processes.

Environment

Windows 11 / windows-amd64, Go 1.26.6, cross-checked on linux/amd64 under WSL2.

@harshitwandhare
harshitwandhare force-pushed the fix/llminternal-resumable-socket-error branch 5 times, most recently from 6705d1f to 2db67d2 Compare September 24, 2026 22:34
@harshitwandhare
harshitwandhare force-pushed the fix/llminternal-resumable-socket-error branch 2 times, most recently from e64e697 to 7aa1887 Compare September 29, 2026 02:01
@harshitwandhare

harshitwandhare commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor Author

Gentle ping on this one when someone has time. All checks pass. Happy to change the approach if you'd rather classify the dropped connection differently.

@harshitwandhare
harshitwandhare force-pushed the fix/llminternal-resumable-socket-error branch 2 times, most recently from afd481e to 837f184 Compare October 2, 2026 07:20
…or text

RunLive's reader and sender goroutines both report into the same errChan
and the flow acts on whichever arrives first, so the two must agree on
whether a dropped connection is resumable. They did not agree on Windows.

isResumable matched substrings. Its list covers the reader's websocket
close text ("close 1006 ... unexpected EOF") and the POSIX sender text
("write: broken pipe"), but not the Windows sender text ("wsasend: An
established connection was aborted by the software in your host
machine."). When the sender won the race on Windows, the same connection
loss that would have resumed was pushed to the caller as fatal and the
session stopped after one connection.

errors.Is against the POSIX constants does not close the gap, because Go
does not map WSA error numbers onto them. The chain is *net.OpError ->
*os.SyscallError -> syscall.Errno(10053).

Match the transport failure by type instead. Any *net.OpError reaching
this path is a failed read or write on an already-established live
socket, since the dial is handled separately above, so it is resumable
on every platform. The substring checks stay for the websocket-level
cases (1006, 1008, GoAway), which are not *net.OpError.

isResumable moves from a closure inside RunLive to a package-level
function so it can be tested directly. Behaviour is otherwise unchanged.

Fixes google#1602
@harshitwandhare
harshitwandhare force-pushed the fix/llminternal-resumable-socket-error branch from 837f184 to 21a70bf Compare October 3, 2026 01:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

llminternal: a dropped live connection is resumable or fatal depending on which goroutine reports it, so live sessions stop resuming on Windows

1 participant