fix: return reaper handshake errors - #3841
Conversation
✅ Deploy Preview for testcontainers-go ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Summary by CodeRabbit
Walkthrough
ChangesReaper handshake handling
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested labels: Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@reaper.go`:
- Around line 545-548: Update Connect around r.handshake(conn) to apply the
context deadline to the connection before starting the synchronous handshake, so
an unresponsive peer cannot block indefinitely. After a successful handshake,
clear the connection deadline before returning the connection; preserve existing
close-and-error behavior when the handshake fails.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 42e15ab9-e889-405f-8ede-52842dd30e3e
📒 Files selected for processing (2)
reaper.goreaper_test.go
| if err := r.handshake(conn); err != nil { | ||
| conn.Close() | ||
| return nil, fmt.Errorf("handshake reaper %s: %w", r.Endpoint, err) | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect whether the handshake path applies a connection deadline.
rg -n -C 5 'DialContext|SetDeadline|handshake\(conn\)' reaper.go reaper_test.goRepository: testcontainers/testcontainers-go
Length of output: 915
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '--- outline reaper.go ---\n'
ast-grep outline reaper.go --match connect --view expanded || true
printf '\n--- relevant reaper.go sections ---\n'
sed -n '500,580p' reaper.go | cat -n | sed 's/^/reaper.go /'
printf '\n--- handshake implementation references ---\n'
rg -n -C 8 'func \(r \*Reaper\) handshake|handshake\(' reaper.go
printf '\n--- connect/call references ---\n'
rg -n -C 7 'connect\(ctx\)|handshake\(conn\)|DialContext|ConfigContext|Connect' reaper.go reaper_test.go
printf '\n--- Go net Dialer docs/source in local tooling if available ---\n'
go env GOPATH GOROOT 2>/dev/null || true
mkdir -p /tmp/gosrc
GOSRC="$GOROOT/src"
[ -n "$GOSRC" ] && { rg -n 'func \(d \*Dialer\) DialContext|ctx\.Apply|SetDeadline' "$GOSRC/net" || true; }Repository: testcontainers/testcontainers-go
Length of output: 261
🌐 Web query:
Go net.Dialer.DialContext context timeout applies to dial only net.Conn Read Write deadline
💡 Result:
No, the context timeout passed to net.Dialer.DialContext does not apply to Read or Write operations on the established connection [1][2][3]. The context provided to DialContext is used exclusively for the connection establishment phase (the dial) [1][2][4]. Once the connection has been successfully established, the context's expiration or cancellation has no further effect on the connection [1][4]. To control timeouts for Read and Write operations after a connection is established, you must explicitly set deadlines on the net.Conn object using the SetDeadline, SetReadDeadline, or SetWriteDeadline methods [5][3]. Summary of behavior: 1. Dial Phase: The timeout is governed by the provided context and the Dialer's Timeout field [1][4]. If the context expires or is canceled during this time, the dial operation is aborted [1][4]. 2. Established Connection Phase: The context is no longer active for the connection [1][3]. You must manage I/O timeouts manually using the connection's deadline methods [5][3].
Citations:
- 1: https://pkg.go.dev/net
- 2: https://go.dev/src/net/example_test.go
- 3: https://forum.golangbridge.org/t/read-write-to-network-with-context/5758
- 4: https://pkg.go.dev/net@go1.26.3
- 5: https://stackoverflow.com/questions/63870817/does-setting-the-timeout-on-the-net-dialer-and-deadline-on-the-connection-result
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- relevant file and function info ---'
wc -l reaper.go reaper_test.go
rg -n -C 8 'func \(r \*Reaper\) connect|func \(r \*Reaper\) handshake|func \(r \*Reaper\) Connect|DialContext|handshake\(conn\)|SetDeadline|SetReadDeadline|SetWriteDeadline' reaper.go reaper_test.go
echo '--- connect section ---'
sed -n '515,570p' reaper.go
echo '--- handshake section ---'
rg -n 'handshake' reaper.go
awk '/func \(r \*Reaper\) handshake/ {p=1; f=NR} p; NR>=f && /^[[:space:]]*$/ && NR>f+1 {exit}' reaper.go | sed -n '1,160p'Repository: testcontainers/testcontainers-go
Length of output: 4534
🌐 Web query:
Go net.Dialer.DialContext context deadline cancels only dial TCP connection established Read deadline
💡 Result:
The context passed to net.Dialer.DialContext is used exclusively for the connection establishment phase [1][2][3]. Once a TCP connection is successfully established, any expiration or cancellation of that context has no effect on the connection's lifetime or its I/O operations [1][4]. Specifically, the net.Dialer.DialContext mechanism does not bind the context to the connection after it is returned [1]. If the context expires or is canceled after the dial has finished, the established net.Conn remains open and will continue to operate normally [1][4]. Consequently, the context does not automatically enforce any read or write deadlines on the established connection [5][2]. To enforce timeouts on an established connection, you must explicitly manage the connection's deadline using the net.Conn.SetDeadline, SetReadDeadline, or SetWriteDeadline methods [1][6][2]. Because these methods require a concrete connection object, a common pattern for handling timeouts on established connections is to wrap the net.Conn or to invoke these deadline methods directly after obtaining the connection [2][7]. Top results: [1][5][2][4]
Citations:
- 1: https://pkg.go.dev/net
- 2: https://forum.golangbridge.org/t/read-write-to-network-with-context/5758
- 3: net: clarify DialContext's lifetime golang/go#15325
- 4: [question] cancelled context is ignored? gorilla/websocket#825
- 5: net, net/http: Deadline and Timeout Fail to Close Hung Connections. golang/go#48125
- 6: https://stackoverflow.com/questions/63870817/does-setting-the-timeout-on-the-net-dialer-and-deadline-on-the-connection-result
- 7: https://gist.github.com/c4milo/275abc6eccbfd88ad56ca7c77947883a
Bound the synchronous handshake by the context deadline.
Connect applies the five-second timeout only to net.Dialer.DialContext; once the TCP connection is returned, r.handshake(conn) is no longer limited by that context. If Ryuk accepts the connection and does not send an ACK, the handshake can block indefinitely and prevent retries. Set a connection deadline before the handshake and clear it after the handshake succeeds.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@reaper.go` around lines 545 - 548, Update Connect around r.handshake(conn) to
apply the context deadline to the connection before starting the synchronous
handshake, so an unresponsive peer cannot block indefinitely. After a successful
handshake, clear the connection deadline before returning the connection;
preserve existing close-and-error behavior when the handshake fails.
1 participant
This supersedes #3828. The original PR was automatically closed after I accidentally deleted my fork
Related issues
What does this PR do?
This PR makes
Reaper.connectperform the Ryuk handshake synchronously before returning a successful connection.If the handshake fails, the connection is closed and the handshake error is returned to the caller. The existing retry path can then handle the failure instead of treating the reaper connection as successful.
It also adds a regression test using a local TCP listener that returns an invalid ACK, verifying that
connectreturns an error and no termination channel.Why is it important?
Previously, handshake failures were only logged inside the connection goroutine.
Reaper.connectstill returned a non-nil termination channel and nil error, so callers could believe the reaper was connected even though Ryuk had rejected or failed the handshake.Returning the error makes reaper startup failures visible and retryable.
How to test this PR