Corrected from the original report. I first described this as the request hanging. It isn't: the request succeeds and the test body passes. The hang is in test teardown, and it looks macOS-specific.
With a Go dynamic module loaded, ~IntegrationTestServer never returns. sample on the stuck process shows the main thread parked here:
~DynamicModulesIntegrationTest -> ~HttpIntegrationTest -> ~BaseIntegrationTest
-> ~IntegrationTestServerImpl -> IntegrationTestServer::~IntegrationTestServer()
-> Thread::PosixThread::join() -> _pthread_join -> __ulock_wait
and the thread it is waiting on parked here:
thread_start -> _pthread_start -> _pthread_exit -> _pthread_tsd_cleanup
-> _sigtramp -> ??? (in <unknown binary>)
So the server thread is exiting, macOS runs its pthread thread-specific-data destructors, one of them enters the loaded Go module, and it never comes back. pthread_join then blocks forever.
Repro: any test with a direct_response route and a Go module filter, which is what I was adding for #46901. It also reproduces with the in-tree passthrough filter, so it has nothing to do with what the filter does — only that a Go module is loaded. rust, rust_static and cpp on the identical test finish in under a second. Go is the only one of the three SDKs whose runtime installs pthread TSD destructors, which fits.
Two things suggest this is macOS-only and test-only rather than a production problem. A Go dynamic module in a long-running Linux Envoy container handles the same direct_response route fine over both HTTP/1.1 and HTTP/2 — but that process never tears a server thread down with the module loaded, so it does not exercise this path. And CI is Linux, so this likely never shows up there.
Not sure what the right fix is. Possibly the server thread should not be joined while a dynamic module holding thread-local state is still loaded, or the module should be unloaded first, or this is a Go runtime interaction that Envoy can only work around.
Repro harness: https://github.com/derekargueta/envoy — the test is in the #46902 branch, which skips the go param on Apple platforms and cites this issue.
main @ 8ea3b35, macOS arm64.
Corrected from the original report. I first described this as the request hanging. It isn't: the request succeeds and the test body passes. The hang is in test teardown, and it looks macOS-specific.
With a Go dynamic module loaded,
~IntegrationTestServernever returns.sampleon the stuck process shows the main thread parked here:and the thread it is waiting on parked here:
So the server thread is exiting, macOS runs its pthread thread-specific-data destructors, one of them enters the loaded Go module, and it never comes back.
pthread_jointhen blocks forever.Repro: any test with a
direct_responseroute and a Go module filter, which is what I was adding for #46901. It also reproduces with the in-treepassthroughfilter, so it has nothing to do with what the filter does — only that a Go module is loaded.rust,rust_staticandcppon the identical test finish in under a second. Go is the only one of the three SDKs whose runtime installs pthread TSD destructors, which fits.Two things suggest this is macOS-only and test-only rather than a production problem. A Go dynamic module in a long-running Linux Envoy container handles the same
direct_responseroute fine over both HTTP/1.1 and HTTP/2 — but that process never tears a server thread down with the module loaded, so it does not exercise this path. And CI is Linux, so this likely never shows up there.Not sure what the right fix is. Possibly the server thread should not be joined while a dynamic module holding thread-local state is still loaded, or the module should be unloaded first, or this is a Go runtime interaction that Envoy can only work around.
Repro harness: https://github.com/derekargueta/envoy — the test is in the #46902 branch, which skips the
goparam on Apple platforms and cites this issue.main @ 8ea3b35, macOS arm64.