Skip to content

[Go] Verify expansion service readiness before use - #40859

Draft
bvolpato wants to merge 3 commits into
apache:masterfrom
bvolpato:bvolpato/go-portable-service-readiness
Draft

bvolpato wants to merge 3 commits into
apache:masterfrom
bvolpato:bvolpato/go-portable-service-readiness

Conversation

@bvolpato

@bvolpato bvolpato commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Expansion service startup currently accepts any TCP listener on the selected port. Another process can claim that port before the Java jar binds, leaving the helper with an unrelated service address. In this CI run, the failed expansion address also appeared as a Python worker endpoint.

Check the expansion RPC before caching the address, detect an exited jar, and clean up failed startup. A wrong-service response gets a bounded retry on a fresh port. Expand-only services remain supported through a single probe within the startup deadline. The probe uses a fixed reconnect delay because the default gRPC backoff can outlast the startup deadline.

Reproduced the old helper accepting an unrelated gRPC listener and a real jar failing to bind the occupied port. Focused tests, the race detector, vet, and the Windows process-wrapper build passed. Real-jar startup, early exit, and startup-deadline cleanup were also checked. A local measurement with services that bound late in the startup deadline failed often with the default backoff and passed with the fixed delay. The full GoPortable container workflow was not run locally.

@github-actions github-actions Bot added go size/m M: 50-249 lines changed labels Oct 28, 2026
@bvolpato
bvolpato force-pushed the bvolpato/go-portable-service-readiness branch from 672250f to ca16bb3 Compare October 28, 2026 23:57
The readiness probe uses one gRPC channel for all attempts. The default reconnect backoff grows longer than the polling interval, so the probe can miss a service that binds late in the 30 second startup deadline. The TCP poll that the probe replaced did not have this delay.

In a local measurement, services that bound 26 to 28 seconds after the start often failed with the default backoff. The same measurement passed with a fixed 250 ms delay, and the wait after a bind dropped from up to 10 seconds to about 1 second.
The test helper binds the port that GetAddr selects, but the port is not reserved between selection and bind. Another process can take the port first. GetAddr handles this race, but the helper failed the test, so TestExpansionServices_GetAddr_RetriesWrongService failed on a busy machine.

Skip the test in this case because the test did not reach the behavior that it checks.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

go size/m M: 50-249 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant