test: bound vault test-server startup so an unreachable VAULT_ADDR can't hang the suite - #1559
Open
Moep90 wants to merge 3 commits into
Open
test: bound vault test-server startup so an unreachable VAULT_ADDR can't hang the suite#1559Moep90 wants to merge 3 commits into
Moep90 wants to merge 3 commits into
Conversation
Contributor
Author
|
Relates to #1512 — this is the test-isolation/hang half of that test-infra issue (bounding the vault test-server startup). |
…n't hang the suite The vault test helper polled the server in a `while not initialized` loop with no timeout and `except: continue`. When VAULT_ADDR points at an unreachable vault (common locally: a stale env var, VPN off), the whole test suite hangs forever instead of failing. It surfaces via test_cli/test_vault/ test_vault_transit, whose setUpClass builds the shared vault server. - vault_server.py: bound both startup loops with a deadline (KAPITAN_VAULT_SETUP_TIMEOUT, default 60s) and raise VaultServerError on timeout. Cache construction failures so each dependent class skips instantly instead of re-waiting the timeout. - test_cli / test_vault / test_vault_transit: skip cleanly when the vault server is unavailable, the same way reclass_rs tests already skip. - add a regression test asserting an unreachable VAULT_ADDR raises fast. CI is unaffected: vault is reachable there, so the loops exit immediately and nothing skips. Locally the full (non-slow) suite now finishes in ~33s instead of hanging.
Moep90
force-pushed
the
fix/vault-test-server-hang
branch
from
June 19, 2026 17:53
113ba04 to
24bc71b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The vault test-server helper (
tests/vault_server.py) polled the server in awhile not initialized:loop with no timeout andexcept: continue. If the vault it talks to is unreachable, that loop never exits.When
VAULT_ADDRpoints at a vault that doesn't resolve/respond (common locally — a stale env var, VPN off, no Docker), the whole test suite hangs forever instead of failing. It bites intest_cli.py,test_vault.pyandtest_vault_transit.py, whosesetUpClassbuilds the shared vault server.Why
I hit this running the full suite locally: it stalled on
test_cli.py::CliFuncsTest::test_cli_inventoryfor 20+ minutes. The process wasn't deadlocked — it was retrying a health check against an unreachableVAULT_ADDRin an unbounded loop, loggingstatus is {}/NameResolutionErrorendlessly. CI never sees it because vault is reachable there.Approach
vault_server.py: bound both startup loops (container-running wait and the health-check/init wait) with a deadline (KAPITAN_VAULT_SETUP_TIMEOUT, default60s). On timeout raiseVaultServerErrorinstead of looping. Cache construction failures so each dependent test class skips instantly rather than re-waiting the full timeout.test_cli/test_vault/test_vault_transit: catchVaultServerErrorinsetUpClassandSkipTest— the same graceful-skip behaviour the reclass-rs tests already use when their backend is unavailable. Teardown guarded for the skipped case.tests/test_vault_server.py: regression test asserting that an unreachableVAULT_ADDRmakesVaultServer()raise quickly instead of hanging.Verification
VAULT_ADDR: 522 passed, 36 skipped in ~33s (previously hung indefinitely). The 35 vault tests skip with a clear reason.ruff check/ruff formatclean; pre-commit hooks pass.CI is unaffected: vault is reachable there, so the loops exit immediately and nothing skips — behaviour is unchanged on CI.