Skip to content

test: bound vault test-server startup so an unreachable VAULT_ADDR can't hang the suite - #1559

Open
Moep90 wants to merge 3 commits into
masterfrom
fix/vault-test-server-hang
Open

test: bound vault test-server startup so an unreachable VAULT_ADDR can't hang the suite#1559
Moep90 wants to merge 3 commits into
masterfrom
fix/vault-test-server-hang

Conversation

@Moep90

@Moep90 Moep90 commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

What

The vault test-server helper (tests/vault_server.py) polled the server in a while not initialized: loop with no timeout and except: continue. If the vault it talks to is unreachable, that loop never exits.

When VAULT_ADDR points at a vault that doesn't resolve/respond (common locally — a stale env var, VPN off, no Docker), the whole test suite hangs forever instead of failing. It bites in test_cli.py, test_vault.py and test_vault_transit.py, whose setUpClass builds the shared vault server.

Why

I hit this running the full suite locally: it stalled on test_cli.py::CliFuncsTest::test_cli_inventory for 20+ minutes. The process wasn't deadlocked — it was retrying a health check against an unreachable VAULT_ADDR in an unbounded loop, logging status is {} / NameResolutionError endlessly. CI never sees it because vault is reachable there.

Approach

  • vault_server.py: bound both startup loops (container-running wait and the health-check/init wait) with a deadline (KAPITAN_VAULT_SETUP_TIMEOUT, default 60s). On timeout raise VaultServerError instead of looping. Cache construction failures so each dependent test class skips instantly rather than re-waiting the full timeout.
  • test_cli / test_vault / test_vault_transit: catch VaultServerError in setUpClass and SkipTest — the same graceful-skip behaviour the reclass-rs tests already use when their backend is unavailable. Teardown guarded for the skipped case.
  • New tests/test_vault_server.py: regression test asserting that an unreachable VAULT_ADDR makes VaultServer() raise quickly instead of hanging.

Verification

  • Regression test: red→green (infinite hang → raises in ~4s).
  • Full non-slow suite on this branch with an unreachable VAULT_ADDR: 522 passed, 36 skipped in ~33s (previously hung indefinitely). The 35 vault tests skip with a clear reason.
  • ruff check / ruff format clean; pre-commit hooks pass.

CI is unaffected: vault is reachable there, so the loops exit immediately and nothing skips — behaviour is unchanged on CI.

@github-actions github-actions Bot added python Pull requests that update Python code size/m labels Jun 16, 2026
@Moep90 Moep90 self-assigned this Jun 16, 2026
@Moep90
Moep90 requested review from ademariag and ramaro June 16, 2026 10:03
@Moep90

Moep90 commented Jun 19, 2026

Copy link
Copy Markdown
Contributor Author

Relates to #1512 — this is the test-isolation/hang half of that test-infra issue (bounding the vault test-server startup).

…n't hang the suite

The vault test helper polled the server in a `while not initialized` loop with
no timeout and `except: continue`. When VAULT_ADDR points at an unreachable
vault (common locally: a stale env var, VPN off), the whole test suite hangs
forever instead of failing. It surfaces via test_cli/test_vault/
test_vault_transit, whose setUpClass builds the shared vault server.

- vault_server.py: bound both startup loops with a deadline
  (KAPITAN_VAULT_SETUP_TIMEOUT, default 60s) and raise VaultServerError on
  timeout. Cache construction failures so each dependent class skips instantly
  instead of re-waiting the timeout.
- test_cli / test_vault / test_vault_transit: skip cleanly when the vault
  server is unavailable, the same way reclass_rs tests already skip.
- add a regression test asserting an unreachable VAULT_ADDR raises fast.

CI is unaffected: vault is reachable there, so the loops exit immediately and
nothing skips. Locally the full (non-slow) suite now finishes in ~33s instead
of hanging.
@Moep90
Moep90 force-pushed the fix/vault-test-server-hang branch from 113ba04 to 24bc71b Compare June 19, 2026 17:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Pull requests that update Python code size/m

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant