Skip to content

fix(test): make verify --write-verdicts assertion terminal-width independent#85

Merged
VGonPa merged 1 commit into
developfrom
fix/flaky-verify-cli-test
Jul 7, 2026
Merged

fix(test): make verify --write-verdicts assertion terminal-width independent#85
VGonPa merged 1 commit into
developfrom
fix/flaky-verify-cli-test

Conversation

@VGonPa

@VGonPa VGonPa commented Jul 7, 2026

Copy link
Copy Markdown
Owner

Problem

tests/test_cli.py::test_verify_write_verdicts_without_apply_errors passed locally but went CI-red on develop (commit cf99c21):

assert '--apply' in result.output
E  assert '--apply' in "\x1b[33mUsage: \x1b[0mroot verify [OPTIONS]\n ... ╰────╯\n"
1 failed, 1225 passed

Root cause (reproduced locally)

The test asserted the literal substring "--apply" in a Typer/Rich-formatted usage box. When color is enabled — CI runs on a color-capable terminal; pytest's captured output is colorless — Rich styles each flag with its leading dash in a separate ANSI span (\x1b[1;36m-\x1b[0m\x1b[1;36m-apply\x1b[0m), so the literal --apply never appears contiguously. The 80-col box also wraps the message across lines. Hence green locally (no color) / red in CI (color on).

Reproduced deterministically with FORCE_COLOR=1 COLUMNS=80 — same \x1b[33mUsage: … ╰────╯ output as CI. The feature is correct: result.exit_code != 0 already passes (the CLI rightly rejects --write-verdicts without --apply); only the brittle output-substring check broke.

Fix

  • New _plain_output() test helper: strips ANSI escapes (rejoins the split flag), drops Rich box-drawing chrome (│ ╭ ╮ ╰ ╯ ─, so words wrapped across box lines rejoin), and collapses whitespace.
  • The target test now asserts the code-controlled BadParameter message "--write-verdicts requires --apply" on normalized output — terminal-width and color independent.

Sibling sweep

Swept the verify / audit / video-digest / badge CLI tests (test_cli.py, test_verification*.py, test_video_digest.py, test_generate.py). Stress-running the suite under FORCE_COLOR=1 at COLUMNS=80 and COLUMNS=30 surfaced no other width/ANSI-fragile assertion — the audit ValueError messages are echoed plainly (not Rich-panel/flag-styled). One pre-existing width-safe assertion (test_digest_video_vision_model_without_frames_errors) had an inline copy of the same normalization; refactored it to reuse the shared helper (1 sibling touched, quality only — it was already green).

Verification

  • Target test now passes under COLUMNS=80 (the CI condition), COLUMNS=80 FORCE_COLOR=1, COLUMNS=30 FORCE_COLOR=1, and wide terminals.
  • Full gate uv run --extra dev poe check: all 11 checks PASS — 1226 tests, 95% coverage, Radon all A/B, mypy/ruff/bandit/detect-secrets clean.

🤖 Generated with Claude Code

…pendent

test_verify_write_verdicts_without_apply_errors asserted the literal
substring "--apply" in the CLI output. With color enabled (CI has a
color-capable terminal; pytest's captured output does not), Rich styles
each flag with its leading dash in a separate ANSI span
("-" + "-apply"), so "--apply" never appears contiguously, and the
80-col usage box wraps the message across lines. The literal check
therefore passed locally but failed red in CI.

Assert the code-controlled BadParameter message
("--write-verdicts requires --apply") against normalized output: a new
_plain_output() helper strips ANSI escapes, drops Rich box-drawing
chrome, and collapses whitespace, making the check independent of
terminal width and color. Passes under both wide and narrow (COLUMNS=80,
even COLUMNS=30) terminals. The feature is unchanged; only the brittle
output-substring check was hardened.

Also refactored the pre-existing width-safe assertion in
test_digest_video_vision_model_without_frames_errors to reuse the shared
_plain_output() helper (it had an inline copy of the same normalization).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@VGonPa VGonPa merged commit f440ee8 into develop Jul 7, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant