Skip to content

test: guard schema-probe allowlist against format drift - #312

Open
millsmillsymills wants to merge 1 commit into
mainfrom
issue-293-allowlist-drift-guard
Open

test: guard schema-probe allowlist against format drift#312
millsmillsymills wants to merge 1 commit into
mainfrom
issue-293-allowlist-drift-guard

Conversation

@millsmillsymills

Copy link
Copy Markdown
Collaborator

Adds tests/unit/test_schema_probe_allowlist.py, a guard test for the implicit coupling between compute_schema_drift (src/unraid_mcp/clients/unraid.py) and the fail-closed allowlist in the scrub step of .github/workflows/schema-probe.yml. Nothing in either file references the other, so a change to the drift message format desyncs the allowlist and the scrub then withholds every genuine drift report.

The test lifts four shell fragments out of the workflow verbatim — the field_list, allow_missing and allow_type assignments plus the unexpected= line carrying the grep -vE invocation — orders them by position in the file, and runs them under set -euo pipefail with the candidate lines on stdin. The patterns are therefore checked by a real shell rather than by a Python copy of them, and a change to the alternation or to the assignment order fails the test rather than being transcribed away. The CI run on ubuntu-latest is the authoritative one; a local run uses whatever grep is on PATH.

Coverage:

  • Every drift line the shipped SCHEMA_EXPECTATIONS can produce passes the scrub. Three probe results span the output space: no type present, every type present but empty, and every type missing its alphabetically first field.
  • The corpus contains both drift formats, so an all-pass result cannot be vacuous.
  • A type with an empty expected field set (the [] list repr, reachable only through the type-missing branch) passes.
  • Each drift format has its own set of rejected lookalikes. allow_missing rejects every missing fields lookalike on its own, so a corpus of those alone would leave allow_type with acceptance coverage only.
  • A single unexpected line mixed in with valid drift is reported.
  • _scrub_rejects asserts the shell wrote nothing to stderr, because a malformed pattern makes grep exit 2 with empty stdout, which the workflow's || true swallows.

No source or workflow changes.

Closes #293

🤖 Generated with Claude Code

The scrub step in .github/workflows/schema-probe.yml validates probe.log
against two regexes that encode the literal output of
compute_schema_drift, with no reference between the two files. A change
to the drift message format desyncs the allowlist and the scrub then
withholds every genuine drift report.

The test lifts the scrub's shell fragments verbatim from the workflow
and runs them, asserting that every drift line SCHEMA_EXPECTATIONS can
produce is accepted and that lookalikes of both drift formats are
rejected.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Add a drift-format guard test for the schema-probe allowlist

1 participant