Skip to content

process: ignored-test policy + bitrot audit (#826 followup) #839

Description

@zackees

Background

The #826 audit counted ~50+ #[ignore] annotations across the test suite. Most are legitimate (require attached hardware: a real ESP32, a Teensy, a USB CDC device) but some are stale — the underlying reason for the ignore was fixed and nobody removed the annotation. Without a tracking discipline, ignored tests rot until they no longer even compile.

Examples from #826

  • crates/fbuild-build/tests/teensy30_acceptance.rs#[ignore = "requires teensy hardware"] (legitimate; hardware required)
  • crates/fbuild-build/tests/avr_build.rs#[ignore] (reason unclear; needs investigation)
  • crates/fbuild-packages/tests/disk_cache_schema_migration.rs#[ignore] (reason unclear)

(Full list to be enumerated by the script proposed below.)

Why this is NOT a dylint

This is a process concern, not a code-shape concern. The right tool is a periodic audit, not a per-commit lint.

Proposal

  1. Add ci/audit_ignored_tests.py that walks the workspace, finds every #[ignore] (with and without a reason string), and emits a CSV: file,test_name,reason,line.
  2. Add a weekly GitHub Action (cron 0 0 * * MON) that runs the script and posts the diff vs. the previous week to a tracking issue (open one stable issue tracking: ignored-test inventory and update its body).
  3. Define the policy:
    • Every #[ignore] must have a reason string (#[ignore = \"...\"]).
    • The reason must reference either a tracking issue (#NNN) OR a hardware requirement (e.g. requires teensy 3.0).
    • A test that's been ignored for 90+ days without a hardware reason gets escalated to a CodeRabbit review.

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    Status
    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions