Background
The #826 audit counted ~50+ #[ignore] annotations across the test suite. Most are legitimate (require attached hardware: a real ESP32, a Teensy, a USB CDC device) but some are stale — the underlying reason for the ignore was fixed and nobody removed the annotation. Without a tracking discipline, ignored tests rot until they no longer even compile.
Examples from #826
crates/fbuild-build/tests/teensy30_acceptance.rs — #[ignore = "requires teensy hardware"] (legitimate; hardware required)
crates/fbuild-build/tests/avr_build.rs — #[ignore] (reason unclear; needs investigation)
crates/fbuild-packages/tests/disk_cache_schema_migration.rs — #[ignore] (reason unclear)
(Full list to be enumerated by the script proposed below.)
Why this is NOT a dylint
This is a process concern, not a code-shape concern. The right tool is a periodic audit, not a per-commit lint.
Proposal
- Add
ci/audit_ignored_tests.py that walks the workspace, finds every #[ignore] (with and without a reason string), and emits a CSV: file,test_name,reason,line.
- Add a weekly GitHub Action (cron
0 0 * * MON) that runs the script and posts the diff vs. the previous week to a tracking issue (open one stable issue tracking: ignored-test inventory and update its body).
- Define the policy:
- Every
#[ignore] must have a reason string (#[ignore = \"...\"]).
- The reason must reference either a tracking issue (
#NNN) OR a hardware requirement (e.g. requires teensy 3.0).
- A test that's been ignored for 90+ days without a hardware reason gets escalated to a CodeRabbit review.
References
Background
The #826 audit counted ~50+
#[ignore]annotations across the test suite. Most are legitimate (require attached hardware: a real ESP32, a Teensy, a USB CDC device) but some are stale — the underlying reason for the ignore was fixed and nobody removed the annotation. Without a tracking discipline, ignored tests rot until they no longer even compile.Examples from #826
crates/fbuild-build/tests/teensy30_acceptance.rs—#[ignore = "requires teensy hardware"](legitimate; hardware required)crates/fbuild-build/tests/avr_build.rs—#[ignore](reason unclear; needs investigation)crates/fbuild-packages/tests/disk_cache_schema_migration.rs—#[ignore](reason unclear)(Full list to be enumerated by the script proposed below.)
Why this is NOT a dylint
This is a process concern, not a code-shape concern. The right tool is a periodic audit, not a per-commit lint.
Proposal
ci/audit_ignored_tests.pythat walks the workspace, finds every#[ignore](with and without a reason string), and emits a CSV:file,test_name,reason,line.0 0 * * MON) that runs the script and posts the diff vs. the previous week to a tracking issue (open one stable issuetracking: ignored-test inventoryand update its body).#[ignore]must have a reason string (#[ignore = \"...\"]).#NNN) OR a hardware requirement (e.g.requires teensy 3.0).References