fix(ci): the invisible-character gate never matched anything - #3
fix(ci): the invisible-character gate never matched anything#3hyperpolymath wants to merge 1 commit into
Conversation
MEASURED 2026-08-27: this gate's pattern caught 0 OF 6 invisible-character test
cases. It has never detected an NBSP, zero-width space, BOM, soft hyphen, bidi
override or word joiner.
ROOT CAUSE: the pattern used UTF-8 BYTE sequences (\xc2\xa0) while grep -P
matches CHARACTERS. Bytes c2 a0 are ONE character U+00A0; \xc2\xa0 asks for TWO
characters, U+00C2 then U+00A0, which is never present.
grep -P '\xc2\xa0' -> miss
grep -P '\x{a0}' -> MATCH
Only \x00 worked, being single-byte in both readings.
FIXED: codepoint escapes; C0 control characters \x01-\x08,\x0B,\x0C,\x0E-\x1F
added (TAB/LF/CR excluded); and grep -a, without which grep skips any NUL-bearing
file as binary.
The C0 range matters: a stray BACKSPACE byte made a workflow unparseable in
developer-ecosystem, so it never ran, and this linter called it clean.
Canonical fix: hyperpolymath/empty-linter#70. 1 file(s) here.
VERIFIED: YAML re-parsed, and the corrected pattern was confirmed to catch a real
NBSP before the change was kept.
Up to standards ✅🟢 Issues
|
![codacy-production[bot]](https://avatars.githubusercontent.com/in/56611?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull Request Overview
The PR successfully addresses the invisible-character detection issue in the CI pipeline. By switching from UTF-8 byte sequences to Unicode codepoint escapes and incorporating the -a flag, the gate now correctly processes files that were previously skipped (such as those containing null bytes or Byte Order Marks). Codacy analysis indicates the changes are up to standards.
While the implementation aligns with the requirements, there is an opportunity to optimize the scanning process. Batching file paths using the + terminator in the find command will significantly reduce execution time in larger repositories. Furthermore, several critical test scenarios (e.g., verifying detection of NBSP, ZWSP, and BOM) are currently missing from the automated suite and should be considered to prevent future regressions.
About this PR
- The automated test suite currently lacks verification for specific invisible characters (NBSP, ZWSP, BOM, and control characters). It is recommended to add regression tests covering these cases to ensure the gate remains functional as the environment evolves.
Test suggestions
- Identify a file containing a Non-breaking Space (U+00A0)
- Identify a file containing a Zero-width Space (U+200B)
- Identify a file containing a Byte Order Mark (BOM, U+FEFF)
- Identify a file containing a Backspace control character (\x08)
- Verify that a file containing a Null byte (\x00) is scanned and flagged rather than skipped as binary
Prompt proposal for missing tests
Consider implementing these tests if applicable:
1. Identify a file containing a Non-breaking Space (U+00A0)
2. Identify a file containing a Zero-width Space (U+200B)
3. Identify a file containing a Byte Order Mark (BOM, U+FEFF)
4. Identify a file containing a Backspace control character (\x08)
5. Verify that a file containing a Null byte (\x00) is scanned and flagged rather than skipped as binary
TIP Improve review quality by adding custom instructions
TIP How was this review? Give us feedback
| -o -name '*.idr' -o -name '*.zig' -o -name '*.v' -o -name '*.jl' \ | ||
| -o -name '*.gleam' -o -name '*.hs' -o -name '*.ml' -o -name '*.sh' \) \ | ||
| -exec grep -Prl "$PATTERNS" {} \; > /tmp/empty-lint-results.txt 2>/dev/null | ||
| -exec grep -aPrl "$PATTERNS" {} \; > /tmp/empty-lint-results.txt 2>/dev/null |
There was a problem hiding this comment.
🟡 MEDIUM RISK
Suggestion: Using -exec ... {} + instead of -exec ... {} ; is significantly more efficient because it passes multiple file paths to a single grep process, reducing the overhead of spawning a new process for every file. Additionally, the -r (recursive) flag is redundant when using find for traversal, as find already provides individual file paths. The use of the -a flag is excellent as it ensures grep processes files with NUL bytes rather than ignoring them.
| -exec grep -aPrl "$PATTERNS" {} \; > /tmp/empty-lint-results.txt 2>/dev/null | |
| -exec grep -aPl "$PATTERNS" {} + > /tmp/empty-lint-results.txt 2>/dev/null |
1 participant
Measured 2026-08-27: this gate caught 0 of 6 invisible-character test cases. It has never detected an NBSP, zero-width space, BOM, soft hyphen, bidi override or word joiner.
Root cause
The pattern used UTF-8 byte sequences (
\xc2\xa0) whilegrep -Pmatches characters. Bytesc2 a0are one character U+00A0;\xc2\xa0asks for two, U+00C2 then U+00A0 — never present.Only
\x00worked, being single-byte in both readings. The gate ran, passed, and could not see what it exists to see.Fixed
\x01-\x08,\x0B,\x0C,\x0E-\x1Fadded (TAB/LF/CR excluded)grep -a— without it grep skips any NUL-bearing file as binaryThe C0 range matters: a stray backspace byte made a workflow unparseable in
developer-ecosystem, so it never ran — and this linter called it clean.Canonical fix: hyperpolymath/empty-linter#70. 1 file(s) here.
Verified: YAML re-parsed, and the corrected pattern was confirmed to catch a real NBSP before the change was kept.