fix(wired): see the links that fail slowly, and the multi-gig ones that fall to 1G - #33
Merged
drench44 merged 1 commit intoAug 7, 2026
Conversation
…at fall to 1G Two blind spots, both found by a real port that was disconnecting a work computer all day while the tool reported nothing. wired.port_flapping only had a 10-minute and a 1-hour tier, so it could only see a link failing fast. The port in question dropped 55 times in two weeks -- 16 in one day -- but never put 5 transitions in any single 10-minute window, so it never fired once. That slow-burn shape is what a marginal cable or a power-managed NIC actually produces, and it is just as disruptive to the person using the port, so it carries the same severity. Adds a 24 h tier (>=12 transitions), and makes the title name the tightest tier that tripped: the old title always quoted the 10-minute count, so a wide-window finding rendered as "(0 transitions/10m)" and looked like a bug. wired.bad_cable's downshift arm returned early on any speed >= 1000, which was right when gigabit was the ceiling. On a 2.5G/10G port a fall to 1000 is the same broken-pair symptom and was unreportable. It cannot simply compare against the rated ceiling -- 1000 is unremarkable for a 2.5G port carrying a 1G device -- so it compares against the port's own observed history: a link that has held 2500 proves its peer can do 2500. Four things make that inference safe, three of them found by review or by dogfooding rather than by the tests: * Time-held, not row count. record_state_change writes only on change, so a port that ran a week at 2500 carries exactly ONE row saying 2500, dated a week ago. Counting rows inside the window finds nothing, and the commonest shape of all -- damaged once, renegotiated down once, sitting there since -- reports nothing. The value in effect entering the window seeds the timeline. prune() now preserves that boundary row per (entity, attr) for the same reason it already preserves the latest one: both answer "what was the value at time T" for a T still served. Without it the arm silently stopped reporting after 30 days. * Fetched by window, never by row count. A count-limited fetch drops the oldest rows first -- exactly where the seed lives -- so the harder a link flapped, the more certainly its ceiling was truncated away. * Peer age compared against when the speed was HELD, not against the window start. The latter is wrong in both directions: it rejects a peer that demonstrably ran fast mid-window, and on any store younger than the lookback it reads every peer as a newcomer and silently disables the arm site-wide. Where several peers claim one port the newest wins, so a departed device's lingering entity cannot vouch for its replacement. * An observed ceiling overrules the 10/100 device-class list only when it DOMINATES -- held longer than the current speed. Dogfooding caught the alternative: a real G6 Turret, 10/100 by Ubiquiti's spec and working correctly, blipped to 1000 six times inside one 20-minute cabling event and then sat at 100 for five days. Twelve minutes of anomaly must not outvote five days of correct operation and send someone to re-run cable to a healthy camera. Also: the peer-age confounder no longer leaks onto findings whose downshift arm was suppressed (the dossier prints every confounder key unconditionally, so it read as a trap the detector had "tested and rejected" when it had concluded the opposite); the flapping confounder note guards its new clause, so issues predating the tier -- including the demo seed and every resolved issue -- stop rendering "unknown in unknown"; and _as_int/_as_float reject non-finite input, since json.loads accepts Infinity by default and one malformed port field raised OverflowError out of the detector, blinding wired.bad_cable for every port on the site. Verified against the live site DB across all 46 ports on all 6 switches: port_flapping fires P2 on exactly the offending port (38 transitions/24h, short=0, long=2 -- the counts that show why it was missed) and nowhere else; a point-in-time replay of the window when that port sat at 1000 fires bad_cable with observed_speed_max 2500; and no other port on the site reports, the camera ports included. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
drench44
force-pushed
the
fix/wired-flapping-and-downshift-detection
branch
from
August 7, 2026 21:18
b4fc37a to
e6c4959
Compare
gneitzke
added a commit
that referenced
this pull request
Aug 9, 2026
…ect pass Fixes a foreign-key crash (GitHub #34) where deleting an unconfirmed issue that correlation had rooted an incident on aborted the entire daily detection run, every night. Also carries the wired-detection fixes from #33 (slow-burn port flapping, multi-gig downshift) that landed on main after 0.7.13, and makes a pass-level failure record its cause on the poll_runs row instead of only the log. Version synced across all four files (pyproject, both Dockerfiles, addon config). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011zVbf7n8a6XazfxPg1vn2F
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two blind spots in the wired detectors, both found the hard way: a port on a Mini Switch 2.5G was dropping a work computer's ethernet all day, and the tool reported nothing about it.
Review band: three agents (code-reviewer + silent-failure-hunter + pr-test-analyzer) — the diff touches
tests/netadmin/detect/test_unit_convention.py, a convention-enforcing CI gate, andnetadmin/store/repository.py's retention path. Their findings are folded in; the notable ones are called out below.wired.port_flapping— a 24 h tierIt had a 10-minute tier (≥5 transitions) and a 1-hour tier (≥10), so it could only see a link failing fast. The port in question dropped 55 times in two weeks, 16 in one day, and never fired once — no single 10-minute window ever held 5 transitions.
That slow-burn shape is what a marginal cable or a power-managed NIC actually produces, and it is just as disruptive to the person using the port, so the new tier carries the same severity (P2, P1 on infra).
Also fixes the title. It always quoted the 10-minute count, so a long- or daily-tier trip rendered as
(0 transitions/10m)— there is a real issue in a live store that reads exactly that, and it makes a true finding look like a bug. The title now names the tightest tier that actually tripped.wired.bad_cable— an observed-ceiling armThe downshift check returned early on any speed
>= 1000, correct when gigabit was the ceiling. On a 2.5G/10G port a fall to 1000 is the same broken-pair symptom and was simply unreportable.It cannot just compare against the rated ceiling — 1000 is unremarkable for a 2.5G port carrying a 1G device — so it compares against the port's own history: a link that has held 2500 proves its peer can do 2500.
Four things make that inference safe. Three of them came from review or dogfooding, not from the tests:
record_state_changewrites only on change, so a port that ran a week at 2500 carries exactly one row saying 2500, dated a week ago. Counting rows inside the window finds nothing, and the commonest shape of all — damaged once, renegotiated down once, sitting there since — reports nothing. The value in effect entering the window seeds the timeline.prune()now preserves that boundary row per(entity, attr)for the same reason it already preserves the latest one: both answer "what was the value at time T" for a T still served. Without it the arm silently stopped reporting after 30 days.Also fixed
confoundersis the shared list handed to every finding, and the LLM dossier prints every key unconditionally — so an error-rate finding shipped claiming the detector had "tested and rejected" a trap when it had concluded the opposite._as_int/_as_floatreject non-finite input.json.loadsacceptsInfinityby default andport.metacomes straight from stored controller JSON, so one malformed field raised an uncaughtOverflowErrorout of the detector — blindingwired.bad_cablefor every port on the site for that pass.Verification
Run against a live site DB, all 46 ports across 6 switches:
port_flappingfires P2 on exactly the offending port —38 transitions/24h, withtransitions_short: 0, transitions_long: 2in the evidence, the counts that show precisely why it was missed — and on no other port.bad_cablewithobserved_speed_max: 2500.2052 passed, 1 skipped(2024 on main → +28 new tests), 92.53% coverage, black/isort/flake8 clean.transitions_sustainedis added to the same conscious-exemption list that already carriestransitions_short/transitions_long; its siblingwindow_sustained_scarries the_ssuffix and is not exempted.🤖 Generated with Claude Code