Skip to content

fix(wired): see the links that fail slowly, and the multi-gig ones that fall to 1G - #33

Merged
drench44 merged 1 commit into
gneitzke:mainfrom
drench44:fix/wired-flapping-and-downshift-detection
Aug 7, 2026
Merged

fix(wired): see the links that fail slowly, and the multi-gig ones that fall to 1G#33
drench44 merged 1 commit into
gneitzke:mainfrom
drench44:fix/wired-flapping-and-downshift-detection

Conversation

@drench44

@drench44 drench44 commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Two blind spots in the wired detectors, both found the hard way: a port on a Mini Switch 2.5G was dropping a work computer's ethernet all day, and the tool reported nothing about it.

Review band: three agents (code-reviewer + silent-failure-hunter + pr-test-analyzer) — the diff touches tests/netadmin/detect/test_unit_convention.py, a convention-enforcing CI gate, and netadmin/store/repository.py's retention path. Their findings are folded in; the notable ones are called out below.

wired.port_flapping — a 24 h tier

It had a 10-minute tier (≥5 transitions) and a 1-hour tier (≥10), so it could only see a link failing fast. The port in question dropped 55 times in two weeks, 16 in one day, and never fired once — no single 10-minute window ever held 5 transitions.

That slow-burn shape is what a marginal cable or a power-managed NIC actually produces, and it is just as disruptive to the person using the port, so the new tier carries the same severity (P2, P1 on infra).

Also fixes the title. It always quoted the 10-minute count, so a long- or daily-tier trip rendered as (0 transitions/10m) — there is a real issue in a live store that reads exactly that, and it makes a true finding look like a bug. The title now names the tightest tier that actually tripped.

wired.bad_cable — an observed-ceiling arm

The downshift check returned early on any speed >= 1000, correct when gigabit was the ceiling. On a 2.5G/10G port a fall to 1000 is the same broken-pair symptom and was simply unreportable.

It cannot just compare against the rated ceiling — 1000 is unremarkable for a 2.5G port carrying a 1G device — so it compares against the port's own history: a link that has held 2500 proves its peer can do 2500.

Four things make that inference safe. Three of them came from review or dogfooding, not from the tests:

  • Time-held, not row count. record_state_change writes only on change, so a port that ran a week at 2500 carries exactly one row saying 2500, dated a week ago. Counting rows inside the window finds nothing, and the commonest shape of all — damaged once, renegotiated down once, sitting there since — reports nothing. The value in effect entering the window seeds the timeline. prune() now preserves that boundary row per (entity, attr) for the same reason it already preserves the latest one: both answer "what was the value at time T" for a T still served. Without it the arm silently stopped reporting after 30 days.
  • Fetched by window, never by row count. A count-limited fetch drops the oldest rows first — exactly where the seed lives — so the harder a link flapped, the more certainly its ceiling was truncated away.
  • Peer age compared against when the speed was held, not against the window start. The latter is wrong in both directions: it rejects a peer that demonstrably ran fast mid-window, and on any store younger than the lookback it reads every peer as a newcomer and silently disables the arm site-wide. Where several peers claim one port the newest wins, so a departed device's lingering entity cannot vouch for its replacement.
  • The device-class list is overruled only by a dominant ceiling — one held longer than the current speed. Dogfooding caught the alternative: a real G6 Turret, 10/100 by Ubiquiti's spec and working correctly, blipped to 1000 six times inside one 20-minute cabling event and then sat at 100 for five days. Twelve minutes of anomaly must not outvote five days of correct operation and send someone to re-run cable to a healthy camera. Thirty days at gigabit against two hours at 100 still wins.

Also fixed

  • The peer-age confounder no longer leaks onto findings whose downshift arm was suppressed. confounders is the shared list handed to every finding, and the LLM dossier prints every key unconditionally — so an error-rate finding shipped claiming the detector had "tested and rejected" a trap when it had concluded the opposite.
  • The flapping confounder note guards its new clause. Issues predating the tier never get their evidence refreshed, so an unguarded f-string rendered "unknown in unknown" on every resolved issue and on the public demo seed.
  • _as_int/_as_float reject non-finite input. json.loads accepts Infinity by default and port.meta comes straight from stored controller JSON, so one malformed field raised an uncaught OverflowError out of the detector — blinding wired.bad_cable for every port on the site for that pass.

Verification

Run against a live site DB, all 46 ports across 6 switches:

  • port_flapping fires P2 on exactly the offending port — 38 transitions/24h, with transitions_short: 0, transitions_long: 2 in the evidence, the counts that show precisely why it was missed — and on no other port.
  • A point-in-time replay of the window when that port sat at 1000 fires bad_cable with observed_speed_max: 2500.
  • No other port on the site reports, the camera ports included.

2052 passed, 1 skipped (2024 on main → +28 new tests), 92.53% coverage, black/isort/flake8 clean. transitions_sustained is added to the same conscious-exemption list that already carries transitions_short/transitions_long; its sibling window_sustained_s carries the _s suffix and is not exempted.

🤖 Generated with Claude Code

…at fall to 1G

Two blind spots, both found by a real port that was disconnecting a work
computer all day while the tool reported nothing.

wired.port_flapping only had a 10-minute and a 1-hour tier, so it could
only see a link failing fast. The port in question dropped 55 times in
two weeks -- 16 in one day -- but never put 5 transitions in any single
10-minute window, so it never fired once. That slow-burn shape is what a
marginal cable or a power-managed NIC actually produces, and it is just
as disruptive to the person using the port, so it carries the same
severity. Adds a 24 h tier (>=12 transitions), and makes the title name
the tightest tier that tripped: the old title always quoted the
10-minute count, so a wide-window finding rendered as
"(0 transitions/10m)" and looked like a bug.

wired.bad_cable's downshift arm returned early on any speed >= 1000,
which was right when gigabit was the ceiling. On a 2.5G/10G port a fall
to 1000 is the same broken-pair symptom and was unreportable. It cannot
simply compare against the rated ceiling -- 1000 is unremarkable for a
2.5G port carrying a 1G device -- so it compares against the port's own
observed history: a link that has held 2500 proves its peer can do 2500.

Four things make that inference safe, three of them found by review or
by dogfooding rather than by the tests:

* Time-held, not row count. record_state_change writes only on change,
  so a port that ran a week at 2500 carries exactly ONE row saying 2500,
  dated a week ago. Counting rows inside the window finds nothing, and
  the commonest shape of all -- damaged once, renegotiated down once,
  sitting there since -- reports nothing. The value in effect entering
  the window seeds the timeline. prune() now preserves that boundary row
  per (entity, attr) for the same reason it already preserves the latest
  one: both answer "what was the value at time T" for a T still served.
  Without it the arm silently stopped reporting after 30 days.
* Fetched by window, never by row count. A count-limited fetch drops the
  oldest rows first -- exactly where the seed lives -- so the harder a
  link flapped, the more certainly its ceiling was truncated away.
* Peer age compared against when the speed was HELD, not against the
  window start. The latter is wrong in both directions: it rejects a
  peer that demonstrably ran fast mid-window, and on any store younger
  than the lookback it reads every peer as a newcomer and silently
  disables the arm site-wide. Where several peers claim one port the
  newest wins, so a departed device's lingering entity cannot vouch for
  its replacement.
* An observed ceiling overrules the 10/100 device-class list only when
  it DOMINATES -- held longer than the current speed. Dogfooding caught
  the alternative: a real G6 Turret, 10/100 by Ubiquiti's spec and
  working correctly, blipped to 1000 six times inside one 20-minute
  cabling event and then sat at 100 for five days. Twelve minutes of
  anomaly must not outvote five days of correct operation and send
  someone to re-run cable to a healthy camera.

Also: the peer-age confounder no longer leaks onto findings whose
downshift arm was suppressed (the dossier prints every confounder key
unconditionally, so it read as a trap the detector had "tested and
rejected" when it had concluded the opposite); the flapping confounder
note guards its new clause, so issues predating the tier -- including
the demo seed and every resolved issue -- stop rendering "unknown in
unknown"; and _as_int/_as_float reject non-finite input, since
json.loads accepts Infinity by default and one malformed port field
raised OverflowError out of the detector, blinding wired.bad_cable for
every port on the site.

Verified against the live site DB across all 46 ports on all 6
switches: port_flapping fires P2 on exactly the offending port (38
transitions/24h, short=0, long=2 -- the counts that show why it was
missed) and nowhere else; a point-in-time replay of the window when
that port sat at 1000 fires bad_cable with observed_speed_max 2500; and
no other port on the site reports, the camera ports included.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@drench44
drench44 force-pushed the fix/wired-flapping-and-downshift-detection branch from b4fc37a to e6c4959 Compare August 7, 2026 21:18
@drench44
drench44 merged commit 3740ecf into gneitzke:main Aug 7, 2026
5 checks passed
@drench44
drench44 deleted the fix/wired-flapping-and-downshift-detection branch August 7, 2026 21:57
gneitzke added a commit that referenced this pull request Aug 9, 2026
…ect pass

Fixes a foreign-key crash (GitHub #34) where deleting an unconfirmed issue that
correlation had rooted an incident on aborted the entire daily detection run,
every night. Also carries the wired-detection fixes from #33 (slow-burn port
flapping, multi-gig downshift) that landed on main after 0.7.13, and makes a
pass-level failure record its cause on the poll_runs row instead of only the log.

Version synced across all four files (pyproject, both Dockerfiles, addon config).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011zVbf7n8a6XazfxPg1vn2F
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant