Skip to content

fix(check): stop rebooting the router on transient probe failures - #7

Merged
prorochestvo merged 3 commits into
mainfrom
fix/6-connectivity-false-positives
Aug 4, 2026
Merged

fix(check): stop rebooting the router on transient probe failures#7
prorochestvo merged 3 commits into
mainfrom
fix/6-connectivity-false-positives

Conversation

@prorochestvo

Copy link
Copy Markdown
Owner

Closes #6

What the data showed

Analysed a read-only snapshot of the production database (2026-07-14 → 2026-08-04:
6140 runs at a 5-minute cadence, 24 735 checks). Full write-up in
the issue comment.

  • 29 runs were judged "internet down" → 11 reboots executed, 4 reboot attempts failed,
    14 skipped by cooldown.
  • In 28 of the 29 the raw uplink was provably alive: both ip targets connected in
    19–1118 ms, and the ping collector recorded 4/4 ICMP replies from 8.8.8.8, 1.1.1.1
    and the ISP gateway in the same runs. Only the two HTTPS domain probes failed, both
    at exactly the 5 s CHECK_TIMEOUT.
  • The 29th (2026-07-31 02:55) lost every target for a single sample and healed by the
    next run; its reboot attempt failed and was not what fixed it.
  • Only one pair of consecutive down runs occurred in 21 days. Every other trigger was an
    isolated 5-minute sample between healthy runs.
  • A further 17 runs had exactly one domain probe time out and correctly stayed "up". At a
    ~0.6 % per-probe failure rate, independent failures would coincide ~0.2 times in 21
    days; 29 were observed, so the two probes fail from a common cause — they share one DNS
    path, in production through the very router being rebooted.

What changed

Graded verdict (domain.Verdict) — down (every ip target failed) is the only
state a reboot may follow. Every domain target failing while an ip target still
answers is degraded: recorded, announced, never rebooted. With no ip targets
configured the domain group remains the only evidence and still counts as down.

In-run confirmation — a down verdict is re-probed CHECK_CONFIRMATIONS times,
CHECK_CONFIRM_DELAY apart (persisted under a new confirm phase). One healthy sample
ends the run as unconfirmed and corrects the internet_ok written from the first
sample.

Cross-run corroboration — a reboot additionally requires the previous
REBOOT_MIN_STREAK - 1 runs to have seen the outage, via a new
RunRepository.RecentRuns. A run that falls short ends as unsustained. Fail-closed: an
unreadable history withholds the reboot.

Quieter Telegram — neither guard sends a message; the run row and a log line carry the
record. degraded and cooldown-skip notices are edge-triggered, so the five identical
cooldown notices between 01:50 and 03:05 in the issue screenshot become one.

Probe diagnostics — a failed domain probe now carries the request stage it stalled in
(stage=dns|connect|tls|request|response|body) via httptrace, so "context deadline
exceeded" stops being the end of the investigation.

New optional config (defaults shown, all clamped): CHECK_CONFIRMATIONS=2,
CHECK_CONFIRM_DELAY=20s, REBOOT_MIN_STREAK=2. Setting confirmations to 0 and the
streak to 1 restores the previous behaviour exactly.

New runs.outcome values degraded / unconfirmed / unsustained, and exit code 6
for "a failure was observed and deliberately not acted on".

Applied to the recorded history, these guards remove all 29 triggers, including the one
genuine blip that no reboot fixed.

Also in this branch

  • test(httpapi): the overview-window subtest asserted a hardcoded ?since= that
    QueryService floors at now-31d; it began failing on 2026-08-01 and blocked the test
    gate. Now derived from time.Now().

Verification

go vet ./... && go test ./... green (9/9 packages). New coverage: verdict grading table,
confirmation clearing and confirming a verdict, degraded never reaching the rebooter,
streak satisfied/unsatisfied/hard-run/query-error paths, edge-triggered notices, recovery
ending on a degraded verdict, RecentRuns against :memory:, config defaults and
clamping, and the probe stage tags.

Operational follow-up (not in this PR)

The collector resolves DNS through the router itself (systemd-resolved → 192.168.1.1).
Pointing the host at upstream resolvers directly would remove the common-mode dependency
that produces these stalls at all. That is a change on the Pi, outside this repository.

🤖 Generated with Claude Code

prorochestvo and others added 3 commits August 4, 2026 15:19
The subtest asserted that a hardcoded ?since=2026-07-01T00:00:00Z was
forwarded verbatim, but QueryService floors the window at now-31d, so the
case started failing once the wall clock walked past that date. Derive the
timestamp from time.Now() instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every recorded domain-probe failure reads "context deadline exceeded",
which does not say whether name resolution, the TCP connect, or the TLS
handshake was what hung — three faults with three different remedies.
Attach an httptrace.ClientTrace to the probe request and tag the recorded
error with the phase the request died in.

Refs: #6

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Of 29 "internet down" verdicts recorded in 21 days, 28 had both ip targets
connecting in 19-1118ms and ICMP answering 4/4 while only the two HTTPS
domain probes timed out at exactly CHECK_TIMEOUT. Those two probes share
one DNS path — in production, through the very router being rebooted — so
they fail together, and the old rule ("all ip fail OR all domain fail")
turned that into 11 router reboots that cut the LAN off for minutes each.
Only one verdict in the window was a real link loss, and it lasted a single
five-minute sample and healed on its own.

Verdict is now graded: down (all ip targets failed) is the only state a
reboot may follow; a domain-only wipeout while raw connectivity holds is
degraded — recorded, announced once, never rebooted, because four reboots
in seven hours on 2026-07-30 did not stop the same probes from stalling.

Two guards then sit between a down verdict and a reboot. Within the run,
the verdict is re-probed CHECK_CONFIRMATIONS times CHECK_CONFIRM_DELAY
apart, and one healthy sample ends the run as unconfirmed. Across runs, the
previous REBOOT_MIN_STREAK-1 runs must have seen the outage too, or the run
ends as unsustained; an unreadable history withholds the reboot rather than
granting it. Neither guard sends a message — the run row and the log carry
the record. Degraded and cooldown-skip notices became edge-triggered, which
also ends the per-run message storm during a long outage.

Refs: #6

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@prorochestvo
prorochestvo merged commit 4d44361 into main Aug 4, 2026
1 check passed
@prorochestvo
prorochestvo deleted the fix/6-connectivity-false-positives branch August 4, 2026 10:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Review and adjust internet connectivity check logic to avoid false positives at night

1 participant