fix(check): stop rebooting the router on transient probe failures - #7
Merged
Conversation
The subtest asserted that a hardcoded ?since=2026-07-01T00:00:00Z was forwarded verbatim, but QueryService floors the window at now-31d, so the case started failing once the wall clock walked past that date. Derive the timestamp from time.Now() instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every recorded domain-probe failure reads "context deadline exceeded", which does not say whether name resolution, the TCP connect, or the TLS handshake was what hung — three faults with three different remedies. Attach an httptrace.ClientTrace to the probe request and tag the recorded error with the phase the request died in. Refs: #6 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Of 29 "internet down" verdicts recorded in 21 days, 28 had both ip targets
connecting in 19-1118ms and ICMP answering 4/4 while only the two HTTPS
domain probes timed out at exactly CHECK_TIMEOUT. Those two probes share
one DNS path — in production, through the very router being rebooted — so
they fail together, and the old rule ("all ip fail OR all domain fail")
turned that into 11 router reboots that cut the LAN off for minutes each.
Only one verdict in the window was a real link loss, and it lasted a single
five-minute sample and healed on its own.
Verdict is now graded: down (all ip targets failed) is the only state a
reboot may follow; a domain-only wipeout while raw connectivity holds is
degraded — recorded, announced once, never rebooted, because four reboots
in seven hours on 2026-07-30 did not stop the same probes from stalling.
Two guards then sit between a down verdict and a reboot. Within the run,
the verdict is re-probed CHECK_CONFIRMATIONS times CHECK_CONFIRM_DELAY
apart, and one healthy sample ends the run as unconfirmed. Across runs, the
previous REBOOT_MIN_STREAK-1 runs must have seen the outage too, or the run
ends as unsustained; an unreadable history withholds the reboot rather than
granting it. Neither guard sends a message — the run row and the log carry
the record. Degraded and cooldown-skip notices became edge-triggered, which
also ends the per-run message storm during a long outage.
Refs: #6
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #6
What the data showed
Analysed a read-only snapshot of the production database (2026-07-14 → 2026-08-04:
6140 runs at a 5-minute cadence, 24 735 checks). Full write-up in
the issue comment.
14 skipped by cooldown.
iptargets connected in19–1118 ms, and the ping collector recorded 4/4 ICMP replies from
8.8.8.8,1.1.1.1and the ISP gateway in the same runs. Only the two HTTPS
domainprobes failed, bothat exactly the 5 s
CHECK_TIMEOUT.next run; its reboot attempt failed and was not what fixed it.
isolated 5-minute sample between healthy runs.
~0.6 % per-probe failure rate, independent failures would coincide ~0.2 times in 21
days; 29 were observed, so the two probes fail from a common cause — they share one DNS
path, in production through the very router being rebooted.
What changed
Graded verdict (
domain.Verdict) —down(everyiptarget failed) is the onlystate a reboot may follow. Every
domaintarget failing while aniptarget stillanswers is
degraded: recorded, announced, never rebooted. With noiptargetsconfigured the
domaingroup remains the only evidence and still counts asdown.In-run confirmation — a
downverdict is re-probedCHECK_CONFIRMATIONStimes,CHECK_CONFIRM_DELAYapart (persisted under a newconfirmphase). One healthy sampleends the run as
unconfirmedand corrects theinternet_okwritten from the firstsample.
Cross-run corroboration — a reboot additionally requires the previous
REBOOT_MIN_STREAK - 1runs to have seen the outage, via a newRunRepository.RecentRuns. A run that falls short ends asunsustained. Fail-closed: anunreadable history withholds the reboot.
Quieter Telegram — neither guard sends a message; the run row and a log line carry the
record.
degradedand cooldown-skip notices are edge-triggered, so the five identicalcooldown notices between 01:50 and 03:05 in the issue screenshot become one.
Probe diagnostics — a failed domain probe now carries the request stage it stalled in
(
stage=dns|connect|tls|request|response|body) viahttptrace, so "context deadlineexceeded" stops being the end of the investigation.
New optional config (defaults shown, all clamped):
CHECK_CONFIRMATIONS=2,CHECK_CONFIRM_DELAY=20s,REBOOT_MIN_STREAK=2. Setting confirmations to 0 and thestreak to 1 restores the previous behaviour exactly.
New
runs.outcomevaluesdegraded/unconfirmed/unsustained, and exit code6for "a failure was observed and deliberately not acted on".
Applied to the recorded history, these guards remove all 29 triggers, including the one
genuine blip that no reboot fixed.
Also in this branch
test(httpapi): the overview-window subtest asserted a hardcoded?since=thatQueryServicefloors atnow-31d; it began failing on 2026-08-01 and blocked the testgate. Now derived from
time.Now().Verification
go vet ./... && go test ./...green (9/9 packages). New coverage: verdict grading table,confirmation clearing and confirming a verdict, degraded never reaching the rebooter,
streak satisfied/unsatisfied/hard-run/query-error paths, edge-triggered notices, recovery
ending on a degraded verdict,
RecentRunsagainst:memory:, config defaults andclamping, and the probe stage tags.
Operational follow-up (not in this PR)
The collector resolves DNS through the router itself (
systemd-resolved→ 192.168.1.1).Pointing the host at upstream resolvers directly would remove the common-mode dependency
that produces these stalls at all. That is a change on the Pi, outside this repository.
🤖 Generated with Claude Code