Review and adjust internet connectivity check logic to avoid false positives at night

[prorochestvo]

Problem

There are suspicious internet connectivity checks triggering during night and early morning hours. These triggers may be caused by the ISP, but they still pose a risk: many processes depend on internet access, and we do not want to cut off the internet for them due to potentially incorrect connectivity checks.

Context

• Strange connectivity check events are observed only at night or early in the morning.
• It might be related to the internet provider performing maintenance or having brief outages at those times.
• Current behavior risks disabling internet access for processes that critically depend on it.

Tasks / Next Steps

1. Review the existing internet connectivity check mechanism:
   • What exactly is being checked (e.g., ping to specific hosts, HTTP requests, DNS resolution)?
   • What are the timeouts and retry policies?
   • How often checks run, especially at night.
2. Analyze recent logs for night/early-morning periods:
   • Identify when and how often connectivity checks are failing.
   • Confirm whether failures are brief/transient or prolonged.
3. Assess impact on dependent processes:
   • Determine which processes are affected when the system decides that the internet is down.
   • Verify whether any processes have been incorrectly blocked during these night-time events.
4. Propose and implement safer handling:
   • Adjust thresholds, timeouts, or retry counts to better tolerate brief night-time issues.
   • Consider different behavior during known maintenance windows (if applicable) to avoid unnecessary cutoffs.
5. Validate changes:
   • Monitor behavior over several nights.
   • Confirm that legitimate outages are still detected while avoiding unnecessary internet cutoffs for dependent processes.

Acceptance Criteria

• False-positive internet outage detections during night/early-morning hours are reduced or eliminated.
• Dependent processes no longer lose internet access due to brief or spurious connectivity check failures.
• Logs clearly show when and why internet is considered unavailable.

Task: 260801.001

Attachments:

[prorochestvo]

Investigation: what the recorded telemetry actually shows

Analysed a read-only snapshot of the production database (/opt/yarddog/yarddog.sqlite),
window 2026-07-14 → 2026-08-04: 6140 soft runs at a 5-minute cadence, 24 735 checks.
All times below are local (UTC+05).

Trigger counts

	runs
judged "internet down"	29
→ reboot executed	11
→ reboot attempt failed	4
→ skipped by cooldown	14

In 28 of those 29 runs the raw uplink was provably alive.

• Both ip targets (1.1.1.1:443, 8.8.8.8:53) connected successfully, 19–1118 ms.
• Only the two domain probes failed, both at exactly the 5 s CHECK_TIMEOUT, with
  Client.Timeout exceeded while awaiting headers.
• ICMP telemetry recorded in the very same runs: 4/4 replies from 8.8.8.8,
  1.1.1.1 and the ISP gateway at normal RTT (24 ms / 90 ms / 8 ms).

The 29th run (2026-07-31 02:55) was the only one where all four targets failed — and even
there the ping collector still got 3/4 replies from every host, the reboot attempt failed
with a login error, and the next run at 03:00 was healthy again. The link recovered on its
own. There was not a single sustained outage in 21 days.

The failures are isolated blips, not outages. Exactly one pair of consecutive down runs
occurred in the whole window (07-30 02:45 and 02:50). Every other trigger was a single
5-minute sample surrounded by healthy runs.

The domain probes flap continuously in the background. 17 further runs had exactly one
of the two domain probes time out and correctly stayed "up". Per-probe failure rate is
~0.6 %; if the two probes failed independently, coincidences would be expected ~0.2 times in
21 days. Observed: 29. The two failures are therefore common-mode, not independent.

Common mode identified. Both domain targets are HTTPS GETs that require name
resolution; the collector host resolves through systemd-resolved → 192.168.1.1, i.e.
the router itself, over wlan0. The ip targets are IP literals and bypass that path
entirely. Successful domain probes take a very stable ~210 ms at every hour of the day;
failures are hard 5 s stalls with nothing in between — the signature of a stalled resolver,
not of a degrading link.

Timing. Domain-probe failures do cluster at night (63 of 75 initial-phase failures fall
between 00:00 and 08:00), but they also occur at 13:15, 17:50, 19:25 and 21:05. A
time-of-day exception would be both leaky and blind.

Root cause

domain.Verdict (domain/check.go) declares the internet down when all ip targets fail
OR all domain targets fail. The domain arm has a single point of failure — the router's
DNS forwarder — that the ip arm proves healthy at the same instant. Nothing re-confirms the
verdict: one 5-second sample is enough to reboot the router and take the LAN down for every
dependent process for 2–3 minutes.

---

Implementation plan

Task 1 — Record which stage of the domain probe failed

infrastructure/check.go: attach an httptrace.ClientTrace to the probe request and tag the
recorded error with the last stage entered (dns, connect, tls, headers). No behaviour
change; it turns "context deadline exceeded" into evidence and makes the classification in
Task 2 verifiable in the data rather than inferred.

Task 2 — Tier the verdict: a domain-only failure is degraded, not down

domain/check.go: replace Result.Down bool with a tri-state Result.State:

• up — no group fully failed,
• degraded — every domain target failed while at least one ip target succeeded
  (name-resolution / application path broken, raw link alive),
• down — every ip target failed.

Result.IsDown() keeps the call sites readable. Only down may lead to a reboot.
degraded is recorded (runs.outcome = "degraded", internet_ok stays 1 because the link
is up) and notified, never rebooted. The recovery loop treats "no longer down" as restored,
so a post-reboot run does not hang waiting for DNS.

Task 3 — Confirm a down verdict before acting

services/orchestrator.go: on a down initial verdict, re-probe CHECK_CONFIRMATIONS more
times spaced CHECK_CONFIRM_DELAY apart, via the injected Clock (no real sleeping in
tests). Every attempt is persisted with a new domain.PhaseConfirm = "confirm" phase. A
single non-down attempt aborts the reboot path; the run is corrected to
internet_ok = 1, outcome = "unconfirmed", and sends no Telegram message. Requires a new
InternetOK field on domain.RunUpdate.

Defaults: 2 extra attempts, 20 s apart — a decision inside ~45 s, well within the 5-minute
cron slot.

Task 4 — Require a sustained outage across runs

One new port method on services.RunRepository, backed by the hot runs table:

RecentRuns(ctx context.Context, limit int) ([]domain.RunSummary, error) // newest first

A reboot requires the current run and the previous REBOOT_MIN_STREAK - 1 soft runs to
have been down (default 2 ⇒ ≥5 minutes of continuous outage). A down run that does not meet
the streak is recorded as unconfirmed and stays silent. Applied to the recorded data, this
plus Task 2 removes 29 of 29 triggers, including the 2026-07-31 blip that the failed reboot
did not fix.

Task 5 — Edge-triggered notifications

The same RecentRuns query removes the message storm visible in the issue screenshot:
degraded and skipped_cooldown notices are sent only when the previous run was not
already in that state. One message per episode instead of one per 5-minute run.

Task 6 — Configuration and documentation

New optional keys (defaults in parens, clamped like the existing ones):
CHECK_CONFIRMATIONS (2, [0,5]), CHECK_CONFIRM_DELAY (20s, [1s,60s]),
REBOOT_MIN_STREAK (2, [1,10]). Update yarddog.env.example, the README env table, and
the verdict/outcome sections of CLAUDE.md.

Tests

• domain: table-driven Verdict tests over every up/degraded/down combination, including
  the empty-group guard.
• services: fake clock + scripted checker — abort on an unconfirmed verdict, reboot when
  confirmations and streak agree, degraded never reaches the rebooter, notifications fire
  once per episode.
• infrastructure: RecentRuns against :memory:, config parsing/clamping, and an
  httptest server that stalls to assert the stage tag from Task 1.

Gate: go vet ./... && go test ./....

---

Rejected alternatives

1. Night-time maintenance window. False positives also occurred at 13:15, 17:50, 19:25
   and 21:05, and muting the watchdog at night would blind it during exactly the hours a real
   outage is least likely to be noticed manually. Behaviour-based guards dominate a clock.
2. Raising CHECK_TIMEOUT. Healthy domain probes complete in ~210 ms; the failures are
   hard ≥5 s stalls. A larger timeout only postpones the same wrong verdict and lengthens
   every run.
3. Adding an IP-literal HTTPS target to the domain group. It would stop the group from
   ever being "all failed" and hide a genuine DNS degradation instead of classifying it.
4. Rebooting on degraded. On 2026-07-30 four reboots (00:00, 02:10, 04:45, 07:25) did
   not stop the domain-probe stalls, which continued until 07:25. The reboot does not fix
   this failure mode.
5. Dropping the domain group entirely. It is the only signal that catches a DNS or
   application-layer break; it should keep being observed, just not trigger a reboot.

Operational follow-up (host change, not code — needs your approval)

The collector resolves DNS through the router (systemd-resolved → 192.168.1.1). Pointing
the host at upstream resolvers directly (1.1.1.1, 8.8.8.8) would remove the common-mode
dependency that produces these stalls in the first place. That is a change on the Pi outside
this repository, so it is listed here rather than performed.

[prorochestvo]

Implementation is up in #7 (branch fix/6-connectivity-false-positives), CI green.

Shipped: graded verdict (domain-only failure = degraded, never a reboot trigger), in-run re-confirmation of a down verdict, cross-run corroboration before any reboot, edge-triggered Telegram notices, and httptrace stage tags on failed domain probes. New optional keys CHECK_CONFIRMATIONS (2), CHECK_CONFIRM_DELAY (20s), REBOOT_MIN_STREAK (2) — set to 0/1 to restore the previous behaviour.

Applied to the 21 days of recorded history, the guards remove all 29 triggers, including the single genuine blip that no reboot fixed.

Waiting on review and merge.