From fe3bcd0585de37a044213fbfdeea30b0bf290335 Mon Sep 17 00:00:00 2001 From: Tim Denisenko Date: Sat, 18 Jul 2026 15:52:47 +0700 Subject: [PATCH] docs: record WireGuard tunnel recovery --- ROADMAP.md | 65 ++++++++++++++++++++++++++++++------------------------ 1 file changed, 36 insertions(+), 29 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 8cd5cac0..1a4627b8 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -4,24 +4,23 @@ LogEx starts from a recent consensus checkpoint, tracks the live head, reverse-syncs execution history toward genesis, stores compressed verified logs, and serves the dashboard, SQL query API, JSON-RPC, gRPC, and live ERC20 transfer subscriptions. -Current branch: `master`. +Current branch: `fix/wireguard-stale-tunnel-repair`. The nightly Rust CI compatibility failure is fixed and merged through PR #106. Both the PR and merge-triggered `master` workflows pass all four CI jobs, and `master` is the only remaining remote branch. +The Mac mini LogEx process is running from the full synced data directory and is publicly reachable through the VPS at `http://157.245.195.72:18683/`. The July 18 dashboard outage was caused by a stale WireGuard tunnel, not a LogEx process crash; after repairing the tunnel and gracefully restarting LogEx, forward indexing resumed. + ## Completed Since Last Run -- Inspected the latest failed GitHub Actions run on `master`. -- Confirmed that Check, Clippy, and Test all failed during compilation from the same newly denied nightly Rust lint; Format passed. -- Terminated seven expression-position `eyre::bail!` invocations as statements without changing their early-return behavior. -- Validated the full workspace against the newly enforced lint and all four CI commands. -- Searched for additional expression-position `bail!` invocations and found none requiring changes. -- Merged PR #106 after all four GitHub checks passed. -- Verified the merge-triggered `master` workflow also passed all four GitHub checks. -- Removed every non-`master` remote branch, including the temporary CI-fix and roadmap-finalization branches. +- Confirmed LogEx did not crash; the tmux session and process were still alive while the public dashboard was unreachable. +- Identified the outage as a stale WireGuard tunnel: VPS NAT rules were still present, but the VPS could not reach `10.66.0.2` and the peer handshake was stale. +- Repaired full VPS routing on the Mac mini and verified public dashboard access through `157.245.195.72:18683`. +- Updated the ignored local routing scripts and installed Mac LaunchDaemon so the WireGuard watchdog probes tunnel traffic and rebuilds the tunnel when an existing `utun` interface stops passing packets. +- Gracefully restarted LogEx after restoring the tunnel because the live catch-up path remained wedged from the stale network state; storage integrity passed and forward indexing resumed. ## Remaining TODOs -No remaining TODOs for the nightly Rust CI compatibility and remote branch cleanup task. +No remaining TODOs for the stale WireGuard tunnel recovery task. ## Design Decisions @@ -45,6 +44,11 @@ No remaining TODOs for the nightly Rust CI compatibility and remote branch clean - Alternatives considered: reporting request payloads only. That was technically payload-accurate but misleading for user-facing bandwidth. - Tradeoff: Upload is estimated from download traffic unless LogEx is serving larger payloads to peers. +- Treat WireGuard interface presence as insufficient health. + - Why: The outage showed `utun` and routes can remain present after the tunnel stops handshaking. + - Alternatives considered: only reinstalling routes on a timer. + - Tradeoff: The watchdog now performs a lightweight ping probe to the VPS tunnel IP before deciding the tunnel is healthy. + ## Challenges and Resolutions - Challenge: The local nightly compiler predates the GitHub runner compiler that promoted `semicolon_in_expressions_from_macros` to an error. @@ -59,35 +63,38 @@ No remaining TODOs for the nightly Rust CI compatibility and remote branch clean - Resolution: Account EL response sizes as Snappy-compressed RLPx wire-equivalent bytes and include measured lower-layer overhead factors. - Remaining: The value is an estimate, not a packet capture. +- Challenge: The public dashboard looked like a client outage around 04:00 local time, but the LogEx process was still alive. + - Resolution: Compared the LAN status endpoint, VPS WireGuard state, and tunnel reachability; the failure matched a stale WireGuard tunnel rather than a LogEx crash. + - Remaining: None. + +- Challenge: The LaunchDaemon preserved routes and interface state but did not verify actual tunnel traffic. + - Resolution: The installed Mac watchdog and local ignored ops scripts now rebuild WireGuard when tunnel probes fail. + - Remaining: None. + ## Dead Code and Obsolescence Cleanup -- Inspected every `bail!` invocation in the Rust workspace for the newly invalid expression-position pattern. -- Confirmed the remaining invocations are already statement-terminated. -- No dead files, imports, exports, dependencies, or superseded code were introduced or found in the affected request path. +- Inspected the LogEx process, VPS NAT rules, WireGuard daemon state, and local routing scripts. +- No production Rust code was changed. +- The modified routing scripts live under ignored `local-ops/` because they contain machine-specific operational details. ## Git Workflow -- Final branch: `master`. -- Task branch `fix/nightly-bail-ci` was created from the latest `master`. +- Branch: `fix/wireguard-stale-tunnel-repair`. +- New branch created from `master` for the operational recovery note. - Commits made during this run: - - `8d02f434 fix: restore nightly CI compatibility` - - `58c18e72 docs: record CI fix workflow` - - `docs: finalize nightly CI roadmap` -- Pull request status: - - PR #106 fixed CI and was merged into `master`: `https://github.com/tdenisenko/logex/pull/106`. - - PR #107 finalized the roadmap and was merged into `master`: `https://github.com/tdenisenko/logex/pull/107`. -- Merge status: both pull requests were merged after their required checks passed. -- Remote branch cleanup: every remote branch except `master` was deleted after both merges. -- GitHub CLI authentication was expired; the connected GitHub app supplied workflow logs, PR creation, and merge operations, while authenticated SSH handled Git fetch/push operations. + - `docs: record WireGuard tunnel recovery` +- Pull request status: pending. +- Merge status: pending. - Validation run: - - `cargo fmt --all -- --check` - - `RUSTFLAGS=-Dsemicolon_in_expressions_from_macros cargo check --workspace` - - `cargo check --workspace` - - `cargo clippy --workspace -- -D warnings` - - `cargo test --workspace` + - `sh -n local-ops/logex-full-vps-routing.sh` + - `sh -n local-ops/logex-dashboard-only-routing.sh` + - Mac LaunchDaemon inspection for the installed tunnel probe/rebuild logic + - Public `/status` checks through `157.245.195.72:18683` - `git diff --check` ## Known Issues or Risks - Bandwidth metrics are calibrated wire-equivalent estimates, not packet captures. They should track normal sync traffic closely, but exact values can differ during peer churn, retransmits, or unrelated host traffic on the same VPS tunnel. - The repository intentionally follows rolling nightly Rust, so future compiler changes can expose additional source incompatibilities; CI remains the guardrail. +- `local-ops/` is intentionally ignored by Git, so machine-specific routing scripts are not versioned unless explicitly force-added. +- The remote Mac mini checkout used for this operational run is a dirty `perf/historical-sync-throughput-v3` tree, not a clean `master` checkout; evaluate any remote-only warnings against that state before treating them as production regressions.