Skip to content

Record WireGuard tunnel recovery #108

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tdenisenko merged 1 commit into from
Jul 18, 2026
Merged

Record WireGuard tunnel recovery #108

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 36 additions & 29 deletions ROADMAP.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,24 +4,23 @@

LogEx starts from a recent consensus checkpoint, tracks the live head, reverse-syncs execution history toward genesis, stores compressed verified logs, and serves the dashboard, SQL query API, JSON-RPC, gRPC, and live ERC20 transfer subscriptions.

Current branch: `master`.
Current branch: `fix/wireguard-stale-tunnel-repair`.

The nightly Rust CI compatibility failure is fixed and merged through PR #106. Both the PR and merge-triggered `master` workflows pass all four CI jobs, and `master` is the only remaining remote branch.

The Mac mini LogEx process is running from the full synced data directory and is publicly reachable through the VPS at `http://157.245.195.72:18683/`. The July 18 dashboard outage was caused by a stale WireGuard tunnel, not a LogEx process crash; after repairing the tunnel and gracefully restarting LogEx, forward indexing resumed.

## Completed Since Last Run

- Inspected the latest failed GitHub Actions run on `master`.
- Confirmed that Check, Clippy, and Test all failed during compilation from the same newly denied nightly Rust lint; Format passed.
- Terminated seven expression-position `eyre::bail!` invocations as statements without changing their early-return behavior.
- Validated the full workspace against the newly enforced lint and all four CI commands.
- Searched for additional expression-position `bail!` invocations and found none requiring changes.
- Merged PR #106 after all four GitHub checks passed.
- Verified the merge-triggered `master` workflow also passed all four GitHub checks.
- Removed every non-`master` remote branch, including the temporary CI-fix and roadmap-finalization branches.
- Confirmed LogEx did not crash; the tmux session and process were still alive while the public dashboard was unreachable.
- Identified the outage as a stale WireGuard tunnel: VPS NAT rules were still present, but the VPS could not reach `10.66.0.2` and the peer handshake was stale.
- Repaired full VPS routing on the Mac mini and verified public dashboard access through `157.245.195.72:18683`.
- Updated the ignored local routing scripts and installed Mac LaunchDaemon so the WireGuard watchdog probes tunnel traffic and rebuilds the tunnel when an existing `utun` interface stops passing packets.
- Gracefully restarted LogEx after restoring the tunnel because the live catch-up path remained wedged from the stale network state; storage integrity passed and forward indexing resumed.

## Remaining TODOs

No remaining TODOs for the nightly Rust CI compatibility and remote branch cleanup task.
No remaining TODOs for the stale WireGuard tunnel recovery task.

## Design Decisions

Expand All @@ -45,6 +44,11 @@ No remaining TODOs for the nightly Rust CI compatibility and remote branch clean
- Alternatives considered: reporting request payloads only. That was technically payload-accurate but misleading for user-facing bandwidth.
- Tradeoff: Upload is estimated from download traffic unless LogEx is serving larger payloads to peers.

- Treat WireGuard interface presence as insufficient health.
- Why: The outage showed `utun` and routes can remain present after the tunnel stops handshaking.
- Alternatives considered: only reinstalling routes on a timer.
- Tradeoff: The watchdog now performs a lightweight ping probe to the VPS tunnel IP before deciding the tunnel is healthy.

## Challenges and Resolutions

- Challenge: The local nightly compiler predates the GitHub runner compiler that promoted `semicolon_in_expressions_from_macros` to an error.
Expand All @@ -59,35 +63,38 @@ No remaining TODOs for the nightly Rust CI compatibility and remote branch clean
- Resolution: Account EL response sizes as Snappy-compressed RLPx wire-equivalent bytes and include measured lower-layer overhead factors.
- Remaining: The value is an estimate, not a packet capture.

- Challenge: The public dashboard looked like a client outage around 04:00 local time, but the LogEx process was still alive.
- Resolution: Compared the LAN status endpoint, VPS WireGuard state, and tunnel reachability; the failure matched a stale WireGuard tunnel rather than a LogEx crash.
- Remaining: None.

- Challenge: The LaunchDaemon preserved routes and interface state but did not verify actual tunnel traffic.
- Resolution: The installed Mac watchdog and local ignored ops scripts now rebuild WireGuard when tunnel probes fail.
- Remaining: None.

## Dead Code and Obsolescence Cleanup

- Inspected every `bail!` invocation in the Rust workspace for the newly invalid expression-position pattern.
- Confirmed the remaining invocations are already statement-terminated.
- No dead files, imports, exports, dependencies, or superseded code were introduced or found in the affected request path.
- Inspected the LogEx process, VPS NAT rules, WireGuard daemon state, and local routing scripts.
- No production Rust code was changed.
- The modified routing scripts live under ignored `local-ops/` because they contain machine-specific operational details.

## Git Workflow

- Final branch: `master`.
- Task branch `fix/nightly-bail-ci` was created from the latest `master`.
- Branch: `fix/wireguard-stale-tunnel-repair`.
- New branch created from `master` for the operational recovery note.
- Commits made during this run:
- `8d02f434 fix: restore nightly CI compatibility`
- `58c18e72 docs: record CI fix workflow`
- `docs: finalize nightly CI roadmap`
- Pull request status:
- PR #106 fixed CI and was merged into `master`: `https://github.com/tdenisenko/logex/pull/106`.
- PR #107 finalized the roadmap and was merged into `master`: `https://github.com/tdenisenko/logex/pull/107`.
- Merge status: both pull requests were merged after their required checks passed.
- Remote branch cleanup: every remote branch except `master` was deleted after both merges.
- GitHub CLI authentication was expired; the connected GitHub app supplied workflow logs, PR creation, and merge operations, while authenticated SSH handled Git fetch/push operations.
- `docs: record WireGuard tunnel recovery`
- Pull request status: pending.
- Merge status: pending.
- Validation run:
- `cargo fmt --all -- --check`
- `RUSTFLAGS=-Dsemicolon_in_expressions_from_macros cargo check --workspace`
- `cargo check --workspace`
- `cargo clippy --workspace -- -D warnings`
- `cargo test --workspace`
- `sh -n local-ops/logex-full-vps-routing.sh`
- `sh -n local-ops/logex-dashboard-only-routing.sh`
- Mac LaunchDaemon inspection for the installed tunnel probe/rebuild logic
- Public `/status` checks through `157.245.195.72:18683`
- `git diff --check`

## Known Issues or Risks

- Bandwidth metrics are calibrated wire-equivalent estimates, not packet captures. They should track normal sync traffic closely, but exact values can differ during peer churn, retransmits, or unrelated host traffic on the same VPS tunnel.
- The repository intentionally follows rolling nightly Rust, so future compiler changes can expose additional source incompatibilities; CI remains the guardrail.
- `local-ops/` is intentionally ignored by Git, so machine-specific routing scripts are not versioned unless explicitly force-added.
- The remote Mac mini checkout used for this operational run is a dirty `perf/historical-sync-throughput-v3` tree, not a clean `master` checkout; evaluate any remote-only warnings against that state before treating them as production regressions.
Loading