Channel health monitoring with alerts and auto-restart - #207
Open
gluk-w wants to merge 3 commits into
Open
Conversation
Control plane now polls each running instance's OpenClaw gateway
(channels.status WS RPC over the existing Gateway tunnel) to detect
channels that are disconnected, wedged (stale sockets), or not running,
catching the known failure mode where the process is alive but a Slack
Socket Mode / Telegram poller connection is silently dead.
- internal/channelhealth: background monitor (60s interval, bounded
concurrency), health evaluation with 30m stale threshold, persisted
per-channel status + in-memory snapshot
- GET /api/v1/instances/{id}/channels/health + channel_health summary
on instance responses
- Channel Health panel on Agent detail Settings tab and warning
indicator on the agent list for unhealthy/unreachable instances
- CLAWORC_CHANNEL_HEALTH_ENABLED / CLAWORC_CHANNEL_HEALTH_INTERVAL
- docs/channel-monitoring.md spec + end-user page
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CfW5Dn8yEsULgbKzmcjbZ7
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CfW5Dn8yEsULgbKzmcjbZ7
… auto-restart
The channel health monitor now feeds every snapshot to an escalator that
tracks consecutive failing checks (overall unhealthy/unreachable) per
instance and acts on sustained outages instead of only displaying them:
- Webhook alert (channel_failure) after 3 consecutive failing checks,
once per incident; recovery notification with outage duration when the
incident closes. Payloads include a human-readable text field so Slack
incoming webhooks work as-is; optional bearer token encrypted at rest.
- Opt-in auto-restart (default off) after 5 consecutive failing checks,
reusing the existing async restart flow, capped at 3 restarts per
rolling hour per instance with a 10m post-restart cooldown.
- channel_health_events audit table + GET /instances/{id}/channels/health/events.
- Settings: Channel Health Alerts card (Misc tab) with webhook URL,
token, toggles, and a Send Test button (POST /settings/channel-alerts/test).
- Thresholds via CLAWORC_CHANNEL_HEALTH_{ALERT_THRESHOLD,RESTART_THRESHOLD,
RESTART_MAX_PER_HOUR,RESTART_COOLDOWN}.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CfW5Dn8yEsULgbKzmcjbZ7
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Detects when an OpenClaw instance's chat channels (Slack, Telegram, Discord, ...) stop responding — including the common failure mode where the process is alive but a Slack Socket Mode / Telegram poller connection is silently wedged — and escalates instead of failing silently.
channels.statusover the existing Gateway SSH tunnel) and classifies per-channel health (healthy / stale / disconnected / not running), persisted plus in-memoryGET /instances/{id}/channels/healthendpoint andchannel_healthsummary on instance responseschannel_health_eventsaudit table doubling as outage historyCLAWORC_CHANNEL_HEALTH_*env vars for interval/thresholds; alerts and auto-restart toggles are UI-editable settingsTest plan
go build/go vet/go testpass;vite buildpasses🤖 Generated with Claude Code