Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,9 @@ around the opposite principle — **prove it is safe, then act**:
config validation, locks, processes, preflight, inventory, SLA and events.
- **Notifications** to email, Slack, Teams and webhook sinks (ntfy/Telegram/
Gotify) with a templated default message.
- An optional **interactive Telegram report bot** (read-only): ask it `/status`,
`/services`, `/sla` and it replies with live reports — long polling only, no
inbound port, answering allow-listed chats.
- A **daemon-wide panic switch** to pause all automatic remediation instantly.
- **Guided wizards** for common setups (service, docker, vm, mount, volume, net,
uplink).
Expand Down
30 changes: 28 additions & 2 deletions cmd/sermod/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ import (
"sermo/internal/rules"
"sermo/internal/servicemgr"
"sermo/internal/state"
"sermo/internal/telegrambot"
"sermo/internal/web"
)

Expand Down Expand Up @@ -368,13 +369,20 @@ func run(args []string) int {
}
}

botCfg := telegrambot.ParseConfig(config.SectionMap(cfg.Global.Raw, config.SectionTelegramBot))

var webHolder *app.WebBackendHolder
var webDone chan struct{}
addr, webDisabledReason := webListenAddr(cfg)
if addr != "" {
// The web backend feeds both the dashboard and the report bot; build it when
// either is enabled, even if the HTTP server itself stays off.
if addr != "" || botCfg.Enabled {
var webWarnings []string
webHolder, webWarnings = app.NewWebBackendHolder(ctx, cfg, deps)
app.LogBuildNotices(logger, "build web backend", webWarnings)
}

var webDone chan struct{}
if addr != "" {
auth := webAuth(cfg)
server := &web.Server{
Addr: addr,
Expand Down Expand Up @@ -410,6 +418,21 @@ func run(args []string) int {
logger.Warn("web ui disabled; no port will be opened", logFieldReason, webDisabledReason)
}

// Interactive read-only report bot (long polling; no inbound socket). It
// reads the same web backend the dashboard serves and replies to commands
// from allow-listed chats only.
var botDone chan struct{}
if botCfg.Enabled {
bot := telegrambot.New(app.NewTelegramReporter(webHolder, store, time.Now), botCfg, logger)
deps.TelegramBot = bot
botDone = make(chan struct{})
go func() {
defer close(botDone)
bot.Run(ctx)
}()
logger.Info("telegram report bot enabled", "allowed_chats", len(botCfg.AllowedChats))
}

pruneDone := startOldHistoryPrune(ctx, logger, store, time.Now().Add(-state.DefaultHistoryRetention))

logger.Info("sermod starting", logFieldBackend, detection.Backend, logFieldServices, len(workers), logFieldWatches, len(watches))
Expand Down Expand Up @@ -453,6 +476,9 @@ func run(args []string) int {
if webDone != nil {
<-webDone
}
if botDone != nil {
<-botDone
}
if !drainOrTimeout(pruneDone, shutdownPruneDrainTimeout) {
logger.Warn("history prune still running at shutdown; closing the store without it")
}
Expand Down
69 changes: 68 additions & 1 deletion docs/configuration.es.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ directorio equivocado. La configuración distribuida la omite.
- [Notificaciones](#notificaciones)
- [Plantillas de notificación](#plantillas-de-notificación)
- [Selección por defecto y precedencia](#selección-por-defecto-y-precedencia)
- [Bot de informes de Telegram](#bot-de-informes-de-telegram)
- [Host watches](#host-watches)
- [then.expand — crecimiento de volumen (watch de storage)](#thenexpand--crecimiento-de-volumen-watch-de-storage)
- [Control manual de reconstrucción RAID](#control-manual-de-reconstrucción-raid)
Expand Down Expand Up @@ -1200,10 +1201,19 @@ notifiers:

- **`telegram`** — envía a través de un **bot de Telegram** (`sendMessage`).
- **`token`** — el token del bot de `@BotFather`. Queda dentro de la URL de la
API y nunca aparece en el dashboard.
API y nunca aparece en el dashboard. Prefiere `${env:...}`; si queda vacío
(la variable no está definida) el notifier queda inactivo en lugar de fallar
la carga de configuración.
- **`chat_id`** — el id numérico del chat/grupo o un nombre `@canal`. El
asunto es la línea principal y el detalle (los campos `SERMO_*`) sigue como
texto plano.
- **`parse_mode`** *(opcional)* — `MarkdownV2`, `Markdown` o `HTML` para
renderizar el mensaje como texto con formato (negrita, código, enlaces) en
lugar de texto plano. Omítelo para texto plano.
- **`silent`** *(opcional)* — `true` entrega el mensaje en silencio, sin sonido
ni vibración (`disable_notification` de la Bot API).
- **`message_thread_id`** *(opcional)* — un id entero de tema (forum topic),
para publicar en un tema concreto de un grupo.

```yaml
# /etc/sermo/notifiers/telegram.yml
Expand All @@ -1212,6 +1222,9 @@ notifiers:
type: telegram
token: "123456789:AAF...XXXX"
chat_id: -1001234567890
# parse_mode: MarkdownV2 # opcional: formatea el texto del mensaje
# silent: true # opcional: entrega sin sonido
# message_thread_id: 42 # opcional: publica en un tema del grupo
```

- **`tty`** — escribe directamente en las sesiones de terminal Linux activas, similar a
Expand Down Expand Up @@ -1346,6 +1359,60 @@ solo-alerta (estado de disparo + eventos en la interfaz y el log, pero sin accio
herencia de los globales). Consulta la sección de host watches a continuación para el
ejemplo de `check` + `for` desnudo.

## Bot de informes de Telegram

La sección opcional de nivel superior **`telegram_bot`** ejecuta un bot de Telegram
interactivo y de **solo lectura** dentro de `sermod`. Mientras que un notifier
`telegram` *empuja* alertas, el bot permite al operador *pedir* informes bajo demanda:
envíale `/status` y responde con el resumen actual del fleet. Nunca puede cambiar el
host — solo lee el mismo estado que sirve el dashboard web.

Recibe comandos mediante **long polling** de la Bot API (`getUpdates`), así que no
necesita puerto entrante, ni exposición pública, ni proxy inverso — en línea con la
postura de solo-salida de Sermo. El token del bot queda dentro de la URL de la API y se
depura de logs y errores, igual que el notifier `telegram`.

```yaml
# /etc/sermo/sermo.yml (o un fragmento drop-in)
telegram_bot:
token: "${env:TELEGRAM_BOT_TOKEN}" # token del bot de @BotFather
allowed_chats: # obligatorio: solo se responde a estos chats
- 123456789
- -1001234567890
# poll_interval: 30s # timeout opcional del long-poll getUpdates
# enabled: false # opcional; omitido => habilitado
```

- **`token`** — el token del bot de `@BotFather` (usado tanto para `getUpdates` como
para las respuestas). Prefiere `${env:...}` para no escribirlo en un archivo. Opcional:
si queda vacío (la variable de entorno no está definida) el bot simplemente queda
inactivo y el resto de la configuración se carga igual.
- **`allowed_chats`** — la lista de ids numéricos de chat que el bot responde. Un
mensaje de cualquier otro chat se ignora, nunca se responde. Este es el control de
acceso: limítalo a los operadores/grupos que pueden consultar el daemon.
- **`poll_interval`** *(opcional)* — el timeout del long-poll, por defecto `30s`,
acotado a `1s`–`10m`.
- **`enabled`** *(opcional)* — ponlo en `false` para conservar la sección pero detener
el polling.

Comandos (todos de solo lectura):

| Comando | Respuesta |
| --- | --- |
| `/status` | resumen del fleet: services ok/fallando, monitorizados/pausados, errores recientes, uptime del host |
| `/services [name]` | la lista de services, o el estado y salud de un service concreto |
| `/watches` | estados de host watches y watches de servicio |
| `/sla <service>` | ventanas de disponibilidad (hora…año) de un service |
| `/events [count]` | los eventos más recientes (por defecto 10, máximo 50) |
| `/help` | la lista de comandos |

Recargar (`sermoctl daemon reload` / `SIGHUP`) aplica cambios en `token`,
`allowed_chats` y `poll_interval` sin reiniciar. Como la goroutine solo se arranca al
inicio cuando la sección está presente, **habilitar el bot por primera vez requiere un
reinicio** (la misma regla que sigue la interfaz web para su puerto). Al arrancar, el bot
descarta cualquier comando encolado mientras estuvo caído, así que un reinicio nunca
reproduce solicitudes antiguas.

## Host watches

Los `watches` monitorizan recursos a nivel de host independientemente de cualquier
Expand Down
68 changes: 67 additions & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ configuration omits it.
- [Notifications](#notifications)
- [Notification templates](#notification-templates)
- [Default selection and precedence](#default-selection-and-precedence)
- [Telegram report bot](#telegram-report-bot)
- [Host watches](#host-watches)
- [then.expand — volume growth (storage watch)](#thenexpand--volume-growth-storage-watch)
- [Manual RAID reconstruction control](#manual-raid-reconstruction-control)
Expand Down Expand Up @@ -1155,10 +1156,19 @@ notifiers:

- **`telegram`** — sends through a **Telegram bot** (`sendMessage`).
- **`token`** — the bot token from `@BotFather`. It stays inside the API URL
and never appears on the dashboard.
and never appears on the dashboard. Prefer `${env:...}`; when it resolves
empty (the variable is unset) the notifier stays inactive instead of failing
config load.
- **`chat_id`** — the numeric chat/group id or an `@channel` name. The
subject is the lead line and the detail (the `SERMO_*` fields) follows as
plain text.
- **`parse_mode`** *(optional)* — `MarkdownV2`, `Markdown` or `HTML` to render
the message as formatted text (bold, code, links) instead of plain text. Omit
for plain text.
- **`silent`** *(optional)* — `true` delivers the message quietly, with no
sound or vibration (Bot API `disable_notification`).
- **`message_thread_id`** *(optional)* — an integer forum-topic id, to post
into a specific topic within a group.

```yaml
# /etc/sermo/notifiers/telegram.yml
Expand All @@ -1167,6 +1177,9 @@ notifiers:
type: telegram
token: "123456789:AAF...XXXX"
chat_id: -1001234567890
# parse_mode: MarkdownV2 # optional: format the message text
# silent: true # optional: deliver without sound
# message_thread_id: 42 # optional: post into a forum topic
```

- **`tty`** — writes directly to active Linux terminal sessions, similar to
Expand Down Expand Up @@ -1296,6 +1309,59 @@ alert-only behaviour (firing state + events in the UI and log, but no actions
and no inheritance of globals). See the host watches section below for the
bare `check` + `for` example.

## Telegram report bot

The optional top-level **`telegram_bot`** section runs an interactive,
**read-only** Telegram bot inside `sermod`. Where a `telegram` notifier *pushes*
alerts, the bot lets an operator *ask* for reports on demand: send it `/status`
and it replies with the current fleet summary. It can never change the host — it
only reads the same state the web dashboard serves.

It receives commands over Bot API **long polling** (`getUpdates`), so it needs
no inbound port, no public exposure and no reverse proxy — matching Sermo's
outbound-only posture. The bot token stays inside the API URL and is scrubbed
from logs and errors, exactly like the `telegram` notifier.

```yaml
# /etc/sermo/sermo.yml (or a drop-in fragment)
telegram_bot:
token: "${env:TELEGRAM_BOT_TOKEN}" # bot token from @BotFather
allowed_chats: # required: only these chats are answered
- 123456789
- -1001234567890
# poll_interval: 30s # optional getUpdates long-poll timeout
# enabled: false # optional; omitted => enabled
```

- **`token`** — the bot token from `@BotFather` (used for both `getUpdates` and
replies). Prefer `${env:...}` so it is not written in a file. Optional: when it
resolves empty (the env var is unset) the bot simply stays inactive and the
rest of the config still loads.
- **`allowed_chats`** — the list of numeric chat ids the bot answers. A message
from any other chat is ignored, never answered. This is the access control:
keep it to the operators/groups that may query the daemon.
- **`poll_interval`** *(optional)* — the long-poll timeout, default `30s`,
clamped to `1s`–`10m`.
- **`enabled`** *(optional)* — set `false` to keep the section but stop polling.

Commands (all read-only):

| Command | Reply |
| --- | --- |
| `/status` | fleet summary: services ok/failing, monitored/paused, recent errors, host uptime |
| `/services [name]` | the service list, or one named service's state and health |
| `/watches` | host and service watch states |
| `/sla <service>` | availability windows (hour…year) for a service |
| `/events [count]` | the most recent events (default 10, max 50) |
| `/help` | the command list |

Reloading (`sermoctl daemon reload` / `SIGHUP`) applies changes to `token`,
`allowed_chats` and `poll_interval` without a restart. Because the goroutine is
only started at boot when the section is present, **enabling the bot for the
first time requires a restart** (the same rule the web UI follows for its port).
On startup the bot discards any commands queued while it was down, so a restart
never replays old requests.

## Host watches

`watches` monitor host-level resources independently of any service and run a
Expand Down
16 changes: 16 additions & 0 deletions docs/sermo-all.yml
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,13 @@ notifiers:
enabled: false # keep configured but skip delivery
type: teams
webhook: "https://prod-01.westeurope.logic.azure.com:443/workflows/x"
ops-telegram:
type: telegram
token: "${env:TELEGRAM_TOKEN:-123456789:AAF...XXXX}"
chat_id: "-1001234567890"
parse_mode: MarkdownV2 # optional: HTML, Markdown or MarkdownV2
silent: true # optional: deliver without sound
message_thread_id: 42 # optional: post into a forum topic
tty:
type: tty
users: [root] # optional; omit to notify every active terminal
Expand All @@ -130,6 +137,15 @@ notifiers:
# delivery per site (a watch whose only action is [none] is monitor-only).
notify: [ops-email]

# Interactive read-only Telegram report bot (long polling; no inbound port).
# Distinct from a `telegram` notifier: it answers /status, /services, /sla, ...
# from allow-listed chats only, and can never change the host.
telegram_bot:
enabled: true
token: "${env:TELEGRAM_BOT_TOKEN:-123456789:AAF...XXXX}"
allowed_chats: [123456789, -1001234567890]
poll_interval: 30s

# Host watches: one entry per concern. This reference groups several examples
# only to keep the schema readable; real config stores each watch as its own
# `name:` document under any directory listed in paths.watches. Storage checks
Expand Down
10 changes: 10 additions & 0 deletions examples/notifiers/ops-telegram.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
notifiers:
ops-telegram:
# enabled: false # keep configured but skip delivery attempts
type: telegram
# template: default-alert
token: "${env:TELEGRAM_TOKEN}" # bot token from @BotFather; kept inside the API URL, never surfaced
chat_id: "-1001234567890" # numeric chat/channel id or @channelname
# parse_mode: MarkdownV2 # HTML, Markdown or MarkdownV2; omit for plain text
# silent: true # deliver quietly, without sound or vibration
# message_thread_id: 42 # post into a specific forum topic within a group
4 changes: 4 additions & 0 deletions internal/app/daemon.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import (
"sermo/internal/rules"
"sermo/internal/servicemgr"
"sermo/internal/state"
"sermo/internal/telegrambot"
"sermo/internal/web"
)

Expand Down Expand Up @@ -288,6 +289,9 @@ type Deps struct {
Events *EventLog
// DiagnosticLog exports scheduled diagnostics to engine.diagnostics when set.
DiagnosticLog *DiagnosticLog
// TelegramBot is the interactive read-only report bot. Optional: nil when the
// `telegram_bot` section is absent. It is refreshed on reload via UpdateConfig.
TelegramBot *telegrambot.Bot
// SystemFreshness caches system metrics so concurrent workers in one cycle
// share a computation; it must be below the scheduler interval.
SystemFreshness time.Duration
Expand Down
4 changes: 4 additions & 0 deletions internal/app/monitor.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import (
"sermo/internal/notify"
"sermo/internal/process"
"sermo/internal/rules"
"sermo/internal/telegrambot"
)

// Monitor runs service workers and host watches in reloadable generations.
Expand Down Expand Up @@ -196,6 +197,9 @@ func (m *Monitor) installGenerationLocked(ctx context.Context, newCfg *config.Co
m.deps.DiagnosticLog.UpdateConfig(newCfg)
go m.deps.DiagnosticLog.Export()
}
if m.deps.TelegramBot != nil {
m.deps.TelegramBot.UpdateConfig(telegrambot.ParseConfig(config.SectionMap(newCfg.Global.Raw, config.SectionTelegramBot)))
}
LogBuildNotices(m.Logger, "reload build", warnings)

m.startGenerationLocked(ctx, false)
Expand Down
Loading
Loading