diff --git a/docs/ch14-zabbix-maintenance/housekeeper-events-tables.md b/docs/ch14-zabbix-maintenance/housekeeper-events-tables.md new file mode 100644 index 0000000000..c178a0348b --- /dev/null +++ b/docs/ch14-zabbix-maintenance/housekeeper-events-tables.md @@ -0,0 +1,341 @@ +``` +description: | + A deep dive into the events and event_tag tables: the five event types + Zabbix stores, how the housekeeper deletes them per source, why internal + events can flood a large database, and how a bloated events table slows + down the frontend and history widgets. Includes a SQL check to monitor + table sizes from Zabbix itself, a safe batched procedure to purge internal + events when the housekeeper falls behind, and notes on reclaiming disk + afterwards with pg_repack and autovacuum tuning. +tags: [expert] +``` + +# Housekeeping, the events tables, and internal events + +In small and medium installations the housekeeper quietly does its job and we +never think about it. In large environments, or after a single misbehaving +template starts generating millions of rows a day, the `events` table can grow +faster than the housekeeper can clean it. Once that happens the symptoms are +rarely obvious: the *Monitoring → Problems* page loads slowly, history widgets +on dashboards time out, autovacuum runs constantly, and in the worst case the +database server runs out of memory. + +This chapter explains what actually lands in the `events` and `event_tag` +tables, how the housekeeper removes it, how to monitor table growth from +Zabbix itself, and what to do when the housekeeper can no longer keep up. + +!!! info "Where this knowledge comes from" + Much of what follows is **not part of the official Zabbix documentation**. It + is based on operational experience running large environments and on reading + the Zabbix server source code (event processing) and the PostgreSQL internals. + Treat it as field-tested guidance rather than a specification: table names, + internal behaviour, and defaults can change between versions, so verify + against your own version before acting, and always take a backup before any + bulk database operation. + +--- + +## The five event types + +Everything in the `events` table belongs to one of five **sources**. The +`source` column (and the related `object` column) tells us where the event +came from: + +| `source` | Event source | Generated when… | +|:--------:|:------------------ |:-------------------------------------------- | +| 0 | Trigger | A trigger changes state (problem / resolved) | +| 1 | Network discovery | A discovery rule finds/loses a host or service | +| 2 | Autoregistration | An active agent registers itself | +| 3 | Internal | An item becomes *not supported*, or a trigger goes to *unknown* | +| 4 | Service | A business service changes state | + +The `object` column refines this further. For internal events (source `3`) the +object distinguishes an **item** that became not supported (`object = 4`) from +a **trigger** that became unknown (`object = 0`) or an **LLD rule** that failed +(`object = 5`). + +For most environments, trigger events (source `0`) are the meaningful signal, +and there are not many of them, perhaps a few tens of thousands per day. The +source that tends to explode is **internal** (source `3`). Every poll of a +broken item writes a *not supported* event, and every trigger that depends on +that item writes an *unknown* event right after it. A single template applied +to hundreds of hosts that polls one OID the hardware does not implement, or one +file that does not exist, can produce millions of internal events per day, all +of them noise. + +!!! warning "Do not run `count(*)` on a large events table" + It is tempting to check the split with + `SELECT source, count(*) FROM events GROUP BY source;`, but on a table with + tens of millions of rows that is a full sequential scan and can hurt an + already busy server. Use the planner's estimate instead (see the *SQL check* + further down): it reads `reltuples` and the most-common-values statistics and + costs almost nothing. If `source = 3` dominates, you have an internal-event + problem, not a monitoring problem. + +--- + +## The events and event_tag tables + +The events themselves live in `events`, but they are never alone: + +- **`event_tag`** stores the tags attached to each event. Because a single + event can carry many tags (inherited from the trigger, the template, and the + host), this table is almost always **several times larger than `events` + itself**, both in rows and in index size. It is usually the single biggest + table in a busy Zabbix database. +- **`event_recovery`** links a problem event to its recovery event. +- **`problem`** and **`problem_tag`** hold the currently open problems and + their tags (a working set that the frontend reads constantly). + +These tables are connected by foreign keys. Deleting a row from `events` +cascades automatically to its `event_tag` and `event_recovery` rows through +`ON DELETE CASCADE`. A few references, such as the `cause_eventid` used for +problem correlation, are **not** cascaded, which matters when we purge events +manually (see below). + +The practical consequence: when you delete one million internal events, you are +really deleting that plus several million `event_tag` rows and maintaining their +indexes. This is why bulk cleanup of the events tables is heavier than the row +count suggests, and why it is the housekeeper's most expensive job. + +--- + +## How the housekeeper cleans events + +The retention of events is configured under **Administration → Housekeeping**, +and crucially the *Events and problems* section has **separate storage periods +per source**: Trigger, Internal, Network discovery, and Autoregistration. The +default for internal events is short (often `1d`), while trigger events are kept +for much longer. This is deliberate: internal events are high-volume and +low-value, so Zabbix expects to throw them away quickly. + +Two server parameters in `zabbix_server.conf` govern how the housekeeper runs: + +- **`HousekeepingFrequency`** (default `1h`) is how often the housekeeper runs. +- **`MaxHousekeeperDelete`** (default `5000`) caps how many rows are removed per + cycle for a given cleanup rule, so a single cycle cannot lock the database for + too long. + +For each table the housekeeper finds the rows older than the configured period +and deletes them in bounded batches. You can watch it work in the server log: + +```text +housekeeper [deleted 0 hist/trends, 8560 items/triggers, 221730 events, +297920 problems, 796 sessions, 0 records in 4711.454 sec, idle for 1 hour(s)] +``` + +The key numbers are `events` and `problems`, and the time it took. If a cycle +runs for almost as long as `HousekeepingFrequency`, or you see the deleted +counts staying high cycle after cycle without the table shrinking, the +housekeeper is **falling behind**: it is deleting as fast as it can but new +internal events arrive faster. + +!!! warning + A housekeeper that never catches up is a symptom, not the disease. The + disease is the rate of internal events being generated. Purging helps once, + but unless you stop the source (see *The real fix* below) the table will + simply fill again. + +--- + +## Why a bloated events table hurts the frontend + +It is easy to think of the events tables as write-only archives, but the +frontend reads them constantly: + +- **Monitoring → Problems** queries `problem` and `event_tag` on every refresh. + When those tables are bloated with dead rows, the `SELECT` slows down, and + several operators opening the page at once can pile up long-running queries. +- **Item value** and **history/SVG graph** widgets on dashboards read from + `events` (and history) to draw their data and markers. On a bloated table + these widgets become slow or time out, which users experience as "the + dashboard is broken". +- On PostgreSQL, a table with hundreds of millions of dead rows keeps + **autovacuum** busy for hours and inflates planner estimates, making + unrelated queries slower too. + +In other words, an events table that has been allowed to grow unchecked does +not just waste disk; it degrades the interactive experience of the whole +product. + +--- + +## Monitoring table sizes from Zabbix (large environments) + +In a large environment you want to know the events tables are growing *before* +the frontend slows down. You can monitor this from Zabbix itself with a single +low-cost SQL check, using estimates rather than expensive `count(*)` scans. + +On PostgreSQL, sizes come from the catalog and row counts from the planner's +`reltuples` estimate, both essentially free: + +```sql +SELECT relname AS name, + pg_total_relation_size(c.oid) AS bytes, + round(c.reltuples)::bigint AS approx_rows +FROM pg_class c +JOIN pg_namespace n ON n.oid = c.relnamespace +WHERE n.nspname = 'public' + AND relname IN ('events', 'event_tag', 'event_recovery', 'problem'); +``` + +To see the per-source breakdown without scanning the table, read the +most-common-values statistics that `ANALYZE` already collects: + +```sql +SELECT s.val AS source, + round(c.reltuples * s.freq)::bigint AS approx_rows +FROM pg_class c +CROSS JOIN LATERAL unnest( + (SELECT most_common_vals::text::int[] FROM pg_stats + WHERE tablename = 'events' AND attname = 'source'), + (SELECT most_common_freqs FROM pg_stats + WHERE tablename = 'events' AND attname = 'source') +) AS s(val, freq) +WHERE c.relname = 'events'; +``` + +Wire this into Zabbix with the **Zabbix agent 2** PostgreSQL plugin +(`Plugins.PostgreSQL.CustomQueriesPath`) or a `UserParameter`, returning the +result as JSON from a single master item. Then create dependent items that +extract each table size and each per-source row count, and add triggers such as: + +- `events` table larger than a chosen ceiling, +- internal-event rows above a threshold (the housekeeper is losing the race), +- daily growth rate of `events` exceeding what the housekeeper deletes per day. + +The point is to turn "the database feels slow" into a graph you can see trending +days in advance. + +--- + +## Purging internal events when the housekeeper cannot keep up + +Sometimes you inherit a database with a backlog of hundreds of millions of +internal events, far more than the housekeeper can clear at `MaxHousekeeperDelete` +rows per cycle. In that case a one-off, controlled bulk purge is appropriate. + +The rules for doing this safely are: + +1. **Take a backup first**, and ideally test on a copy. +2. **Delete in small batches with a commit per batch.** A single + `DELETE` of tens of millions of rows generates enormous WAL, bloats the table + instantly, and can take the database down. Batches of 50k–100k rows keep each + transaction short. +3. **Clear the non-cascading references first** (`cause_eventid`), then delete + from `events` and let `ON DELETE CASCADE` remove the `event_tag` and + `event_recovery` children. +4. **Watch replication.** If you run replicas, pause when they fall behind. +5. **Run it close to the database** (on the DB host or as an in-cluster job), + not over a slow remote connection. + +The following is a **bash** script (not SQL) that drives `psql` in a loop, +following these rules. Run it from a shell on, or close to, the database server: + +```bash +SRC=3 # 3 = internal events +DAYS=1 # keep the last day, delete everything older +BATCH=100000 +CUTOFF=$(psql -qtAc "SELECT extract(epoch from now())::int - $DAYS*86400") + +while : ; do + out=$(psql -v ON_ERROR_STOP=1 -c "BEGIN; + CREATE TEMP TABLE _d ON COMMIT DROP AS + SELECT eventid FROM events + WHERE source=$SRC AND clock < $CUTOFF + LIMIT $BATCH; + UPDATE problem SET cause_eventid = NULL + WHERE cause_eventid IN (SELECT eventid FROM _d); + DELETE FROM events + WHERE eventid IN (SELECT eventid FROM _d); + COMMIT;") + n=$(echo "$out" | grep -oE 'DELETE [0-9]+' | awk '{print $2}' | tail -1) + [ "${n:-0}" -eq 0 ] && break # nothing left to delete + sleep 0.1 # let other work through +done +``` + +The `LIMIT` plus the loop is what makes this safe: each iteration is a short +transaction, the temporary table is dropped on commit, and the cascade keeps the +child tables consistent without us touching them directly. Adjust `SRC` and +`DAYS` to target a different source or retention. + +!!! danger + This deletes data permanently. Double-check `SRC` and the `clock` cutoff + before running, and never point it at trigger events (`source = 0`) unless + you really mean to discard your problem history. + +--- + +## After the purge: reclaiming disk + +Deleting rows does **not** return disk space to the operating system. PostgreSQL +marks the rows dead; the space is reused by future inserts but the files stay the +same size. After a large purge you will see the row count drop while +`pg_total_relation_size` stays flat. + +To actually shrink the files: + +- **`pg_repack`** rebuilds a table and its indexes online, with only brief locks, + and returns the freed space to the OS. This is the right tool on a production + database for `event_tag`, `events`, and `event_recovery`. +- **`VACUUM FULL`** does the same but takes an exclusive lock for the whole + operation, which on a multi-hundred-gigabyte table means hours of downtime. + Avoid it on production. + +Two tuning notes learned the hard way: + +- A plain `VACUUM` after a huge delete can run for many hours because the default + `autovacuum_work_mem` / `maintenance_work_mem` is too small to hold all the dead + tuple identifiers at once, forcing the vacuum to scan every index many times + over. Raising `autovacuum_work_mem` (a few GB) lets it do a single index pass. +- A long-running vacuum holds back the oldest transaction id, which can trip the + "oldest XID" health alert in the PostgreSQL template. That alert is usually + cosmetic in this situation: it clears as soon as the vacuum finishes and + releases its snapshot. + +--- + +## The real fix is upstream + +Purging and repacking treat the symptom. The lasting fix is to stop generating +internal events in the first place: + +- **Do not collect what you cannot read.** Tighten low-level discovery filters so + virtual or absent entities are never discovered, and use the *Check for not + supported value* preprocessing step with *Discard value* for the errors you + cannot filter out. This is covered in detail in the preprocessing section of + [Chapter 4](../ch04-zabbix-collecting-data/preprocessing.md). +- **Use trigger dependencies for downstream noise.** A dependent trigger whose + master is already in a problem state does **not** generate a problem event at + all, so no row is written. This is fundamentally different from maintenance + suppression, which still writes the event (and a row in `event_suppress`) and + merely hides it from view. If your goal is to keep the `events` table small, + dependencies reduce what is written; suppression does not. + +Get the source rate under control and the housekeeper, on its default settings, +will keep the events tables healthy on its own. + +--- + +## Questions + + * What is stored in the `events` table, and how do the five `source` values + differ? + * Why is `event_tag` usually larger than `events`, and what happens to it when + you delete an event? + * Where do you configure how long internal events are kept, and why is that + period shorter than for trigger events? + * How can you tell from the server log that the housekeeper is falling behind? + * Why does deleting millions of rows not shrink the database on disk, and what + do you use to actually reclaim the space? + * Why does a trigger dependency reduce the size of the `events` table while a + maintenance period does not? + +--- + +## Useful URLs + + * [https://www.zabbix.com/documentation/current/en/manual/web_interface/frontend_sections/administration/housekeeping](https://www.zabbix.com/documentation/current/en/manual/web_interface/frontend_sections/administration/housekeeping) + * [https://www.zabbix.com/documentation/current/en/manual/config/events](https://www.zabbix.com/documentation/current/en/manual/config/events) + * [https://github.com/reorg/pg_repack](https://github.com/reorg/pg_repack) diff --git a/mkdocs.yml b/mkdocs.yml index 81dc21b512..13f8b004d1 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -117,6 +117,7 @@ nav: - ch14-zabbix-maintenance/chapter.md - maintenance: ch14-zabbix-maintenance/maintenance.md - Internal health: ch14-zabbix-maintenance/internal-health.md + - Housekeeping and the events tables: ch14-zabbix-maintenance/housekeeper-events-tables.md - Backups: ch14-zabbix-maintenance/taking-backups.md - Upgrades: ch14-zabbix-maintenance/upgrades.md - "Chapter 15 : Real world examples":