monitoring

Pre-Alpha. This page describes behavior that may change.

Ze gives you three different ways to watch a running daemon: an auto-refreshing peer dashboard for the operator at the keyboard, a live event stream for the engineer chasing a bug, and a Prometheus endpoint for everything else. They all read the same in-process state, so a number you see in the dashboard matches a metric you scrape from Prometheus.

The live peer dashboard

ze cli monitor bgp

Auto-refreshing every two seconds. Sortable, color-coded peer table with router identity, state, uptime, and update rates. Navigate with j and k, sort with s or S, hit Enter to drop into a peer's detail page, and Esc to exit.

This is the screen you leave open on a side monitor when you are about to touch something risky. It is also the fastest way to confirm at a glance that a session you just brought up is actually moving messages.

The event stream

ze cli monitor event

The raw firehose. Filter it before it overwhelms you.

ze cli monitor event peer upstream event update direction received
ze cli monitor event event state                # Just up and down

The filters compose. peer <selector> narrows to one peer, event <type>[,<type>] selects which event types to include, and direction received or direction sent narrows to one side of the conversation. There is no exclude capability; you specify which types to include via event. The recognised event types are update, open, notification, keepalive, refresh, state, negotiated, eor, and rpki.

For scripts, pipe the stream through | json and parse it. Every event uses the same envelope: a peer block, a message block with id, direction, and type, and the per-event payload (update carries next-hop, as-path, local-preference, and the per-family NLRI lists; state carries the new FSM state).

Prometheus

Ze exposes a Prometheus endpoint when you set telemetry { prometheus { ... } } in the config. Metrics refresh every ten seconds.

The metrics you actually need most of the time are these.

Metric	Why you care
ze_peer_state{peer}	3 means Established. Anything else is a graphable problem.
ze_peer_messages_received_total{peer,type}	Update rate, broken down by message type.
ze_peer_messages_sent_total{peer,type}	Same for outbound.
ze_bgp_prefix_count{peer,family}	Current per-family prefix count for a peer.
ze_bgp_prefix_ratio{peer,family}	count / maximum. Alert above 0.9.
ze_bgp_prefix_warning_exceeded{peer,family}	1 if the warning threshold has been crossed.
ze_bgp_pool_used_ratio	Forwarding pool utilisation. Anything above 0.8 means you are close to congestion teardown.
ze_uptime_seconds	The reactor's uptime.
ze_info{version,router_id,local_as}	Tag for joins.

There are also histograms for connection timing (ze_peer_dial_seconds, ze_peer_connect_attempt_seconds, ze_peer_backoff_seconds), per-peer overflow counters, and prefix-limit teardown counters. The full list lives in the in-tree monitoring guide.

Counters that name something silently dropped

Each of these counts an action Ze took that nothing else reports to an operator. A non-zero rate is worth an alert.

Metric	Labels	What an increment means
ze_bgp_announce_dropped_oversize_total	rail, stage	An announce did not fit its build buffer and never reached one peer. Ze queries the encoded size before it writes, so the announce is abandoned whole and no truncated UPDATE goes out. The peer is not notified and the session is not disturbed.
ze_bgp_wellknown_community_suppressed_total	community	A route carrying NO_EXPORT, NO_ADVERTISE, or NO_EXPORT_SUBCONFED was withheld from one destination peer. One observation is one destination, so a route withheld from 20 peers counts 20. The suppression is not configurable and is never logged per route.
ze_bgp_update_modify_failed_total	reason	A configured egress modification (a next-hop rewrite, a community strip, a private-ASN removal) did not fit the route it applied to. Treat any increment as a policy that did not take effect.
ze_bgp_send_refused_total	process, type	A program asked to generate a message toward a peer its config does not attach it to. The peer was dropped from that command. A steady rate usually means a program widened its selector to peer *; a step at reload usually means an attach block was removed while the program kept announcing.
ze_bgp_open_in_established_total	peer	A peer tried to re-negotiate mid-session. Ze answers with a Cease and closes the connection.
ze_bgp_update_span_spill_total	peer	A received UPDATE carried more path attributes than the inline span capacity of 8, so the remainder went on the heap. A public-internet corpus of 112M routes has 99.9% at 8 or fewer.

Session lifecycle

Metric	Type	Labels
ze_peer_sessions_established_total	counter	peer
ze_peer_session_flaps_total	counter	peer
ze_peer_state_transitions_total	counter	peer, from, to
ze_peer_notifications_sent_total	counter	peer, code, subcode
ze_peer_notifications_received_total	counter	peer, code, subcode
ze_peer_session_duration_seconds	gauge	peer
ze_bgp_connect_retry_counter	gauge	peer

ze_bgp_connect_retry_counter is RFC 4271 Section 8.1.1 mandatory session attribute 2: the number of times this peer has tried to establish a session. The FSM raises it by one on each teardown that Section 8.2.2 gives an increment clause. Two events set it back to zero, and only these two: the operator starts the peer, and the operator stops it. A reconnect does not.

It is a gauge, not a counter, because those operator resets make the value go down, and a Prometheus counter that goes down reads as a counter reset to rate(). Use the value directly. The same number is on show bgp peer <address> detail as connect-retry-counter.

Plugin event subscription

Plugins are first-class consumers of the same event stream. A plugin binding under a peer declares which events it wants:

peer upstream {
    attach process my-plugin {
        receive [ update state ]
    }
}

The attach process <name> block is the whole binding. receive is what the program is fed from this peer, and send is the permission to originate toward it. A program that addresses a peer no attach process block links it to has the peer dropped from that command, and ze_bgp_send_refused_total counts it.

The plugin gets each event through its OnEvent callback. The set of recognised event types is the same as the CLI filter list, plus any extra types a plugin registers itself.

OS metrics (Netdata-compatible)

Ze includes Netdata-compatible OS collectors that expose system metrics through Prometheus. These are enabled automatically when the telemetry prometheus block is configured.

Collector	Metrics
CPU	Per-core user/system/idle/iowait/steal, load averages, context switches.
Memory	Total, available, buffers, cached, swap, slab, hugepages.
Network	Per-interface rx/tx bytes/packets/errors/drops, carrier state.
Disk	Per-device reads/writes/io-time, filesystem usage.
Conntrack	Entries, max, searches, found, new, invalid, expect.
IPv6	Per-interface in/out/forwarded/discarded counts.
Softirqs	Per-CPU softirq counts by type.
Sockstat6	IPv6 socket statistics.

Host hardware metrics

When the host inventory plugin is active, hardware metrics are exposed under the ze_host_* namespace:

Metric	Description
ze_host_memory_*	DIMM size, ECC errors, type.
ze_host_cpu_*	Core count, frequency, model.
ze_host_nic_*	Speed, carrier, firmware version.
ze_host_thermal_*	Temperature readings per sensor.
ze_host_storage_*	SMART health status, capacity, wear level.

Continuous monitoring commands

monitor ping

monitor ping <target>
monitor ping <target> interval 500ms

Continuous ICMP ping with live statistics: Sent, Recv, Loss%, Last, Min, Avg, Max, StDev. Use | log for scrollback output suitable for correlation with other events.

monitor traceroute

monitor traceroute <target>
monitor traceroute <target> | log | origin
monitor traceroute <target> | log | resolve

mtr-style continuous traceroute with per-hop loss and latency statistics. | log | origin appends one line per round and annotates hops with ASN names. | log | resolve adds reverse DNS hostnames.

monitor system netlink

monitor system netlink all
monitor system netlink link
monitor system netlink route

Streams live kernel events as JSON. Replaces ip monitor. Useful for observing route additions/deletions, link state changes, and address changes in real time.

monitor interface rate

monitor interface rate

Live per-interface rate tracking (rx/tx bytes, packets, errors, drops).

Live packet capture

show capture interface eth0 tcp port 179 count 10 format text
show capture interface eth0 tcp port 179 duration 10s format pcap

Replaces tcpdump. Uses AF_PACKET with BPF filters. Output in text or pcap format.

Diagnostics

Ze includes runtime diagnostics for production debugging. See Production Diagnostics for the full symptom-based guide.

Feature	CLI
BGP FSM history	Global event ring and per-peer state transition history.
L2TP FSM history	Per-session event ring with disconnect reasons.
Capture rings	BGP, L2TP, and BFD packet capture rings, pcap export.
VPP trace	VPP per-node packet trace capture.
TCP connectivity	show tcp-check <ip> <port> for port probing.
Path tracing	show traceroute <target> for single-shot ICMP trace.
Socket state	show system sockets replaces ss.
Kernel log	show system kernel-log replaces dmesg.
Goroutine dump	show system goroutines for leak detection.
FD monitoring	show system file-descriptors for exhaustion detection.
Profiling	show system profile cpu/heap for pprof profiles.
DNS diagnostics	show dns lookup/cache for DNS resolution and cache inspection.
Crash capture	show crashes for enriched panic reports (build/runtime metadata, ring-buffer context).
SMART disk health	show storage smart for per-device health, temperature, and self-test schedule.
PKI certificates	show pki certificate <name> pem / bundle pem / fingerprint for export and inspection.
Structured log query	Slog ring buffer query for recent structured log entries.
Component health	show health and /health endpoint with per-component status (bgp, fib, firewall, iface, plugins, vpp, ipsec, pki, l2tp).

See also

• Show commands for the read-only snapshot view.
• Production Diagnostics for symptom-based troubleshooting.
• Health Checks for the health registry and ze doctor.
• Plugins for the event-bus consumer model.
• Logging for log levels and backends.

Adapted from main/docs/guide/monitoring.md.