production diagnostics

Pre-Alpha. This page describes behavior that may change.

Symptom-based troubleshooting using Ze's built-in diagnostic commands. All commands work on gokrazy appliances without external Linux tools: no ss, no dmesg, no lsof, no dig, no tcpdump, no mtr.

Quick reference

Symptom	First command
BGP session won't establish	show tcp-check <peer-ip> 179
Path/routing issue	show traceroute <dest> or monitor traceroute <dest>
BGP session flapping	show system kernel-log level warning count 50
High CPU	show system profile cpu duration 10s
Memory leak	show system profile heap
FD exhaustion	show system file-descriptors summary
Goroutine leak	show system goroutines summary
DNS failure	show dns lookup <name> type A
Latency/reachability	monitor ping <target>
Process killed	show system kernel-log level err
Route/link/addr changes	monitor system netlink all
Packet-level debugging	show capture interface eth0 tcp port 179 count 10 format text
Config commits and nothing happens	ze doctor --json, look for doctor-config-root-unclaimed
IPsec peers never establish	ze doctor --json, look for doctor-ipsec-iface

Diagnostic commands

TCP connectivity probe

show tcp-check <peer-ip> 179

If "refused": peer is not listening. If "timeout": firewall or routing issue.

Path tracing

show traceroute <peer-ip>
monitor traceroute <target>
monitor traceroute <target> | log | origin

show traceroute runs a single trace. monitor traceroute is an mtr-style continuous trace with per-hop loss and latency statistics. | log | origin annotates each hop with ASN names. | log | resolve adds reverse DNS hostnames.

Continuous ping

monitor ping <target>
monitor ping <target> interval 500ms

Shows live stats: Sent, Recv, Loss%, Last, Min, Avg, Max, StDev. Use | log for scrollback output suitable for correlation with other events.

Kernel log inspection

show system kernel-log level warning count 50
show system kernel-log level err

Replaces dmesg. Reads from /dev/kmsg on Linux.

CPU and memory profiling

show system profile cpu duration 10s
show system profile heap

Returns base64-encoded pprof data. Decode with go tool pprof.

Goroutine inspection

show system goroutines summary
show system goroutines blocked
show system goroutines full

A steadily increasing count indicates a leak. Large numbers stuck in "chan receive" or "select" with the same stack suggest leaked goroutines.

File descriptor usage

show system file-descriptors summary
show system file-descriptors detail

If total is close to soft-limit, the process is near exhaustion.

Socket state

show system sockets tcp state ESTABLISHED port 179
show system sockets

Replaces ss. Shows TCP/UDP socket state.

DNS resolution and cache

show dns lookup <hostname> type A
show dns cache stats
show dns cache list
show dns cache record <hostname>
clear dns cache

Cache listing shows all entries with remaining TTL. Selective delete and full flush are available.

Live packet capture

show capture interface eth0 tcp port 179 count 10 format text
show capture interface eth0 tcp port 179 duration 10s format pcap

Replaces tcpdump. Uses AF_PACKET with BPF filters. Output in text or pcap format.

Netlink monitoring

monitor system netlink all
monitor system netlink link
monitor system netlink route

Streams live kernel events as JSON. Replaces ip monitor. Useful for observing route, link, and address changes in real time.

Process memory map

show system memory
show system memory-map

Reads from /proc/self/status. Compare VmRSS with Go heap-in-use.

System metrics

show system cpu
show system conntrack

Config accepted, feature does nothing

A config block that commits cleanly and changes no behavior usually reaches nobody. Ze picks the plugins for a config change by matching the changed path against the config roots each plugin declares, and a path that matches nothing is stored and delivered to no one.

ze doctor --json
ze explain doctor-config-root-unclaimed

The diagnostic names the subtree. The usual cause is a plugin that is not compiled into this binary or did not start; check with ze plugin list.

IPsec peers never establish

ze doctor --json
ze explain doctor-ipsec-iface

Three distinct faults share this code, and each used to be silent. The vpn ipsec section failing to parse (a malformed esp-group or ike-group) was reported by no layer at all, so ze doctor said "ready": true over a section that cannot be used. An empty interface leaf passes ze config validate and resolves to nothing at runtime, so every peer without an explicit local-address fails to establish with only a debug-level line about a missing IPv4 address. An interface name absent from the host gives the same outcome.

Crash reports

Ze captures a Go panic's stack trace, with the last 64 log entries before it, and forwards it to syslog at LOG_CRIT. After the restart, show crashes lists the saved reports and show crashes latest prints the newest; ze show crashes reads them even with no daemon running. ze.crash.dir overrides the directory and ze.crash.keep (default 5) bounds how many are kept. On an appliance a kernel panic is captured too: see Appliance.

What ze doctor checks

ze doctor runs readiness checks before and after reading the config, and every finding carries a code that ze explain <code> describes. Each check is registered by the package that owns what it reads, so a new subsystem brings its own. Checks added since August 2026 include:

Code	What it catches
doctor-store-integrity	A corrupt frame in the database/ store, or a history finding (dangling entry, wrong-hash object, dangling pointer). See Backup and restore.
doctor-aaa-no-local-fallback	TACACS+ or RADIUS is configured and no local user exists, so an unreachable server or a backend that will not build leaves no way in.
doctor-tacacs-weak-secret	A TACACS+ shared secret shorter than 16 characters.
doctor-gtsm-kernel-state	A GTSM peer (connection ttl) whose host route with hop limit 255, or whose ze_gtsm nftables table, did not install.
doctor-redistribute-unknown-source, doctor-redistribute-unknown-destination	A redistribution rule that loads but can produce nothing, because no component registers its source or no protocol consumes its destination.
doctor-crash-capture-unarmed, doctor-crash-capture-pstore	Appliance kernel crash capture configured but not armed on this boot, or its record store unreadable.

A config that uses a subsystem the running kernel cannot carry (IPsec without XFRM, MPLS without the label modules) is refused at start, naming the subsystem and the missing kernel feature. A capability that could not be determined is a doctor warning, never a refusal.

Platform detection

show system platform reports the runtime platform type and capability flags.

show system platform
show system platform | json

Detected platforms: gokrazy, systemd, container, plain-linux, darwin.

Capability flags: read-only-root, perm-available, systemd-available, gokrazy-update-socket, gokrazy-ui-available, reboot-allowed, persistent-storage-writable, fd-limit-soft-current, fd-limit-hard-max, fd-limit-raisable.

The same platform information feeds ze doctor checks (e.g. gokrazy /perm writability) and is included in ze support archives as the platform module.

Platform notes

Commands that read /proc (sockets, kernel-log, file-descriptors, memory-map) are Linux-only. On other platforms they return "not available on this platform". The remaining commands (tcp-check, traceroute, goroutines, dns, profile) work on all platforms but traceroute requires CAP_NET_RAW.

See also

• Health Checks for the health registry and report bus.
• Monitoring for live event streams and Prometheus.
• Command Reference for the full command list.

Adapted from main/docs/guide/production-diagnostics.md.