feat: per-process network straggler detection (procfs + eBPF)

[shvbsle]

Adds per-process network telemetry to detect network-induced stragglers in distributed training jobs.

What it detects

Fault type	Metric	Source
Latency	net.tcp_rtt_us	eBPF kprobe on tcp_rtt_estimator
Packet loss	net.tcp_retransmits	/proc/<pid>/net/snmp
Bandwidth throttle	net.tcp_out_segs + net.tcp_tx_queue_bytes	/proc/<pid>/net/snmp + /proc/<pid>/net/tcp

How it works

Kitty watches processes that have KITTY_WATCH=1 in their environment. For each watched process:

1. A background ProcessState task (5s refresh) resolves PIDs, parses /proc/<pid>/net/tcp for local ports and tx_queue. Optionally extracts RANK and JOB_NAME from environ as metric labels if available (not required — only KITTY_WATCH=1 is mandatory).
2. The network collector reads /proc/<pid>/net/snmp every 500ms for monotonic TCP counters.
3. The eBPF collector attaches a kprobe to tcp_rtt_estimator and captures raw (unsmoothed) RTT samples, attributed to the correct process via port matching.

Why raw RTT via eBPF

We tried and rejected several alternatives:

• SRTT from /proc/net/tcp: doesn't decay after fault removal (exponential smoothing)
• kprobe on tcp_sendmsg duration: sendmsg returns before qdisc delay is applied
• kprobe on tcp_recvmsg duration: data already in socket buffer by the time recvmsg runs

tcp_rtt_estimator receives the raw RTT sample as its second argument (mrtt_us) before any smoothing is applied. This value jumps immediately when latency is introduced and drops immediately when removed.

Key design decisions

• tcp_rtt_estimator runs in softirq context (not process context), so bpf_get_current_pid_tgid() returns an unrelated process. Instead, the BPF program reads the socket's local port from struct sock, and userspace maps port→pid by discovering NCCL connections from /proc/<pid>/net/tcp (refreshed every 5s in the snapshot). No hardcoded ports — they're whatever NCCL allocates at runtime.
• ProcessState is a shared snapshot that both network and eBPF collectors consume — no duplicated procfs reads
• /proc/<pid>/net/snmp is read per-tick (500ms) because counters must be fresh for rate computation; everything else is cached in the 5s snapshot

Validation

Tested with kfault inject-one on a 4-node GPT-2 DDP training job (g4dn.xlarge, 1x T4 per node):

Fault	Signal	Baseline → During fault → After removal
50ms netem delay	tcp_rtt_us	2ms → 51ms → 3ms
5% netem loss	tcp_retransmits	flat → +510/min → flat
100Mbit tbf	tcp_out_segs	~150k/2s → 0 → resumes

Usage

Add to training pod env:

env:
  - name: KITTY_WATCH
    value: "1"

Metrics appear at :9100/metrics labeled with {pid, rank, job}.