DNS (and other) events unattributed: PEM can't read cgroup.procs (empty container-id in cri-containerd-.scope path) -> upid_to_pod_name empty

[entlein]

Recurring: ~73% of dns_events have empty pod despite a valid upid

Live on rig 6a4ce480 (k3s, node-01). px.DataFrame('dns_events') → px.upid_to_pod_name(df.upid) is empty for ~73% of rows (client-side, remote_addr=coredns clusterIP 10.43.0.10:53, local_addr empty).

Not a capture gap — a metadata gap

The upid is present (e.g. 00000006-0004-543f-0000-0000001acf92), but upid_to_pod_name returns "". Root cause is in the PEM logs:

state_manager.cc:276 Failed to read PID info for pod=<uid>, cid=
  [msg=Failed to open file /sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/
       kubepods-besteffort-pod<uid>.slice/cri-containerd-.scope/cgroup.procs]

Note cri-containerd-.scope — the container-id segment is empty. So the PEM builds a malformed cgroup path, can't enumerate that pod's PIDs, and every event carrying one of those PIDs (DNS heavily, since resolver PIDs churn) is left unattributed. local_addr is also empty for UDP DNS, and remote is the coredns service IP, so there's no query-side rescue either.

Ask

Fix the container-id resolution for the cgroup path (the cri-containerd-<cid>.scope construction) so PID->pod mapping populates. This regresses on every fresh k3s rig. Happy to run any probe query live. — dx-agent

[entlein]

Correction — the cgroup fix is the wrong fix. This must be capture-time, not a query-time join.

The RCA (PID→pod metadata missing) is right, but "fix the cgroup read" can't work: DNS resolver PIDs are short-lived, and upid_to_pod_name runs at query time, long after the process (and its cgroup) is gone. No amount of cgroup-read robustness wins that race.

Why local_addr is empty (the actual capture gap)

• bcc_bpf/socket_trace.c:150 — for UDP the socket laddr.sa_family = PX_AF_UNKNOWN (unset).
• conn_tracker.cc:106 — SetLocalAddr(...) is only called from AddConnOpenEvent, i.e. on a conn_open. UDP DNS is connectionless → no conn_open → local_addr never gets set.

So dns_events end up with: ephemeral upid (unmappable later) + empty local_addr + remote = coredns service IP. Nothing stable to attribute on.

Right fix — attribute at capture time / by stable identity

Two viable directions, both capture-side:

1. Populate local_addr for UDP from the socket's inet_sk source at send (kprobe on udp_sendmsg/udpv6_sendmsg), so dns_events carry the pod IP. Then attribution is px.ip_to_pod_id(local_addr) — a stable value that survives the process. This is the clean one.
2. Resolve upid→pod in the PEM at event time (process still alive) and stamp the pod name onto the record, so no query-time join is needed.

Either removes the dependency on a late upid/cgroup lookup. (1) also fixes the ip_to_pod_id path we tried and lets the AE steer DNS by pod like the other tables. — dx-agent

[entlein]

Best fix: capture ppid and attribute via the long-running parent (validated live)

The short-lived resolver (nslookup/getent/dig) is forked from a long-running container process (shell/app/init) that IS in pixie's live process set. So join on the parent, not the doomed ephemeral child.

Validated live — deployed a kprobe:bprm_execve tracepoint capturing curtask->real_parent->tgid:

comm:sh         pid:668488  ppid:34590    <- child ~668k, parent 34590 (stable, long-running)
comm:k3s-agent  pid:668483  ppid:961      <- parent 961 (stable)

Children churn; parents are stable and attributable.

Pixie captures no ppid today — confirmed grep -rniE '\bppid\b|parent_pid' src = 0 hits. But src/common/system/proc_parser.cc already parses /proc/<pid>/stat for num_threads (field 20) and starttime (field 22) — it just skips ppid (field 4) on the same line. Adding it is trivial.

Proposed fix:

1. In proc_parser.cc, extract field 4 (ppid) alongside the fields already read; carry it into the process/pid metadata.
2. In upid_to_pod_name (or a new upid_to_pod_name_recursive), when a upid doesn't resolve, walk pid → ppid until a tracked ancestor resolves → that pod. Cheap, and it fixes UDP DNS, short-lived recon, and any forked tool — not just DNS.

This beats the eBPF local_addr-for-UDP route: ppid is one field away in code pixie already runs, and it generalizes to all short-lived-process attribution. — dx-agent

[entlein]

Validated live: attribute DNS by joining pixie's pid to kubescape's process tree (no pixie change)

Kubescape's kubescape_logs.RuntimeProcessDetails.processTree already carries {pid, ppid, comm, pcomm} + the pod (via containerID/RuntimeK8sDetails) — the exact parent chain pixie lacks. So join the unattributed DNS upid_to_pid against it.

Proven on rig 6a4d837a — fired getent hosts evil-N.exfil… in the observer pod, then:

pixie dns_events:  pid 882811  pod=""   (upid_to_pod_name fails — the recurring symptom)
pixie dns_events:  pid 882888  pod=""
kubescape tree:    pid 882811  ->  observer-6fbb545fdd-r4dt8
kubescape tree:    pid 882888  ->  observer-6fbb545fdd-r4dt8
=> both DNS rows resolve to observer via the join.

pids are the same host-pid namespace (both ranges reach ~884k), so it's a clean equality join.

Scope / honesty: kubescape captures trees event/profile-scoped (62 pids in a 10-min window here; 2,562 all-time across 7 pods). So it covers app + attack processes it profiles — a broad benign/host-DNS sample matched 0 because that DNS is from processes kubescape never trees. For the evidence DNS (attack + app-pod resolvers, incl. getent/curl) it works today with zero pixie change.

Recommended: dx/AE enriches unattributed DNS via dns.pid -> kubescape_logs process-tree pid -> pod (walk to ppid if the leaf pid isn't a tree node). The proc_parser field-4 ppid add remains the pixie-native path for 100% coverage; this join is the immediate, no-rebuild win. — dx-agent