`-m` monitor mode silently produces empty violation logs for non-root users; documented `setcap` workaround is non-functional

[sigmaSd]

Hi thanks for this project, I just started using it and I noticed that filesystem violations are not recorded, so here is a deepseek analysis:

Following the documented rootless setup for eBPF violation monitoring, fence -m starts without any error but logs zero filesystem violation lines. The documented setcap workaround cannot work, and the monitor's failure to start is silently discarded, leaving users with an empty --fence-log-file and no indication that filesystem monitoring is not running.

Environment

• fence v0.1.66 (Linux x86_64, Fedora 44 kernel)
• bpftrace v0.24.2
• kernel.unprivileged_bpf_disabled = 2

Steps to reproduce

1. Install bpftrace + socat + bwrap.
2. Follow the documented rootless recipe from docs/linux-security-features.md:

   sudo setcap cap_bpf+ep /usr/local/bin/fence
   

3. Configure a sandbox that will produce a violation (e.g. run an app that writes to a path outside allowWrite).
4. Run in monitor mode:

   fence -m --fence-log-file fence.log --settings fence.json <cmd>
   

5. Observe that fence.log stays empty even though the sandboxed app fails with "read-only file system" / EACCES on writes.

Expected

• With -m + working CAP_BPF, denied filesystem syscalls appear as [fence:ebpf] lines in the log (this is what the docs promise: "Blocked filesystem access + syscalls").
• If the eBPF monitor cannot actually run, fence should surface that clearly (not silently), and the docs' rootless instructions should be correct.

Actual

• Non-debug run: fence.log is completely empty, no warning of any kind.
• Debug run (-d) reveals the failure:

  [fence:ebpf] Started eBPF monitoring for PID 1725418
  [fence:ebpf:err] ERROR: bpftrace currently only supports running as the root user.
  

Root cause analysis

1. Caps on the fence binary don't reach bpftrace. The eBPF monitor does not load BPF itself — it shells out to the bpftrace executable (internal/sandbox/linux_ebpf.go, tryBpftrace → exec.LookPath("bpftrace")). The file capabilities on fence live in its effective/permitted sets with an empty inheritable set, so they are never inherited by the bpftrace child process. The documented setcap cap_bpf+ep /usr/local/bin/fence therefore has no effect on the actual BPF workload.

2. bpftrace v0.24.2 hard-requires geteuid() == 0 and ignores capabilities entirely. check_is_root() in src/run_bpftrace.cpp (v0.24.2) is:

   void check_is_root()
   {
     if (geteuid() != 0) {
       LOG(ERROR) << "bpftrace currently only supports running as the root user.";
       exit(1);
     }
   }

   There is no capability-based acceptance path. Even setcap cap_bpf,cap_perfmon,cap_sys_admin+ep on the bpftrace binary is rejected (verified: same error). So "grant CAP_BPF" cannot work for filesystem violation monitoring with the current bpftrace, and sudo is the only real option.

3. The monitor failure is silently swallowed. cmd/fence/main.go:

   linuxMonitors, _ = sandbox.StartLinuxMonitor(execCmd.Process.Pid, ...)

   The error is discarded. EBPFMonitor.Start() only logs its failures via fencelog.Printf gated on m.debug, and tryBpftrace returns nil once bpftrace starts (the child dies asynchronously). Result: -m reports success, the monitor is dead, and — because nothing else is denied — the log file stays empty with no diagnostic.

Suggested fixes (any subset)

1. Detect and warn (not debug-gated): before/after starting the eBPF monitor, verify the bpftrace binary is present and usable (e.g. run bpftrace --version or an attach smoke-test), and emit an unconditional [fence:ebpf] warning when filesystem monitoring will not be active. Also stop discarding StartLinuxMonitor's error in cmd/fence/main.go.
2. Make rootless work or say it can't: since bpftrace ≥ v0.24.2 gates on geteuid()==0, either (a) run bpftrace through a rootless-capable path (e.g. invoke via a setuid/sudo wrapper, or exec bpftrace under the granted capabilities by setting the inheritable set), or (b) update docs/linux-security-features.md to state plainly that filesystem violation monitoring on Linux requires running fence as root, and that the setcap cap_bpf+ep fence recipe only enables... (nothing observable today).
3. Docs fix: correct the "grant CAP_BPF" section (Enabling eBPF Monitoring) so it doesn't promise a working rootless path, and note the --linux-features "CAP_BPF available" status is misleading because it checks fence's own caps, not the bpftrace child's ability to run.
4. Consider not requiring root: replace the bpftrace dependency with an in-process eBPF tracepoint program (fence already uses libbpf-style BPF for its seccomp filter), which would make CAP_BPF on the fence binary actually sufficient.

Notes

• Network denials ([fence:http]/[fence:socks]) log fine without root; only filesystem/syscall violations are affected.
• --linux-features currently reports eBPF monitor ... ok (CAP_BPF available) even when this failure will occur, since it checks the fence process's own capabilities rather than end-to-end monitor viability.

[sigmaSd]

Update — root cause confirmed. The rootless setcap recipe in the docs cannot work with the bpftrace fence shells out to:

1. Caps set on the fence binary are not inherited by the bpftrace child (fence's inheritable set is empty).
2. bpftrace ≤ v0.24.2 hard-gates on geteuid() == 0 (check_is_root() in src/run_bpftrace.cpp) and ignores capabilities entirely. bpftrace #4773 (merged Oct 2025) replaced this with a capability check, so newer bpftrace builds do run rootless — but it requires CAP_BPF, CAP_PERFMON, CAP_DAC_READ_SEARCH, and CAP_DAC_OVERRIDE on the bpftrace binary itself.

The two actionable items from this report still stand:

• UX bug: when the eBPF monitor can't actually run, fence -m logs nothing and never tells the user (cmd/fence/main.go discards the StartLinuxMonitor error; EBPFMonitor failures are debug-gated). Should emit an unconditional warning.
• Docs: "Enabling eBPF Monitoring" should instruct setcap on bpftrace with the four caps above (plus a minimum bpftrace version), not on the fence binary.

[sigmaSd]

After debugging more , it seems like even when setting up bpftrace correctly I don't see denied paths. for context I just wanted to run fence --settings ~/dev/fence/grok.json -m --fence-log-file ~/dev/fence/fence.log grok and see in the logs that it got denied access to ~/.grok etc , but it seems this is a limitation of fence

deepseek:
it's a fence limitation — and it's worse than "no paths"
Layer 1 — no paths (fence's design): its bpftrace script hooks sys_exit_openat, and exit tracepoints only carry the return value. The filename exists only on the enter side. Fence never reads it. Documented limitation, low priority for them.

Layer 2 — the monitor can't see the sandbox at all (real bug): fence's sandbox runs bwrap with --unshare-pid. bpftrace reports the sandboxed process's inner-namespace pid (2–50), which never satisfies fence's pid >= SANDBOX_PID filter. So every event it logs — localsearch-ext, crashhelper, ps — is an unrelated host process (noise), and actual sandbox violations are invisible. Fence's docs even admit the leak side of this.

Layer 3 — this kernel fights back: on your kernel (7.1), the per-syscall tracepoints (sys_enter_openat etc.) barely fire at all under bpftrace (1 event vs 361 for raw_syscalls on the same workload), and str() on tracepoint string args silently errors (must use the __data_loc field). My wrapper worked around 1 and 2, but layer 3 makes it flaky.

[sigmaSd]

The workaround for now is

strace -f -e trace=%file -o /tmp/grok.strace \
      fence --settings ~/dev/fence/grok.json -- grok

grep -E 'EROFS|EACCES' /tmp/grok.strace | hx