Skip to content

feat(fw): move per-source rate limiting into the XDP datapath#158

Merged
v0l merged 1 commit into
masterfrom
feat/fw-inkernel-src-rate
Jul 13, 2026
Merged

feat(fw): move per-source rate limiting into the XDP datapath#158
v0l merged 1 commit into
masterfrom
feat/fw-inkernel-src-rate

Conversation

@v0l

@v0l v0l commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Moves the per-source rate machine — counting, rate calculation, and the blocking decision — out of the userspace control loop and into the XDP datapath.

Why

The userspace design polled the entire per-source counter map (up to 256k LRU entries, 2 syscalls/entry) every 500 ms tick to compute rate deltas. That produced a chain of real production incidents:

  • cumulative-counter seeding bug (benign sources blackholed on first observation),
  • a hidden second pps threshold (src_rate_pps) invisible next to the dest limit,
  • control-loop CPU that scaled with flood history instead of live traffic (33% CPU sustained after attacks ended).

Each was patched, but the class of bug is structural: the decision was made from sampled copies of state the kernel already owns. Now the decision uses the same state the packets update.

Design

  • V4/V6_SRC_STATE: LruHashMap<ip, SrcState{window_start_ns, count, blocked_until_ns}> — fixed 1 s window, in-line per packet under mitigation: window roll → reset; count; crossing max_per_windowblocked_until = now + cooldown (re-extended each window still over-rate = natural hysteresis). Blocked sources keep being counted (rate measured pre-drop).
  • Drops gated on the dest's SOURCE_BLOCK escalation (ladder unchanged, driven by escalate_pass_pps); counting is not.
  • SRC_RATE_CFG array written by userspace at startup and on PUT /limits (max_per_window precomputed — no division in the datapath).
  • Deleted: V4/V6_SRC_COUNTERS, auto V4/V6_CIDR_SRC tries, SourceTracker/advance_source/step_sources, CIDR aggregation, spoof gate (~600 lines of policy code + tests). A spoofed high-cardinality flood churns the LRU without ever tripping a per-source limit; the port-filter/SYN-proxy layers remain the defense there.
  • New batch.rs: raw BPF_MAP_LOOKUP_BATCH reader (one syscall per ~4k entries) for all remaining scans (dest/tx counters, GC, display snapshots), with per-entry fallback latched off on old kernels.
  • Idle source states swept after 60 s by the GC timer.

Compatibility

  • Legacy escalation config keys (agg-fanout, src-exit-pct, max-real-sources, …) still parse and are ignored (regression test) — a config parse failure after a self-upgrade restart would take the firewall down.
  • src_exit_pct removed from the Limits API (serde defaults keep old PUT bodies working); dashboard drops the cooling state and src exit % field.
  • On upgrade, in-flight auto blocks (old trie) vanish; the kernel re-blocks active offenders within one window.
  • Dest/prefix detection deliberately stays in userspace: bounded (16k entries), multi-axis policy incl. carpet-bomb prefix aggregation, and enforcement state persists in DEST_STATE even if the daemon dies.

Testing

  • 43 lib + 2 bin + 15 API tests green; clippy clean.
  • Full netns e2e suite green (21 root tests: smoke 4, learning 3, mitigation 4, escalation 2, carpet_bomb 1, syn_proxy 3, gre_decap 2, scoping 2), including two new scenarios:
    • kernel_blocks_over_rate_source_and_releases_after_cooldown — kernel blocks an over-rate source and releases it after cooldown with zero userspace control ticks; low-rate neighbour in the same /24 untouched.
    • source_drops_gated_on_source_block_flag — over-rate source is counted/marked but not dropped until SOURCE_BLOCK escalates.

Work file: work/fw-inkernel-src-rate.md. Suggest releasing as v0.4.0.

The per-source rate machine (counting, rate calculation, and the blocking
decision) now lives entirely in the kernel. The eBPF datapath tracks each
source over a fixed 1s window in V4/V6_SRC_STATE and blocks over-rate
sources in-line (blocked_until, re-extended each window still over-rate);
drops engage once the destination escalates to SOURCE_BLOCK. Userspace
only arms SRC_RATE_CFG (startup + PUT /limits) and batch-reads the state
for display.

Deleted: cumulative V4/V6_SRC_COUNTERS, the auto V4/V6_CIDR_SRC block
tries, SourceTracker/advance_source/step_sources, CIDR aggregation
(plan_blocks), and the spoof gate. The seeding/counter-lifecycle bug
class is structurally gone; blocking latency is packet-exact instead of
up-to-one-500ms-tick; control-loop CPU no longer scales with flood
history.

Also: raw BPF_MAP_LOOKUP_BATCH reader (batch.rs) for all remaining map
scans (dest/tx counters, GC sweeps, state snapshots) with per-entry
fallback on old kernels — one syscall per ~4k entries instead of two per
entry; idle source states GC'd after 60s.

Compat: legacy escalation config keys (agg-fanout, src-exit-pct, spoof
gate, …) still parse and are ignored (regression-tested) so existing
config files keep loading across self-upgrades. src_exit_pct is removed
from the Limits API; the dashboard drops the cooling state and src exit
field.

Verified: 43 lib + 2 bin + 15 API tests and the full netns e2e suite
(21 root tests) pass, including two new scenarios proving the kernel
blocks and releases an over-rate source with zero userspace ticks.

Dest/prefix detection deliberately stays in userspace: bounded (16k),
multi-axis policy with prefix aggregation, and mitigation state persists
in DEST_STATE regardless of the daemon.
@v0l
v0l merged commit 989fa58 into master Jul 13, 2026
7 of 8 checks passed
@v0l
v0l deleted the feat/fw-inkernel-src-rate branch July 13, 2026 09:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant