[bfdsyncd]: BFD hardware offload for BGP sessions - #4588
Draft
vaishnav-nexthop wants to merge 1 commit into
Draft
Conversation
Add a new bfdsyncd daemon that bridges FRR's bfdd to BfdOrch via APP_DB,
enabling BGP BFD sessions to run entirely on the ASIC instead of in
software. Offloading the session to hardware removes the per-session
CPU/jitter cost and is what makes sub-second timers (down to ~300 ms)
viable across a large BGP neighbor count.
End-to-end split:
FRR bfdd --BFDDP/TCP--> bfdsyncd --APP_DB--> BfdOrch --SAI--> ASIC
(state machine, (this PR, (already in (probes
user space) bgp container) swss container) on wire)
bfdd keeps the BFD state machine and protocol logic; bfdsyncd is the
BFDDP<->Redis bridge; BfdOrch is the Redis<->SAI bridge.
bfdsyncd/ -- new daemon, runs in the bgp container alongside FRR
(bfdd is launched with --dplaneaddr ipv4c:127.0.0.1 so it talks
to bfdsyncd over BFDDP/TCP on 127.0.0.1). Opens the BFDDP control
channel, parses BFD_DP_SESSION_ADD / DELETE / UPDATE messages, and
translates them into APP_BFD_SESSION_TABLE writes. Reads back state
transitions from STATE_DB and pushes them to bfdd. Warm-restart
aware via the existing warmRestartHelper.
orchagent/bfdorch.cpp -- on a BFD session state change, read the
remote-side SAI attrs (REMOTE_DISCRIMINATOR, REMOTE_MULTIPLIER,
REMOTE_MIN_RX, REMOTE_MIN_TX) and surface them in STATE_DB so the
operator can see negotiated timers. create_bfd_session() now treats
a re-create on an existing key as an attribute update
(set_bfd_session_attribute) instead of a hard error, which lets bfdd
retune timers without bringing the session down. A missing
bfd_session_lookup entry at state-change time logs and continues
instead of dereferencing.
BgpGlobalStateOrch -- the SYSTEM_DEFAULTS|software_bfd override is now
surfaced at WARN with a detailed message so operators see when the
override is actually taking effect.
tests/mock_tests/bfdsyncd/test_bfdlink.cpp -- gtest coverage for
BFDDP frame parsing, key composition, and APP_DB writes.
How verified:
* Unit tests: `make check` (tests_bfdsyncd: 8/8 pass) on both bookworm
(GCC 13) and trixie (GCC 14.2) slave images.
* Hardware (Broadcom DNX): brought up BGP neighbors with BFD enabled,
set min_tx=300 min_rx=300 multiplier=3 in BFD_SESSION_TABLE.
Confirmed session in ASIC_DB with SAI_BFD_SESSION_ATTR_HW_LOOKUP_VALID
true, MIN_TX/MIN_RX=300000, MULTIPLIER=3. Link flap declares Down
within multiplier x min_rx ~= 900 ms. Renegotiating timers (push
min_rx=500) does not bring the session down -- syncd log shows
set_bfd_session_attribute, not delete + create.
Signed-off-by: vaishnav-nexthop <vaishnav@nexthop.ai>
|
Collaborator
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
Hi, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What I did
Add a new
bfdsyncddaemon that bridges FRR'sbfddtoBfdOrchviaAPP_DB, so BGP BFD sessions can run entirely on the ASIC instead of in software.End-to-end split:
bfdsyncd/— new daemon. Built here in swss but runs in thebgpcontainer alongside FRR. Opens the BFDDP control channel (FRR'sbfddis launched with--dplaneaddr ipv4c:127.0.0.1so it talks tobfdsyncdover BFDDP/TCP on127.0.0.1). ParsesBFD_DP_SESSION_ADD / DELETE / UPDATEmessages and translates them intoAPP_BFD_SESSION_TABLEwrites. Pushes BFD state transitions fromSTATE_DBback tobfdd. Warm-restart aware via the existingwarmRestartHelper.orchagent/bfdorch.cpp. On a state change, read the SAI remote-side attrs (REMOTE_DISCRIMINATOR,REMOTE_MULTIPLIER,REMOTE_MIN_RX,REMOTE_MIN_TX) and surface them inSTATE_DBso the operator can see negotiated timers.create_bfd_session()now treats a re-create on an existing key as an attribute update (set_bfd_session_attribute) instead of a hard error, sobfddcan retune timers without bringing the session down. A missingbfd_session_lookupentry at state-change time logs and continues instead of dereferencing.BgpGlobalStateOrch. TheSYSTEM_DEFAULTS|software_bfdoverride is surfaced at WARN with a detailed message so operators see when the override is actually taking effect.tests/mock_tests/bfdsyncd/test_bfdlink.cpp— gtest coverage for BFDDP frame parsing, key composition, and APP_DB writes.Why I did it
Hardware-offloaded BFD lets the ASIC drive BFD probes on the wire instead of FRR's
bfdddoing it from user space. Offloading the session to hardware removes the per-session CPU/jitter cost and is what makes sub-second timers (down to ~300 ms) actually viable across a large BGP neighbor count.How I verified it
make check(targettests_bfdsyncd) — 8/8 pass on bothbookworm(GCC 13) andtrixie(GCC 14.2)sonic-slaveimages.BFD_SESSION_TABLEwithmin_tx=300 min_rx=300 multiplier=3.SAI_BFD_SESSION_ATTR_HW_LOOKUP_VALID=true,MIN_TX/MIN_RX=300000,MULTIPLIER=3.multiplier × min_rx ≈ 900 ms.min_rx=500) does not tear down the session —syncdlog showsset_bfd_session_attributeonly, nodelete_bfd_session/create_bfd_sessionpair.Details if related
bfdsyncdruns in thebgpcontainer; FRRbfddneeds--dplaneaddr ipv4c:127.0.0.1to use it. The corresponding supervisord template change (gatingbfdsyncdoff whenSYSTEM_DEFAULTS|software_bfd=enabled) lives insonic-buildimage(dockers/docker-fpm-frr/).bfdsyncd/bfdd/bfddp_packet.his FRR's BFDDP wire-protocol definition (NetDEF, MIT-style license) included verbatim so we don't take a FRR build-time dependency just for the message layout.sonic-buildimageandsonic-mgmtchanges are being prepared.