From 251e1a955d1f5e55ee2e1f9853c178359fba1f46 Mon Sep 17 00:00:00 2001 From: Baorong Liu <96146196+baorliu@users.noreply.github.com> Date: Sun, 4 Feb 2024 10:02:02 -0800 Subject: [PATCH 01/14] [bfd] BFD HW offload for BGP sessions Initial version, a proposal for BFD hardware offload for BGP sessions --- doc/bfd/BFD HW Offload for BGP session HLD.md | 354 ++++++++++++++++++ doc/bfd/images/bfdsyncd.png | Bin 0 -> 45680 bytes 2 files changed, 354 insertions(+) create mode 100644 doc/bfd/BFD HW Offload for BGP session HLD.md create mode 100644 doc/bfd/images/bfdsyncd.png diff --git a/doc/bfd/BFD HW Offload for BGP session HLD.md b/doc/bfd/BFD HW Offload for BGP session HLD.md new file mode 100644 index 00000000000..9f1adef3fcf --- /dev/null +++ b/doc/bfd/BFD HW Offload for BGP session HLD.md @@ -0,0 +1,354 @@ +# BFD HW Offload for BGP session + +# Table of Content +- [BFD HW Offload for BGP session](#bfd-hw-offload-for-bgp-session) +- [Table of Content](#table-of-content) + - [Revision](#revision) +- [About this Manual](#about-this-manual) +- [Definitions/Abbreviation](#definitionsabbreviation) + - [Table 1: Abbreviations](#table-1-abbreviations) +- [1 Requirements Overview](#1-requirements-overview) + - [1.1 Functional requirements](#11-functional-requirements) + - [1.2 CLI requirements](#12-cli-requirements) + - [1.3 Scalability and Default Values](#13-scalability-and-default-values) + - [1.4 Warm Restart requirements](#14-warm-restart-requirements) +- [2 Modules Design](#2-modules-design) + - [2.1 bfdsyncd](#21-bfdsyncd) + - [2.2 BFD DP message format](#22-bfd-dp-message-format) + - [2.3 bfddp state change format](#23-bfddp-state-change-format) + - [2.4 Orchestration Agent](#24-orchestration-agent) + - [BfdOrch](#bfdorch) + - [2.5 BFD state\_db format example](#25-bfd-state_db-format-example) + - [2.6 Local Discriminator handling](#26-local-discriminator-handling) + - [2.7 Control plane BFD](#27-control-plane-bfd) + - [2.8 CLI](#28-cli) +- [3 start the daemons](#3-start-the-daemons) + - [3.1 start bfdsyncd](#31-start-bfdsyncd) + - [3.2 start bfdd](#32-start-bfdd) + - [3.3 create bfd session from vtysh cli](#33-create-bfd-session-from-vtysh-cli) + +###### Revision + +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 0.1| 02/04/2024 | Baorong Liu | Initial proposal | + + +# About this Manual +This document describes a design to use HW offload BFD to monitor BGP session. The system can be configured to use BFD to monitor BGP neighbor. When BFD state goes down, it asks BGP to shutdown the connection with its neighbor and, when it goes back up, notify BGP to try to connect to it. + +HW offloaded BFD can provide a faster failure detection and more BFD sessions, compare to existing SW BFD. + +# Definitions/Abbreviation +###### Table 1: Abbreviations + +| | | +|--------------------------|--------------------------------| +| BFD | Bidirectional Forwarding Detection | +| HW | Hardware | +| SW | Software | +| BGP | Border Gateway Protocol | +| FRR | Free Range Routing https://frrouting.org/ | +| SWSS | Switch state service | + +# 1 Requirements Overview + +## 1.1 Functional requirements + +At a high level the following should be supported: + +- Create HW offload BFD sessions upon request from frr/BGP +- Notifications on BFD session state change to frr/BGP + +## 1.2 CLI requirements +- using existing sonic CLI to show the BFD session and the corresponding status +- use existing frr BFD CLI to show bfd peer +- bfddp_request_counters/bfddp_session_counters are not supported for the first phase + + +## 1.3 Scalability and Default Values + +The BGP BFD HW offload session shares a total number of **4000** with all other features in the SONiC system. + +if any BFD session configuration not provided, will rely on bfdorch default value when create BFD session. +The default values for BFD configs if not specified explicitly is: + +| Attribute | Value | +|--------------------------|--------------------------------| +| BFD_SESSION_DEFAULT_TX_INTERVAL | 1 Sec | +| BFD_SESSION_DEFAULT_RX_INTERVAL | 1 Sec | +| BFD_SESSION_DEFAULT_DETECT_MULTIPLIER | 3 | + +## 1.4 Warm Restart requirements +No special handling for Warm restart support. + +# 2 Modules Design + +## 2.1 bfdsyncd + +SONiC BGP is based on FRR zebra framework. the protocol daemon is running as a zebra client around zebra, including a bfdd daemon to provide BFD service. + +The bfdd daemon provides a software based BFD service, it also provides an interface to external bfd dataplane (distributed BFD) through socket, follow a set of message format. + +A new component bfdsyncd will do the synchronization between bfdd and BfdOrch through bfd dataplane message and redis appl_db, to support BFD hardware offload for BGP sessions. + +The bfdsyncd listens the socket for the BFD session creation/deletion message (from bfdd), and converts the message to BFD transaction, write +the BFD transaction to redis appl_db to notify orchagent(bfdorch) to create/delete a HW offload BFD session. + +The bfdsyncd monitors BFD session state change from redis state_db, sends the BFD state change back to bfdd using bfd dataplane message. The BFD session state will be sent to BGP eventually. If BFD state is down, BGP shutdown the connection with its neighbor. If BFD state becomes UP, BGP will try to connect to its neighbor. + +![Alt text](images/bfdsyncd.png) + +## 2.2 BFD DP message format +``` +/** BFD data plane protocol version. */ +#define BFD_DP_VERSION 1 + +/** BFD data plane message types. */ +enum bfddp_message_type { + /** Ask for BFD daemon or data plane for echo packet. */ + ECHO_REQUEST = 0, + /** Answer a ECHO_REQUEST packet. */ + ECHO_REPLY = 1, + /** Add or update BFD peer session. */ + DP_ADD_SESSION = 2, + /** Delete BFD peer session. */ + DP_DELETE_SESSION = 3, + /** Tell BFD daemon state changed: timer expired or session down. */ + BFD_STATE_CHANGE = 4, + + /** Ask for BFD session counters. */ + DP_REQUEST_SESSION_COUNTERS = 5, + /** Tell BFD daemon about counters values. */ + BFD_SESSION_COUNTERS = 6, +}; +/** + * The protocol wire message header structure. + */ +struct bfddp_message_header { + /** Protocol version format. \see BFD_DP_VERSION. */ + uint8_t version; + /** Reserved / zero field. */ + uint8_t zero; + /** Message contents type. \see bfddp_message_type. */ + uint16_t type; + /** + * Message identification (to pair request/response). + * + * The ID `0` is reserved for asynchronous messages (e.g. unrequested + * messages). + */ + uint16_t id; + /** Message length. */ + uint16_t length; +}; + +/** + * The protocol wire messages structure. + */ +struct bfddp_message { + /** Message header. \see bfddp_message_header. */ + struct bfddp_message_header header; + + /** Message payload. \see bfddp_message_type. */ + union { + struct bfddp_echo echo; + struct bfddp_session session; + struct bfddp_state_change state; + struct bfddp_control_packet control; + struct bfddp_request_counters counters_req; + struct bfddp_session_counters session_counters; + } data; +}; +``` +## 2.3 bfddp state change format + +``` +/** + * `BFD_STATE_CHANGE` data payload. + */ +struct bfddp_state_change { + /** Local discriminator. */ + uint32_t lid; + /** Remote discriminator. */ + uint32_t rid; + /** Remote configurations/bits set. \see bfd_remote_flags. */ + uint32_t remote_flags; + /** Remote minimum desired transmission interval. */ + uint32_t desired_tx; + /** Remote minimum receive interval. */ + uint32_t required_rx; + /** Remote minimum echo receive interval. */ + uint32_t required_echo_rx; + /** Remote state. \see bfd_state_values.*/ + uint8_t state; + /** Remote diagnostics (if any) */ + uint8_t diagnostics; + /** Remote detection multiplier. */ + uint8_t detection_multiplier; +}; +``` + +## 2.4 Orchestration Agent + +### BfdOrch + +The frr 'show bfd peers' CLI needs to show BFD peer information for hardware offloaded BFD sessions: + - remote discriminator + - remote detect multiplier + - remote receive interval + - remote transmission interval + +``` +root@sonic:/# vtysh + +Hello, this is FRRouting (version 8.2.2). +Copyright 1996-2005 Kunihiro Ishiguro, et al. +... +sonic# show bfd peers +BFD Peers: + peer 192.168.1.29 multihop local-address 192.168.1.24 vrf default + ID: 1230045420 + Remote ID: 0 + Active mode + Minimum TTL: 254 + Status: up + Uptime: 11 second(s) + Diagnostics: ok + Remote diagnostics: ok + Peer Type: configured + Local timers: + Detect-multiplier: 3 + Receive interval: 300ms + Transmission interval: 300ms + Echo receive interval: 50ms + Echo transmission interval: disabled + Remote timers: + Detect-multiplier: 0 + Receive interval: 0ms + Transmission interval: 0ms + Echo receive interval: disabled +``` + +The above information is missing in SONiC database today, it might be available in SDK or hardware side. So need to add the following attributes support in SAI attribute get API to get the information for 'show bfd peers' CLI. + + - SAI_BFD_SESSION_ATTR_REMOTE_DISCRIMINATOR + - SAI_BFD_SESSION_ATTR_REMOTE_MULTIPLIER + - SAI_BFD_SESSION_ATTR_REMOTE_MIN_RX + - SAI_BFD_SESSION_ATTR_REMOTE_MIN_TX + +Propose to do it in the bfd session state change handling in BfdOrch. When the BFD session state becomes UP, get these attribute values and write it into BFD session state DB. The bfdsyncd monitors BFD session state change and update the values to bfdd, so the values can be shown in 'show bfd peers'. + +The get_bfd_session_attribute API need bfd session id, which is available only in BfdOrch. + +In the case of SDK or hardware does not support the above attribute, BfdOrch should not crash when the error returned from the above get attribute SAI API call + +If the above values are not written into BFD session state DB, bfdsyncd will use value 0 when send the information to bfdd. + +## 2.5 BFD state_db format example +``` + "BFD_SESSION_TABLE|default|default|192.168.1.29": { + "expireat": 1705044089.302891, + "ttl": -0.001, + "type": "hash", + "value": { + "local_addr": "192.168.1.24", + "local_discriminator": "1", + "multihop": "false", + "multiplier": "10", + "remote_discriminator": "2", + "remote_min_rx": "300", + "remote_min_tx": "300", + "remote_multiplier": "10", + "rx_interval": "300", + "state": "Up", + "tx_interval": "300", + "type": "async_active" + } + }, +``` + +## 2.6 Local Discriminator handling + +In SONiC, bfdd and BfdOrch manage the bfd local discriminator separately. +Local discriminator starts from 1 in BfdOrch. + +In bfdd,local discriminator is not less than 0x10000. +``` +static uint32_t ptm_bfd_gen_ID(void) +{ + uint32_t session_id; + + /* + * RFC 5880, Section 6.8.1. recommends that we should generate + * random session identification numbers. + */ + do { + session_id = ((frr_weak_random() << 16) & 0xFFFF0000) + | (frr_weak_random() & 0x0000FFFF); + } while (session_id == 0 || bfd_id_lookup(session_id) != NULL); + + return session_id; +} +struct bfd_session *bs_registrate(struct bfd_session *bfd) +{ + /* Registrate session into data structures. */ + bfd_key_insert(bfd); + bfd->discrs.my_discr = ptm_bfd_gen_ID(); +``` +so bfdsyncd need to do local discriminator mapping between bfdd and BfdOrch, map the bfd session key to bfdd local discriminator. When bfdsyncd get bfd session state update, so it can lookup the bfdd local discriminator using bfd session key. bfdd use its local discriminator to update the information to correct bfd session. + + +## 2.7 Control plane BFD + +A control plane BFD approach is to use FRR SW BFD. If use HW offload BFD for FRR, it is expected that all FRR BFD sessions are offloaded to HW, with a global flag to switch when start bfdd. + +## 2.8 CLI + +no new CLI introduced for this feature + +# 3 start the daemons + +## 3.1 start bfdsyncd +run bfdsyncd inside bgp container, default port number for bfd is 50700 +``` +sonic@sonic:$ docker exec -it bgp bash +root@sonic# bfdsyncd & +``` +## 3.2 start bfdd +run bfdd with option dplaneaddr inside bgp container, it connects to default bfd port. +``` +sonic@sonic:$ docker exec -it bgp bash +root@sonic# /usr/lib/frr/bfdd --dplaneaddr ipv4c:127.0.0.1 +``` +## 3.3 create bfd session from vtysh cli +``` +sonic@sonic:/var/tmp$ docker exec -it bgp bash +root@sonic:/# bfdsyncd& +[1] 262 +root@sonic:/# Waiting for bfd-client connection... v28 port 50700 + +root@sonic:/# /usr/lib/frr/bfdd --dplaneaddr ipv4c:127.0.0.1 & +[2] 270 +root@sonic:/# bfd-client connected + +root@sonic:/# vtysh + +Hello, this is FRRouting (version 8.2.2). +Copyright 1996-2005 Kunihiro Ishiguro, et al. + +sonic# config +sonic(config)# bfd +sonic(config-bfd)# peer 192.168.1.29 local-address 192.168.1.24 +sonic(config-bfd-peer)# + +``` + +``` +sonic@sonic:/var/tmp$ show bfd sum +Total number of BFD sessions: 1 +Peer Addr Interface Vrf State Type Local Addr TX Interval RX Interval Multiplier Multihop Local Discriminator +------------ ----------- ------- ------- ------------ ------------ ------------- ------------- ------------ ---------- --------------------- +192.168.1.29 default default Down async_active 192.168.1.24 300 300 3 false 1 +sonic@sonic:/var/tmp$ +``` diff --git a/doc/bfd/images/bfdsyncd.png b/doc/bfd/images/bfdsyncd.png new file mode 100644 index 0000000000000000000000000000000000000000..10858be6e1f873ae291172e125f56a514136efe9 GIT binary patch literal 45680 zcmeFZcTf~vw>L_ZB%&Z7L2{6+AUO<55|BLPC^_dG1qB2IB#UGOBD_zx+H0@1ek<%DLRDFg5T6ns4GoP@{)MzU z8rm%|8X7ta?k(WXeW?Hh8rp3M8!0JOc_}G6RaYlV8+!{hv=HulFKc)jZIS-eRwOqyzb?yW(d|V%~j?&Sipz zOYPYHj)Ii*RyJV<%JINa(#cwTFJBL}i>fI2y1MCxE}2WGyt(dnOecYbX4FR-L4Zc5 z;8RnC&A=8$XdUq?s^d&+3;CW9w%O+m8b8@a$<2!izFD4I2nTwn*t&`SE1A{9SQPOHaNkNOjG(c z%1ZdUVSQa8%v7hdAfEY7Ox!TV^q$u6nkt)CaHGmcntOOvEZwwNVyU3+N6d=E)I8^M zgpF`c9lmVLD#qAVTKp>X?%>YFcaO2NGU;*6rEY8EiYTde#JtfVUy(r|Uh9O8kR7f) zkm0PAkQy{)zi^|j;w*nUjVZ4zrjdgZi5k(`GAowRb&0sEK_|d-pGjJ>6u(L;P}#n) z)5<`NjKlQ19A`w_hSMFF+r|&pSZ8|i`MQIj;G@P{5jk*E{c5aTTx4Lc)zPvRjIXX5o zic_&B9{d`!n~vmHa9aokW!$5kd!1&Sk1&FscjLgjRuSfH{;Ig}impK6LljI_)Ct6P<ldi&bRg@p2i+rzy0!7xs$Qe<2m`yGp1)%7hgELo{>)y zfhQS|y?Q^ARVT2|yV|lvz~4Na159E?yM$gY;>daib1KF))Uuys)uII6(C&wI#_qn5 z?PL7jZoZQxS3F<>lV}}L&zF(m+A}0`Fv>_W#sO9w81D5wr>$WHj?@F4)!Cg3nPu&cES3N{%xUv6 z_bNKRlGnp;jj<#$xYk78=ZXr82xHPA+rJwdJLOtH38lXvMFmkrXsvp;aByP%H=paF z%^uvMMHL*R^J4~zB%%w=bSJ$dSf}O|zxzY~IiZA&ZwKy2bUPth)*tv{!92KV^mN4C z!RC+eS!Q9+G(C8U&G%JI6{EkyweprdTH05EN`hT%97#+eeB}_n9}JDR%ipv(pSr&x z&!%m~%({m~PY-?@wnhK#9kyw>5dH0LDm=-~4-c3qh2DC?dyb73= zOMf@vEaFI{?fjS~*6on(!rmXR9UU4vv+8i6iJ)JO4PsQqyM-khf)lLtg29})fLH-{ z{q4iIU2i9T>B3lA8HpoAq&ae_{K*m`=DI1mnY*LAMON@kSr4DMyq0Fzj@x}L$f?>Vq((!r1V6LB=y9%3Ox$z3N1-79ENJX1t+Pc%Jp=99~k?@ zj@*xUk33!Z;KWU7h}7I?Rk_}3E05+9eG)wq_4;l4#rvr&EG#rEyw%!b3(~?egdCj;&PR+R@*XQf?{Z#Bpw(_U>1*!yvibYeC8k0DaM8!`vaWw1M zRXk#BLv7n&q*E!j=KKWw1@qE%40Wz`Z){U0FDLt8$uLfjm`nRp{8N_GG#Do3)95FZ zag;ukicdZ5(r32Dm~8ZHZ&@EbCAW5*5X*;u3>zD?=A=v!$g0oq{4`xAGg&=uYG+}W zWV1H~8->kGPkK%(k7gDa<^;TKQN)?3ntZjh2IHI%&UTa9oWSk6H+TZb$C8S#dmCCCbsMQKq3`*rOMpJo8d-FicA~O#9z999EgUae?Ym9YPg~Gm7m8a7qiRG650CNxPSPX98poDg+## z1S-!I@NGApc$^6MLL!3YD(<9(Y@fB?ONd^keU36KU-j&5F?-tk_B|WTgxE%1LK(?SQRnu&4s5og%I%v# zOu)MR=AhT|n*TuP2JwFNX#UiCNl9ACY~6I-%+Mm&P}%q%i&WIT9-U61O=ahEx2)B` zN#y7761m`B^jL`4p17Nsm*3g@8p>YUhJZyg9u1zD8LbL#r)IBD=VS-mv?R1KDoG=Z z#(c(^2X~-WFR3gA+=!0zvp#3d9#lg8l5ye>}^!F|jmJ zG70*iIQl-D27QnyT7a`KlH(Xb-%HE2@Rc z=^;;qlai|93#CG3@2st?M&2q88~5?jyr{m*zL2$V_th1i_s1j#AEK|$ygyL(kmr(b zd-Bz`vlefY6{a$n;cXv3&orF(#^!BpckPmmu~1w(`>>>lXd7=^*1$^ndixLfk=C&< z+z63`u%a-c;NbJ+ZR6FpIJ1PCCt021>4Xr5XwQ9}ORIryN^k5AQVc2$mJMk1D-PHZ z+v2;ubx4S2QAt~5t0qZ$*IeNJQkO5^n3{b#E_O}jyA$VV2AP#)aONUz$Z$v+P6!Z=H@X-#N4$q5FrDjO9c*M-5W4kqP z-WposkLBgDT0ZEB>EVci4y&X?TRWV;E@H<9;a%nzNY_alxVq(Rb`C}Tk_nM{7}d*d z1i8>>t2lX{YNqQmm%_ZrOdj~~c&FJIC&iL;PiLdqe!_Ld_2&^4hriaYUQt!G_4=yj z&k~sSgmy}KV2kDWr{neE^|TrFX@zOeG6MaX#$|ux(W$Y3xkIDTPi-}A8`Y-9Bc$x| zH;y!a!6`?9MT2_y1liYQef>A})7rXPNluK89E)e}ew}AEkSBp9%Q=wG_PsM1of(EL zrk-`0Et=W~0<&$Ux60xsPxj@P+LJnuO`l*LU(<$Pw7@o z_UNOelEQ7m6?ft;w@;CTx`c+&ebE_8^h#wznM2{RLez6UlWW#JPxj0PVl-lSgd}Fw zCwZNpyXPL5Z}jz+CFmVwDu{TLaW$8%x^Hc}Sz(N%iwuY@xDM|ujvdo%rg1;!ern*} zrUTs{54`Y7ff9K&dQKXlt9~FLuRtI8igA0r}8gdS!cD2iMRu<7kJa?U`=1cm{lGGVH13HomE4F>Me?UL_kHDptL0Q4z zDGG%DH93x#zl>MdNeU29L|W*`TPi7`u>;q*Xjtf!XxP9NI&es!Q~l?)Eczof%-{Dh z(9ptd(6IjXj52V(`TGDIH+}w`G2_F~aDZ>bz~P;Z@gGm$0%v3X$2B?&(1!L*LrPvA zIBS@>T39%`Sv$Gw8ayQeZs0k;&~-yYBV)cf(B;*cegWgp*noB1b(CI;m^nFczcP0+ zwcz%4aK0G_P0U*axOA{^e?{l*VDIQA;w{eb`w0=?`sOwd1KsaO-0j2}bd*%-q?}wW z=mY?1ddwh!Pe(^5=4x&!qAo4_ukOG%aRzI5cV`hE9xpF1ZZAG=Cs!+;C&I$QJdb&K zczL;iC%D{v9Nk}eb2+*({u$&y#*wygGjp|ZcDHeIq`MjSm8p}5yEp^G&5Qo??+=|8 z-Zp=|$QbJN1}g!?hie~t}w6}!1BqH5!9VXrG~;{fmsc!vb9@Kdqh&;PfU zzux$_o;rW^d?LX6`0ri+*7X1Gs_ACoD&^z=ywhFcul4#@=f5}ptD_jt&D8(Kia&h* zeHUP~1il#0e=eE?e%sQhHL#AhY20nAM_InN=|NSFC-$o29_S;xkcO}u#|IG(o zBwDb-eL5D#|F3bd*x&!V{=WzQXORES-G6xU4^M9N_aEK)@0R5sp8Ug;e{ko&q4GbF z@xKEZ>INoL$+7Ei>QKr2`YCTXJe8YTle!L$%xI z9I{WuN`fXW2mh$09DyD&%($Np<|Yg-jN@bC3Qup<=R&ophY_gJW7RtNy}Nmguy}xG+M#e9w&A zecxY{G(XY!P2F8*EPR5f_3p#xeM3VjZ*XE1fA*QqtX~d%>sKjn#bo?5V!ZNeN#<>Z z{#*2aSrXN{jMX5nE;#R>cA<&kCA^!;>SAM7iRbKqV?Ept%g3P0Idbv z)|!(28LpF2GMhM4Z%pHF^uEV#<be_^@09a-!oOS$$p z>4b7pT(1^ANH;P}12xtny6uYE21AF3d~XY7>5%)Mn(YyJ5ChN5iN?B_{b zX?L2UX{k@CN5Erdd89Ya?(0whGReq7x&Jn^AK@KkDz`moB}W|E5u+BVzgeG;-MSD| zgfj&a+sGeJxOn*6Lk%7UIRCcZ-JSX`hcy^RP&$89wRqY6SdA$AV7bQTl!}-2k7NA_ z!%l0@jTQ{@7>pV}F1`Mwm_zZW_0TbrMw4S={^n$>1_ZkV(Qs0E zmL|qzP$?pB)fGcO2>%&s^uCa9o}@s4r|PiHf+7tB?lF`EG|@P&Mf}YGdG+-VnbC{TXllM=2y%039JV z#`OJ5{$9U60G6(uqz2oe7A1Rpf|m1m*i-xU|7gowML=u~PIDXneEl8YSmWI- zw5}S%y_sK!>TAiKyB{`wH){b6I+}Mx-u!0%pA`TBZ@6x(2LFCTF+f7`*(>&&8Avt) zjOo)CvNBjW;4ft6SeMVXsP7wWy1 z%(ta2_$}7|5FPw~8>e4u2KNv1xLHh`NmU0p#q5fHWDSqqPz-#QlP+SBM3Hf*n=MNi zH>gK3UKm*L)WJYyUSK3X#-S$Pqb5wEBDH6Mv(g{#nhFYP=hUs1o~|(F5%t=<%^x^vh_u`k)%N6WuHZgAxpsJNyPCd9Ro4!*`U=Tie-x` z6id4&7JEbmx;S80-Ga?}mHWpSvEx&`)Q?o|_xFV`u(5k8Q#vnqP&ZUF3!wrsGqt2_ zrxj8L=PA@HjFgI?*~yb-`Zd)}909SC`RZiG(6x&N?dAATH-G)-_lk{YK_P6~ya%*+ zL|}Ynvc45kQH#0erRsA@qsOEcTsilxm-8}D_U6^OcbD{Rr(P1R3yf~hVjUeFtsVB# zrdVYKXib+JQt6v*D6<7SE;aXkk7Kmhovj)UTANZVa%k`{n+tL%dmn%_MG=;>hy+J^ ztsH%HTRD2`wNm}ACOJ@xOB{I;F)%+IG0;rO%8lHm8jz(^!{|{qsYaEUx&G>K5q4>P zthQv*mXBpS)A!+<66`yL)pXKRR_7v^OtyNGGNQl*@ergq!R+F1$Nm6ywo7{`IAv(c zyGx5l2qxoPVmaWPV`n{5p}jj*lE${DCy-30RFfzF*>^nK$YHbFC#;Zz%sW5kbu&ZJ z5Az)AK2ds{JDYu(0m9V|=g;~?lMfgshX+3#ob1r0WJ0+;RMnjoS(QF9K~Ts8W>(hT zrAw_cy*^ECLuPVA_qdbYIgR-`J62s?-Qll(dib!seU=iA_{op6^$uyJdUXn9TzX8X z%NdCIOt~R3(+UY;x~an{*0%BwH}e9^2HoXVHX(S3Qa3fVs{3o?eJ-qn z{R12y7SsJV`?Dv!@Kx!ZYsgOXZEC(7+cKyg)`)yb89Ho0eM>3}UR~%1@A}0e8K3aW zZy*`zgCm>kWIuF;;J zdEYsYXt;iGXs++nL0hRQhl#YVW^1XrIUfq;2u3JHRN2zYjS_Et6 zhbFhp=7qqo*yP_o$4c~@NSLDg(ek?ax#MPxf$+vI4s5CePirM>BV;$kpdsqf(pE4A=lJm$9$>5 z_;&CrQLv^lm>v2Vw1Nw9Syn75lhL?XCq5Iv608^WCTuuDaZ`Xhz44@kvWja)++}-l zf9u6Qzyuh@t+IKoz!lzlW9VV3srY%) z3I}iqZ0_DJ^h{ipcU5-=D1#m%HQdv6%6tj^v_Pb=mJ;p*Qj2Ro|f(f)i;(+}jXH zplu*=B_~6o55?QLzNeF&TmBM$mCo;oygaw%31~h>5a{E$eY~vm9}<9juai5b1bH)S z7r!m?%)csb7k2I<6l`=_3PK@WXAHfPG6xTEt?>~1UXX;AgZ2dnQL}Hi%fVyn@0Z^p z1)J}=b#r%IQ?=-N=z+-!6NQ0QQ~09P5;moK|CmX~q_^Hc_{XBHTw8xu<4flm9dQg< zjC{=H4XzzUx6L`~)=tJsuMxGmmaD>rpX0$i!3zF%-w(a~#oy>r_BMQdQ4@rgIGT^O3lE zB{aKb8PpH6{lcu6{Oe_s(9Cdx4L>!Gfaz5ujIo8UW3nt5 zk^zx|z;>WT|vro>TkF~fA0xVI{ zPHhS}?E4V|r#FfMOwo9gE=5ZOyzPIZBhSjSJm~ML1dj;NIRB0yR*F7-wD4&_sS_0+ z_$!m#5d2W+hTz;Af`g6#g6D44tyS*Qw&So9Bs(SuLQw{jm|H>4r*JVo=E8d)O?T8r z!0_JP{QWnuil)W{17C@TM1ronAD`O9FS_(T%%zcp_@3HkrV@wchE~SHGSkIDN zs2^{L`u7^3G@WIY6$87FvneJDzwT=)XmRo#u!L7Tb|wJopPbVwE*Mx%Y?gmdS5z2! zYco{x!EeLg;zi>^D+$ftp?I~3b!nh%($$OvXwa{B{4uSilqt9!3)SGK@ z&`J%x)Sx;e3Nm^sG)uJN6A~CJAlJ<|Mm%TXmIyUXux^cI-;b|vm^nBU zJ@@9BPhnF8Dw|xpY*!VbGV&|jRj6+jjYiok8H?&b)HNPWl5$9*ycjP~>Y;2PLR73> z=P>_1Xq8wSZ@QFG)xck2x_q8hM&?Ui*W6tE)FTb6C$&{C={B9)?U(#tj8E7#)@d#= zs*dpx_d+i{1?tx>1ll!Uq25(k<))B@A-iTIds;;D9R|L+4mJ2OHPe)Wu(6i?z3S|t z0jiA(J=dmwP?`r%tv?K=^7UTdndllNFz{JV;`Q1$_{mOstX_@!L{^Smi z19_o0YlDyrg*?0eL5hUST$z=|X0h?+PwdvaxDl_tJQUuPz7%dT^L*i}Bv`^&=`%qu z9G_s!%Mq-ACo>2E3@TshDWOn%v~k2NXvNBAAl@Dd@+SndfUa_O#Xs~;vDLr-c7H3} zD@oy)<9#d}>J))r z+deU13ae0S7owhg3;NFfR1o~Cpze|JWahc{pbN6XT?DT-O<-WeB;X-2LO>nKzN4}C z?|PqQ(aJAZ`8Qd3gKU;cU*X%kTnML`A#T_Mli zJpcOZ*OR45EK9ymqaTwyh3lH^!2(>XvuBsEFi6`JToGlPWMNv=&`;VC$gaiFY3AA1 z8=}1%u3+<3p$7AinzFhaWBEswKUPP~mkPue&jwcJeN)V0DESo%l+r9+RJQf1@!r#j z>t*`F*c`7cRCUgbF~vSn0?12w*c9C+nO|_6-sC7g^3<44u~OM&uef3@@0faEcvB#* z*~3n61|;%U(g*#((^zS^l+j_{GAU~iDoIwygZjm#ntJ_>UH23&(S+#ac2y3yte*}GWX|dMR%YDa+C!CR^`(G!rA0i! znV<_IFBNt>PTR;wCNtnn7w{saF9hFl;YN3yY)@FMeUEcp@|fux;DJzo3S5{}J2Orz zj?)Wxh2pOOB=)KB=ki*cUd`5x+%v9MH}F!4deDHZu6TO#D5PVWE1Y?A6whWsLXlFi zh~yRb@|~KFub90SSLDL`EO!gHG~-vD#2X&hu3SvFD(up3gw=C@-9UCVNcQ*z_K%ia zey5l_btof5Dm)KHhR5_PqV%}Mv86Ac zXlky+zl2bGIquYtHI=3L-UDb${x`I(m-mu=k3c+cviUB^T%#5CM+!a|t3C<}fMXVA zq|!sEyljXaUtI_mseP8cF=9AGG>Nil!1wE-mF0_P=oU7I9Ql znG51#myPrYMb5PTFv{+XP;^t&Px+D2xF*bAuk|i4iCm{`MUjnNgjw8IFzMu^_9y`^ ziiZ^$8EJ7E8L*!W@RBs~-_d(kbnv}MgTXA&J7Ti%m7UF&HZOjm%Xv4Hym{4eybLP; zP3bmLApgr3MIcZ-rd5%NPyQAp6xyCmKaa-0KY5|CsvHe6sv@XFeRGtJy1=bOeG>J( z2@{6691KIeR(QqR!rVeWEpRXBv0k3mbwsM1MGB#i(=95;70|VYJQAI%#iE*H2GubR zu3o6G<16c)!!n!pcFL?;6|go6;zGwD<*M(XoVyHu@BocNp%86#KX=YQ6aJAYZ;z_?__@s}c5;Iw0kwTWJz3IQ!tTdlLrGEap~rAo?FPr z3^prVU(u(2++7_x;JsUT!DZij)JBqm#qK%z#t6*S;axV*s$1ae!J;}=@~9UC*ee^E zfFbtKtkkT%GKVsr;K)73@N1jj8D#9~@6$xx9cVgg&#@l_St5#leCdqUtFyJiW@o=o zAz=T}ez{c_Hd9d%x>Wf}3LsZ$QnE+iyY$#^uk(VB^jj7XCa4~2%w$^lv2x=9Np?K( z2+j`gX#0eh8m%HE0173o(uy}!*55xjfGdytMJztD7<39?u`oS0smjB?_bi zq2MFTjEtXe=QS(Mux9zM80qHDpYAO*nEOr?D6Nml(Q?5TuGX72GZVCp12i{AvUuA3 z&OM#(oedaYt5XU(8f=rYR}CMOv8DzVbY2|S7-6vo!scu254xFyBgOf1SmUrCRKz*N z+eB(x0Zv=7E8_DeS+dsTOsV=sDtvBve)HeiPw zuj|_Pc$>MoIbP=<9r2JU9uijNL?FkYP@tRv3u7zZdy7Ys2LS*6v35U5ig_A7P|6u` z{Q_XVRTC~4FutZsv2##5W| zJabnywK%_|etoAl&5fLss$t{m4kiTijbK_Z{%;HBF$O8AeIlem5E{KSbi<2PTE6XKATH zQ#FFA2w+g=5HH{=Qiypw0}S$fHS~#}gQT!qBq%hLWsIum@il^z{Qn2VF4T8bN)sr< zn5}%(brX>DTJ$I4PZX0B7M5sCI&@8P^@yoBJv)a6L4az4xCI8lq4VUV_aN~KKp2@< zH{k6`H^n*qDvaL>OS+y-qK5ULLhW_DdG&9F4f&rc>~0_|Ss$ilbmKn*K~^(SCggZn zbZNnPeRFdFP`tHNtIRk;DuEZCo}PG?&z=Pr(rx?o+(iMUb#IqxPt5GU=8ZZlLWMILiZz8z_4V`Ml()J;i)d5N)zCdLxkvDZ7CUOP*;j zGeo9j=Bq8D3QN?$-ZzHy+^WHG$^6@U3fHqas}#xN$4o%(1N=*EuB4j75C|pKme1_Q za%GqY9488*2h)W$xSO{a1i=)%=I>fh#ufJ&%rYR=flOU~eG=D*@_WIN&`wWI=x_5c zC%z>!g;6A8G-7vdJl?WL%00|6aLv@zd#;Ut+DlJh-nHLzw2R}VB_hu0^al_h{1-q3 zrn&kp+70qpXYL^tL*3zOfc}nR_KIAcu4U&by$jLTJnI&1yd5&rLXD%N52@MxSPmL@V`{87*m+l{v zX(v6>PWEEz(AQ=2m8(%p@OZtvpwHS~n)y|5o$=?yONXP%o@yJYtIyF=nqnJOyK&nM zh)%`)7Z9zeo`yk!^lPo@cV;TQAPDU_v1e^ob^&4R}yjS z#)D;%W0y8Y8F=6mg`UIMOnn_aHRBqEpH>^%x^|~8M*w;4;)46LX+TMC`bLk+YDVIqt^P&# z?`OR8G26e~e`y1*NTrrI%fC-bu8{GGkf}`e)8gs;_QHm=o{0wpZ@(_qBd{Wa3-sGj zst3Of1T%ktt@7&u#Sz1g1wOx>CCdLQ8}MlWEOQ-YkU*VD5p|6mP`QI}x{(j!3ssZQ zXTVkmtejHF9l*moZFrabiWuR>@tVJ2T(o zNO0=tOvPO&b`W|QM>%m`EAf&&I0j@CUg%WccJ{O-c_Lay=x z682qJ%rjf-Hf-S5$Ai2J*KNm+3GI6oiSu=^0k+LZ(bJ}rk#!=OR@43Fc1Zkw8HD-N zPB0mf(7P6eELl&L@%o+>##!nd+W)!+!V2B?{5CFccM*27KW;{CW)fDf2 zF(Lp>W;xXtZeUSLSjCo0FEN_#``Ws61gpkT97_H8Mo}w7e$*U?eLAXqiKDo z2z;hnw;#}@7zZ{L>We^uI!I7A!d8D zvdyU1gsZ2bhEDDwG4^U-_tG_0`aAYPaUq?n)!q|!%93MxzQWIXF35Q}=dPg!$m*x% zP<4sLry5#&A9*}%`9DO75XHqPU42KKj`Zy&#U>4w1X8%onYm7&>>x!7HCTytNNo3I?3aFa)9q!L^JBsTc+%%U_epc?O7RlGEkgH{Sa(6_# z{o-+4>Fjk3RpE)PTa&xZ6Uj5$CAJ{{g0MZ`K==)Uw~K z$lbI?%xVr|CLrfdGzO}ruB*|4{d^;0tByYas4+)e9IET+JTm+CKCYyqy?HQfBvV|U z#&7q@P0;c3)6>3yz%QS2ng`Cw$ZmA~^td~MWS>te%Wq$pLde-5+e}E^pkOZP~>TQIw>0`5(I!xM88Fz+=OpTrC6mxx6sqRmEfyQZ4R<*9$U|M>M!aOBtsJqGQ3KtS_`+?J02k^ z4~L+l+7SjULB4M9sLTs(wB4_01;yYKyE)7UMduR}yyl%h=G1xkN}gxu7Wna;Qhd~{ zaZC#Cf2fV@!c(*Tj#(4#GqjimYWI#O!L3V_cViD!PELLD8q_LsMOs)oi- z4J2!7mFoPwZc+Eu%MDa<4`6f<3|3&GSSG4_Z46i@_5GFu2|MbOk5vZRqM8 zEnwDGo1!@fjBXau;m;zzhOY9wOJIn@3ejoc3Xb%>71_KuBx;A!t`DqJ5E0m6Ov$)n z+?*EaP1pA|G~8g?cwK`d8r-itT;|H{;zV)QSKpw3>lS<91tBy#3OiOTg<@6gBVhU~ z85w9>oiV81wuc||HX#`cXhr(4TF~b&24l!x*G@d|>!*G%I;BW&oG^v$xUUEG__I%q zSc&vG73XPAes=! z&sXe&(6*_DwIJ@mQ6W{hXA0HYr)AVkfH!i}nA-!S4H2rG-?Ng{8^V9)5gLQ_D@=RS z8;70!Me6lg*Z5B9b$d8v+{Z}>ba|k{q~~CjwgQEC`5x_*-%X_!Dtge$sBPd9T6k;) z6b=)VGeqRC&h~)#u?uhvkS9hr9+gp>Z{N?Z@TI#2!2XB*JRk|+bq(ZEb(>uC-M2=+ z+A&md8uLeb1UkvE--pfDmAp(BQVS#Gpc?f%W?Hez-J!Bc!viz){R%9^UU^vP{Vejb z$OuN!g$r2%2?_5KURKxEbpbQ1qTrHd!mtqD&!G7@+h%P(Zq zvp!TYtEy?|EQ%hppVOe0-&yNy-c}>ZFm7|p5o@?;SYBWTO>se#p_H4w%m+y*=4q#p zUJ=Oo?FXm!A$WWhoFoWy5$)o|tG1vf?Rmt(RYS^wUUty$L;!|`mBl9vfoN^gLIgZR zOBU}akD@kxLhy^LlB|M2E4UsT}w@KMKy^R=n<1oOURLH4I=+C^eQ?r|>Jxnzr9 zkaPh4&M0~=S*p)8!DI|=j;G@Gcpbj-q;D4P7FF(NLw~`jDw3L*QVQ7g- zC1C$ZF@-znp{VEgMwit^y7ioJn)sN9qWwOb8LUlBO%@y4LEY8<8E{Ri)5WATr*>WN zmWBwH`UEk&I;TC!0TirY!QM*l8|;nEzW(Mu_UGqe-Z;1GgG=@_v{$?LLD9=tKD~Yj||<*t10M#!1z|vRb4P?0X94g$YiT|+;P>{)xN;C zr>vO51Mjz=d7LYJ=r0#4Dy4))lAZeGWDcocPnEvHX<98Ne~(FediPS}OQa%M1xA*U z15e6VXrs)yZQg)kg`pZRmm0*|LjdVjUmes!0=K{&(J;*CMefP9f|hu!3qdQSrQOFD zU-A+VrqO@~aLD8$C&@$GBnr9IMqa2j3^s4lY-&});k-`Et-Mr1K$hY>`1k;VYOl4K z_(sSh;JEnc%`L)sa$|orBftURJ??Er0P&YrL3z0v{isUNrN#jlN1$(geE|Ix3ix#+ z3Kc(x1w})XM7#NQ*=`0->ty%*H)G#T)kGs~^zBFnO;yge`67TclKG@q}ah?%GH?G{xR;CfLHo4dE-@wixi zLTW4qlD~!l?v38w)j<#{?dQj&hav~yS2m%9Bhli)V27Z_tUCL-@h3mP>zfa#`J6*v zQ|&oj!zg%9Qf4fIs|0uq%P(RV1J|WC3|*i!N5^(y4o+D8p-MW7)g@DmN#MX!szmi2 zZ&ZS)oRLpe7ZZihzSdl;m1fvVNs1$rp?)h%Tf1%g&soSPqQvJuQGHYp1V`7Cw$=4J zkiG^tobRRlpe@5-kC^!i>eaut7+h6~epm{asgbtvw6n0&E0+PEkW_LCOse@G?4O~_ss7syzVgeiCa zR+QP=c=OxHi|Fy_^`Zm)0bsvQNX(Mo_JVrdTIjR${A`l( z+tT#w?c)&eZ3)tZ-`7| zPf_K{6qRXXv#asZJ?GEJZNDDCnF_$f6<7`8RNgmp{G}oLi`9Kq=p?g1qP9 z@Ar4*!E!BROkv6a&&d#{c41_rd9Y~LCAfIr$S*}+>L^)NA6PT3`PuvvUeBL?6yqtJ z2D-itNZ!KLaTt95`U$~u65c?V<)hV00-AUNj=CgPNDD((mG}PW!<2|4H?f`mYJ^G$ z!5+U=%p(4p;vQ?C90bG=gk)^a6gZ=~eNU;{YJeNn7Fhb@VC(~1Y+td1o94Xzx9g25 zk^ZE*{rIVc>kqMCPSmdGy~TV_o#))FOInY2#MwdV1UdPEa4lJhx%Dn6O>&aH)j&i- zFZdIN&p1=sA#L))CkY-O*Np@s)^jjK%7I<2G55sgs*Ya0BH4>&kuf}WPkWQ}yN0x{ z0UX5WfbQcfop#$aMuaT}Wew6t>ZeUX9eMn3=R64uyq|M!wg=X6IYOA#zgbvj`Vkfc z4`lxqaL4QW1@}UOQwlM&!i)3$rTie&bv%!;ghfY=RCuD~P3oLNQ~>D|R&PIFYcpPJ zqfw;aP|m4SDY@r3njMS)Hf8eq9GN$*#q*7P)jDqQZLcqKSg0>vl%ov*9~=$NNAEC_ zCFNuj6BG~f{lqH8XTOY%-M4=a4whcbW}G9AMP7a*E#hqHj2sr6@|InBU<11Tc$2Jb zt32Hrld85Fq1guSH%J2O%?IT`JgPME-OdjYnQik7p2GtXI7WKIs?2)VZnpo;)mW9d zl{alrQarU*?LPi2{Mv8qXni2x_Y{T)8QyO@H=+>tb^kEuwmGax1bZf`aT1_HBLiAG+s`LI5kVl_0C*`vqSjrErk(rSGK8@aW=B{f) z96{_|&Lm5Jwb@GzGxtGj3W2N8j%#WAtr$go_K55e-ZJ7B?tX0rfnH6u-^%U{UHeWS zQ^>YG`_;G;ubpRlI3IHLwNHbUNqe}JJ@! zOrcxc2BQ?y_Vf`UbFm2)o^1Riw_n8KTx$@WLu2&M8skKY61ie2+qaSQEVR7)8F$s_ z`qRY%)Lp60sQF||TLr6Ho~&5fG}^B;t6T}|J!Ad^St^vQh4++HUmn=!9qrFYaIL8Z`vj0^J8Q@OAHplC>T z$US<%a3@&~ePa8G#egU4ZQ_n!4?!F$vKhDgt0sias%J~0`@q*zZjx=V2eM^1dsWZ& z7RcR)bcUcX_#^-2Rv*`!-I&0}#XRe=+-LQ^Kf@Dt7?aO5R9#Z{Hhk}u7YkdjN zgg-OZ8o(pRFF#S`+twY2#B9?BIW5wGqTxmbFRwMP4eDQH%#_NMbih-LmbEgPP2t3F z=TSQmT@%Af)mKGoH8BQ$Qmj&>r_*=KUdOr2ztd8Ef;#N$e+s&G=`U8+eK=yD7%5yf zHLP2)e0Rv>sPn8)?eio#entCbaW)Bs)UzfbR+BHt&=&0rwY$>dbj4deu(~LB&x`G& zMIER4T6ytWSgT{>&s%UHA~;dex5MQ3C4&D)^? zXb_4E$f!O!P?&Sz*yEWWe$eqU=?=xxSi-v)&zDU}$rW$KT=u+RirZ5Qn|(j}h>C_G ztxfpfm$;ybM?J#{M|HS{ouS7;#Ki|l27J1C0=A*pK^0JCAuZq%C|h@&H8&kwNO)jX1OziYMT*! zv%Enu(YbD>=rcF1Ag40K-r0EXT}nO6&)^eb9eq-*QZdxW0r_r$Wx;RJR9{4*W`I%) zDLWX~PPQ+ec|911<^xb7;@v%z-n;!<%=Ru)4!&6Fy?YpFrQXsI>QxWGP(P4TO?(%! z-NL}Nr8w%jokl%>*0at6?B))nKuuV~ zFOO03snzRwx=|h<*3WzCQ$7Xk6V^H~x6!YWO+svi z)(pclTJy$vexX^|TH*>lFL~$f$Zbl0@umFSIO_~0Sch+BrM&72i=YbWe?uhk$f@l# z*~I-jkm1=WC`vxoYxAw+G;9VxHn=Kp*;n6YD@_0TK`FVI+U9Hp`Fw*FDi-WYg|;) zab5P&?5|{3-63xE5_`92INR-vq88Ve#%r6*w4t9qetwh+3v|$j&ARqk0bYRbw&Qzr70%x-=3U7?-yYJNb5YqW)yooNa;46=`pQ4 z4iuLAQu(ZOH15!d6&6r-C@wiPEU{qX+_{||uJ-C@n7CX5NVa%?u}Nb+^cD0GEl<*e zR~kj?%=~RSnPNT|7NL$Gg=V z&o{GzitjxY5*$AvEHHmEH&}|4xza40tMt{R>?=9qg$^Dpx=SUThxi%}A}+I0fR{ z5z1y#i+M{9J2|_I9y*8ME9*c!>kOtZ#B|%3#S0BpbMM3Nu@{=3_Ofzt2JVXu6+4bm z&j?mp5A$b{6uD__0uCqY9D?do0fj;WsrYS~h{R7mP(;mN!%8a5fBpE%en(nQ@6%Jq zMIAz~Z(?a1GN8tX!miR1h+k6tb@xo=CJWW1dg3}#PIDgie5|(ajXCj2JFF8Iae#WS z5>g1e<|T4yD+7KGv(l%hc=v0p#~9!%Q9P52skk4snp`cmCyNpmT?e>|R|TC-Aprk?lW$AfZ$Zq|PLCn}g^4%a|NKs7fuvzcJ#p=@gC^iRNE zsm{1G9(}$C&aK0Nn}Jn;`YMJD5N0SvfPHc7y94mhiH8ON4_{MK23KzbJN+xkimF)3 z=$TamqwGSu?L%H85^yaoqPifPI7mhK)t$H>&BrLlk@u~#XSDL<31!nJKZYjOhm5Zs zy0djm`HOf)RHOquJpQpN67POb%xco7o7@Q$11gO zy#OBR_ObRawp0?x6Ad^}uCVBQi>D3=3~0K9m8=8h!(`K*NQ(zKAzn)F@j)K2M1?>g zHC%kIoT1%#`tBYh`2S(=ErY6#-hWXgl$20PT0~&eC0(MRba!`3cga?|JEXfirIqgP zkZzG?lV@#z|8wr#H|PH6&fFJw=DcuZhSAx3t@VA@^L*;-Dbj3nwA7^f*L$f*nVu8* zZ)v?)xX%lwa@(7%+Ld7sw}z9ShXaFcof2Hx)Rtm( z!zeTb8K+n0k4O3wqkb9PPO@Wdwmh8L6e_lf_>a-4`Ke zohT>41_siOL>=#SpLZb0ss0&_P5ObU)MU=EApx zGSn>U-4O>mgkH7m1q!k|oqX2&$iOi`75WBd5@^Xr@Sg+hnB2a)!vd1N#Bm|{9cWRT)~ILT>0oFqi5`s^7H zTf9Pgve5k_k_G2nI`3i+iXQzI{cu|~R6bMY!JCHhwPM#hKfJqJ!*c;6pfZ8w&=Eb=OsV2@do5D zWSDZTq{eLQCt;tvMi-&i*_im$ix1>Jw|2G&A#EkIHrp-y8V)~UpsQNlrM|x7XjWb% zmPA-Bi%^RvuwH(lltpcP%#Q-`c}XSd=oypBSJ!utq-ZoVOnNe5gP!A;S(xA^E>6)QM4SmlyKfjw<&?gkYD~S&SJiFYPaACefjPvS$2A$hzW$4qHGyxBP%$zX&e(tAO!4X3Du8}*q zO16pQi7!zMFv$##|0YX?z_6;~o+`5l6L)jC_Ei{hN_1(7dmU!|G4bAVSX}Pj2?5&S zqPL!79|%xe$$oKuoqYu)@v6s)a(U`<9Y?@KR!~@KJET{{$Pie60(1_2JliSRd6qX4osE|tIS6^zfJ!%&~LKiC+*7Bvu zM*h8E)22-dY@6SkAMS4~_{bh!lt#)zxIV>XZJu^#bFU=e%u|YrbhSzznO#IHG2+~o z2%0Xb5gqpwU$BSCODx$XNkRLcuhd09BM@Mk3VZb1&4ECv(naNSDwSxyU|A<4dw-dj zXPSa}T&~`-0pxzzo2olZ=NUO41!9lWv=HL{1;Its{3esUc{rKS>wRa2Xn8R0EV|R;NRo!NaZ5ub<8eo#L9;CHidx#FAlW!NF zs5@B9VINEfN?;I6W8YYk5H>Zmk6Uy4B zet})v3OGC^QjoKI#9W<&t@PIVTJq&vi!vELBdNLC&xa70*SKrr-rj!h_s*!Dn%$X5g~k9{R3`&iz5(m(pz;yrpEC;@JC-dT$MVIz;2km^N(8oSo@_c92QM;_{(C<+$M`ZHu|PeGax>_-W*>R>PZk7Z z^gP;81i$+*^8pGiVKdlF+puHRs%s527|#RTDf!-Sm3m)l-e5>NcjloIlN?Mw^lSxuGP}fD38Hsr3V}`^( zPm8-~z4@AcjuIyjPzR|ryp4Y7n?FR)wdi|dKj&88ha-CHYO*l}y`ptDSK8zimE-Bb z{-}X6{!n>8SW|NI>?jg-KyWI*=)5JN#V}N`h`H?!7RRu_A>IE%K((xBNjV6|4?}({ zO!p63^2vT4x`*U?5m#^ecT-wQB&bBoNolC7aF|h#Z`jG&4m%aSz9>L>dn?xiHfM}Q zm56@)RpC5}l(ftV;Y*h}Dn7API2RK+NW67JBUW5XVX?ZCdizI*i6Gsfbdq60sx|#4 zSW2CUgy#9Vwa-{~)$nhGF=PX@wYJ~4DLnSX#@=*P+qt?ovRT!c-cn1 zsy94Dd$=RK6w{gs^zOYd1*?R**1^C&sm=bf0Hm809zr*2~? z5TJxipf789=QZk|7?@c2cw%Ucl$iXHf2-y9(-Dh>qC1)8f$FLc;IX>npjRM@=mCEl z?t;*3gzM`HkB5-Buop@O@82%Q>e5;~w6>g#deuwDIfVTVmTfDMIGQcXJ*X{d2z@>Qe7YixwCFj1xSM-VeI9i302Bks`ZL}9s z=|xnw256tnL~?(((iL6&Kx9Fdcg^b8_}K9>MAaq`TnP`qpG3&WOa1yv*K#&Tn9pNs|dKBR> zH(M%Ho}|s~bHg8ZN#w1~pQ}|5*o8KFu6>`(JQmZ{w(UR$j;=)v6c#oFMT|t`IDnW2 zlkr4R`d0ob%rM~eVJ!Il0sHdXXfBZdLY5_0TD_(|vYj5AA>kEYBqHO5o{8wWw|ql` zT&-bIS!JoeLoPQPpA`_-ZA-sBD`GY7O?vjO36?9tyF_{V*MqAVLL(~fQFJvdBjD6y z%yy-Y4oNgPY@?blilRvVwCZV!3LR!<6VvtNJw$EctLTo^-GzRv&?DvUm~F_KPL<$y zGnW#l_TMq&36&?4)#VaH+xRpM8CeA(H%T0>L>xafeG3=3{(kCJ!mbb1*gT~uV zQFTs7Q2-79iT&l-lkxVt{Bm-tUnVUS)zi1wvk#-nqB%8`D3~NJ#a0XTf)#D5`y-P( zG511wx;@;HT&f~&7yfo^rb`i!^!C1PF#tj!x!rw6`Jf?Jsi^CaRyiw^ckvQkurE6Q z!l9r&o#EV}iR}8QF72I?(8H|PefU<(=$;3};8Z`BO|ThNo23ZJAV5(B8Md~b$=r6U zAnhh#-2dWS1S{{&ucyJ8RDGg{y$AKmH>u4DrY29r%^yzww`Vgt1GS)J{hfL%+z+WJaDPw&hgm{6!paJ@!GGpjsc@$r~s>0<&ZmQjFgujbL z{xMRMyEHor+Op))nPCMFkjZS=z{G-y2W_?#LC${d*zP8E=(L^cQnb6rVu5kk z41j!fG|1(mL&Qr;{NXm|lNl)5$u9zDl#D`x*>C3U6?(kYqZ!HSs1WCa%2x$D%n17A z2&3N(C-l>gzpijkD-1oN^=6Q8|U*V}yU zBe}%7&+eF9k=WixYCnD32S9V=;jV7w+6U{<&_;Pw$!lRz!wm@!!_L9%Uz)vz6T2d5 z=d30A`+J8z1-rd}7i7GmnBpbbYG>5!Pdt|AM|czY@4XExZ|uGe_Y`ZLt!Nfs_}wo} z?~11ToZXwvj5QNzc*%A-xNi@qZvNOW8_;`_HYRVf!`=+*G!rZM?Ik@!YR(ElKI*!n zz^0n^xjR>Nh>x!;oj=-W;iv!QNDA}bJG`7Ix$RxAq||E_G*E=LXwenSuE!e}4I;%I zuX{&w>AJ~?-9_L3yuZoO-MIHzVQ7<5w~|_kJ4SazgQ%|!+rBHv$dav?cPpA$m3*A4 zawHj78z-YW)VT;Nx=fnUX`xKEuK5vG?(kQS%VA536N6|M$MR7n(X#t~sjzdp=Sk3Y zvdMIbIzZ!n+wVUTUw1HaB-RO%W3>YGUiq|l&kH2ne|2ck`x2=8*cZ8uraB`_eRrSl zrSD@X+I+(bffuQXN`&o9l?;KU$mAj{qOc8{kgPHY?i6VV2Rd#6W4NSWIRzI5gLj9w z)!(2z%5xRI`7ffrWh(MtUN0vRWgF8rX_9t6C0`Ftfc7 zW@H?4YqzPjXFH*7KQSPP3$G-tBll;ZH zAASjo2Ca@{&V{L}nux8|-MMM=o?b3VJ)e=Ici-sPXFtQD)Z%}$oclL4F-xoEYRFgm zV&VxaC+p+nvulVKFUct?xE+~YuB2O2`6$f31r8%-D_y&!M2pCE96yQE7@(edknLjV zvk5imGr6tzdUyiU8}|w2hJ1Iz_jTH#jYDt??q%l+jXT=Hh%_&5MNbI;h+%)6e+HDU zgGE7{IdDv2D3Jbi5o=~&JE`n&nrlLeluQbYkJuj+XlA^h;k^T#4dJf+O&pdRQ2m)k ze|#5QrkAr#?!{m1!~q7#6~*KG8({k`BEqcknt#UcvA#4LlmRU?E>PqTafsZ%y~lkMF&l_=|66Q-Zp469T&qbV?fA zkQJeRDSRiko8rKl<32n<=>*n>iy-6(a;GCcM)r?sU90 zUvf2ZRQMb%FP&(u=-5xsQGXRH0NTZmWIOI5i=&y^%A)PLuchr_u+wULB1oVs&E2)^GF@P?zWk zi{IOx^(64AQPgCdlKZw)e(~7hFv7-W;YC*Tpu}3*4K|O_djS8gd`%4yQR16X7!Uv8 zGp>x(8?RWjy`&`9*tyyiBBA<)r($u!v7siuAAQFluCRU!R@veJ2OQg$fB1ul+!xm` z3G(!?05nJ@vfgXVGJ$3WgAOmG%a}oYBp8W+dkt8W*=PYmFMM7lz(t4aUq3?A8 zM(Xu994&$MgkEgIGQPHeGa~O#Pa?s=mDRrdI;g`Z*3BeEusf6#`(H%(*9>SPUA}a- z2n!@v$4{j{{9w+$ps-~SwsS}jCJzaDE2EID|AWIzrzp;f@Is3oiY8XdQ9ey)CHO<@ zbY8}5XxFeZRqBuv-!qz8_HbKH$`16x{pLovM2Z`hsXB!7x(-R z-UZn(ZM=zzHB!Qna#x!6rytZJ!ID$?c%tOa6@10NuQU+quQ=5qE9Lp|EIEvabVYW; z+;#JA@__hrPk^w=jJPKQiY^**++QuLBkfAhpc-?xJW);#szB=c#bxPv9b*12q|YgE zF6a8(OO3kF1gW5hgIW@QC`Lt9hY^d=cWZyrG9z3%olX9;b-JK*AkPDn%M!D!P`aSw zbh8j%eKoEUM~vA$_pXDYT`by_(e!T7oY6&(bh@FA$GN{*-B^EJ_*ZNu%+^4xFdJxq zuxIV*xFmDoQz_45OnD@9GD32oDLtnaSQF1Me2xzNupZ)(qM9v+S=-HDpYC;$L~-G( z^EVDV?{9Dog}J7K2LPpVGc^hN z=2ta|%ViInCwbfi%)Qt?cKkY$HHRa?gi#WtY(HXXz$CXmwRNi&{-;6eiO@a{WZ|Mb zVKkBYuk?yD)#mT9mXkeMZMFu;OA+M9>z1*f8Dzayct5@tF}c*au2W)k(~r!6HZaQ8 z66fX-@&TcYToPO;6AL@&wKP6jJrwErizu?AIkO94zO{{k)2@)Rz9ah9VcRr3kzCKp z)100-@_~T1^H}5WdB(lPX7>vIvkpm88uxy@Cbg|ZY@r>`iVx4bp2Zk7%Fj?Ll1KP* zE)P~|RlXcQSUkfL8m)W#N29Eh$3}yA8FIK?a=)d4qteVRK|Q}6VdnQjntdMBZoz4l z74)b#^Os5~A;{G)f2+T!ls@CL0NDq|oq;$|7bAYv^=#^9%RS-qUAkWnAyFHIB4NlQ z_gBK$Fk$YFQ+_0ZK#tk&CP=@1)q6`Ju~Ja=1PqFJqgh9?t)7B|eAH$%I}GBYVjR+^F32WXH_=>!qa z-fPDbOapaCU%VNw?yuY_Sl-JKIeb?=KVbxUMA8_*f1?Ue|FV&){TlEwkaj0_ih(`F z%&w>7>r>>3$kogjK7Snm{K5M4A*C$o!^HioCF0SGJ6;K7Cp0{XQmp}%tHvqFUnkOU zD7O(fn%G>gq?4Ym>djRd?;$_SE@O#Pu2Fof_YI#c%T7RV5=AHN9eGaaMmciapg}da zu~8RUlkuv?T(etsiAJ2@(z8FxS1Y%8nM=*3dDZjxS-Z0zbDt5RK~ywTghpma6TAtO zSTK+?dE_N35T$-U+54nEhm~-Dz>Y`1v^?iCm*-p+~s7j)=u9% zCr2USvtNspD0dO2a68*ut=b+{aoe8pz_2rkIJSz@rr@7sUXif&xmx%IQ9(1viKW$& zzplBPPxQ8`ZDwC3qk>`J#RSiiKa3AkIrWeL*`v8^P*8`6)UfWTzXGiqttgC zwQQXwO)otZwgsaCNgkaN(?%Zt;w*I+sb>IID*;Q zGKYE`(cJ0-?h1LGmoa29gBD(h{#~}eEIdeO&67*!->NRN!=^A#u?b8k{=_}@|Gcp) zM)p6sy7?Wrj77KXUG$O}mMnnPi0673vA{7$!n2mE`6*EY+x0YTnf5WIiZfLHI0jSm zx*yv;C@XEvHz=BQJKZXD(FmmY%)FDK)8t&5Oh#~2yoH6fL^}|E_qjDe6VBMNyD!4& z+>rxd5{ylL1)EaURVHuvzCMt~Lx0MM5}IiDs2Xi!=(}o*=&BcEv8xcaBFhwqVA+6G zQS+c=#G!oWmFMXY*A&B>D{O35$@R(xaFnh89p$ZUR0I`dJZr}sA(?AeNSFfZ%x)2n z+OFr6`Jg3S3G&vS&kc3f+-LO20;r+aW=Mq_)*r^EFsQg#4m(I)7AbAo z?&^|@D3mgSm}t^+QD?1v)KTdR$0@V<+;5l1$`(Xx_6?Bx2`5CSv1#b%%cCP&4YWrqvi3Bjs7In@O191>7LMY@$TNxsf8`%TclNe z;rv2K-Y1|Xq+0x#V1O)9EKe5bmvt)Eb1oj^AyFT11AwP&1eTNcVREc_sC-7D!M=eA zQJWjPG0_VZ`-!Xun=L_0h>w3ucRhKF1D-U;u(XFdDXn{Tz!068k+hs{6t0hZs5zWJ zr_yCXc!F-VFTmhM+~%;=%jojfLp`jfgEeQsI#41XqiJ-yEqP8hx1&+5WRgqZgVjUr zybj-6&&9tmayFMGZdJ_vr3-q)jUF~m>hHxwe7zVGjjPM&gT0nU4^i6B>?J%E4o!=F zY^rfpn$P6ln~c14_Iq*2(=;N)D=@SCMz063W&-Pqe7D0n&z7NAzC=hYhLOr5$XO^R zcpV31$msYO!`%J4U{|vnAxZ_<#=%$V&OmlktS(Di&{-g!GgG|F47qgrjN@}u)fr2p zrz$u0`{$kAc#nZ&-_C(R^|Vjy;FGysPtN?A1-Rg}aDS+l3x(U}gw=g6|z+ z>e_~e^Uo1tMv4oS-}|JUifE~VgCt)BFH~NeqRPzcnxwLUZH_5|q8aEfdLG9p*%CQ#M{Q|Qwu%*%-Jm6(Qly4`+fXJw~KwR93fB10-*j;(KC z9XIeRS*VngAzYKl>9X*m+FW;RQ=Km+05GgM;Gt0Xa#I!m9Ltp7St-knIh$N?Eq+ep zwW$X^SqYC&{=4Aefxgp-;jflQxVCG$t?358P2X;6aa|{*pg;oAzl|vot{S*xW<*D& ze7_$`Y`PO5$uvkORdi4MqA2n~b?CO|(~+c~JmklO^32?)?c(8I%@=!^V^&o4F$qAU zE4xB$YyI1&jUhqDy%6nor_+S%3g(xf&BsdpcPe~u#j>Xv=h6*yguI%oR1vY<5SiiRU1C^Nlg%W+V z-HQ9Ru8+!EFrm3|yiJ9SW7XRbQZH&B!dj}8yH-1OW-Y{cFK1FhhC`cO%=)reNyC^f zu|Bcr^qz_jvF91_IU#qM!*8y)773P=38^mMQ-t89k;)!%?B8_q`4O3iY5(Uc&41 zY+*p2OgK{FGm6DiEcp@W5ORk%!~v0!%J2GRVFt-go_Edt*XR4ibU(?wuk<-voEmT- z?54x9aP>L%a{_b&y;pTxfKo20#wYuqtgFf0WKKI1`4L3{!xN>o#4`b! zb&02$83uVBhdGqL<1?~+cZ6hk|9m0tuOD6rvGn@&E}tIhHBD;y>R$e-taJG4-mq5? z&2N*NUlls_Q&TX0d8qOwGy2o10HSwCQ7>X^X8Xyl4W(}dJn+0OQ z>&#)105NYL9#AXDW%<4!T& zxH~9thJ*PiUQ`@F$?{D@lR;;Z9zJo$8Th=1lE)9U#$t2y{jPXiCaq3765^lM!jrXB4C^)X^ z#Ox+ga)auD*Z@3B2qk2FckyS5zJA5qqaDTbW5A<7Tbn?=6$g640n751yv`RnAKEmd zUVTuyjTvBa?-S?17xV-`;X9HVaxMW4|Rrj;l~>m^eaO#8=|LVl4Sx z@1{vE>(9Bz>2|V_h(ebf0a%H7!Lpd|j~3F|AEMLQy`>IAg&F44@mWVvPl_muUfTyb z1%wU@RfkJ*?XT4RX@L$)cv6SY;-ma-(cf0U6TV&GxnqCfB)?g_x#xaK zXE_6VTOl_RhgAti1v3pF$O9V{g?^#Z)qsYtYbT7q@kIQE(w9G(#r_}7MI#B}UWqyz zuP-L$)9Hm{gm*s8JQcDw@A-Zn0I|Hn&)%mo9&oyAXTxGCclWeg;E2u5-0|t0ou1(+ z>n)+{sUzo_(bO~#^eoUVW)4c%hDMaFw0yOxo0N)wLHI3-U_mxk-k;^dZ(NhKQ^o?SC$X+R#uSGJBT8S~qjntD@-iwEGwI6&LXm_qUU+=}o z9q?ILHi1U^y#|FdMe7!D+~>L;ulY`Za$G3pBAm$bPl*z1F;!F;MHluG+2@KAByMsl z7=xiq0qB^7b=(mr0y<1kX#dF%P;y2(0$+xT`LZB14Wj2&cK4E?@Mc2$o83;~22v>7 z8T2YMQ0TOq*c8dlYp`!#`hmBSih~h3PVUuh>I)H;%vY|?0Dxu6~bOaSXzJlLl$ zBF)~Xz!Gj#jtwnLkqqYUM}hQTVm;Bx-$?n?)v>ozQLb_K=kj>N9%j)m9N$XH0t)B% zz(b-Wz8;dj@RU-dhaD*{_j?}U#QtqYI45s@YKp1n6qX3GZ|zVBI5JVw3LMVam6tcIh+@)C_oiUqQngTrUq1YM=#EE$w3tkpJ=brP!hVJ6y6<639Zz+ zBknpTcT{;h?+J*rBzpWK2A;;;Q~A6+y)mr3_NyJ1M2EC7z|_c^*F-r)mZr62eT>*&3 z-TnmZ|H4bsPmj4yW}om5kIOJ%q9>}4fx&2q?u=_?&dQYyI<}koWYHmnsq#M@cYi}e zb$#w(`C5)+A1owFN52Am62y8w52VV>mX0J4E2Iz%ar-8bz@(i621Q(%FCxh<%WmRf zIWSPokJ-(9GuS{-59gBV=;Mh9SwYx)K(!ITfPzulgMB6Rb%9<#`$j6Z@Y|h)Ks+f=4T13%w)gL}>(VnNbH6IWyoj zfXXV>d0qZ6fM8*5|Fc9Uj8}o`aw5NBpjs8t!PsB3IO}bZNG+TLeA1YkBrJo?{mcw+ zU~@*-XDCu|{23Z#*NFQawF3I8x9;j!vOOYM`!fybwJbB}3J6U0Mt`FzIP6om<$t&?S}wq+>LNuf;5 zX>8(~`rS{BZA1}dRs zsz08Q9(Y77{0+g`B}|2y|6<8atkX1|EY}rs+bhh6BH@ubAloTm|6XoBSZ>km;_fJj zd}}?Qh*fxK2ABG9740@-Pi!`y4i7~308N__=uyXul#AP0!4lJg@{x?h6&>^l8PCQ; z@nJP^>=BQRYT7h)3cQgZi2P3@0sqJq>Z;Y-E5;mgV>Y}_)eQfx^EK2hD|Gvx@ zfUZP%K&k3Ie@pe(BP|8C@vCo_Rf=V)TO_6;ic9tJ?(!buoQLaoz&tdt-c9KnWBmCv zGr9k{)53zDIkfdtIEhff#l8?FGxs89BYTBBs_j1MgnrW5Df$>| zFTj-{_4^rn-XF-TBUw*j@@T4VN1X=Uz0KBvWxT-Gh~3Ls2g6YU`5XfEU$rblu{&EX zW^zSiGYra(?n;+qg1anKxz(b6xNxF?+S zISMO0$rw%I0pv_0L1JlS$>Ma5gd<%$O$q_^@p3F$$j5u#4OY_(FmQ_+RIx(%OB}jB zh%~CP6d?l!T=1Xl@S;?wMWag)#CwEd5Wkf=oOpm|uUWslO5(Q5QYulCzk5OxOYnpL z?rh=&AY^@DJfj4uA#wp720+9M6+WHO@hF8)fEGgBK*}g!G0)yt`A*=8_buRBjluPC15n^j-Tvr(NVY7C0wIB2S+uGr zMmG-rtU6E$pjqitJQV8#>O?e06N>@MIO{h%4W83iKRf^~Z*jTFdBnnd@`ETfPTgeQtb=e#%i|*scKTen-58q?l~L7APe_^ zh0c}L+!e5UjJ5+)`rz7qWunC@$pt~ydnjga3CS)5oS`O zNvxnU#otAzK6-yzc}?Um;@*w+Tgp8%M`{eBd4g{OjiJbjx&#OGrLGEaDydAlINlQwi3+(AabG&HMaRd-k=}i3pkDk)5X#U|qYDBSQ8&Ue zjsgY7OVrt^`$4t>B}FWtCe_m1wutcbT7eF?h%l5E7?dM$-GHW=fc9m!?tHxs`4#C8 zR#0Owjx{O#^*S+6zUnD9yV{+9Rx8uuaa#+bQM!#M;j!=aMSPkgm-)_@0Y?%pOW1P( z%q?=Z`Pm5|SC!E=X4EJ% zVxm6ARg!p7M(Z{2&G`MY1n#0W3`P6o)5oCaX}+}H|3IXxOAwhaUW1Cuid~PS8^2zB z%DJ#z9IQfT*Sc5pKHv=!;ict6Eklb~75cFr4VA#4WVf6_aV>2zc;5@dD`(3ao6izU-nmh2KwD4q ze{_WMsLkYtALelxY@6R@On%VXo@inhV;7-mxhVQe+LX{Du&zyRd9A9i!fPqqPvp4t zRjN&X>-@+IrFVs5mtfu=vpd;WKPzV+>55*7XVjbw#0jJ&+|3-YH?ecKkjm$@S?1@g zda~>A=n=yZJPY7u*R3EvolR((ZzL4?C*0*tm1<%60Jr|yA~gX}l_CzJjP_F7kY14h z*L+Lo`>H=Q7=73nf0;^Hq{~B)RdWNFn&!kl(=fw#rj@#F=V}Nfk+Hn@1V#$n(hO({ zt0ys5Kipv4^hY9nnKmAZ*00rIo5>HXCVTX>Xb_6%X31tGJkEDr*dL#fE?Ii^X}_12 zbr`=o;i*4H=E@s(voUa)t6;zLlbi7WJyzPauh5a`FZQ&gheMw@w9rm4agmCmf9JH+ zQ(rsVkxGQZlHv62?W=TYNpe$|5pxvVW=I{5dT9amMicIOkr1am5_wll`v2BmyV zbOoJIrc*~SmlrR*Z%InOU`NLDxL-Z37x?V`3H}(pfbpzvpHITQV|eBDB=e>r;4apj zU5oE6=IU&CtS5p>4SF1mHu5X|;TgC~G2Q3BP|Q=$wa(48>dlwcp(f`;1)&k%eTMUM zoN;|uw(-OO*?kT*LnPF#x-OFXk24@(uR`}oPiHPd@cJVvBvF#UoFZJQOiR5;wVW)` zVLGLL`E3MApM+dcT`RB6q(@gUoT~Q$>?wi3=k!W2O zFBP{p*jiKG&M(!uZT%vuWA(z!%j~G~JUwf#(Ib+%HHD)ERGZJu+lE*A|1C#^Qd!I$ z{D=lW50)ZkMzAVBew8|FUy}~RN~d4R)7lQFLJNhz+&fJ_gZIYe`4>e#bOo+0o1=o7 zNVEmBKk@6|gORlx^AWlPE&L>0wG3nVUSiW?L4DmsSX#JNJQXpuyS+pE3jh@O8$MJc z=XRu2q72V~OpA7OtdNZaA8z#_)BENi=WkMjW_4?<5(2DH(Pb(;Z+u!Uj>ctl@; zxYz7uyqa*8f6H+SR$KjZ(_tM(o!+ueXlQ0e)+g$1UL2@|ee{`gsbdc{Xq;|>rgNX` zc}R!FXMj4{mDbjBe`{T?B9)6e+LrvSg%dWnzLlW#Z-?7zDCW)K-Iip-ns*hL)$E>j zAUuBuzx14$Pj7&k1nKeZD@K%q@OJ2j5d1iy`zz6x9A?R4X%R=j=O&KFiPQ!r`gRg!JE*F26q^2K_@J+U+T*}=R`%TwD)upjLuK>PEw>xwluI# zO!^B{VzL0&J_~$eRNszw>oV|9Y+ruz^=M$7SC}ms*U3dk?!wU! z<)YqXPD`X+_+2(DWI9YjohFqoC}8G@Cet1mSE^b6WOxdSb|Gp;r&gi&o6w+MsYvP5 z+}UP=ZvPP13JJG=7~njM&+m5g({liKuGY)C zcmFN}HU7n2R`8^eIeZKq9D|QUZ%@jqX(mr$B za(XS23((LLppW}5tGrOEQ5%i={7r9+tRNj&0(9OqFs|3O(zXHN+&@qLVxz;?{-0Q( ziydc%FO=9aJ{6I^P>SwP#i7rJm(BO@I}yoVwO{N}ns$!9_;*YxJkG%}wN8Y@KDr~8 z=$yAXsqP-IG4+^gd!ckY9c^Z#1S-XQKRSIHw{!nQ777OLGB~=mdI4!WUhf&eQ`O3Z z1hgL~{-hs2>3l9*zztDB!V9$+d%VqBOOSV9F_z^=!sjf7z0(4etvKQZdP;-fibGKR z*1;4iN2k-=z;+Dq+xVNKHlg0de_PQ7!*;It7T(R{ecPay%+dK3#8CWyyAA9;D87ko z;0;uGqSTCx(m_OUFd25bO)rm>&5HdL`B#2?-17PyZU3BOu2y z;81s={(Arw8rXN~J1&yXm{jm4;m_a5isyZ0&k%Frj;?9iDE{z&HjAgCc3O!u^ zWi{yc^%Sii07eE-YNzuB$p_b#JGUTZZcW;PZ`skFH}EnWw{dFCHXV4#Nz{^DDFZLF zY*j%fSO7fA+*ok{-@ldWzxn+U+a~g52f&dv28Pd~Dxm7YxlIQnt!W&W0@ZRactuOB znnI14JSV3bXfLY=FF{GFvfq@LYxQc1W!BY!sFpE+7qbnsX>-u1l``ky|9YW!)uvWy zAi+&1mm$pYf%=w2sHW|kXn*G;G)PkWr!3KbyNAoe8C3l9BDr&zQ(5{Z2^x$V>?bYw zp!Q|+mm&baWd56NQ6^&i}LFPXz+~Yv%_j|mjp9f*-3&9-eB=j zMaDN=@Zb5T{}Ua*<^L@8|5@t)Kc4Ea zq>PhZ%k|AO^C%A+bIQ3YwXedmJ4yXyzp;bbSS+g8 z7sHpY>LuC~9oar+NVAc>sz_~T&?1|0~!f1vMtdF9{! zXQ?R{RM}`?j_|lRI1XFVVT+w9PiT(PsQY{E^v4ENETU`5r%QHy&n$#oL*6naw@qF3 zZq`f|>i)di@gvuuHVol*JBejTZj)9H1H}gG6t+OdC6u#vTI#@jd~G{CTFm1Ld;>`t zY_`3Qb5*Aqj#dl8!w`+p;?_*qXelKD`q)Z}cfUS$+C z_?=4@yglRS0j=^$DR!nLVTyb-Q!#BQ7*XFW23^pN69GF{M_Y zSv&(Iw|Kaf*i)xZ<;kC#K2iwl!Ny7LH~yj)dh6KrB?zhN32{XSw6BWxY>pW7Df@xp z^^_iT-FBy<5WyAROWbm-ZH$)WuLu1 zo*CF#QI!}f%FW&Enrhx7o{#&6^kztlQc~4A-ZD9zo{_mI@f+uvPaHgU8f5&%!`~kY z+RxwrS@Y3S@WuFDE6$EqbjjQ{aQ?SzUiERTSz$Njl@4c&bCmFx_PWxi#uMFl3bzNI zAWTW)Kc8He_@aOKFyQ}b&57&(#LNh#QzM^jM=jWxZ78pdmc4$>vT;LX|94b=cdRip zmR_A3)pF+nGL*`v3Shu@beeS~R*MZ|55V5-DVm|xXb(pd2U2((ascMsTDc(U57s=I zOJM*XtodCja@?BfX2J;`EOaBK;L`re@=kEBaXm7YPW7WiEW>XBL~|U}Z`j;MQcr7t zCO(6x{y9)6mI;c?zD>eFif5RA(-V=ZpbRugBiWw!bv;8^g;$>Yi;~TPDeT|`>HweD zn}T*C-f)TU__ewAjcYpRScej?5G=X=#8$=-uzLq%c@}Lq0x&cO7`BI>;!nJ9drk<% zty(TiPNC^*P%5{(ACIEc)n2vl&)}8?(pZX6tCo={6#tpS8Cie)fU8zz+e9Mdtx!aC zOnbsk=I2A2WDvj44T&p$5m)odO`+tXPkX7wBfNipstGX;aQDLcZeK@M;-lOIAE(9GPyx-Vd_;AyYiO9|-V8|V87?@jcB9+m4#HoQtJLB>mI4<86j^z;yR(PHxs zw$Vvv0Ps!K%qdC-a%&`5QwqifMb(TK2@cy7tFRzF(hW z716c9WkV#nYuZB8A;#DH{xvlmoSUk?2q{mxf^6LGcr$xyR8aq3*Ge#)z)VDP<@B@H zPqdu1#PT-VsESozk|x}Q+y7vGXk^-C6m;6FVNh?7231_Tk}F8#E4~}hq33yqfDc@G zq$lXJV7lEE>)GlwL*Fd8xS6!Gcx&s`sU+u%ct#tSWe(fiDrSzhn!`QBk53Opl1ayh zSqxguwOnG0=U?@9T+>4V%Fv(nA>=hEyMk3|dDnjJKz)z{y7($Ev4zXnjhQ!lYk%u2 zd4>D?oERsp6j4KNELD>v*8n}6=6MnyljRhnKEL8|?m$dCfje-VD1&?kub%*26zhC~(^00P@ z9-n~lzsf^!?!<=gLlZ5sMGC*N)p$-207A5ZR02Zo`B&sbDU|aV z84>937wH0?ktD2{ZpWP9jW71c!(M-B|F1eh?bD&9Q0i`%B;*oO1%(DXnEr(=@ceqe z-M+mr>-J?v{9v}5-n31N+zveJ*H+#qQeP1pHcg>|&7ING*rbJ)=Cg0|75pax3ZWUTzrKc|DleUpOXtc>X z|KWa(jJ6T&VEH0Kw}T-FN$ZX6&ZwQb{^Iw}9WGksyo^|g_UI;VqBcK@)tskrofR*I z0Y+@G$zsgO_Q(g3t`C1b@11c>5USQrt*tUouLBuZ)VBZR?^7{B(Y`!XT7`JBI#(LV z%2Q8EvpAj17@%2rN`9ZW?{Ldi$Upg{G%+Idp!FFDIF+e_A)hv8kyJlCu=*3h4MdKA z7I`}GsZEc;_n9aUD#I&ZnRRVvC@2UYFc#~{t&Dxk%OqK~&h~cHFlApQd)=O$mnkfF z-ffpOft4wGE=E`i^~=lA^PH*>z<&-Ze^=gj$@-#J<< z5Z#kg&tI+m-a_hOlMQFf&95DApVpv+UU)ZVGZLVdHrb#2!lLP}?*PVVv}B*;(+cIL zm=7h}t$D#c-(4X$a^Wu3$M-=s=D>b2y5LN;3eJXg`8wg?xTrR0pBq?@RRUN)q-)J< z=3}Pb#jks+Z4lk(H<6)=EzEAtX4=|)Pzpg;Iz(iSuI`UI>0Ipquh0mOL{K7LetS4C zZdyYfc86hWbP?gzTbCU2R{jZJWm!GOb(E&0G{)A^ zmnC=pBtKArF(2JtWaLTOcu-QQaMN~gu%J?<%JPSntogItJ8!MdChCIM zyzF;tO3Z>|DU-Yv=0eV4e^y{z zbDn9$dXfsGKte3G5+uYPO_m)uLgOD*6K{vP*5W5BexV6GD)9)-Lhk}{TAk~S?gM(( zOkTZ$Vr!N)R%M7Zj`)#UYa@wqK$Z-5aoD#bemsOJ2a5@AGP4rqR-J3@+PRAM=asth zhYI`TH<35yEPv= z-rXckaJ!pAU3ERPDuGbG?%tJG*@xn4*y#8TyAQWB?h+fU~pHz9(3ly-cpw8@?wIgrK>;*hElXAYXGm5l6##>8@6XWe{+A!wV zM>!yf>AsrmkDBms-5hI?)zN1Sc7Un;^WHQYcWi!7YQ=H3PyZ>yt(!dg*N_x2(61`> zRQ{1#pRRFvWGbq2+bw0N`xbzgcJ-j6Xu%2L#_ebM>@)(HpB@I=v2kZ7Yr2E0nD24i zUIjBp6W97pfiY(~AHPyHAM?;D^wz^2cUI|kE_wpeg(>RXG^$UZg*tZ?vmnS>aEDQ0d!fVIqP?raI&p}3Q z!&lQY_jjkrSAf4UR*bDSUC*^vQjU41uIaOGlu|MZAxm93(&XD;N-OcXe}eK_%@}Ka zE`u`d*fB;ot&%|+kTM5M@@=l0QFHf8@+sKwiX3$X{5sbC7fyHOt+u1dDM&CS$pWw5 zw}BK14J=5-J|rc_p0d29KyXg7jFKgSO|^%?av*lJoT`Avb-BT>Gn@PY$Q?vU?|AY` z72T7nxGp|BVtAsm1MVIn?=)UXPKeb{h7awL(0V36Vb0srtG=CAZyGj{m%a_Yq50rr z#0uDr*;DdXhQRP=hPU#fKpH3_D&U}s1_*X5!`d`$@u$5cTl1U1m`27Uz2Y&5*@JCA zg$9G1<2h3&S(yZhPbF_txHr&)@&o#*))d(_bW?|$Bj&d!Q@WpA34t5RN~v>Wyx5OG zXA+WZh(;hK5&maDJ}t-?w*mTql=M>o%f;l(dt^l)#n%{&q-g|+EayI|O@M)du>gq< zQe`Sf8`KF;(jZu%XwIE+ti?cZU}}s%5qdWRZo_~htU-mLdleocf^TFf6nYjW5UT#E zbb~3}Aro#75_)JO1D3W>8#{}&A>s1Y0BV>gM;wzNYT|B5)hV!@byFy0Zk+gq^XF?-&f>{WE5Tb zWB?o+l63+NN!kn$vW#ErrwSo7gaVEOUII0!`p41`2=XYwU8rp_&_im3gcK*q-$;)r z)SDHcCa6qPLVc1NVMBu??Vl|+g`jE!gq-FwuQn0?Noe^V-hR{us{XNLLh1cLgB%O! zNdf{e%!@qlL=@^x8KQRMPO9*bq(2w%qkcl6AIygcB(-*e0Ft!)jZ=_N zQV3z};ck`)#Q+-UB$U^YxX2a^xEXM~xG`tTpCJPwQILscc2uy0dc#4~&R%rdw@C1C z2OR*MQ`cP$xe;Lqs-PB3#Ml|L}LqIo!CP3q6V+fU9(2xFyra_Rw_U{UN zQEx!sQz~vC2!ANe%_ZPe7h#h`OyR?-arEf_j?TjTFScLUEcly_MbZT8|0w5Qju+YkNtE_^7gFzYKY;SP zTL5<;^? zyEp$wOJooVY=}awBCUuBc=Rgr>{vUY50Rd&EEM3ysKXwwP1_eAkqx@ z=Hn{=ElWXu9Uk5^+j`H*?czK7yOO*UMtXZ7mUMJ>!HnhgV=5S zAVlepy!Hq>(&Vr%RLZ=EInuo$2Cep!tB z96m9c!F8Y={4}JSChW`p4kGk?Szr7at_#q=sPl_D|ATuh?}5K~=jWR9m&X2o6g#Ir zyy&-(k&uvpmCav@r%T&rcNAnWzy=^(GDAWV2A5Vq5|-VhOB*}bMN`>-^R>cLxEeO2 z&f~Ln0ky$h4LJe0q_oeGhL{o1gOu?J)VhD;D@KBb$#d8_z1quREdc#CT=IdX`ieuv SigQ3L3F2 Date: Tue, 4 Jun 2024 13:31:38 -0700 Subject: [PATCH 02/14] Update BFD HW Offload for BGP session HLD.md --- doc/bfd/BFD HW Offload for BGP session HLD.md | 230 +++++++++++++++--- 1 file changed, 194 insertions(+), 36 deletions(-) diff --git a/doc/bfd/BFD HW Offload for BGP session HLD.md b/doc/bfd/BFD HW Offload for BGP session HLD.md index 9f1adef3fcf..4538d71e8b3 100644 --- a/doc/bfd/BFD HW Offload for BGP session HLD.md +++ b/doc/bfd/BFD HW Offload for BGP session HLD.md @@ -20,8 +20,9 @@ - [BfdOrch](#bfdorch) - [2.5 BFD state\_db format example](#25-bfd-state_db-format-example) - [2.6 Local Discriminator handling](#26-local-discriminator-handling) - - [2.7 Control plane BFD](#27-control-plane-bfd) - - [2.8 CLI](#28-cli) + - [2.7 IPv6 link local address support](#27-ipv6-link-local-address-support) + - [2.8 Control plane BFD](#28-control-plane-bfd) + - [2.9 CLI](#29-cli) - [3 start the daemons](#3-start-the-daemons) - [3.1 start bfdsyncd](#31-start-bfdsyncd) - [3.2 start bfdd](#32-start-bfdd) @@ -32,12 +33,13 @@ | Rev | Date | Author | Change Description | |:---:|:-----------:|:------------------:|-----------------------------------| | 0.1| 02/04/2024 | Baorong Liu | Initial proposal | +| 0.2| 06/03/2024 | Baorong Liu | Add IPv6 link local address support | # About this Manual -This document describes a design to use HW offload BFD to monitor BGP session. The system can be configured to use BFD to monitor BGP neighbor. When BFD state goes down, it asks BGP to shutdown the connection with its neighbor and, when it goes back up, notify BGP to try to connect to it. +This document describes a design to use HW offload BFD to monitor BGP peer reachability. The system can be configured to use BFD to monitor BGP neighbor. When BFD detect that the neighbor is not reachable, the bfd session state goes down, then BFD notifies BGP about this event. BGP reset its state to IDLE after get BFD DOWN event. -HW offloaded BFD can provide a faster failure detection and more BFD sessions, compare to existing SW BFD. +HW offloaded BFD can provide a faster failure detection and supports more BFD sessions, comparing to software BFD. # Definitions/Abbreviation ###### Table 1: Abbreviations @@ -63,24 +65,24 @@ At a high level the following should be supported: ## 1.2 CLI requirements - using existing sonic CLI to show the BFD session and the corresponding status - use existing frr BFD CLI to show bfd peer -- bfddp_request_counters/bfddp_session_counters are not supported for the first phase +- bfddp_request_counters/bfddp_session_counters are not supported ## 1.3 Scalability and Default Values The BGP BFD HW offload session shares a total number of **4000** with all other features in the SONiC system. -if any BFD session configuration not provided, will rely on bfdorch default value when create BFD session. +if any BFD session configuration not provided, will rely on frr/bfdd default value when create BFD session. The default values for BFD configs if not specified explicitly is: | Attribute | Value | |--------------------------|--------------------------------| -| BFD_SESSION_DEFAULT_TX_INTERVAL | 1 Sec | -| BFD_SESSION_DEFAULT_RX_INTERVAL | 1 Sec | +| BFD_SESSION_DEFAULT_TX_INTERVAL | 300 millisecond | +| BFD_SESSION_DEFAULT_RX_INTERVAL | 300 millisecond | | BFD_SESSION_DEFAULT_DETECT_MULTIPLIER | 3 | ## 1.4 Warm Restart requirements -No special handling for Warm restart support. +No special handling for Warm restart support for first phase. # 2 Modules Design @@ -92,10 +94,10 @@ The bfdd daemon provides a software based BFD service, it also provides an inter A new component bfdsyncd will do the synchronization between bfdd and BfdOrch through bfd dataplane message and redis appl_db, to support BFD hardware offload for BGP sessions. -The bfdsyncd listens the socket for the BFD session creation/deletion message (from bfdd), and converts the message to BFD transaction, write +The bfdsyncd listens the socket for the BFD session creation/deletion message from bfdd, and converts the message to BFD transaction, write the BFD transaction to redis appl_db to notify orchagent(bfdorch) to create/delete a HW offload BFD session. -The bfdsyncd monitors BFD session state change from redis state_db, sends the BFD state change back to bfdd using bfd dataplane message. The BFD session state will be sent to BGP eventually. If BFD state is down, BGP shutdown the connection with its neighbor. If BFD state becomes UP, BGP will try to connect to its neighbor. +The bfdsyncd monitors BFD session state change from redis state_db, sends the BFD state change back to bfdd using bfd dataplane message. The BFD session state will be sent to BGP eventually. If BFD state becomes DOWN, BGP reset it's state to IDLE. ![Alt text](images/bfdsyncd.png) @@ -160,6 +162,41 @@ struct bfddp_message { struct bfddp_session_counters session_counters; } data; }; + +/** + * `DP_ADD_SESSION`/`DP_DELETE_SESSION` data payload. + * + * `lid` is unique in BFD daemon so it might be used as key for data + * structures lookup. + */ +struct bfddp_session { + /** Important session flags. \see bfddp_session_flag. */ + uint32_t flags; + struct in6_addr src; + struct in6_addr dst; + + /** Local discriminator. */ + uint32_t lid; + uint32_t min_tx; + uint32_t min_rx; + uint32_t min_echo_tx; + uint32_t min_echo_rx; + /** Amount of milliseconds to wait before starting the session */ + uint32_t hold_time; + + /** Minimum TTL. */ + uint8_t ttl; + /** Detection multiplier. */ + uint8_t detect_mult; + /** Reserved / zeroed. */ + uint16_t zero; + + /** Interface index (set to `0` when unavailable). */ + uint32_t ifindex; + /** Interface name (empty when unavailable). */ + char ifname[64]; +}; + ``` ## 2.3 bfddp state change format @@ -199,24 +236,49 @@ The frr 'show bfd peers' CLI needs to show BFD peer information for hardware off - remote receive interval - remote transmission interval +The above information is missing in SONiC database today, it might be available in SDK or hardware side. If SDK support the following attributes, better to add the following attributes in SAI API to get the information for 'show bfd peers' CLI. + + - SAI_BFD_SESSION_ATTR_REMOTE_DISCRIMINATOR + - SAI_BFD_SESSION_ATTR_REMOTE_MULTIPLIER + - SAI_BFD_SESSION_ATTR_REMOTE_MIN_RX + - SAI_BFD_SESSION_ATTR_REMOTE_MIN_TX + +Propose to do it in the bfd session state change handling in BfdOrch. When the BFD session state becomes UP, get these attribute values and write it into BFD session state DB. The bfdsyncd monitors BFD session state change and update the values to bfdd, so the values can be shown in 'show bfd peers'. + +The get_bfd_session_attribute API need bfd session id, which is available only in BfdOrch. + +In the case of SDK or hardware does not support the above attribute, BfdOrch should not crash when the error returned from the above get attribute SAI API call + +If the above values are not written into BFD session state DB, bfdsyncd will use value 0 when send the information to bfdd. + +Examples: ``` -root@sonic:/# vtysh +sonic@sonic:~$ vtysh -Hello, this is FRRouting (version 8.2.2). +Hello, this is FRRouting (version 8.5.1). Copyright 1996-2005 Kunihiro Ishiguro, et al. ... -sonic# show bfd peers +sonic(config-router)# neighbor FOO peer-group +sonic(config-router)# neighbor FOO remote-as external +sonic(config-router)# neighbor FOO disable-connected-check +sonic(config-router)# neighbor FOO ebgp-multihop 255 +sonic(config-router)# neighbor FOO update-source Loopback27 +sonic(config-router)# neighbor FOO bfd +sonic(config-router)# neighbor 10.200.200.201 peer-group FOO +... +sonic# show bfd peer BFD Peers: - peer 192.168.1.29 multihop local-address 192.168.1.24 vrf default - ID: 1230045420 - Remote ID: 0 + peer 10.200.200.201 multihop local-address 10.200.200.200 vrf default + ID: 300650336 + Remote ID: 1 Active mode - Minimum TTL: 254 + Minimum TTL: 1 Status: up - Uptime: 11 second(s) + Uptime: 51 second(s) Diagnostics: ok Remote diagnostics: ok - Peer Type: configured + Peer Type: dynamic + RTT min/avg/max: 0/0/0 usec Local timers: Detect-multiplier: 3 Receive interval: 300ms @@ -224,26 +286,63 @@ BFD Peers: Echo receive interval: 50ms Echo transmission interval: disabled Remote timers: - Detect-multiplier: 0 - Receive interval: 0ms - Transmission interval: 0ms + Detect-multiplier: 3 + Receive interval: 300ms + Transmission interval: 300ms Echo receive interval: disabled -``` - -The above information is missing in SONiC database today, it might be available in SDK or hardware side. So need to add the following attributes support in SAI attribute get API to get the information for 'show bfd peers' CLI. - - SAI_BFD_SESSION_ATTR_REMOTE_DISCRIMINATOR - - SAI_BFD_SESSION_ATTR_REMOTE_MULTIPLIER - - SAI_BFD_SESSION_ATTR_REMOTE_MIN_RX - - SAI_BFD_SESSION_ATTR_REMOTE_MIN_TX +sonic# exit +sonic@sonic:/var/tmp$ show bfd sum +Total number of BFD sessions: 1 +Peer Addr Interface Vrf State Type Local Addr TX Interval RX Interval Multiplier Multihop Local Discriminator +-------------- ----------- ------- ------- ------------ -------------- ------------- ------------- ------------ ---------- --------------------- +10.200.200.201 default default Up async_active 10.200.200.200 300 300 3 true 1 +sonic@sonic:/var/tmp$ -Propose to do it in the bfd session state change handling in BfdOrch. When the BFD session state becomes UP, get these attribute values and write it into BFD session state DB. The bfdsyncd monitors BFD session state change and update the values to bfdd, so the values can be shown in 'show bfd peers'. +from peer system: +sonic(config-router)# neighbor FOO peer-group +sonic(config-router)# neighbor FOO remote-as external +sonic(config-router)# neighbor FOO disable-connected-check +sonic(config-router)# neighbor FOO ebgp-multihop 255 +sonic(config-router)# neighbor FOO update-source Loopback27 +sonic(config-router)# neighbor FOO bfd +sonic(config-router)# neighbor 10.200.200.200 peer-group FOO +... +sonic# show bfd peer +BFD Peers: + peer 10.200.200.200 multihop local-address 10.200.200.201 vrf default + ID: 2538789519 + Remote ID: 1 + Active mode + Minimum TTL: 1 + Status: up + Uptime: 5 second(s) + Diagnostics: ok + Remote diagnostics: ok + Peer Type: dynamic + RTT min/avg/max: 0/0/0 usec + Local timers: + Detect-multiplier: 3 + Receive interval: 300ms + Transmission interval: 300ms + Echo receive interval: 50ms + Echo transmission interval: disabled + Remote timers: + Detect-multiplier: 3 + Receive interval: 300ms + Transmission interval: 300ms + Echo receive interval: disabled -The get_bfd_session_attribute API need bfd session id, which is available only in BfdOrch. +sonic# exit +sonic@sonic:/var/tmp$ show bfd sum +Total number of BFD sessions: 1 +Peer Addr Interface Vrf State Type Local Addr TX Interval RX Interval Multiplier Multihop Local Discriminator +-------------- ----------- ------- ------- ------------ -------------- ------------- ------------- ------------ ---------- --------------------- +10.200.200.200 default default Up async_active 10.200.200.201 300 300 3 true 1 +sonic@sonic:/var/tmp$ -In the case of SDK or hardware does not support the above attribute, BfdOrch should not crash when the error returned from the above get attribute SAI API call -If the above values are not written into BFD session state DB, bfdsyncd will use value 0 when send the information to bfdd. +``` ## 2.5 BFD state_db format example ``` @@ -299,11 +398,70 @@ struct bfd_session *bs_registrate(struct bfd_session *bfd) so bfdsyncd need to do local discriminator mapping between bfdd and BfdOrch, map the bfd session key to bfdd local discriminator. When bfdsyncd get bfd session state update, so it can lookup the bfdd local discriminator using bfd session key. bfdd use its local discriminator to update the information to correct bfd session. -## 2.7 Control plane BFD +## 2.7 IPv6 link local address support + +It needs special handling to use link local address for bgp bfd hardware offload. + +When bgp use link local address to peer with remote system, it needs to specify interface index (and interface name) when create the bfd session. + +because link local address is not a routable ip address, so bfdsyncd has to provide source mac and destination mac address to let bfdorch create a bfd session with inject-down mode (construct layer 2 packet). + +SONiC does not set source mac address for a port with link local address only. But the source mac address can be found at /sys/class/net/'interface_name'/address + +One way to get destination mac address is to read neighbor table, for example, using cmd ip -6 neighbor. +``` +sonic@sonic:/var/tmp$ ip -6 neighbor get fe80::7a6b:17ff:fe5a:7000 dev Ethernet0 +fe80::7a6b:17ff:fe5a:7000 dev Ethernet0 lladdr 78:6b:17:5a:70:00 router REACHABLE +sonic@sonic:/var/tmp$ +``` + +To make sure the the destination mac address is available before bgp creating bfd session, do not use this bgp configuration: "neighbor xxx disable-connected-check", let bgp check connection first before it try to create bfd session. so peer system mac address will be available if bfdsyncd issue a PING before try to read the neighbor table. + +example for bfd session with link local address: +``` +bgp configuration: +sonic(config-router)# neighbor FOO peer-group +sonic(config-router)# neighbor FOO remote-as external +sonic(config-router)# neighbor FOO ebgp-multihop 1 +sonic(config-router)# neighbor FOO bfd +sonic(config-router)# neighbor Ethernet0 interface peer-group FOO + +sonic@sonic:/var/tmp$ show bfd sum +Total number of BFD sessions: 1 +Peer Addr Interface Vrf State Type Local Addr TX Interval RX Interval Multiplier Multihop Local Discriminator +------------------------- ----------- ------- ------- ------------ ------------------------- ------------- ------------- ------------ ---------- --------------------- +fe80::7a6b:17ff:fe5a:7000 Ethernet0 default Up async_active fe80::7a3e:86ff:fe14:8400 300 300 3 false 2 +sonic@sonic:/var/tmp$ vtysh -c "show bfd peer" +BFD Peers: + peer fe80::7a6b:17ff:fe5a:7000 local-address fe80::7a3e:86ff:fe14:8400 vrf default interface Ethernet0 + ID: 2602968139 + Remote ID: 1 + Active mode + Status: up + Uptime: 26 second(s) + Diagnostics: ok + Remote diagnostics: ok + Peer Type: dynamic + RTT min/avg/max: 0/0/0 usec + Local timers: + Detect-multiplier: 3 + Receive interval: 300ms + Transmission interval: 300ms + Echo receive interval: 50ms + Echo transmission interval: disabled + Remote timers: + Detect-multiplier: 3 + Receive interval: 300ms + Transmission interval: 300ms + Echo receive interval: disabled + +``` + +## 2.8 Control plane BFD A control plane BFD approach is to use FRR SW BFD. If use HW offload BFD for FRR, it is expected that all FRR BFD sessions are offloaded to HW, with a global flag to switch when start bfdd. -## 2.8 CLI +## 2.9 CLI no new CLI introduced for this feature From b93c6dec6b3bef037c0bcf79265153cceb60a6b1 Mon Sep 17 00:00:00 2001 From: Baorong Liu <96146196+baorliu@users.noreply.github.com> Date: Tue, 4 Jun 2024 15:27:14 -0700 Subject: [PATCH 03/14] Update BFD HW Offload for BGP session HLD.md --- doc/bfd/BFD HW Offload for BGP session HLD.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/bfd/BFD HW Offload for BGP session HLD.md b/doc/bfd/BFD HW Offload for BGP session HLD.md index 4538d71e8b3..0e8af4e9d3f 100644 --- a/doc/bfd/BFD HW Offload for BGP session HLD.md +++ b/doc/bfd/BFD HW Offload for BGP session HLD.md @@ -37,9 +37,7 @@ # About this Manual -This document describes a design to use HW offload BFD to monitor BGP peer reachability. The system can be configured to use BFD to monitor BGP neighbor. When BFD detect that the neighbor is not reachable, the bfd session state goes down, then BFD notifies BGP about this event. BGP reset its state to IDLE after get BFD DOWN event. - -HW offloaded BFD can provide a faster failure detection and supports more BFD sessions, comparing to software BFD. +This document describes a design to use HW offload BFD to monitor BGP neighbor reachability. The system can be configured to use BFD to monitor BGP neighbor. When BFD detect that the neighbor is not reachable, the bfd session state goes down, then BFD notifies BGP about this event. Usually, BGP resets its state to IDLE after receiving a BFD DOWN event, but the BGP behavior following a BFD state change is outside the scope of this document. HW offloaded BFD can provide a faster failure detection and supports more BFD sessions, comparing to software BFD. # Definitions/Abbreviation ###### Table 1: Abbreviations @@ -479,6 +477,8 @@ run bfdd with option dplaneaddr inside bgp container, it connects to default bfd sonic@sonic:$ docker exec -it bgp bash root@sonic# /usr/lib/frr/bfdd --dplaneaddr ipv4c:127.0.0.1 ``` +The logic for starting frr/bfdd with different parameters is outside the scope of this document. + ## 3.3 create bfd session from vtysh cli ``` sonic@sonic:/var/tmp$ docker exec -it bgp bash From 4a18c3c355c9034f699ad91b7531447784d8fcca Mon Sep 17 00:00:00 2001 From: Baorong Liu <96146196+baorliu@users.noreply.github.com> Date: Wed, 5 Jun 2024 13:32:56 -0700 Subject: [PATCH 04/14] Update BFD HW Offload for BGP session HLD.md --- doc/bfd/BFD HW Offload for BGP session HLD.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/doc/bfd/BFD HW Offload for BGP session HLD.md b/doc/bfd/BFD HW Offload for BGP session HLD.md index 0e8af4e9d3f..0df9e2142bf 100644 --- a/doc/bfd/BFD HW Offload for BGP session HLD.md +++ b/doc/bfd/BFD HW Offload for BGP session HLD.md @@ -18,6 +18,8 @@ - [2.3 bfddp state change format](#23-bfddp-state-change-format) - [2.4 Orchestration Agent](#24-orchestration-agent) - [BfdOrch](#bfdorch) + - [Source mac address handling](#source-mac-address-handling) + - [Remote discriminator and timer handling for frr show bfd peer command](#remote-discriminator-and-timer-handling-for-frr-show-bfd-peer-command) - [2.5 BFD state\_db format example](#25-bfd-state_db-format-example) - [2.6 Local Discriminator handling](#26-local-discriminator-handling) - [2.7 IPv6 link local address support](#27-ipv6-link-local-address-support) @@ -228,6 +230,17 @@ struct bfddp_state_change { ### BfdOrch +#### Source mac address handling +If specify interface when creating BFD session (see section 2.7, link local address case), source mac address and destination mac address need to be provided. +Current bfdorch gets source mac address from the port: + + attr.id = SAI_BFD_SESSION_ATTR_SRC_MAC_ADDRESS; + memcpy(attr.value.mac, port.m_mac.getMac(), sizeof(sai_mac_t)); + attrs.emplace_back(attr); + +But in the link local address case, the mac address of the port is empty. So the design here need to be modified, let application provide source mac address. + +#### Remote discriminator and timer handling for frr show bfd peer command The frr 'show bfd peers' CLI needs to show BFD peer information for hardware offloaded BFD sessions: - remote discriminator - remote detect multiplier @@ -338,9 +351,9 @@ Peer Addr Interface Vrf State Type Local Addr T -------------- ----------- ------- ------- ------------ -------------- ------------- ------------- ------------ ---------- --------------------- 10.200.200.200 default default Up async_active 10.200.200.201 300 300 3 true 1 sonic@sonic:/var/tmp$ +``` -``` ## 2.5 BFD state_db format example ``` From 0624435a8be02887dd9f77df72baa23f470b108c Mon Sep 17 00:00:00 2001 From: Baorong Liu <96146196+baorliu@users.noreply.github.com> Date: Fri, 7 Jun 2024 10:22:25 -0700 Subject: [PATCH 05/14] Update BFD HW Offload for BGP session HLD.md --- doc/bfd/BFD HW Offload for BGP session HLD.md | 116 ++++++++++++++---- 1 file changed, 89 insertions(+), 27 deletions(-) diff --git a/doc/bfd/BFD HW Offload for BGP session HLD.md b/doc/bfd/BFD HW Offload for BGP session HLD.md index 0df9e2142bf..f87ac181f37 100644 --- a/doc/bfd/BFD HW Offload for BGP session HLD.md +++ b/doc/bfd/BFD HW Offload for BGP session HLD.md @@ -3,32 +3,33 @@ # Table of Content - [BFD HW Offload for BGP session](#bfd-hw-offload-for-bgp-session) - [Table of Content](#table-of-content) - - [Revision](#revision) + - [Revision](#revision) - [About this Manual](#about-this-manual) - [Definitions/Abbreviation](#definitionsabbreviation) - - [Table 1: Abbreviations](#table-1-abbreviations) + - [Table 1: Abbreviations](#table-1-abbreviations) - [1 Requirements Overview](#1-requirements-overview) - - [1.1 Functional requirements](#11-functional-requirements) - - [1.2 CLI requirements](#12-cli-requirements) - - [1.3 Scalability and Default Values](#13-scalability-and-default-values) - - [1.4 Warm Restart requirements](#14-warm-restart-requirements) + - [1.1 Functional requirements](#11-functional-requirements) + - [1.2 CLI requirements](#12-cli-requirements) + - [1.3 Scalability and Default Values](#13-scalability-and-default-values) + - [1.4 Warm Restart requirements](#14-warm-restart-requirements) - [2 Modules Design](#2-modules-design) - - [2.1 bfdsyncd](#21-bfdsyncd) - - [2.2 BFD DP message format](#22-bfd-dp-message-format) - - [2.3 bfddp state change format](#23-bfddp-state-change-format) - - [2.4 Orchestration Agent](#24-orchestration-agent) - - [BfdOrch](#bfdorch) - - [Source mac address handling](#source-mac-address-handling) - - [Remote discriminator and timer handling for frr show bfd peer command](#remote-discriminator-and-timer-handling-for-frr-show-bfd-peer-command) - - [2.5 BFD state\_db format example](#25-bfd-state_db-format-example) - - [2.6 Local Discriminator handling](#26-local-discriminator-handling) - - [2.7 IPv6 link local address support](#27-ipv6-link-local-address-support) - - [2.8 Control plane BFD](#28-control-plane-bfd) - - [2.9 CLI](#29-cli) -- [3 start the daemons](#3-start-the-daemons) - - [3.1 start bfdsyncd](#31-start-bfdsyncd) - - [3.2 start bfdd](#32-start-bfdd) - - [3.3 create bfd session from vtysh cli](#33-create-bfd-session-from-vtysh-cli) + - [2.1 bfdsyncd](#21-bfdsyncd) + - [2.2 BFD DP message format](#22-bfd-dp-message-format) + - [2.3 bfddp state change format](#23-bfddp-state-change-format) + - [2.4 Orchestration Agent](#24-orchestration-agent) + - [BfdOrch](#bfdorch) + - [Source mac address handling](#source-mac-address-handling) + - [Remote discriminator and timer handling for frr show bfd peer command](#remote-discriminator-and-timer-handling-for-frr-show-bfd-peer-command) + - [2.5 BFD state\_db format example](#25-bfd-state_db-format-example) + - [2.6 Local Discriminator handling](#26-local-discriminator-handling) + - [2.7 IPv6 link local address support](#27-ipv6-link-local-address-support) + - [2.8 Control plane BFD](#28-control-plane-bfd) + - [2.9 CLI](#29-cli) +- [3 Configuration](#3-configuration) +- [4 mannual testing](#4-mannual-testing) + - [4.1 start bfdsyncd](#41-start-bfdsyncd) + - [4.2 start bfdd](#42-start-bfdd) + - [4.3 create bfd session from vtysh cli](#43-create-bfd-session-from-vtysh-cli) ###### Revision @@ -476,23 +477,84 @@ A control plane BFD approach is to use FRR SW BFD. If use HW offload BFD for FRR no new CLI introduced for this feature -# 3 start the daemons +# 3 Configuration +To support hardware bfd offload for bgp,need to run two daemons inside bgp container: + bfdd --dplaneaddr ipv4c:127.0.0.1 + bfdsyncd +But the logic to launch is outside of the scope of this document. +Here is an example to configure it in config_db: -## 3.1 start bfdsyncd +/etc/sonic/config_db.json +``` + "FEATURE": { + "bgp": { + "bfd_hw_offload": "true", +``` +dockers/docker-fpm-frr/frr/supervisord/supervisord.conf.j2 +``` +{% if FEATURE.bgp.bfd_hw_offload is defined and FEATURE.bgp.bfd_hw_offload == "true" %} +[program:bfdsyncd] +command=bfdsyncd -d +priority=6 +autostart=false +autorestart=false +startsecs=0 +stdout_logfile=syslog +stderr_logfile=syslog +dependent_startup=true +dependent_startup_wait_for=bgpd:running +{% endif %} + +{% if FEATURE.bgp.bfd_hw_offload is defined and FEATURE.bgp.bfd_hw_offload == "true" %} +[program:bfdd] +command=/usr/lib/frr/bfdd --dplaneaddr ipv4c:127.0.0.1 +priority=4 +stopsignal=KILL +autostart=false +autorestart=false +startsecs=0 +stdout_logfile=syslog +stderr_logfile=syslog +dependent_startup=true +dependent_startup_wait_for=bfdsyncd:running +{% else %} +[program:bfdd] +command=/usr/lib/frr/bfdd -A 127.0.0.1 +priority=4 +stopsignal=KILL +autostart=false +autorestart=false +startsecs=0 +stdout_logfile=syslog +stderr_logfile=syslog +dependent_startup=true +dependent_startup_wait_for=zebra:running +{% endif %} +``` +``` +inside bgp container: +root@sonic:/# ps -ef +UID PID PPID C STIME TTY TIME CMD +root 58 1 0 16:44 pts/0 00:00:00 bfdsyncd -d +frr 69 1 0 16:44 pts/0 00:00:00 /usr/lib/frr/bfdd --dplaneaddr ipv4c:127.0.0.1 +``` + +# 4 mannual testing + +## 4.1 start bfdsyncd run bfdsyncd inside bgp container, default port number for bfd is 50700 ``` sonic@sonic:$ docker exec -it bgp bash root@sonic# bfdsyncd & ``` -## 3.2 start bfdd +## 4.2 start bfdd run bfdd with option dplaneaddr inside bgp container, it connects to default bfd port. ``` sonic@sonic:$ docker exec -it bgp bash root@sonic# /usr/lib/frr/bfdd --dplaneaddr ipv4c:127.0.0.1 ``` -The logic for starting frr/bfdd with different parameters is outside the scope of this document. -## 3.3 create bfd session from vtysh cli +## 4.3 create bfd session from vtysh cli ``` sonic@sonic:/var/tmp$ docker exec -it bgp bash root@sonic:/# bfdsyncd& From 7e73311c6a47e7ea97145bbee4e3be8498db8410 Mon Sep 17 00:00:00 2001 From: Baorong Liu <96146196+baorliu@users.noreply.github.com> Date: Fri, 7 Jun 2024 10:47:57 -0700 Subject: [PATCH 06/14] Update BFD HW Offload for BGP session HLD.md --- doc/bfd/BFD HW Offload for BGP session HLD.md | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/doc/bfd/BFD HW Offload for BGP session HLD.md b/doc/bfd/BFD HW Offload for BGP session HLD.md index f87ac181f37..4e2cdec9f92 100644 --- a/doc/bfd/BFD HW Offload for BGP session HLD.md +++ b/doc/bfd/BFD HW Offload for BGP session HLD.md @@ -30,6 +30,10 @@ - [4.1 start bfdsyncd](#41-start-bfdsyncd) - [4.2 start bfdd](#42-start-bfdd) - [4.3 create bfd session from vtysh cli](#43-create-bfd-session-from-vtysh-cli) +- [5, limitations](#5-limitations) + - [5.1 unsupported commands:](#51-unsupported-commands) + - [5.2 unsupported bfd dataplane message](#52-unsupported-bfd-dataplane-message) + - [5.3 unsupported bfd dataplane message](#53-unsupported-bfd-dataplane-message) ###### Revision @@ -585,3 +589,20 @@ Peer Addr Interface Vrf State Type Local Addr TX In 192.168.1.29 default default Down async_active 192.168.1.24 300 300 3 false 1 sonic@sonic:/var/tmp$ ``` + +# 5, limitations +## 5.1 unsupported commands: +show bfd peers counters + not able to read hardware bfd counter value and show it with this command. + Zero value will be returned if use issue this command. + +## 5.2 unsupported bfd dataplane message + ECHO_REQUEST + ECHO_REPLY + DP_REQUEST_SESSION_COUNTERS + BFD_SESSION_COUNTERS + +## 5.3 unsupported bfd dataplane message + DP_ADD_SESSION + DP_DELETE_SESSION + BFD_STATE_CHANGE From f2290c58bd50c1255db8cd2b7e341a133ec1bc41 Mon Sep 17 00:00:00 2001 From: Baorong Liu <96146196+baorliu@users.noreply.github.com> Date: Fri, 7 Jun 2024 11:56:31 -0700 Subject: [PATCH 07/14] Update BFD HW Offload for BGP session HLD.md --- doc/bfd/BFD HW Offload for BGP session HLD.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/bfd/BFD HW Offload for BGP session HLD.md b/doc/bfd/BFD HW Offload for BGP session HLD.md index 4e2cdec9f92..b46defc9444 100644 --- a/doc/bfd/BFD HW Offload for BGP session HLD.md +++ b/doc/bfd/BFD HW Offload for BGP session HLD.md @@ -33,7 +33,7 @@ - [5, limitations](#5-limitations) - [5.1 unsupported commands:](#51-unsupported-commands) - [5.2 unsupported bfd dataplane message](#52-unsupported-bfd-dataplane-message) - - [5.3 unsupported bfd dataplane message](#53-unsupported-bfd-dataplane-message) + - [5.3 supported bfd dataplane message](#53-supported-bfd-dataplane-message) ###### Revision @@ -602,7 +602,7 @@ show bfd peers counters DP_REQUEST_SESSION_COUNTERS BFD_SESSION_COUNTERS -## 5.3 unsupported bfd dataplane message +## 5.3 supported bfd dataplane message DP_ADD_SESSION DP_DELETE_SESSION BFD_STATE_CHANGE From 167a8b6ac7529723e03004e16bb6848b1466203b Mon Sep 17 00:00:00 2001 From: Baorong Liu Date: Wed, 12 Feb 2025 13:59:45 -0800 Subject: [PATCH 08/14] Update BFD HW Offload for BGP session HLD.md --- doc/bfd/BFD HW Offload for BGP session HLD.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/doc/bfd/BFD HW Offload for BGP session HLD.md b/doc/bfd/BFD HW Offload for BGP session HLD.md index b46defc9444..6e3b14f270c 100644 --- a/doc/bfd/BFD HW Offload for BGP session HLD.md +++ b/doc/bfd/BFD HW Offload for BGP session HLD.md @@ -483,8 +483,10 @@ no new CLI introduced for this feature # 3 Configuration To support hardware bfd offload for bgp,need to run two daemons inside bgp container: +``` bfdd --dplaneaddr ipv4c:127.0.0.1 bfdsyncd +``` But the logic to launch is outside of the scope of this document. Here is an example to configure it in config_db: @@ -492,7 +494,9 @@ Here is an example to configure it in config_db: ``` "FEATURE": { "bgp": { - "bfd_hw_offload": "true", + "bfd_hw_offload": "true" + } + }, ``` dockers/docker-fpm-frr/frr/supervisord/supervisord.conf.j2 ``` From 12070f13497aa99c19c2e516d89171aefbd21b04 Mon Sep 17 00:00:00 2001 From: Baorong Liu Date: Wed, 12 Feb 2025 14:03:53 -0800 Subject: [PATCH 09/14] Update BFD HW Offload for BGP session HLD.md --- doc/bfd/BFD HW Offload for BGP session HLD.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/bfd/BFD HW Offload for BGP session HLD.md b/doc/bfd/BFD HW Offload for BGP session HLD.md index 6e3b14f270c..f69eca1a2be 100644 --- a/doc/bfd/BFD HW Offload for BGP session HLD.md +++ b/doc/bfd/BFD HW Offload for BGP session HLD.md @@ -487,8 +487,8 @@ To support hardware bfd offload for bgp,need to run two daemons inside bgp conta bfdd --dplaneaddr ipv4c:127.0.0.1 bfdsyncd ``` -But the logic to launch is outside of the scope of this document. -Here is an example to configure it in config_db: +But the logic to launch these to daemons is outside of the scope of this document. +Here is an example to do it through config_db and frr supervisord: /etc/sonic/config_db.json ``` From e71602394dac789fe4517b3cc2f8a05d789ad605 Mon Sep 17 00:00:00 2001 From: Baorong Liu Date: Thu, 16 Apr 2026 14:27:08 -0700 Subject: [PATCH 10/14] bfdsyncd: make the mac address for linklocal implementation dependent Signed-off-by: Baorong Liu --- doc/bfd/BFD HW Offload for BGP session HLD.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/doc/bfd/BFD HW Offload for BGP session HLD.md b/doc/bfd/BFD HW Offload for BGP session HLD.md index f69eca1a2be..c619f06d3a8 100644 --- a/doc/bfd/BFD HW Offload for BGP session HLD.md +++ b/doc/bfd/BFD HW Offload for BGP session HLD.md @@ -422,18 +422,22 @@ When bgp use link local address to peer with remote system, it needs to specify because link local address is not a routable ip address, so bfdsyncd has to provide source mac and destination mac address to let bfdorch create a bfd session with inject-down mode (construct layer 2 packet). -SONiC does not set source mac address for a port with link local address only. But the source mac address can be found at /sys/class/net/'interface_name'/address +SONiC does not set source mac address for a port with link local address only. In the data struct bfddp_session, there is no field to carry source mac address and destination mac address. So bfdsyncd is not able to get these mac addresses from frr/bfdd without changing the existing design. -One way to get destination mac address is to read neighbor table, for example, using cmd ip -6 neighbor. +**How to get source mac address and destination mac address for IPv6 link local address is outside of the scope of this HLD, the implementation need to find a way to get these information.** + +Here is one example to get the mac addresses: + +Source mac address can be found at /sys/class/net/'interface_name'/address +For destination mac address, it can be read from neighbor table: ``` sonic@sonic:/var/tmp$ ip -6 neighbor get fe80::7a6b:17ff:fe5a:7000 dev Ethernet0 fe80::7a6b:17ff:fe5a:7000 dev Ethernet0 lladdr 78:6b:17:5a:70:00 router REACHABLE sonic@sonic:/var/tmp$ ``` -To make sure the the destination mac address is available before bgp creating bfd session, do not use this bgp configuration: "neighbor xxx disable-connected-check", let bgp check connection first before it try to create bfd session. so peer system mac address will be available if bfdsyncd issue a PING before try to read the neighbor table. +[Notes]To make sure the the destination mac address is available before bgp creating bfd session, do not use this bgp configuration: "neighbor xxx disable-connected-check", let bgp check connection first before it try to create bfd session. so peer system mac address will be available if bfdsyncd issue a PING before try to read the neighbor table. -example for bfd session with link local address: ``` bgp configuration: sonic(config-router)# neighbor FOO peer-group From 3a5ea4e4565abb2e256d474996a68babc9ed7bde Mon Sep 17 00:00:00 2001 From: Baorong Liu Date: Fri, 17 Apr 2026 13:23:22 -0700 Subject: [PATCH 11/14] bfdsyncd: address review comments Signed-off-by: Baorong Liu --- doc/bfd/BFD HW Offload for BGP session HLD.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/doc/bfd/BFD HW Offload for BGP session HLD.md b/doc/bfd/BFD HW Offload for BGP session HLD.md index c619f06d3a8..2b41a654065 100644 --- a/doc/bfd/BFD HW Offload for BGP session HLD.md +++ b/doc/bfd/BFD HW Offload for BGP session HLD.md @@ -436,8 +436,6 @@ fe80::7a6b:17ff:fe5a:7000 dev Ethernet0 lladdr 78:6b:17:5a:70:00 router REACHABL sonic@sonic:/var/tmp$ ``` -[Notes]To make sure the the destination mac address is available before bgp creating bfd session, do not use this bgp configuration: "neighbor xxx disable-connected-check", let bgp check connection first before it try to create bfd session. so peer system mac address will be available if bfdsyncd issue a PING before try to read the neighbor table. - ``` bgp configuration: sonic(config-router)# neighbor FOO peer-group From 49e8a1ccb03042b2d1023cf962fea90b28f04d4d Mon Sep 17 00:00:00 2001 From: Baorong Liu Date: Fri, 17 Apr 2026 14:30:44 -0700 Subject: [PATCH 12/14] bfdsyncd: add testplan section Signed-off-by: Baorong Liu --- doc/bfd/BFD HW Offload for BGP session HLD.md | 51 +++++++++++++++---- 1 file changed, 42 insertions(+), 9 deletions(-) diff --git a/doc/bfd/BFD HW Offload for BGP session HLD.md b/doc/bfd/BFD HW Offload for BGP session HLD.md index 2b41a654065..96487581367 100644 --- a/doc/bfd/BFD HW Offload for BGP session HLD.md +++ b/doc/bfd/BFD HW Offload for BGP session HLD.md @@ -549,22 +549,24 @@ root 58 1 0 16:44 pts/0 00:00:00 bfdsyncd -d frr 69 1 0 16:44 pts/0 00:00:00 /usr/lib/frr/bfdd --dplaneaddr ipv4c:127.0.0.1 ``` -# 4 mannual testing +# 4 Testplan -## 4.1 start bfdsyncd +## 4.1 Setup + If the bfdsyncd was not launched by the OS, here is the manual steps to launch bfdsyncd and bfdd +### 4.1.1 Start bfdsyncd run bfdsyncd inside bgp container, default port number for bfd is 50700 ``` sonic@sonic:$ docker exec -it bgp bash root@sonic# bfdsyncd & ``` -## 4.2 start bfdd +### 4.1.2 Start bfdd run bfdd with option dplaneaddr inside bgp container, it connects to default bfd port. ``` sonic@sonic:$ docker exec -it bgp bash root@sonic# /usr/lib/frr/bfdd --dplaneaddr ipv4c:127.0.0.1 ``` -## 4.3 create bfd session from vtysh cli +### 4.1.3 Validate the setupby creating a bfd session from vtysh cli ``` sonic@sonic:/var/tmp$ docker exec -it bgp bash root@sonic:/# bfdsyncd& @@ -596,19 +598,50 @@ Peer Addr Interface Vrf State Type Local Addr TX In sonic@sonic:/var/tmp$ ``` -# 5, limitations -## 5.1 unsupported commands: -show bfd peers counters +## 4.2 Unit tests +### 4.2.1 Mock tests + bfddp message DP_ADD_SESSION handling + bfddp message DP_DELETE_SESSION handling + BFD_STATE_CHANGE handling +### 4.2.2 SONiC mgmt tests +#### 4.2.2.1 Basic BGP neighbor(IPv4) with bfd monitoring +Configuration example: +``` +sonic(config-router)# neighbor FOO peer-group +sonic(config-router)# neighbor FOO remote-as external +sonic(config-router)# neighbor FOO disable-connected-check +sonic(config-router)# neighbor FOO ebgp-multihop 255 +sonic(config-router)# neighbor FOO update-source Loopback27 +sonic(config-router)# neighbor FOO bfd +sonic(config-router)# neighbor 10.200.200.201 peer-group FOO +``` +Test both BFD session state UP and DOWN cases, check bgp session state. + +#### 4.2.2.2 Basic BGP neighbor(IPv6) with bfd monitoring +#### 4.2.2.3 Basic BGP neighbor(IPv6 link local address) with bfd monitoring +Example for IPv6 link local, only interface provided in the bgp neighbor configuration +``` + neighbor FOO peer-group + neighbor FOO remote-as external + neighbor FOO ebgp-multihop 1 + neighbor Ethernet0 interface peer-group FOO + neighbor FOO bfd +``` + +# 5 Limitations +## 5.1 Unsupported commands: + show bfd peers counters + not able to read hardware bfd counter value and show it with this command. Zero value will be returned if use issue this command. -## 5.2 unsupported bfd dataplane message +## 5.2 Unsupported bfd dataplane message ECHO_REQUEST ECHO_REPLY DP_REQUEST_SESSION_COUNTERS BFD_SESSION_COUNTERS -## 5.3 supported bfd dataplane message +## 5.3 Supported bfd dataplane message DP_ADD_SESSION DP_DELETE_SESSION BFD_STATE_CHANGE From fe63c568d8bebd46bb4e75249690e2c0a4b99499 Mon Sep 17 00:00:00 2001 From: Baorong Liu Date: Fri, 17 Apr 2026 15:28:16 -0700 Subject: [PATCH 13/14] bfdsyncd: update testplan section Signed-off-by: Baorong Liu --- doc/bfd/BFD HW Offload for BGP session HLD.md | 65 ++++++++++++------- 1 file changed, 43 insertions(+), 22 deletions(-) diff --git a/doc/bfd/BFD HW Offload for BGP session HLD.md b/doc/bfd/BFD HW Offload for BGP session HLD.md index 96487581367..2254e826ffc 100644 --- a/doc/bfd/BFD HW Offload for BGP session HLD.md +++ b/doc/bfd/BFD HW Offload for BGP session HLD.md @@ -551,22 +551,20 @@ frr 69 1 0 16:44 pts/0 00:00:00 /usr/lib/frr/bfdd --dplanead # 4 Testplan -## 4.1 Setup - If the bfdsyncd was not launched by the OS, here is the manual steps to launch bfdsyncd and bfdd -### 4.1.1 Start bfdsyncd -run bfdsyncd inside bgp container, default port number for bfd is 50700 +Pre-requisite: + +run bfdsyncd inside bgp container (if the OS does not launch it), default port number for bfd is 50700 ``` sonic@sonic:$ docker exec -it bgp bash root@sonic# bfdsyncd & ``` -### 4.1.2 Start bfdd run bfdd with option dplaneaddr inside bgp container, it connects to default bfd port. ``` sonic@sonic:$ docker exec -it bgp bash root@sonic# /usr/lib/frr/bfdd --dplaneaddr ipv4c:127.0.0.1 ``` -### 4.1.3 Validate the setupby creating a bfd session from vtysh cli +Verify the setup by creating a bfd session from vtysh cli ``` sonic@sonic:/var/tmp$ docker exec -it bgp bash root@sonic:/# bfdsyncd& @@ -598,13 +596,27 @@ Peer Addr Interface Vrf State Type Local Addr TX In sonic@sonic:/var/tmp$ ``` -## 4.2 Unit tests -### 4.2.1 Mock tests - bfddp message DP_ADD_SESSION handling - bfddp message DP_DELETE_SESSION handling - BFD_STATE_CHANGE handling -### 4.2.2 SONiC mgmt tests -#### 4.2.2.1 Basic BGP neighbor(IPv4) with bfd monitoring +## 4.1 Mock tests +verify the following messages: +- Bfddp message DP_ADD_SESSION handling +- BFD_STATE_CHANGE handling +- Bfddp message DP_DELETE_SESSION handling + +| Step | Goal | Expected results | +|-|-|-| +|Prepare a BFD DP message with DP_ADD_SESSION type |BFD session creation in APPL DB |BFD session created in APPL_DB, and the fields value are correct | +|Call handleBfdStateUpdate with the above BFD session key| State change |DP message with BFD_STATE_CHANGE type is sent| +|Prepare a BFD DP message with DP_DELETE_SESSION type |BFD session deletion in APPL DB |BFD session removed from APPL_DB| + + +## 4.2 Mgmt tests +### 4.2.1 Basic BGP neighbor(IPv4) with bfd monitoring +| Step | Goal | Expected results | +|-|-|-| +|Configure the bgp using the following example|BFD session creation|HW offload bfd session created, check with show bfd summary command in SONiC CLI| +|Bring the interface down|BGP sessino state change|BGP session down| +|Bring the interface UP and waiting for BFD session up|BGP sessino state change|BGP session UP| + Configuration example: ``` sonic(config-router)# neighbor FOO peer-group @@ -617,15 +629,24 @@ sonic(config-router)# neighbor 10.200.200.201 peer-group FOO ``` Test both BFD session state UP and DOWN cases, check bgp session state. -#### 4.2.2.2 Basic BGP neighbor(IPv6) with bfd monitoring -#### 4.2.2.3 Basic BGP neighbor(IPv6 link local address) with bfd monitoring -Example for IPv6 link local, only interface provided in the bgp neighbor configuration -``` - neighbor FOO peer-group - neighbor FOO remote-as external - neighbor FOO ebgp-multihop 1 - neighbor Ethernet0 interface peer-group FOO - neighbor FOO bfd +### 4.2.2 Basic BGP neighbor(IPv6) with bfd monitoring +Similar to the above test (4.2.1), but use IPv6 neighbor + +### 4.2.3 Basic BGP neighbor(IPv6 link local address) with bfd monitoring +Example for IPv6 link local, IPv6 link local address enabled for the corresponding interface, use interface name only in the bgp neighbor configuration +| Step | Goal | Expected results | +|-|-|-| +|Configure the DUT use link local IP address|IPv6 link local address|Check if the interface has IPv6 link local address assigned| +|ping neighbor's link local IP address with source interface name (ping6 -c 3 -I ) |update neighbor mac address to neigbor table|Check if the neighbor's mac address is available| +|configure BGP neighbor using the following example|BFD HW offload session creation|check if the BFD HW offload session created after apply the configuration| +|Bring the interface down|BGP sessino state change|BGP session down| +|Bring the interface UP and waiting for BFD session up|BGP sessino state change|BGP session UP| +``` +sonic(config-router)# neighbor FOO peer-group +sonic(config-router)# neighbor FOO remote-as external +sonic(config-router)# neighbor FOO ebgp-multihop 1 +sonic(config-router)# neighbor Ethernet0 interface peer-group FOO +sonic(config-router)# neighbor FOO bfd ``` # 5 Limitations From edd09da2f8d6a3b42275dfcb47eabb96b60b0169 Mon Sep 17 00:00:00 2001 From: Baorong Liu Date: Fri, 17 Apr 2026 15:38:27 -0700 Subject: [PATCH 14/14] bfdsyncd: update table of content Signed-off-by: Baorong Liu --- doc/bfd/BFD HW Offload for BGP session HLD.md | 56 ++++++++++--------- 1 file changed, 29 insertions(+), 27 deletions(-) diff --git a/doc/bfd/BFD HW Offload for BGP session HLD.md b/doc/bfd/BFD HW Offload for BGP session HLD.md index 2254e826ffc..6d4c95fb376 100644 --- a/doc/bfd/BFD HW Offload for BGP session HLD.md +++ b/doc/bfd/BFD HW Offload for BGP session HLD.md @@ -2,38 +2,40 @@ # Table of Content - [BFD HW Offload for BGP session](#bfd-hw-offload-for-bgp-session) -- [Table of Content](#table-of-content) - - [Revision](#revision) +- [Table of Contents](#table-of-contents) +- [Revision](#revision) - [About this Manual](#about-this-manual) - [Definitions/Abbreviation](#definitionsabbreviation) - - [Table 1: Abbreviations](#table-1-abbreviations) + - [Table 1: Abbreviations](#table-1-abbreviations) - [1 Requirements Overview](#1-requirements-overview) - - [1.1 Functional requirements](#11-functional-requirements) - - [1.2 CLI requirements](#12-cli-requirements) - - [1.3 Scalability and Default Values](#13-scalability-and-default-values) - - [1.4 Warm Restart requirements](#14-warm-restart-requirements) + - [1.1 Functional requirements](#11-functional-requirements) + - [1.2 CLI requirements](#12-cli-requirements) + - [1.3 Scalability and Default Values](#13-scalability-and-default-values) + - [1.4 Warm Restart requirements](#14-warm-restart-requirements) - [2 Modules Design](#2-modules-design) - - [2.1 bfdsyncd](#21-bfdsyncd) - - [2.2 BFD DP message format](#22-bfd-dp-message-format) - - [2.3 bfddp state change format](#23-bfddp-state-change-format) - - [2.4 Orchestration Agent](#24-orchestration-agent) - - [BfdOrch](#bfdorch) - - [Source mac address handling](#source-mac-address-handling) - - [Remote discriminator and timer handling for frr show bfd peer command](#remote-discriminator-and-timer-handling-for-frr-show-bfd-peer-command) - - [2.5 BFD state\_db format example](#25-bfd-state_db-format-example) - - [2.6 Local Discriminator handling](#26-local-discriminator-handling) - - [2.7 IPv6 link local address support](#27-ipv6-link-local-address-support) - - [2.8 Control plane BFD](#28-control-plane-bfd) - - [2.9 CLI](#29-cli) + - [2.1 bfdsyncd](#21-bfdsyncd) + - [2.2 BFD DP message format](#22-bfd-dp-message-format) + - [2.3 bfddp state change format](#23-bfddp-state-change-format) + - [2.4 Orchestration Agent](#24-orchestration-agent) + - [BfdOrch](#bfdorch) + - [Source mac address handling](#source-mac-address-handling) + - [Remote discriminator and timer handling for frr show bfd peer command](#remote-discriminator-and-timer-handling-for-frr-show-bfd-peer-command) + - [2.5 BFD state_db format example](#25-bfd-state_db-format-example) + - [2.6 Local Discriminator handling](#26-local-discriminator-handling) + - [2.7 IPv6 link local address support](#27-ipv6-link-local-address-support) + - [2.8 Control plane BFD](#28-control-plane-bfd) + - [2.9 CLI](#29-cli) - [3 Configuration](#3-configuration) -- [4 mannual testing](#4-mannual-testing) - - [4.1 start bfdsyncd](#41-start-bfdsyncd) - - [4.2 start bfdd](#42-start-bfdd) - - [4.3 create bfd session from vtysh cli](#43-create-bfd-session-from-vtysh-cli) -- [5, limitations](#5-limitations) - - [5.1 unsupported commands:](#51-unsupported-commands) - - [5.2 unsupported bfd dataplane message](#52-unsupported-bfd-dataplane-message) - - [5.3 supported bfd dataplane message](#53-supported-bfd-dataplane-message) +- [4 Testplan](#4-testplan) + - [4.1 Mock tests](#41-mock-tests) + - [4.2 Mgmt tests](#42-mgmt-tests) + - [4.2.1 Basic BGP neighbor(IPv4) with bfd monitoring](#421-basic-bgp-neighboripv4-with-bfd-monitoring) + - [4.2.2 Basic BGP neighbor(IPv6) with bfd monitoring](#422-basic-bgp-neighboripv6-with-bfd-monitoring) + - [4.2.3 Basic BGP neighbor(IPv6 link local address) with bfd monitoring](#423-basic-bgp-neighboripv6-link-local-address-with-bfd-monitoring) +- [5 Limitations](#5-limitations) + - [5.1 Unsupported commands](#51-unsupported-commands) + - [5.2 Unsupported bfd dataplane message](#52-unsupported-bfd-dataplane-message) + - [5.3 Supported bfd dataplane message](#53-supported-bfd-dataplane-message) ###### Revision