HA - What is the way forward?

[BSpendlove]

The HA feature is quite an important step for osvbng in becoming production ready.

This discussion is a temporary place to dump all my ideas into regarding the future of the HA setup for osvbng, ideally I want to release the initial basic working implementation for v0.3.0.

The proposal for the HA architecture at a high level which I will be approaching this (as I am the only core developer right now), is:

1. Firstly, I want the initial stability on a single node deployment to be more stable, including recovery and monitoring (aka a watchdog like feature) mostly for the dataplane crashing, FRR crashing and being able to crash the control plane while leaving existing programmed subscribers (IPoE and PPPoE) working in the dataplane (with the exception of control plane functions like LCP echos, DHCP requests that touch the control plane, not unicast renewals to a dhcp relay server once we introduce this concept)
2. Introduce/extend the event bus to allow us to listen for events which HA will want to know about, eg. interfaces going down because we want to introduce some ideas around HA that allow us to switchover the primary BNG
3. Before we introduce the HA implementation in phase 4, we want to take a different approach to the config model so it can be easily integrated into an automated deployment / gitops, this just takes the generic idea that other apps use (eg. nginx, cillium) to have a config directory where all the config files are then merged for the runtime config. (conf.d) so that people can deploy common configurations (eg. VRFs, logging, system config, OSPF/ISIS/BGP common configs) within this directory and have a specific file rendered for the BNG node specific config which is different
4. Introduce a way to better advertise/withdraw networks from FRR without depending on vtysh/frr-reload.py because ideally this should be fairly snappy, but vtysh may be the only option for now without looking into enabling the gRPC implementation for FRR. This will support the need to advertise BGP networks to the core facing devices from the current active BNG.

Most if not all deployments I've seen of vendor BNGs in the real world have been 2 max from our perspective of a subscriber group (inspired from TR-459). So we do not plan on supporting more than 2 BNGs within a cluster for each subscriber group (each subscriber group determines the state of a set of subscribers for active/standby configuration).

PPPoE

PPPoE redundancy across BNGs will not be covered anytime soon in a public release, this requires extensive work and testing to become production ready, while networks are still using PPPoE, the majority of the industry on the access network provider level are typically providing IPoE-first services, I am not denying the fact that PPP-based handover still exist with LAC<>LNS environments, there are some wholesale advantages to this type of deployment but it simply doesn't fit our use case with access network providers typically transporting QinQ ethernet packets across their own MPLS core, handover point (E-NNI) typically being in an aggregation site/datacenter where the ISP also takes services from other access network providers.

State persistence and syncing

opdb was introduced a few commits back to provide an easy persistent store (sqlite provider is the only implementation right now) for when either both dataplane and/or control plane crashes. osvbng can reprogram the dataplane (or reload subscriber state into control plane in-memory cache) and get back to a working state. This is not extended for the HA capabilities because its local-scoped to the BNG deployment, however the idea is to introduce a gRPC based connection between active/standby to sync subscriber state required to program the dataplane, and continue with the control plane functions like DHCP synthetic responses, ARP replies, ICMPv6 RS/NS, PPP LCP echos, etc... The local BNG after receiving the bulk/single subscriber sync will then store within its own opdb, but on restart it must request a opdb refresh and consolidate all the existing session data with the new received data from the peer BNG.

Configuration

Some consideration around the configuration may require some work before (or after) to better align with being able to automated the deployment of osvbng within a network either from an OSS perspective or just gitops itself. The configuration yaml file right now is a single file which is starting to get large, but also extremely repetitive for common deployments where multiple BNGs in the network should have a common configuration like NTP servers, BGP peer groups, route-maps/prefix-sets/community-sets, base IGP implementation, and more...

It would be great to introduce a config.d directory where all configurations are merged during the initialization of osvbng, so that operators can deploy common configurations which are the same across multiple BNGs, but more specific BNGs like interfaces with IP addresses on, or BGP neighbor statements within different countries/regions can be deployed separately into the directory, keeping a nice structured version controlled environment for the operator.

To VRRP or not to VRRP

Most BNG implementations for HA typically deploy a VRRP like mechanism on the access interfaces (eg. virtual mac-address) but will typically run the heartbeat/state over a unicast connection between the 2 BNGs (l3 is a hard requirement in our implementation, we shouldn't limit to L2 domains only)

We will implement a VRRP-like functionality for the access side of IPoE (active/standby, active only advertises the virtual MAC) without depending on the access network itself like typical keepalived/vrrp does.

Split-brain scenarios is obviously a concern here which need to be outlined but initially it will be basic, if a pair of BNGs end up in split-brain scenario, both BNGs should be able to make the decision independently if they should be advertising the virtual MAC downstream / responding to control plane packets and upstream functions like advertising BGP prefixes. We could probably introduce a similar election process like EVPN uses for forwarding BUM traffic (rfc8584)

We also need to consider that the virtual MAC address is used for both dataplane and control plane packets, some investigation is required on the IPv6 side of things to ensure we are going in the right direction with virtual mac vs physical mac for ICMPv6 ND/NS/RA/RS, etc...

Trackers

A tracker is like an IP SLA, but more generic that can act upon events like BGP neighbor state/events, IGP neighbor state/events, link up/down events, these can be attached to a subscriber group configuration to track the active/standby failover decision. A grace period should be introduced (or use preempt term) to avoid the flapping between BNGs, only after x amount of seconds/checks the original primary BNG can take over again to serve dataplane traffic. VPP has an event based way to listen for interfaces but FRR doesn't (as far as I can see) so we could either fork FRR and build it into it or just poll every minimum seconds (both ways has its advantages and disadvantages, forking FRR could be quite annoying to track upstream changes)