graceful restart

Pre-Alpha. This page describes behavior that may change.

Graceful Restart (RFC 4724) and Long-Lived Graceful Restart (RFC 9494) let a BGP session preserve forwarding state across a restart of one side, instead of dropping every route and waiting for reconvergence. Ze implements both through the bgp-gr plugin, which pairs with bgp-rib for the in-memory side and (optionally) bgp-persist for the disk side.

The operator story is on the upgrade and restart page. This page is the configuration reference.

The short version

plugin {
    internal gr  { use bgp-gr; }
    internal rib { use bgp-rib; }
}

bgp {
    peer upstream {
        connection { remote { ip 10.0.0.1; } }
        session {
            asn { remote 65001; }
            capability {
                graceful-restart {
                    restart-time 120;
                }
            }
        }
        attach process gr {
            receive [ open-received state eor ];
            send    [ update ];
        }
        attach process rib {
            receive [ update state refresh ];
            send    [ update ];
        }
    }
}

The bgp-gr plugin needs the peer's OPEN, its state changes (to know when the session dropped) and EOR markers (to know when the new session has converged). The bgp-rib plugin needs updates, state, refresh, and the update send path. The engine starts bgp-rib first because bgp-gr declares it as a dependency. Both run in the daemon process through internal and need no plugin { external } block.

The attach process block is authoritative since 2026-08-15. A peer that attaches neither program feeds neither one any peer-scoped event, so neither one acts on that peer's session. See Attached programs.

Keep the bgp-gr engine in the daemon process

Load the plugin with internal <name> { use bgp-gr; }, as every example on this page does. run "ze plugin bgp-gr" starts the plugin engine in a child process, and that changes what the daemon sends to every peer.

The RFC 9494 egress filter runs in the daemon. The peer capability state it reads is written by the plugin engine, so with run that state stays empty for the life of the daemon process. The filter then treats every neighbour as one from which the LLGR Capability was never received: it withdraws stale routes from external peers and attaches NO_EXPORT with LOCAL_PREF 0 to the routes it sends to internal peers. Peers that negotiated LLGR with the child engine get the same treatment.

ze doctor reports the arrangement as doctor-bgp-gr-out-of-process. Run ze explain doctor-bgp-gr-out-of-process for the full text.

Configuration reference

Path	Type	Default	Description
graceful-restart/restart-time	uint16	120	Seconds to hold stale routes during restart. Range 0 to 4095.
graceful-restart/long-lived-stale-time	uint32	none	LLGR period per family in seconds. Range 0 to 16777215.

The graceful-restart container uses YANG presence, so to disable GR for a peer, simply omit the graceful-restart container.

A typical starting point:

capability {
    graceful-restart {
        restart-time          120;      # 2 minutes for a plain GR
        long-lived-stale-time 3600;     # Plus 1 hour of LLGR
    }
}

How standard GR flows

The sequence is deterministic.

1. Normal session. Session establishes, GR capability negotiated in OPEN, routes flow normally.
2. Peer goes down. The bgp-gr plugin sends retain-routes to the RIB.
3. RIB marks routes as stale. Routes stay in forwarding but are tagged.
4. Restart timer starts. Countdown from restart-time seconds.
5. Peer reconnects. A new session is established, fresh routes are received.
6. Fresh routes replace stale ones. Each new route implicitly clears the stale flag on its prefix.
7. End-of-RIB received. The GR plugin sends purge-stale to the RIB.
8. Any remaining stale routes are removed. These are prefixes that existed before the restart but did not come back.

If the peer does not reconnect within restart-time seconds, every stale route is purged. A 5-second safety margin is added to account for processing delays.

The RIB safety net

If the bgp-gr plugin crashes or fails to issue purge-stale, the RIB has its own timeout: stale routes are expired automatically restart-time + 5s after the session dropped. This is the fail-safe that prevents a stuck GR from keeping routes around forever.

Long-Lived GR

LLGR extends standard GR with a second, much longer stale window. When the standard GR restart-time expires without the peer reconnecting, instead of purging every stale route, LLGR keeps them attached with the LLGR_STALE community and a deprioritised position in best-path selection. Routes lose to any non-stale route but keep traffic flowing while the peer is away.

capability {
    graceful-restart {
        restart-time          120;
        long-lived-stale-time 3600;
    }
}

LLGR only activates when both peers negotiate it. Ze advertises LLGR capability (code 71) in OPEN when long-lived-stale-time is configured. LLGR requires the GR capability (code 64) to also be present: LLGR without GR is ignored per RFC 9494.

The LLGR state flow

1. GR period expires. The peer has not reconnected within restart-time seconds.
2. LLGR begins. For each family with long-lived-stale-time > 0:
   • Routes carrying the NO_LLGR community (0xFFFF0007) are deleted.
   • The LLGR_STALE community (0xFFFF0006) is attached to every remaining stale route.
   • Routes are marked as stale level 2 (deprioritised in best-path selection).
   • The per-family LLST timer starts.
3. During LLGR. LLGR-stale routes lose to any non-stale route in best-path selection. Between two LLGR-stale routes, normal tiebreaking applies.
4. LLST timer expires. Stale routes for that family are purged.
5. Peer reconnects during LLGR. Standard RFC 4724 procedures apply. Families with F-bit 0 or missing from the new OPEN are purged.

Stale levels

Ze uses a graduated stale level system for route prioritisation.

Level	Meaning	Best-path behaviour
0	Fresh	Normal selection.
1	GR-stale	Normal selection (not deprioritised).
2+	LLGR-stale	Loses to any route with level less than 2.

Readvertising a stale route to a peer that does not speak LLGR

RFC 9494 sections 4.3 and 4.6 govern what happens when an LLGR-stale route is advertised onward to a neighbour that did not negotiate LLGR.

Destination	Action
LLGR was received from it	Advertise unchanged.
Internal, LLGR not received	Advertise with NO_EXPORT and LOCAL_PREF 0 (section 4.6).
External, LLGR not received	Withdraw the route (section 4.3).

The stale level, not the restart state, decides this. A route can become stale without any restart having happened: request bgp rib mark-stale, run by an operator or by the GR timer, marks routes stale on a session that never went down. Those routes take the same depreference on readvertisement as ones staled by a peer restart.

The filter fails closed. When it cannot read the peer capability state, it answers "LLGR was not received" for every destination, raises one warning per process, and then withdraws or depreferences. An unread state used to accept, which put a long-lived stale route into a neighbour that never agreed to hold one. The one arrangement that leaves the state permanently unread is run "ze plugin bgp-gr".

Skipping GR, going straight to LLGR

If you set restart-time 0 and a non-zero long-lived-stale-time, the standard GR period is skipped entirely. On session drop, LLGR begins immediately.

restart-time	long-lived-stale-time	Behaviour
0	non-zero	Skip GR, enter LLGR immediately.
non-zero	0	GR only, no LLGR.
0	0	Neither.
non-zero	non-zero	GR, then LLGR.

The well-known communities

Community	Value	Purpose
LLGR_STALE	0xFFFF0006	Attached to stale routes during the LLGR period.
NO_LLGR	0xFFFF0007	Routes with this community are deleted on LLGR entry.

Verifying the capability is negotiated

ze# peer upstream capabilities
capabilities {
    four-octet-asn    yes
    graceful-restart  yes
    long-lived-gr     yes
    families [ ipv4/unicast ipv6/unicast ]
}

ze cli -c "rib routes received" shows routes with the stale flag when applicable.

Decode an LLGR capability blob from hex:

ze plugin bgp-gr --capa 00010180000e10

This prints per-family LLST values and F-bit flags.

Without Graceful Restart

When GR is not configured, or when the peer does not advertise the capability, routes are withdrawn immediately on session drop. No stale state, no restart timer. If you do not configure graceful-restart, nothing in this page applies.

See also

• Upgrade and restart for the operator workflow around GR and LLGR.
• Plugins: resilience for bgp-gr and bgp-route-refresh.
• In-tree GR guide for the authoritative reference.

Adapted from main/docs/guide/graceful-restart.md.