[ZTP] Re-assert front panel port config after ZTP profile removal

[gord1306]

Why I did it

When ZTP completes, ztp-profile.sh remove deletes ZTP|mode from CONFIG_DB and then restarts interfaces-config:

287|             sonic-db-cli CONFIG_DB DEL "ZTP|mode" > /dev/null
290|         updateActivity "Restarting network configuration"
292|         systemctl restart interfaces-config

interfaces-config regenerates /etc/network/interfaces from interfaces.j2, whose ZTP in-band block is gated on ZTP['mode'] being defined:

35| {% if (ZTP_DHCP_DISABLED is not defined) and (ZTP is defined) and (ZTP['mode'] is defined) %}
50| {% if ZTP['mode']['inband'] == 'true' %}
51| {% for port in PORT %}
54| auto {{ port }}
55| allow-hotplug {{ port }}
56| {% if PORT_DATA['PORT_TABLE:'+port]['value']['oper_status'] == 'up' %}
58| iface {{ port }} inet dhcp

With that table gone the block is no longer rendered, so the regenerated file contains no Ethernet stanza at all. interfaces-config.sh:70 then runs systemctl restart networking, which performs ifdown -a followed by ifup -a. ifupdown2 tears down every front panel port it had adopted for ZTP in-band DHCP, and ifup -a does not bring them back because they are no longer listed. The netdev is left administratively down with its MTU reset to the kernel default of 1500, while CONFIG_DB, APPL_DB and the ASIC all continue to report the port as up.

portmgrd only acts on CONFIG_DB events and keeps no desired state (portmgr.cpp:336 erases the event once applied; the 1 s tick in portmgrd.cpp:58 calls Consumer::drain(), which returns immediately when m_toSync is empty). It therefore never notices nor corrects the drift, and the port stays broken until an operator runs config interface shutdown/startup — which works only because that writes CONFIG_DB and thereby produces the event portmgrd needs.

Because lldpd sends and receives on the Linux netdev, LLDP stops working on the affected port while show interfaces status keeps reporting up/up, since that command reads APPL_DB. Observed on a customer AS4630-54PE:

dump/lldp.statistics          before bounce      after bounce
Ethernet50                    TX 0   / RX 0      TX 23 / RX 72
Ethernet49                    TX 61  / RX 152    TX 96 / RX 245

dump/ip.link  before:  62: Ethernet50: <BROADCAST,MULTICAST>             mtu 1500  state DOWN
              after :  62: Ethernet50: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100  state UP

How many ports are hit depends only on which ones happened to be oper up when interfaces-config.sh:34 sampled APPL_DB with redis-dump, so the failure looks intermittent. Two real cases from the same customer unit:

boot A   E49 and E50 came up 0.26 s apart -> both adopted -> BOTH broken
         lldpmgrd logged "is not up, continue" 140 times for each port,
         and lldpd recorded zero MSAP events for the whole boot
boot B   E50 up at 21:09:12, E49 at 21:09:29 -> only E50 adopted -> only E50 broken

Note that waiting for all ports to come up before the snapshot would make this worse, not better: it would turn an intermittent single port failure into a deterministic all ports failure.

How I did it

Re-assert the front panel port configuration after the interfaces-config restart.

Writing any field of CONFIG_DB PORT|<port> back with its current value produces a keyspace event, and SubscriberStateTable::pops() delivers the entire hash to the consumer:

// common/subscriberstatetable.cpp
if (!m_table.get(key, kfvFieldsValues(kco)))   // whole hash, not just the changed field

so portmgrd re-applies both mtu and admin_status even though only admin_status was written.

Verified on an AS4630-54PE. A single no-op HSET of admin_status produced:

portmgrd log count for Ethernet50:  0 -> 2
  portmgrd: doTask: Configure Ethernet50 MTU to 9100
  portmgrd: doTask: Configure Ethernet50 admin status to up

and ip -o link show Ethernet50 was byte for byte identical before and after, confirming the operation is idempotent and does not flap a port that is already up. The keyspace event fires even though the value is unchanged.

interfaces-config.service and networking.service are both Type=oneshot and neither restart uses --no-block, so ifdown/ifup have completed by the time the loop runs.

How to verify it

On a unit doing inband ZTP, after ZTP reaches "ZTP successfully completed":

ip -o link show | grep Ethernet | grep -c 'mtu 1500'   # expect 0
show lldp table                                        # expect all connected uplinks present
grep 'Re-asserting front panel port configuration' /var/log/syslog
grep 'portmgrd.*Configure Ethernet.* admin status' /var/log/syslog

Before this change at least one front panel port is left state DOWN mtu 1500 and missing from show lldp table, recoverable only by a manual config interface shutdown/startup.

Notes and limitations

• This is a remediation, not a root fix. The port is still torn down for roughly two seconds before being restored; the change stops that from becoming permanent. Preventing the teardown itself would require keeping front panel ports out of ifupdown2's scope, which is a much larger change.
• The loop issues one KEYS plus two sonic-db-cli calls per port. On a 54 port unit that is around 110 process spawns. ZTP teardown is not time critical, but a reviewer may prefer batching this into a single call. I kept the plain loop because that is the exact form I was able to verify on hardware.
• The config-fallback sub-path already performs config reload, which re-applies everything, so the loop is redundant but harmless there.
• No automated test coverage: tests/test_ztp_engine.py does not exercise ztp-profile.sh.

Related

• SONIC-14580 — customer visible LLDP symptom (Zendesk 50820), full analysis in the ticket comments