[flexcounterorch] Port counters silently unavailable indefinitely after warm/fast-reboot when a port never becomes ready

[AnantKishorSharma]

Description

After a warm/fast-reboot performed under a scaled configuration, show interfaces counters can return no output — port counters are missing — and the condition persists for an extended period, even though:

• the data plane keeps forwarding (traffic unaffected),
• BGP sessions and routes recover normally,
• there is no log explaining why counters are missing.

This is purely a loss of observability (counters/monitoring); forwarding is fine, which makes it easy to miss and hard to diagnose.

Steps to reproduce

1. Bring up a scaled control plane (e.g. dozens of SVI-based eBGP sessions, capability extended-nexthop for IPv6 routes over IPv4 sessions, and several thousand IPv4 + IPv6 routes) so orchagent has substantial work to reconcile.
2. Trigger fast-reboot. Under scale the pre-shutdown RESTARTCHECK may fail; if the reboot is then forced (fast-reboot -f), the warm-boot snapshot can be left inconsistent.
3. After the device comes back, run show interfaces counters.

Observed: empty output, persisting long after boot. Data plane unaffected; BGP recovers.
Expected: port counters available (or cleanly reset but functional) shortly after reboot.

Root cause

Port flex counters are only enabled once FlexCounterOrch::doTask() passes its readiness gate:

// orchagent/flexcounterorch.cpp  (FlexCounterOrch::doTask)
if (!m_delayTimerExpired)        // 60s FLEX_COUNTER_DELAY on warm/fast boot
    return;
...
if (gPortsOrch && !gPortsOrch->allPortsReady())   // <-- silent, no timeout
    return;

On warm/fast-reboot, flex-counter processing is deliberately deferred so it does not slow data-plane restoration (introduced by #3326 and #3562 — a legitimate optimization). PortsOrch::allPortsReady() is:

return m_initDone && m_pendingPortSet.empty();

and a port remains in m_pendingPortSet until its buffer configuration is applied (BufferOrch::isPortReady).

The defect is that this wait has no timeout, is silent, and is all-or-nothing:

• If even one port is slow to become ready (e.g. buffer reconciliation does not complete promptly after an inconsistent forced warm-boot), allPortsReady() stays false until it does
• FlexCounterOrch::doTask() then returns on every invocation with no log, so port flex-counter polling is not enabled for ANY port until that port becomes ready.
• Net effect: all port counters are missing, with no diagnostic indicating which port (or what) is blocking.

The deferral works as designed — the bug is not the deferral itself, but that it has no timeout, no observability, and no per-port isolation, so a single slow port can keep all port counters missing for an extended period with no indication why.

This gate logic is identical on 202505, 202511, and master, so all three are affected.

Impact

• Loss of port-counter visibility for an extended period after warm/fast-reboot (monitoring, ECMP/load-balancing validation, traffic verification), while the device otherwise looks healthy.
• No log message points at the cause, making the issue very hard to root-cause in the field.

Proposed fix

Keep the deferral (preserve the fast-boot optimization) but make it observable and bounded:

1. Once the warm/fast-reboot delay timer has expired and ports are still not ready, emit a throttled warning naming the ports still pending (or noting that PortInitDone has not been received). (PR linked below.)
2. Follow-ups worth discussing: enable counters per-ready-port so one stuck port does not suppress all port counters; and ensure a port cannot remain permanently in m_pendingPortSet after a forced warm-boot.