TC_FAN_3_2.py (test_TC_FAN_3_2): step 6's exact FanMode report-count assertion is inherently timing-fragile, not tolerant of legitimate report coalescing

[Luligu]

Description

test_TC_FAN_3_2 in src/python_testing/TC_FAN_3_2.py iteratively writes SpeedSetting from 1 to
SpeedMax with no delay between writes, subscribes to FanMode/PercentSetting/SpeedSetting/
SpeedCurrent/PercentCurrent, and then asserts that the number of FanMode attribute reports received is
exactly len(self.fan_modes) - 1 (i.e. exactly one report per Low/Medium/High-style transition). This
assertion has no tolerance for the Matter report engine legitimately coalescing several rapid, sub-reporting-
interval attribute changes into a single report of the latest value — behavior explicitly allowed by the
spec (a subscriber is only guaranteed eventual consistency, not delivery of every transient value), and
already tolerated elsewhere in this same test file: TC_FAN_3_1.py's equivalent check
(verify_number_of_fan_mode_reports) only asserts that the FanMode report count equals the
PercentSetting report count — a relative comparison that holds regardless of how much coalescing occurs,
since both attributes change together in the same transaction — rather than asserting an absolute number.

Steps to reproduce

1. Run TC_FAN_3_2.py against a DUT that supports the MultiSpeed feature and processes each SpeedSetting
   write (and its cascaded FanMode/PercentSetting updates) fast enough that consecutive writes land within
   the same reporting interval.
2. Step 6 (testing_scenario_update_speed_setting) writes SpeedSetting 1..SpeedMax back-to-back with no
   delay.
3. verify_attribute_progression fails:

   AssertionError: [FC] FanMode attribute report count (2) does not match expected count (3)
   

   (exact counts vary by run/order; the DUT's actual FanMode value sequence is correct throughout — every
   intermediate SpeedSetting write round-trips exactly and FanMode/PercentSetting settle correctly at
   each step when checked directly against the device — only the subscription report count is fewer than
   expected.)

Root cause

test_TC_FAN_3_2 (TC_FAN_3_2.py, verify_attribute_progression) hardcodes an absolute expected count:

if len(speed_setting_sub.attribute_queue.queue) >= len(fan_mode_sub.attribute_queue.queue):
    expected_fan_mode_qty = len(self.fan_modes) - 1
    asserts.assert_equal(len(fan_mode_sub.attribute_queue.queue), expected_fan_mode_qty,
                         f"[FC] FanMode attribute report count ({len(fan_mode_sub.attribute_queue.queue)}) does not match expected count ({expected_fan_mode_qty})")

Confirmed this is a report-delivery artifact, not a value-correctness issue, by reproducing the identical
symptom independently of SpeedSetting/FanControl-specific code: writing PercentSetting directly (via a
throwaway script, bypassing all SpeedSetting-cascade logic) at the same rapid, no-delay pace against a
running DUT produces the same reduced report count for FanMode, using an AttributeSubscriptionHandler to
observe the raw report stream. A DUT that processes writes fast enough to trigger this coalescing — arguably
the better-performing DUT — fails this assertion purely because of test timing, not because any reported
value was ever wrong.

TC_FAN_3_1.py's own verify_number_of_fan_mode_reports avoids this exact problem for the same kind of
FanMode cascade (driven via PercentSetting there) by comparing two live subscription counts against each
other instead of against a hardcoded absolute number:

asserts.assert_equal(fan_mode_report_qty, percent_setting_report_qty,
                     "[FC] Number of FanMode reports doesn't match the number of PercentSetting reports")

Suggested fix

Relax test_TC_FAN_3_2's assertion to tolerate coalescing the same way TC_FAN_3_1.py already does — either:

• Compare FanMode report count against another live subscription count (e.g. assert it does not exceed
  SpeedSetting's report count, mirroring the existing PercentSetting-vs-SpeedSetting check just below
  it in the same function), or
• Change the hardcoded-count check from assert_equal to an upper-bound assert_less_equal against
  expected_fan_mode_qty — a real regression producing more reports than the theoretical maximum would
  still be caught, while legitimate coalescing producing fewer would not.

Environment

• connectedhomeip checkout: 1.6.0 branch (via luligu/matterbridge:chip-test Docker image)
• Test run against -- Running tests against Specification version 1.6
• Reproduced across multiple runs; exact report counts vary by run (coalescing depends on write/report
  timing), but the underlying assertion's intolerance of any coalescing is the same in every case. Directly
  verified the coalescing itself (not this test) by writing PercentSetting at the same pace with no
  SpeedSetting/FanControl-specific code involved at all, and observing the identical reduced report count.

[Apollon77]

The problem is (again) invalid assumptions around the delivery of subscribed values. The test does writes in a loop and assumes that the subscription is sending each value as an own report. As soon as the Subscription would have any retransmission or delay in it, or - in case of matter.js reports are collected for 50ms before sent out - then this will never have the same results.

We had other tests where same issues around "expectations of DataReport delivery" were included. So the test should wait for the data report produced by a write before it sends the next write in case of a success or such.

I propose to flag Cert-blocker

[raul-marquez-csa]

Addressed in PR
TC-FAN-3.2 - Fix false failures from coalesced attribute reports- #73629