Cherry-pick #346 + #358: mandatory test sideloading + duplicate-test fix - #360
Cherry-pick #346 + #358: mandatory test sideloading + duplicate-test fix#360rquidute wants to merge 2 commits into
Conversation
Cherry-picked from v2.15.1-develop. There was/is a small quirk in the logic that would prevent mandatory tests from the list in python_test_parser.py from being added when custom tests were collated.
) Cherry-picked from v2.15.1-develop. PR #346 ("Allow mandatory tests to be sideloaded", cherry-picked in the prior commit) replaced the mandatory: bool flag in __parse_python_tests() with a 3-valued CollectionType enum, but the LEGACY-suite fallback branch lost its python_test_type != MANDATORY guard in the process: elif collection_type != CollectionType.MANDATORY: suites[SuiteType.LEGACY].add_test_case(test_case) This only checks the target collection, not the test's own type, so when building the NON_MANDATORY ("SDK Python Tests") collection, any test case whose type is MANDATORY falls through the COMMISSIONING/NO_COMMISSIONING branches and lands in the LEGACY ("Old script format") suite anyway. The same test case is also correctly added to SuiteType.MANDATORY when the MANDATORY collection is built, so it ends up listed - and executed - in both collections. Restore the type guard so mandatory tests are excluded from the LEGACY suite when populating the NON_MANDATORY collection, while still landing in SuiteType.MANDATORY for CollectionType.ALL (sideload) per PR #346's intent. Fixes project-chip/certification-tool#1087
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Tick the box to add this pull request to the merge queue (same as
|
1 participant
Summary
Cherry-picks two commits from
v2.15.1-developontov2.16-develop:CollectionTypeenum (NON_MANDATORY/MANDATORY/ALL) so mandatory tests can be sideloaded via custom test collections.python_test_type != MANDATORYguard, causing mandatory tests (e.g.TC-IDM-10.2,TC-IDM-10.3,TC-IDM-10.4,TC-IDM-10.5,TC-IDM-12.1) to be duplicated into bothMandatory SDK Python TestsandSDK Python Tests(Old Script Format), and executed twice via the TH CLI.v2.16-developnever had #346 merged, so it was unaffected by the #1087 bug on its own — this PR brings in both the original feature and its fix together so the branch ends up in the same corrected state asv2.15.1-develop, without ever passing through the broken intermediate state.Fixes project-chip/certification-tool#1087 (on this branch)
Changes
Both commits applied cleanly with no merge conflicts.
test_collections/matter/sdk_tests/support/python_testing/sdk_python_tests.py—CollectionTypeenum + corrected classification guardtest_collections/matter/sdk_tests/support/tests/python_tests/test_sdk_python_collection.py— regression testtest_collections/matter/sdk_tests/support/tests/python_tests/test_python_script/mandatory_tests_info.json— new fixture for the regression test (doesn't touch the sharedpython_tests_info.jsonfixture used by other tests)Testing
flake8, isort, and black all pass on the touched files. Docker isn't available in this environment to run the full pytest suite; please confirm CI passes.