Allow mandatory tests to be sideloaded - #346
Merged
antonio-amjr merged 2 commits intoAug 6, 2026
Merged
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request refactors the SDK Python test parsing logic by replacing the boolean mandatory flag with a new CollectionType enum, allowing for more granular control over test suite selection. Feedback on the changes points out that the CollectionType enum values are defined inconsistently, using single-element tuples for some members and a plain integer for another, and suggests standardizing them as simple integers.
antonio-amjr
approved these changes
Aug 5, 2026
Contributor
|
LGTM @greens, Just need to point out one correction: please change the base of the PR to the target TH version. We usually just pull done releases to the main AFAIK. |
|
Tick the box to add this pull request to the merge queue (same as
|
antonio-amjr
merged commit Aug 6, 2026
030e171
into
project-chip:v2.15.1-develop
7 of 9 checks passed
rquidute
added a commit
that referenced
this pull request
Aug 26, 2026
PR #346 ("Allow mandatory tests to be sideloaded") replaced the mandatory: bool flag in __parse_python_tests() with a 3-valued CollectionType enum, but the LEGACY-suite fallback branch lost its python_test_type != MANDATORY guard in the process: elif collection_type != CollectionType.MANDATORY: suites[SuiteType.LEGACY].add_test_case(test_case) This only checks the target collection, not the test's own type, so when building the NON_MANDATORY ("SDK Python Tests") collection, any test case whose type is MANDATORY falls through the COMMISSIONING/NO_COMMISSIONING branches and lands in the LEGACY ("Old script format") suite anyway. The same test case is also correctly added to SuiteType.MANDATORY when the MANDATORY collection is built, so it ends up listed - and executed - in both collections. Restore the type guard so mandatory tests are excluded from the LEGACY suite when populating the NON_MANDATORY collection, while still landing in SuiteType.MANDATORY for CollectionType.ALL (sideload) per PR #346's intent. Adds a dedicated fixture file (mandatory_tests_info.json) with a mandatory-shaped test case, so the new regression test doesn't perturb the existing python_tests_info.json fixture shared with test_python_parser.py's hardcoded test-count assertion. Fixes project-chip/certification-tool#1087
rquidute
added a commit
that referenced
this pull request
Aug 26, 2026
) PR #346 ("Allow mandatory tests to be sideloaded") replaced the mandatory: bool flag in __parse_python_tests() with a 3-valued CollectionType enum, but the LEGACY-suite fallback branch lost its python_test_type != MANDATORY guard in the process: elif collection_type != CollectionType.MANDATORY: suites[SuiteType.LEGACY].add_test_case(test_case) This only checks the target collection, not the test's own type, so when building the NON_MANDATORY ("SDK Python Tests") collection, any test case whose type is MANDATORY falls through the COMMISSIONING/NO_COMMISSIONING branches and lands in the LEGACY ("Old script format") suite anyway. The same test case is also correctly added to SuiteType.MANDATORY when the MANDATORY collection is built, so it ends up listed - and executed - in both collections. Restore the type guard so mandatory tests are excluded from the LEGACY suite when populating the NON_MANDATORY collection, while still landing in SuiteType.MANDATORY for CollectionType.ALL (sideload) per PR #346's intent. Adds a dedicated fixture file (mandatory_tests_info.json) with a mandatory-shaped test case, so the new regression test doesn't perturb the existing python_tests_info.json fixture shared with test_python_parser.py's hardcoded test-count assertion. Fixes project-chip/certification-tool#1087
rquidute
added a commit
that referenced
this pull request
Aug 28, 2026
…fix (#360) * Allow mandatory tests to be sideloaded (#346) Cherry-picked from v2.15.1-develop. There was/is a small quirk in the logic that would prevent mandatory tests from the list in python_test_parser.py from being added when custom tests were collated. * Fix mandatory Python tests duplicated into Old Script Format suite (#358) Cherry-picked from v2.15.1-develop. PR #346 ("Allow mandatory tests to be sideloaded", cherry-picked in the prior commit) replaced the mandatory: bool flag in __parse_python_tests() with a 3-valued CollectionType enum, but the LEGACY-suite fallback branch lost its python_test_type != MANDATORY guard in the process: elif collection_type != CollectionType.MANDATORY: suites[SuiteType.LEGACY].add_test_case(test_case) This only checks the target collection, not the test's own type, so when building the NON_MANDATORY ("SDK Python Tests") collection, any test case whose type is MANDATORY falls through the COMMISSIONING/NO_COMMISSIONING branches and lands in the LEGACY ("Old script format") suite anyway. The same test case is also correctly added to SuiteType.MANDATORY when the MANDATORY collection is built, so it ends up listed - and executed - in both collections. Restore the type guard so mandatory tests are excluded from the LEGACY suite when populating the NON_MANDATORY collection, while still landing in SuiteType.MANDATORY for CollectionType.ALL (sideload) per PR #346's intent. Fixes project-chip/certification-tool#1087
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes project-chip/certification-tool#1073
There was/is a small quirk in the logic that would prevent mandatory tests from this list: https://github.com/project-chip/certification-tool-backend/blob/main/test_collections/matter/sdk_tests/support/python_testing/models/python_test_parser.py#L36 from being added when custom tests were collated.
This is the lightest-touch fix. I think possibly there could be a total re-evaluation of the mandatory field that shows up in so many places, but for now this will fix the issue.