[WIP] Replace static inventory tests with dynamic subtest-based validation#5614
Draft
OhadRevah wants to merge 6 commits into
Draft
[WIP] Replace static inventory tests with dynamic subtest-based validation#5614OhadRevah wants to merge 6 commits into
OhadRevah wants to merge 6 commits into
Conversation
Remove 4 tests that compare cluster components against hardcoded lists and fail when new items appear. These tests generate recurring maintenance overhead every release without catching real defects — new components are always intentional and the fix is always adding the entry to the list. Removed tests: - test_no_new_cnv_deployments_added (CNV-8289) - test_no_new_cnv_pods_added (CNV-7261) - test_no_new_cnv_crds (CNV-8508) - test_no_new_hco_related_objects (CNV-9843) Add pytest-subtests dependency for upcoming conversion of matrix-parametrized validation tests to dynamic subtest-based tests. assisted by: claude code claude-opus-4-6 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Ohad <orevah@redhat.com>
Replace matrix-parametrized validation tests with dynamic cluster discovery using pytest-subtests. Tests now discover all CNV components from the cluster at runtime and validate each one as a subtest. - Add discovered_cnv_deployments, discovered_cnv_pods, discovered_cnv_daemonsets session-scoped fixtures that query the cluster and validate base components exist - Convert deployment tests (priority class, container image) to subtests - Convert pod tests (priority class, resource request, container image) to subtests - Convert SCC annotation tests to use dynamic discovery - Convert SNO deployment test to subtests - Convert relationship labels tests to subtests - Convert CRD must-gather test to subtests - Convert HCO related objects reconciliation test to subtests - Remove unused matrix fixtures (cnv_deployment_by_name, cnv_pods_by_type, cnv_deployments_excluding_hpp_pool, kubevirt_crd_by_type, etc.) assisted by: claude code claude-opus-4-6 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Ohad <orevah@redhat.com>
Remove cnv_deployment_matrix, cnv_pod_matrix, cnv_daemonset_matrix, cnv_crd_matrix, cnv_related_object_matrix from global_config.py and global_config_sno_hpp.py. Remove unused matrix fixtures (cnv_daemonset_by_name, cnv_pods_by_type, related_object_from_hco_status) from IUO conftest. Convert SNO daemonset test to subtests. assisted by: claude code claude-opus-4-6 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Ohad <orevah@redhat.com>
Add GRADUATED_KUBEVIRT_FEATUREGATES_CNV_5 and GRADUATED_HCO_FEATUREGATES_CNV_5 sets containing feature gates that graduated to GA or were removed in CNV 5.0. Update the expected_value fixture to subtract graduated gates when running on CNV 5.0+ clusters, so tests pass on both 4.22 and 5.0: - KubeVirt: ExpandDisks, VMExport, LiveUpdateNADRef, WithHostModelCPU - HCO: disableMDevConfiguration assisted by: claude code claude-opus-4-6 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Ohad <orevah@redhat.com>
Contributor
|
Important Review skippedIgnore keyword(s) in the title. ⛔ Ignored keywords (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
uv.lock is managed separately, revert to main. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Ohad <orevah@redhat.com>
openshift-virtualization-qe-bot
added
the
tox:verify-tc-requirement-polarion:running
Dependency additions are managed separately. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Ohad <orevah@redhat.com>
openshift-virtualization-qe-bot
added
tox:verify-tc-requirement-polarion:passed
Contributor
Author
|
/build-and-push-container |
openshift-virtualization-qe-bot
added
tox:verify-tc-requirement-polarion:running
|
New container for quay.io/openshift-cnv/openshift-virtualization-tests:pr-5614 published |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Static constants (ALL_CNV_DEPLOYMENTS, ALL_CNV_PODS, etc.) drove
both "no new X" change-detection tests and parametrized validation tests.
Every new CNV component required manually updating these lists which causing
false CI failures (10 in a CNV 5.0 run, zero real bugs) and leaving new
components unvalidated until constants were updated.
In order to solve this problem and make these tests more robust, these things done in this PR:
deployments/pods/daemonsets, validate base components exist, and log extras
from static matrix parametrization to dynamic subtests over discovered resources
New components are now automatically validated without constants updates.
Missing base components trigger assertion failures.
Which issue(s) this PR fixes:
Fixes false failures from rootcoz analysis bde47d45-b3ac-4640-b273-1d0855226c13
(test-pytest-cnv-5.0-iuo-ocs-dualstream #1): test_no_new_cnv_deployments_added,
test_no_new_cnv_pods_added, test_no_new_cnv_crds, test_no_new_hco_related_objects,
test_default_featuregates_by_resource, test_managed_cr_featuregate_reconcile
Special notes for reviewer:
jira-ticket: