Generate coverage for Simulator based integration tests & ui tests in github actions by vishesh92 · Pull Request #7784 · apache/cloudstack

vishesh92 · 2023-07-27T11:50:02Z

Description

This PR generates coverage for Simulator based integration tests in github actions workflow and uploads the report to codecov.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

codecov · 2023-07-27T12:26:31Z

Codecov Report

Merging #7784 (fbf5dfc) into main (8db8aa4) will increase coverage by 15.85%.
Report is 61 commits behind head on main.
The diff coverage is n/a.

❗ Current head fbf5dfc differs from pull request most recent head c944865. Consider uploading reports for the commit c944865 to get more accurate results

@@              Coverage Diff              @@
##               main    #7784       +/-   ##
=============================================
+ Coverage     13.42%   29.27%   +15.85%     
- Complexity     9383    30265    +20882     
=============================================
  Files          2747     5098     +2351     
  Lines        258827   357598    +98771     
  Branches      40318    52212    +11894     
=============================================
+ Hits          34748   104703    +69955     
- Misses       219694   238360    +18666     
- Partials       4385    14535    +10150

Flag	Coverage Δ
simulator-marvin-tests	`25.29% <ø> (?)`
uitests	`4.85% <ø> (?)`
unit-tests	`14.38% <ø> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

see 3628 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

vishesh92 · 2023-07-28T09:17:20Z

I found these tests that are not included in the ci.yml's test matrix.

broken/maint
component/maint
component/test_acquire_specified_public_ip
component/test_advancedsg_networks
component/test_affinity_groups
component/test_base_image_updation
component/test_blocker_bugs
component/test_browse_templates
component/test_browse_templates2
component/test_browse_volumes
component/test_edgezone_supportedoperations
component/test_escalations_instances
component/test_haproxy
component/test_hypervisor_capabilities
component/test_implicit_planner
component/test_interop_xd_ccp
component/test_interpod_migration
component/test_ldap
component/test_migrate_vol_to_maintained_pool
component/test_multiple_nic_support
component/test_multiple_physical_network_creation
component/test_multiple_subnets_in_isolated_network
component/test_multiple_subnets_in_isolated_network_rvr
component/test_multiple_subnets_in_vpc
component/test_multiple_subnets_in_vpc_rvr
component/test_network_mtu
component/test_organization_states
component/test_portable_ip
component/test_ps_domain_limits
component/test_ps_limits
component/test_ps_project_limits
component/test_redundant_router_cleanups
component/test_redundant_router_services
component/test_redundant_router_upgrades
component/test_resource_count_running_vms
component/test_shared_networks
component/test_storage_motion
component/test_template_access_across_domains
component/test_volume_destroy_recover
component/test_vpc_network_internal_lbrules
component/test_vpc_vms_deployment
plugins/ldap
plugins/linstor
plugins/scaleio
plugins/test_quota
smoke/test_gateway_on_shared_networks
smoke/test_guest_os
smoke/test_host_control_state
smoke/test_human_readable_logs
smoke/test_list_ids_parameter#
smoke/test_nonstrict_affinity_group
smoke/test_vm_autoscaling
testpaths/testpath_custom_disk_offering
testpaths/testpath_netmask
testpaths/testpath_queryAsyncJobResult
testpaths/testpath_stopped_vm
testpaths/testpath_usage
testpaths/testpath_uuid_event

DaanHoogland · 2023-07-28T09:27:58Z

I found these tests that are not included in the ci.yml's test matrix.

I would not include any effort on those in this PR @vishesh92 . Best would be to create a tracking issue to decide which ones to delete/fix/add.

borisstoyanov

LGTM

DaanHoogland

clgtm

DaanHoogland · 2023-08-02T07:29:33Z

@blueorangutan package

blueorangutan · 2023-08-02T07:30:03Z

@DaanHoogland a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-08-02T08:39:05Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6626

yadvr · 2023-08-03T05:31:26Z

@vishesh92 good work and proposed PR, we may need a different way to track this perhaps? All tests in smoke should be ideally run against simulator/ci.yml, but other tests arne't included as they fail or amount of effort to stablise them would be huge.

smoke/test_gateway_on_shared_networks
smoke/test_guest_os
smoke/test_host_control_state
smoke/test_human_readable_logs
smoke/test_list_ids_parameter#
smoke/test_nonstrict_affinity_group
smoke/test_vm_autoscaling

yadvr · 2023-08-03T05:32:54Z

      - name: Start CloudStack Management Server with Simulator
        run: |
-          export MAVEN_OPTS="-Xmx4096m -XX:MaxPermSize=800m -Djava.security.egd=file:/dev/urandom"
+          export MAVEN_OPTS="-Xmx4096m -XX:MaxPermSize=800m -Djava.security.egd=file:/dev/urandom  -javaagent:jacoco/lib/jacocoagent.jar=address=*,port=36320,output=tcpserver"


perhaps we can experiment this with our smoketests against actual environment and gather/publish the jacoco file separately cc @vishesh92 @borisstoyanov @vladimirpetrov @kiranchavala @DaanHoogland - so we've a unit test coverage stats and functional/marvin test coverage stats (discuss - if you think, combined is also okay?)

@rohityadavcloud We can get the coverage from Blue Orangutan jobs. I can do that as well, but I don't have much idea about the setup as of now. So, I will need some time to look into it.

I would certainly like to have them separately, but nobody dies if we don't.

yadvr · 2023-08-14T11:04:41Z

@vishesh92 Does this handle reports from multiple jobs and aggregate them somewhoe; could you look into (when you're free/have time): https://about.codecov.io/blog/uploading-code-coverage-in-a-separate-job-on-github-actions/

vishesh92 · 2023-08-14T11:17:35Z

@rohityadavcloud As per my understanding, codecov groups the coverage by flags.

See uploaded files for this PR here: https://app.codecov.io/gh/apache/cloudstack/commit/c944865bd84a1fab86755bc164123c64291adce1

DaanHoogland · 2023-08-16T09:04:15Z

@blueorangutan package

blueorangutan · 2023-08-16T09:06:04Z

@DaanHoogland a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-08-16T10:08:18Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6797

yadvr

LGTM - changes are in the github actions

boring-cyborg Bot added the component:integration label Jul 27, 2023

vishesh92 changed the title ~~Generate coverage for Simulator based integration tests~~ Generate coverage for Simulator based integration tests in github actions Jul 27, 2023

Generate coverage for Simulator based integration tests

943ae16

vishesh92 force-pushed the ci-coverage branch from 7820321 to 943ae16 Compare July 27, 2023 11:53

vishesh92 requested a review from borisstoyanov July 27, 2023 12:40

vishesh92 added 2 commits July 27, 2023 23:07

Add flag for unit tests

8ebe3b1

Add coverage for uitests

c944865

boring-cyborg Bot added the component:UI label Jul 27, 2023

vishesh92 closed this Jul 27, 2023

vishesh92 reopened this Jul 27, 2023

vishesh92 closed this Jul 27, 2023

vishesh92 reopened this Jul 27, 2023

vishesh92 requested a review from DaanHoogland July 28, 2023 05:40

vishesh92 changed the title ~~Generate coverage for Simulator based integration tests in github actions~~ Generate coverage for Simulator based integration tests & ui tests in github actions Jul 28, 2023

vishesh92 added the component:ci-cd label Jul 28, 2023

vishesh92 marked this pull request as ready for review July 28, 2023 08:17

DaanHoogland closed this Jul 28, 2023

DaanHoogland reopened this Jul 28, 2023

borisstoyanov approved these changes Aug 1, 2023

View reviewed changes

DaanHoogland approved these changes Aug 2, 2023

View reviewed changes

yadvr reviewed Aug 3, 2023

View reviewed changes

vishesh92 force-pushed the ci-coverage branch from fbf5dfc to c944865 Compare August 16, 2023 07:02

DaanHoogland closed this Aug 16, 2023

DaanHoogland reopened this Aug 16, 2023

yadvr added this to the 4.19.0.0 milestone Sep 11, 2023

yadvr approved these changes Sep 11, 2023

View reviewed changes

yadvr merged commit 3ec3007 into apache:main Sep 11, 2023

yadvr deleted the ci-coverage branch September 11, 2023 08:44

Uh oh!

Conversation

vishesh92 commented Jul 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

codecov Bot commented Jul 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

vishesh92 commented Jul 28, 2023

Uh oh!

DaanHoogland commented Jul 28, 2023

Uh oh!

borisstoyanov left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland commented Aug 2, 2023

Uh oh!

blueorangutan commented Aug 2, 2023

Uh oh!

blueorangutan commented Aug 2, 2023

Uh oh!

yadvr commented Aug 3, 2023

Uh oh!

yadvr Aug 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vishesh92 Aug 3, 2023

Choose a reason for hiding this comment

Uh oh!

DaanHoogland Aug 3, 2023

Choose a reason for hiding this comment

Uh oh!

yadvr commented Aug 14, 2023

Uh oh!

vishesh92 commented Aug 14, 2023

Uh oh!

DaanHoogland commented Aug 16, 2023

Uh oh!

blueorangutan commented Aug 16, 2023

Uh oh!

blueorangutan commented Aug 16, 2023

Uh oh!

yadvr left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

vishesh92 commented Jul 27, 2023 •

edited

Loading

codecov Bot commented Jul 27, 2023 •

edited

Loading

yadvr Aug 3, 2023 •

edited

Loading