Add lock mechanism considering template id, pool id, host id in PowerFlex Storage by harikrishna-patnala · Pull Request #8233 · apache/cloudstack

harikrishna-patnala · 2023-11-14T12:06:43Z

Description

Observed a failure to start new virtual machine with PowerFlex storage. Traced it to concurrent VM starts using the same template and the same host to copy. Second mapping attempt failed.

While creating the volume clone from the seeded template in primary storage, adding a lock with the string containing IDs of template, storage pool and destination host avoids the situation of concurrent mapping attempts with the same host.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

harikrishna-patnala · 2023-11-14T12:07:15Z

@blueorangutan package

DaanHoogland

clgtm

DaanHoogland · 2023-11-17T08:34:43Z

@harikrishna-patnala can you look at the locking used?

harikrishna-patnala · 2023-11-20T03:19:33Z

@blueorangutan package

codecov · 2023-11-20T04:07:47Z

Codecov Report

Attention: 17 lines in your changes are missing coverage. Please review.

Comparison is base (1b56a8e) 13.10% compared to head (e2fc4ff) 13.10%.

Files	Patch %	Lines
...e/cloudstack/storage/volume/VolumeServiceImpl.java	0.00%	17 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.18    #8233      +/-   ##
============================================
- Coverage     13.10%   13.10%   -0.01%     
  Complexity     9122     9122              
============================================
  Files          2720     2720              
  Lines        257638   257653      +15     
  Branches      40168    40171       +3     
============================================
  Hits          33753    33753              
- Misses       219619   219634      +15     
  Partials       4266     4266

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

harikrishna-patnala · 2023-11-20T05:59:56Z

@blueorangutan package

blueorangutan · 2023-11-20T06:02:03Z

@harikrishna-patnala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-11-20T07:01:46Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 7800

yadvr · 2023-11-20T07:38:31Z

@blueorangutan test

blueorangutan · 2023-11-20T07:40:04Z

@rohityadavcloud a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2023-11-20T19:01:08Z

[SF] Trillian test result (tid-8374)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 39367 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8233-t8374-kvm-centos7.zip
Smoke tests completed. 108 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_list_cpvm_vm	`Failure`	0.04	test_ssvm.py
test_04_cpvm_internals	`Failure`	0.04	test_ssvm.py

harikrishna-patnala · 2023-12-05T08:39:43Z

@blueorangutan package

blueorangutan · 2023-12-05T08:40:03Z

@harikrishna-patnala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-12-05T09:40:16Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 7923

borisstoyanov · 2023-12-06T08:19:05Z

@blueorangutan test

yadvr · 2023-12-06T08:23:03Z

@blueorangutan test

blueorangutan · 2023-12-06T08:30:05Z

@rohityadavcloud a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

borisstoyanov

LGTM, lets wait for latest test and I guess we could merge this

blueorangutan · 2023-12-06T21:18:42Z

[SF] Trillian test result (tid-8480)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 44561 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8233-t8480-kvm-centos7.zip
Smoke tests completed. 107 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_08_arping_in_ssvm	`Failure`	5.19	test_diagnostics.py
test_01_redundant_vpc_site2site_vpn	`Failure`	382.21	test_vpc_vpn.py

yadvr · 2023-12-07T07:45:51Z

@blueorangutan test

blueorangutan · 2023-12-07T07:48:03Z

@rohityadavcloud a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

DaanHoogland · 2023-12-07T08:20:17Z

@blueorangutan test

I don't think re-running the tests without rebuild will add value here.

@JoaoJandre are your conserns met?

blueorangutan · 2023-12-07T08:22:04Z

@DaanHoogland [SL] unsupported parameters provided. Supported mgmt server os are: centos7, centos6, suse15, alma8, ubuntu18, ubuntu22, ubuntu20, rocky8, alma9. Supported hypervisors are: kvm-centos6, kvm-centos7, kvm-rocky8, kvm-alma8, kvm-alma9, kvm-ubuntu18, kvm-ubuntu20, kvm-ubuntu22, kvm-suse15, vmware-55u3, vmware-60u2, vmware-65u2, vmware-67u3, vmware-70u1, vmware-70u2, vmware-70u3, vmware-80, vmware-80u1, xenserver-65sp1, xenserver-71, xenserver-74, xcpng74, xcpng76, xcpng80, xcpng81, xcpng82

harikrishna-patnala · 2023-12-07T10:06:26Z

@DaanHoogland , just to answer for question from my side, @JoaoJandre mentioned about missing lock() method call, which I've added already.

JoaoJandre

CLGTM, didn't test it.

blueorangutan · 2023-12-07T20:11:13Z

[SF] Trillian test result (tid-8496)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 43093 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8233-t8496-kvm-centos7.zip
Smoke tests completed. 107 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_08_migrate_vm	`Error`	45.92	test_vm_life_cycle.py
test_hostha_enable_ha_when_host_in_maintenance	`Error`	302.80	test_hostha_kvm.py

yadvr · 2023-12-08T07:51:00Z

Last two smoketests rounds had different intermittent failures, LGTM, merging this.

…Flex Storage (apache#8233) Observed a failure to start new virtual machine with PowerFlex storage. Traced it to concurrent VM starts using the same template and the same host to copy. Second mapping attempt failed. While creating the volume clone from the seeded template in primary storage, adding a lock with the string containing IDs of template, storage pool and destination host avoids the situation of concurrent mapping attempts with the same host.

boring-cyborg Bot added the component:storage label Nov 14, 2023

harikrishna-patnala requested a review from DaanHoogland November 14, 2023 12:07

harikrishna-patnala changed the title ~~Add lock mechanism considering template id, pool id, host id~~ Add lock mechanism considering template id, pool id, host id in PowerFlex Storage Nov 14, 2023

harikrishna-patnala added the component:powerflex label Nov 14, 2023

DaanHoogland approved these changes Nov 14, 2023

View reviewed changes

JoaoJandre reviewed Nov 16, 2023

View reviewed changes

Comment thread engine/storage/volume/src/main/java/org/apache/cloudstack/storage/volume/VolumeServiceImpl.java

harikrishna-patnala added 2 commits November 20, 2023 08:48

Add lock mechanism considering template id, pool id, host id

cae911a

Added missing lock

e2fc4ff

harikrishna-patnala force-pushed the fixPowerflexSDCfailureWhenMultipleTemplateCopies branch from 0d4d51a to e2fc4ff Compare November 20, 2023 03:19

yadvr added this to the 4.18.2.0 milestone Nov 20, 2023

yadvr approved these changes Nov 20, 2023

View reviewed changes

borisstoyanov approved these changes Dec 6, 2023

View reviewed changes

JoaoJandre approved these changes Dec 7, 2023

View reviewed changes

yadvr merged commit 7eb3636 into apache:4.18 Dec 8, 2023

yadvr deleted the fixPowerflexSDCfailureWhenMultipleTemplateCopies branch December 8, 2023 07:51

Uh oh!

Conversation

harikrishna-patnala commented Nov 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Uh oh!

harikrishna-patnala commented Nov 14, 2023

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

DaanHoogland commented Nov 17, 2023

Uh oh!

harikrishna-patnala commented Nov 20, 2023

Uh oh!

codecov Bot commented Nov 20, 2023

Codecov Report

Uh oh!

harikrishna-patnala commented Nov 20, 2023

Uh oh!

blueorangutan commented Nov 20, 2023

Uh oh!

blueorangutan commented Nov 20, 2023

Uh oh!

yadvr commented Nov 20, 2023

Uh oh!

blueorangutan commented Nov 20, 2023

Uh oh!

blueorangutan commented Nov 20, 2023

Uh oh!

harikrishna-patnala commented Dec 5, 2023

Uh oh!

blueorangutan commented Dec 5, 2023

Uh oh!

blueorangutan commented Dec 5, 2023

Uh oh!

borisstoyanov commented Dec 6, 2023

Uh oh!

yadvr commented Dec 6, 2023

Uh oh!

blueorangutan commented Dec 6, 2023

Uh oh!

borisstoyanov left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Dec 6, 2023

Uh oh!

yadvr commented Dec 7, 2023

Uh oh!

blueorangutan commented Dec 7, 2023

Uh oh!

DaanHoogland commented Dec 7, 2023

Uh oh!

blueorangutan commented Dec 7, 2023

Uh oh!

harikrishna-patnala commented Dec 7, 2023

Uh oh!

JoaoJandre left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Dec 7, 2023

Uh oh!

yadvr commented Dec 8, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

harikrishna-patnala commented Nov 14, 2023 •

edited

Loading