api,server: purge expunged resources

[shwstppr]

Description

Feature spec: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Purge+Expunged+Resources

This PR introduces the functionality of purging removed DB entries for CloudStack entities (currently only for VirtualMachine). There would be three mechanisms for purging removed resources:

• Background task - CloudStack will run a background task which runs at a defined interval. Other parameters for this task can be controlled with new global settings.
• API - New admin-only API purgeExpungedResources. It will allow passing the following parameters - resourcetype, batchsize, startdate, enddate. Currently, API is not supported in the UI.
• Config for service offering. Service offerings can be created with purgeresources parameter which would allow purging resources immediately on expunge.

Following new global settings have been added:

• expunged.resources.purge.enabled: Default: false. Whether to run a background task to purge the expunged resources
• expunged.resources.purge.resources: Default: (empty). A comma-separated list of resource types that will be considered by the background task to purge the expunged resources. Currently only VirtualMachine is supported. An empty "value will result in considering all resource types for purging
• expunged.resources.purge.interval: Default: 86400. Interval (in seconds) for the background task to purge the expunged resources
• expunged.resources.purge.delay: Default: 300. Initial delay (in seconds) to start the background task to purge the expunged resources task.
• expunged.resources.purge.batch.size: Default: 50. Batch size to be used during expunged resources purging.
• expunged.resources.purge.start.time: Default: (empty). Start time to be used by the background task to purge the expunged resources. Use format yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.
• expunged.resources.purge.keep.past.days: Default: 30. The number of days in the past from the execution time of the background task to purge the expunged resources for which the expunged resources must not be purged. To enable purging expunged resource till the execution of the background task, set the value to zero.
• expunged.resource.purge.job.delay: Default: 180. Delay (in seconds) to execute the purging of an expunged resource initiated by the configuration in the offering. Minimum value should be 180 seconds and if a lower value is set then the minimum value will be used.

Documentation PR: https://github.com/apache/cloudstack-documentation/pull/397

Types of changes

• [ ] Breaking change (fix or feature that would cause existing functionality to change)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Bug fix (non-breaking change which fixes an issue)
• [ ] Enhancement (improves an existing feature and functionality)
• [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
• [ ] build/CI

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• [ ] Major
• [ ] Minor

Bug Severity

• [ ] BLOCKER
• [ ] Critical
• [ ] Major
• [ ] Minor
• [ ] Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

API usage:

> purge expungedresources startdate=2024-04-15 enddate=2024-04-20 resourcetype=VirtualMachine
{
  "purgeexpungedresourcesresponse": {
    "resourcecount": 6
  }
}

Verified new smoke test,

[root@pr8999-t6925-kvm-centos7-marvin marvin]# nosetests --with-xunit --xunit-file=results.xml --with-marvin --marvin-config=./pr8999-t6925-kvm-centos7-advanced-cfg -s -a tags=advanced --hypervisor=KVM tests/smoke/test_purge_expunged_vms.py
/usr/local/lib/python3.6/site-packages/paramiko/transport.py:32: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography. The next release of cryptography will remove support for Python 3.6.
  from cryptography.hazmat.backends import default_backend

==== Marvin Init Started ====

=== Marvin Parse Config Successful ===

=== Marvin Setting TestData Successful===

==== Log Folder Path: /marvin/MarvinLogs/May_01_2024_08_09_34_TNMKB9 All logs will be available here ====

=== Marvin Init Logging Successful===

==== Marvin Init Successful ====
=== TestName: test_01_purge_expunged_api_vm_start_date | Status : SUCCESS ===

=== TestName: test_02_purge_expunged_api_vm_end_date | Status : SUCCESS ===

=== TestName: test_03_purge_expunged_api_vm_start_end_date | Status : SUCCESS ===

=== TestName: test_04_purge_expunged_api_vm_no_date | Status : SUCCESS ===

=== TestName: test_05_purge_expunged_vm_service_offering | Status : SUCCESS ===

{'mgtSvrIp': '10.1.33.150', 'port': 8080, 'user': 'root', 'passwd': 'P@ssword123', 'hypervisor': 'kvm', 'useHttps': 'False', 'certCAPath': 'NA', 'certPath': 'NA', 'apiKey': 'YZDWI356FRnLh8vEHROd_ZUVD5U8JeGG-mObb7wPhyL5OQqi6df3KCmF5AlVWxPsZvsvNY2-z9BbyqSZGPaBCg', 'securityKey': 'U4LP9-ZdTuRl4_hJl3x2KN6nTNZCYAWhcMXmFKBSBYTNoUghjj3Pb5-0s2hN_tVG0l8oK6jDrmp953NOP59GAg'}
====Trying SSH Connection: Host:10.1.33.150 User:root                                   Port:22 RetryCnt:60===
===SSH to Host 10.1.33.150 port : 22 SUCCESSFUL===
{Cmd: service cloudstack-management stop via Host: 10.1.33.150} {returns: ['Redirecting to /bin/systemctl stop cloudstack-management.service']}
{Cmd: service cloudstack-management start via Host: 10.1.33.150} {returns: ['Redirecting to /bin/systemctl start cloudstack-management.service']}
====Trying SSH Connection: Host:10.1.35.104 User:root                                   Port:22 RetryCnt:60===
====Trying SSH Connection: Host:10.1.35.104 User:root                                   Port:22 RetryCnt:60===
===SSH to Host 10.1.35.104 port : 22 SUCCESSFUL===
===SSH to Host 10.1.35.104 port : 22 SUCCESSFUL===
{Cmd: service cloudstack-management stop via Host: 10.1.35.104} {returns: ['Redirecting to /bin/systemctl stop cloudstack-management.service']}
{Cmd: service cloudstack-management start via Host: 10.1.35.104} {returns: ['Redirecting to /bin/systemctl start cloudstack-management.service']}
{'mgtSvrIp': '10.1.33.150', 'port': 8080, 'user': 'root', 'passwd': 'P@ssword123', 'hypervisor': 'kvm', 'useHttps': 'False', 'certCAPath': 'NA', 'certPath': 'NA', 'apiKey': 'YZDWI356FRnLh8vEHROd_ZUVD5U8JeGG-mObb7wPhyL5OQqi6df3KCmF5AlVWxPsZvsvNY2-z9BbyqSZGPaBCg', 'securityKey': 'U4LP9-ZdTuRl4_hJl3x2KN6nTNZCYAWhcMXmFKBSBYTNoUghjj3Pb5-0s2hN_tVG0l8oK6jDrmp953NOP59GAg'}
====Trying SSH Connection: Host:10.1.33.150 User:root                                   Port:22 RetryCnt:60===
===SSH to Host 10.1.33.150 port : 22 SUCCESSFUL===
{Cmd: service cloudstack-management stop via Host: 10.1.33.150} {returns: ['Redirecting to /bin/systemctl stop cloudstack-management.service']}
{Cmd: service cloudstack-management start via Host: 10.1.33.150} {returns: ['Redirecting to /bin/systemctl start cloudstack-management.service']}
====Trying SSH Connection: Host:10.1.35.104 User:root                                   Port:22 RetryCnt:60===
====Trying SSH Connection: Host:10.1.35.104 User:root                                   Port:22 RetryCnt:60===
===SSH to Host 10.1.35.104 port : 22 SUCCESSFUL===
===SSH to Host 10.1.35.104 port : 22 SUCCESSFUL===
{Cmd: service cloudstack-management stop via Host: 10.1.35.104} {returns: ['Redirecting to /bin/systemctl stop cloudstack-management.service']}
{Cmd: service cloudstack-management start via Host: 10.1.35.104} {returns: ['Redirecting to /bin/systemctl start cloudstack-management.service']}
=== TestName: test_06_purge_expunged_vm_background_task | Status : SUCCESS ===

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 55.68581% with 378 lines in your changes missing coverage. Please review.

Project coverage is 15.42%. Comparing base (a9caee3) to head (93edde5). Report is 15 commits behind head on main.

Files	Patch %	Lines
...loudstack/resource/ResourceCleanupServiceImpl.java	50.51%	220 Missing and 20 partials :warning:
...c/main/java/com/cloud/utils/db/GenericDaoBase.java	0.00%	35 Missing :warning:
.../cloud/configuration/ConfigurationManagerImpl.java	0.00%	25 Missing :warning:
...udstack/resourcedetail/ResourceDetailsDaoBase.java	0.00%	10 Missing :warning:
...mand/admin/resource/PurgeExpungedResourcesCmd.java	72.72%	9 Missing :warning:
...tack/engine/orchestration/NetworkOrchestrator.java	0.00%	9 Missing :warning:
...in/java/com/cloud/storage/dao/SnapshotDaoImpl.java	0.00%	9 Missing :warning:
...a/com/cloud/vm/snapshot/dao/VMSnapshotDaoImpl.java	55.00%	9 Missing :warning:
...om/cloud/network/lb/dao/ElasticLbVmMapDaoImpl.java	0.00%	9 Missing :warning:
.../main/java/com/cloud/vm/dao/VMInstanceDaoImpl.java	75.00%	1 Missing and 3 partials :warning:
... and 8 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8999      +/-   ##
============================================
+ Coverage     15.32%   15.42%   +0.10%     
- Complexity    11686    11808     +122     
============================================
  Files          5459     5464       +5     
  Lines        477298   478139     +841     
  Branches      60202    61057     +855     
============================================
+ Hits          73135    73765     +630     
- Misses       396084   396247     +163     
- Partials       8079     8127      +48     

Flag	Coverage Δ
uitests	4.19% <ø> (ø)
unittests	16.18% <55.68%> (+0.11%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✖️ debian ✔️ suse15. SL-JID 9464

[shwstppr]

@blueorangutan test

[blueorangutan]

@shwstppr a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian Build Failed (tid-10080)

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9471

[blueorangutan]

[SF] Trillian Build Failed (tid-10089)

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9490

[shwstppr]

@blueorangutan test

[blueorangutan]

@shwstppr a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian Build Failed (tid-10101)

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9494

[shwstppr]

@blueorangutan test matrix

[blueorangutan]

@shwstppr a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

[shwstppr]

Manually tested newly added smoke test,

[root@pr8999-t6925-kvm-centos7-marvin marvin]# nosetests --with-xunit --xunit-file=results.xml --with-marvin --marvin-config=./pr8999-t6925-kvm-centos7-advanced-cfg -s -a tags=advanced --hypervisor=KVM tests/smoke/test_purge_expunged_vms.py
/usr/local/lib/python3.6/site-packages/paramiko/transport.py:32: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography. The next release of cryptography will remove support for Python 3.6.
  from cryptography.hazmat.backends import default_backend

==== Marvin Init Started ====

=== Marvin Parse Config Successful ===

=== Marvin Setting TestData Successful===

==== Log Folder Path: /marvin/MarvinLogs/May_01_2024_08_09_34_TNMKB9 All logs will be available here ====

=== Marvin Init Logging Successful===

==== Marvin Init Successful ====
=== TestName: test_01_purge_expunged_api_vm_start_date | Status : SUCCESS ===

=== TestName: test_02_purge_expunged_api_vm_end_date | Status : SUCCESS ===

=== TestName: test_03_purge_expunged_api_vm_start_end_date | Status : SUCCESS ===

=== TestName: test_04_purge_expunged_api_vm_no_date | Status : SUCCESS ===

=== TestName: test_05_purge_expunged_vm_service_offering | Status : SUCCESS ===

{'mgtSvrIp': '10.1.33.150', 'port': 8080, 'user': 'root', 'passwd': 'P@ssword123', 'hypervisor': 'kvm', 'useHttps': 'False', 'certCAPath': 'NA', 'certPath': 'NA', 'apiKey': 'YZDWI356FRnLh8vEHROd_ZUVD5U8JeGG-mObb7wPhyL5OQqi6df3KCmF5AlVWxPsZvsvNY2-z9BbyqSZGPaBCg', 'securityKey': 'U4LP9-ZdTuRl4_hJl3x2KN6nTNZCYAWhcMXmFKBSBYTNoUghjj3Pb5-0s2hN_tVG0l8oK6jDrmp953NOP59GAg'}
====Trying SSH Connection: Host:10.1.33.150 User:root                                   Port:22 RetryCnt:60===
===SSH to Host 10.1.33.150 port : 22 SUCCESSFUL===
{Cmd: service cloudstack-management stop via Host: 10.1.33.150} {returns: ['Redirecting to /bin/systemctl stop cloudstack-management.service']}
{Cmd: service cloudstack-management start via Host: 10.1.33.150} {returns: ['Redirecting to /bin/systemctl start cloudstack-management.service']}
====Trying SSH Connection: Host:10.1.35.104 User:root                                   Port:22 RetryCnt:60===
====Trying SSH Connection: Host:10.1.35.104 User:root                                   Port:22 RetryCnt:60===
===SSH to Host 10.1.35.104 port : 22 SUCCESSFUL===
===SSH to Host 10.1.35.104 port : 22 SUCCESSFUL===
{Cmd: service cloudstack-management stop via Host: 10.1.35.104} {returns: ['Redirecting to /bin/systemctl stop cloudstack-management.service']}
{Cmd: service cloudstack-management start via Host: 10.1.35.104} {returns: ['Redirecting to /bin/systemctl start cloudstack-management.service']}
{'mgtSvrIp': '10.1.33.150', 'port': 8080, 'user': 'root', 'passwd': 'P@ssword123', 'hypervisor': 'kvm', 'useHttps': 'False', 'certCAPath': 'NA', 'certPath': 'NA', 'apiKey': 'YZDWI356FRnLh8vEHROd_ZUVD5U8JeGG-mObb7wPhyL5OQqi6df3KCmF5AlVWxPsZvsvNY2-z9BbyqSZGPaBCg', 'securityKey': 'U4LP9-ZdTuRl4_hJl3x2KN6nTNZCYAWhcMXmFKBSBYTNoUghjj3Pb5-0s2hN_tVG0l8oK6jDrmp953NOP59GAg'}
====Trying SSH Connection: Host:10.1.33.150 User:root                                   Port:22 RetryCnt:60===
===SSH to Host 10.1.33.150 port : 22 SUCCESSFUL===
{Cmd: service cloudstack-management stop via Host: 10.1.33.150} {returns: ['Redirecting to /bin/systemctl stop cloudstack-management.service']}
{Cmd: service cloudstack-management start via Host: 10.1.33.150} {returns: ['Redirecting to /bin/systemctl start cloudstack-management.service']}
====Trying SSH Connection: Host:10.1.35.104 User:root                                   Port:22 RetryCnt:60===
====Trying SSH Connection: Host:10.1.35.104 User:root                                   Port:22 RetryCnt:60===
===SSH to Host 10.1.35.104 port : 22 SUCCESSFUL===
===SSH to Host 10.1.35.104 port : 22 SUCCESSFUL===
{Cmd: service cloudstack-management stop via Host: 10.1.35.104} {returns: ['Redirecting to /bin/systemctl stop cloudstack-management.service']}
{Cmd: service cloudstack-management start via Host: 10.1.35.104} {returns: ['Redirecting to /bin/systemctl start cloudstack-management.service']}
=== TestName: test_06_purge_expunged_vm_background_task | Status : SUCCESS ===

[blueorangutan]

[SF] Trillian test result (tid-10109) Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7 Total time taken: 91139 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8999-t10109-xenserver-71.zip Smoke tests completed. 129 look OK, 3 have errors, 0 did not run Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_events_resource	Error	352.70	test_events_resource.py
test_01_purge_expunged_api_vm_start_date	Failure	3904.59	test_purge_expunged_vms.py
test_02_purge_expunged_api_vm_end_date	Error	4248.79	test_purge_expunged_vms.py
test_03_purge_expunged_api_vm_start_end_date	Error	4209.00	test_purge_expunged_vms.py
test_04_purge_expunged_api_vm_no_date	Error	4268.95	test_purge_expunged_vms.py
test_05_purge_expunged_vm_service_offering	Error	3605.92	test_purge_expunged_vms.py
test_06_purge_expunged_vm_background_task	Error	301.48	test_purge_expunged_vms.py
test_01_restore_vm	Error	0.14	test_restore_vm.py
test_02_restore_vm_allocated_root	Error	0.12	test_restore_vm.py
ContextSuite context=TestRestoreVM>:teardown	Error	1.20	test_restore_vm.py

[blueorangutan]

[SF] Trillian test result (tid-10111) Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7 Total time taken: 95985 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8999-t10111-kvm-centos7.zip Smoke tests completed. 129 look OK, 3 have errors, 0 did not run Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_events_resource	Error	417.66	test_events_resource.py
test_01_purge_expunged_api_vm_start_date	Failure	3861.52	test_purge_expunged_vms.py
test_02_purge_expunged_api_vm_end_date	Error	4266.43	test_purge_expunged_vms.py
test_03_purge_expunged_api_vm_start_end_date	Error	4270.05	test_purge_expunged_vms.py
test_04_purge_expunged_api_vm_no_date	Error	4269.15	test_purge_expunged_vms.py
test_05_purge_expunged_vm_service_offering	Error	3666.85	test_purge_expunged_vms.py
test_06_purge_expunged_vm_background_task	Error	3907.16	test_purge_expunged_vms.py
test_01_restore_vm	Error	0.21	test_restore_vm.py
test_02_restore_vm_allocated_root	Error	0.13	test_restore_vm.py
ContextSuite context=TestRestoreVM>:teardown	Error	1.21	test_restore_vm.py

[blueorangutan]

[SF] Trillian test result (tid-10110) Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8 Total time taken: 101993 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8999-t10110-vmware-67u3.zip Smoke tests completed. 129 look OK, 3 have errors, 0 did not run Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_events_resource	Error	338.69	test_events_resource.py
test_01_purge_expunged_api_vm_start_date	Failure	3936.51	test_purge_expunged_vms.py
test_02_purge_expunged_api_vm_end_date	Error	4217.48	test_purge_expunged_vms.py
test_03_purge_expunged_api_vm_start_end_date	Error	4264.67	test_purge_expunged_vms.py
test_04_purge_expunged_api_vm_no_date	Error	4266.45	test_purge_expunged_vms.py
test_05_purge_expunged_vm_service_offering	Error	3605.18	test_purge_expunged_vms.py
test_06_purge_expunged_vm_background_task	Error	3964.82	test_purge_expunged_vms.py
test_01_restore_vm	Error	0.23	test_restore_vm.py
test_02_restore_vm_allocated_root	Error	0.28	test_restore_vm.py
ContextSuite context=TestRestoreVM>:teardown	Error	1.38	test_restore_vm.py

[shwstppr]

@blueorangutan test keepEnv

[blueorangutan]

@shwstppr a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[blueorangutan]

[LL] Trillian Build Failed (tid-6926)

[blueorangutan]

[SF] Trillian test result (tid-10145) Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7 Total time taken: 102908 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8999-t10145-kvm-centos7.zip Smoke tests completed. 128 look OK, 4 have errors, 0 did not run Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_events_resource	Error	416.19	test_events_resource.py
test_01_purge_expunged_api_vm_start_date	Failure	3867.81	test_purge_expunged_vms.py
test_02_purge_expunged_api_vm_end_date	Error	4209.15	test_purge_expunged_vms.py
test_03_purge_expunged_api_vm_start_end_date	Error	4270.12	test_purge_expunged_vms.py
test_04_purge_expunged_api_vm_no_date	Error	4214.86	test_purge_expunged_vms.py
test_05_purge_expunged_vm_service_offering	Error	3667.20	test_purge_expunged_vms.py
test_06_purge_expunged_vm_background_task	Error	3911.95	test_purge_expunged_vms.py
test_01_restore_vm	Error	0.24	test_restore_vm.py
test_02_restore_vm_allocated_root	Error	0.17	test_restore_vm.py
ContextSuite context=TestRestoreVM>:teardown	Error	1.26	test_restore_vm.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	Failure	453.99	test_vpc_redundant.py
test_05_rvpc_multi_tiers	Failure	519.47	test_vpc_redundant.py
test_05_rvpc_multi_tiers	Error	519.49	test_vpc_redundant.py