KVM incremental snapshot feature

[JoaoJandre]

Description

This PR solves issue #8907.

Currently, when taking a volume snapshot/backup with KVM as the hypervisor, it is always a full snapshot/backup. However, always taking full snapshots of volumes is costly for both the storage network and storage systems. To solve the aforementioned issues, this PR extends the volume snapshot feature in KVM, allowing users to create incremental volume snapshots using KVM as a hypervisor.

To give operators control over which type of snapshot is being created, a new global setting kvm.incremental.snapshot has been added, which can be changed at zone and cluster scopes; this setting is false by default. Also, the snapshot.delta.max configuration, used to control the maximum deltas when using XenServer, was extended to also limit the size of the backing chain of snapshots on primary/secondary storage.

This functionality is only available in environments with Libvirt 7.6.0+ and qemu 6.1+. If the kvm.incremental.snapshot setting is true, and the hosts do not have the required Libvirt and qemu versions, an error will be thrown when trying to take a snapshot. Additionally, this functionality is only available when using file based storage, such as shared mount-point (iSCSI and FC that require a shared mount-point storage file system for KVM such as OCFS2 or GlusterFS), NFS, and local storage. Other storage types for KVM, such as CLVM and RBD, need different approaches to enable incremental backups; therefore, these are not currently supported.

Issue #8907 has more details and flowcharts of all the mapped workflows.

Types of changes

• [ ] Breaking change (fix or feature that would cause existing functionality to change)
• [X] New feature (non-breaking change which adds functionality)
• [ ] Bug fix (non-breaking change which fixes an issue)
• [X] Enhancement (improves an existing feature and functionality)
• [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
• [ ] build/CI

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• [X] Major
• [ ] Minor

Bug Severity

• [ ] BLOCKER
• [ ] Critical
• [ ] Major
• [ ] Minor
• [ ] Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Description of tests

During testing, the kvm.incremental.snapshot setting was changed to true and the snapshot.delta.max setting was changed to 3.

Tests with snapshot.backup.to.secondary = false

For the tests in this section, a test VM was created and reused for all tests.

Snapshot creation tests

Test	Result
Access the VM, create any file in it and create volume snapshot 1 while the VM running	Full snapshot created
Access the VM, create a second file in it, create volume snapshot 2 while the VM running	Incremental snapshot created with correct size and backing chain (snapshot 1)
Stop the VM and create volume snapshot 3	Correctly created incremental snapshot
Start the VM again, create volume snapshot 4	Full snapshot created
Migrate the VM and create volume snapshot 5	Incremental snapshot created from snapshot 4
Migrate VM + ROOT volume	Exception

Snapshot restore tests

Test	Result
Access the VM, delete all previously created files, stop the VM, restore snapshot 1 and start the VM again	Restoration correctly performed, the file created in snapshot creation test 1 was present on the volume
Access the VM, delete the file restored in snapshot restore test 2, stop the VM, restore snapshot 2 and start the VM again	Restoration correctly performed, the files created in tests 1 and 2 of snapshot creation were present on the volume

Snapshot removal tests

Test	Result
Delete snapshot 5	Snapshot deleted and removed from storage
Delete snapshot 1	Snapshot deleted and not removed from storage
Delete snapshots 2 and 3	Snapshots deleted and removed from storage; furthermore, snapshot 1 was also removed from storage

Template creation test

#	Test	Result
1	Create template from snapshot 4 and create a VM using the template	Template created correctly, VM had the files created in the original VM

Tests with snapshot.backup.to.secondary = true

All tests performed in the previous sections were repeated with snapshot.backup.to.secondary = false, in addition, two additional tests were performed. For the tests in this section, a test VM was created and reused for all tests.

Snapshot creation tests

N	Test	Result
1	Migrate the VM + ROOT volume and take snapshot 6	Migration carried out and full snapshot created
2	Stop the VM, migrate the volume and take snapshot 7	Volume migration performed and incremental snapshot created from snapshot 6

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 17.54875% with 1184 lines in your changes missing coverage. Please review.

Project coverage is 16.41%. Comparing base (39c5641) to head (51d5f65). Report is 84 commits behind head on main.

Files with missing lines	Patch %	Lines
...ud/hypervisor/kvm/storage/KVMStorageProcessor.java	9.63%	318 Missing and 1 partial :warning:
...om/cloud/storage/snapshot/SnapshotManagerImpl.java	3.27%	111 Missing and 7 partials :warning:
...storage/datastore/db/SnapshotDataStoreDaoImpl.java	3.26%	88 Missing and 1 partial :warning:
...ervisor/kvm/resource/LibvirtComputingResource.java	29.16%	66 Missing and 2 partials :warning:
...oudstack/storage/snapshot/SnapshotServiceImpl.java	22.07%	60 Missing :warning:
...rce/wrapper/LibvirtRemoveBitmapCommandWrapper.java	1.66%	59 Missing :warning:
...he/cloudstack/storage/snapshot/SnapshotObject.java	2.56%	38 Missing :warning:
.../wrapper/LibvirtConvertSnapshotCommandWrapper.java	2.63%	37 Missing :warning:
...apache/cloudstack/storage/to/SnapshotObjectTO.java	12.50%	34 Missing and 1 partial :warning:
...torage/datastore/ObjectInDataStoreManagerImpl.java	0.00%	35 Missing :warning:
... and 31 more

Additional details and impacted files

@@             Coverage Diff             @@
##               main    #9270     +/-   ##
===========================================
  Coverage     16.40%   16.41%             
- Complexity    13590    13622     +32     
===========================================
  Files          5692     5699      +7     
  Lines        501976   503131   +1155     
  Branches      60795    60940    +145     
===========================================
+ Hits          82369    82568    +199     
- Misses       410449   411387    +938     
- Partials       9158     9176     +18     

Flag	Coverage Δ
uitests	4.00% <ø> (ø)
unittests	17.27% <17.54%> (-0.01%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[weizhouapache]

Good job @JoaoJandre

[DaanHoogland]

Good job @JoaoJandre

second that, tnx

[weizhouapache]

@blueorangutan test rocky8 kvm-rocky8

[alexandremattioli]

@JoaoJandre nice one. Just one remark, the following sentence sounds contradictory to me "Additionally, this functionality is only available when using file based storage, such as shared mount-point (iSCSI and FC)", if it supports iSCSI and FC (through a shared mountpoint) it does support block storage, I think the phrasing could cause some confusion as to which types of storage are supported.

[JoaoJandre]

@JoaoJandre nice one. Just one remark, the following sentence sounds contradictory to me "Additionally, this functionality is only available when using file based storage, such as shared mount-point (iSCSI and FC)", if it supports iSCSI and FC (through a shared mountpoint) it does support block storage, I think the phrasing could cause some confusion as to which types of storage are supported.

Hey @alexandremattioli, I understand your confusion. However, when using shared mount-point, as far as ACS is concerned, the storage is file-based, we will not be working with blocks directly, only files (as ACS does already for shared mount point). The mentions on parenthesis are there to give an example of underlying storages that might be behind the shared mount-point.

I have updated the description to add a little more context.

[github-actions[bot]]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[github-actions[bot]]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[DaanHoogland]

hm, seen this a couple of times now; the bot removes and adds the has-conflicts labels in the same second and a PR without conflicts ends out being marked as having so :(

[github-actions[bot]]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[github-actions[bot]]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[JoaoJandre]

@blueorangutan package

[blueorangutan]

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10402

[DaanHoogland]

@blueorangutan LLtest

[blueorangutan]

@DaanHoogland a [LL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[blueorangutan]

[LL] Trillian Build Failed (tid-6960)

[DaanHoogland]

@blueorangutan test

[blueorangutan]

@DaanHoogland a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-10919) Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7 Total time taken: 57970 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9270-t10919-kvm-centos7.zip Smoke tests completed. 112 look OK, 25 have errors, 0 did not run Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_role_account_acls_multiple_mgmt_servers	Error	2.21	test_dynamicroles.py
test_query_async_job_result	Error	95.90	test_async_job.py
test_revoke_certificate	Error	0.01	test_certauthority_root.py
test_configure_ha_provider_invalid	Error	0.01	test_hostha_simulator.py
test_configure_ha_provider_valid	Error	0.01	test_hostha_simulator.py
test_ha_configure_enabledisable_across_clusterzones	Error	0.01	test_hostha_simulator.py
test_ha_disable_feature_invalid	Error	0.01	test_hostha_simulator.py
test_ha_enable_feature_invalid	Error	0.01	test_hostha_simulator.py
test_ha_list_providers	Error	0.01	test_hostha_simulator.py
test_ha_multiple_mgmt_server_ownership	Error	0.01	test_hostha_simulator.py
test_ha_verify_fsm_available	Error	0.01	test_hostha_simulator.py
test_ha_verify_fsm_degraded	Error	0.01	test_hostha_simulator.py
test_ha_verify_fsm_fenced	Error	0.01	test_hostha_simulator.py
test_ha_verify_fsm_recovering	Error	0.01	test_hostha_simulator.py
test_hostha_configure_default_driver	Error	0.01	test_hostha_simulator.py
test_hostha_configure_invalid_provider	Error	0.01	test_hostha_simulator.py
test_hostha_disable_feature_valid	Error	0.01	test_hostha_simulator.py
test_hostha_enable_feature_valid	Error	0.01	test_hostha_simulator.py
test_hostha_enable_feature_without_setting_provider	Error	0.01	test_hostha_simulator.py
test_list_ha_for_host	Error	0.01	test_hostha_simulator.py
test_list_ha_for_host_invalid	Error	0.01	test_hostha_simulator.py
test_list_ha_for_host_valid	Error	0.01	test_hostha_simulator.py
test_01_host_ping_on_alert	Error	0.09	test_host_ping.py
test_01_host_ping_on_alert	Error	0.09	test_host_ping.py
test_01_browser_migrate_template	Error	5.25	test_image_store_object_migration.py
test_01_invalid_upgrade_kubernetes_cluster	Failure	223.59	test_kubernetes_clusters.py
test_02_upgrade_kubernetes_cluster	Failure	228.34	test_kubernetes_clusters.py
test_03_deploy_and_scale_kubernetes_cluster	Failure	224.14	test_kubernetes_clusters.py
test_04_autoscale_kubernetes_cluster	Failure	224.31	test_kubernetes_clusters.py
test_05_basic_lifecycle_kubernetes_cluster	Failure	201.72	test_kubernetes_clusters.py
test_06_delete_kubernetes_cluster	Failure	222.07	test_kubernetes_clusters.py
test_08_upgrade_kubernetes_ha_cluster	Failure	325.86	test_kubernetes_clusters.py
test_10_vpc_tier_kubernetes_cluster	Failure	236.19	test_kubernetes_clusters.py
test_11_test_unmanaged_cluster_lifecycle	Error	94.17	test_kubernetes_clusters.py
test_01_add_delete_kubernetes_supported_version	Error	0.16	test_kubernetes_supported_versions.py
ContextSuite context=TestListIdsParams>:setup	Error	0.00	test_list_ids_parameter.py
login_test_saml_user	Error	3.08	test_login.py
test_01_deployVMInSharedNetwork	Error	74.25	test_network.py
test_03_destroySharedNetwork	Failure	1.07	test_network.py
ContextSuite context=TestSharedNetwork>:teardown	Error	2.17	test_network.py
test_oobm_issue_power_cycle	Error	3.33	test_outofbandmanagement_nestedplugin.py
test_oobm_issue_power_off	Error	3.28	test_outofbandmanagement_nestedplugin.py
test_oobm_issue_power_on	Error	3.29	test_outofbandmanagement_nestedplugin.py
test_oobm_issue_power_reset	Error	3.33	test_outofbandmanagement_nestedplugin.py
test_oobm_issue_power_soft	Error	3.32	test_outofbandmanagement_nestedplugin.py
test_oobm_issue_power_status	Error	2.22	test_outofbandmanagement_nestedplugin.py
test_oobm_background_powerstate_sync	Failure	20.67	test_outofbandmanagement.py
test_oobm_background_powerstate_sync	Error	20.67	test_outofbandmanagement.py
test_oobm_configure_default_driver	Error	0.06	test_outofbandmanagement.py
test_oobm_configure_invalid_driver	Error	0.06	test_outofbandmanagement.py
test_oobm_disable_feature_invalid	Error	0.05	test_outofbandmanagement.py
test_oobm_disable_feature_valid	Error	1.16	test_outofbandmanagement.py
test_oobm_enable_feature_invalid	Error	0.05	test_outofbandmanagement.py
test_oobm_enable_feature_valid	Error	1.11	test_outofbandmanagement.py
test_oobm_enabledisable_across_clusterzones	Error	10.88	test_outofbandmanagement.py
test_oobm_enabledisable_across_clusterzones	Error	10.88	test_outofbandmanagement.py
test_oobm_issue_power_cycle	Error	4.35	test_outofbandmanagement.py
test_oobm_issue_power_cycle	Error	4.35	test_outofbandmanagement.py
test_oobm_issue_power_off	Error	3.33	test_outofbandmanagement.py
test_oobm_issue_power_off	Error	3.33	test_outofbandmanagement.py
test_oobm_issue_power_on	Error	4.38	test_outofbandmanagement.py
test_oobm_issue_power_on	Error	4.38	test_outofbandmanagement.py
test_oobm_issue_power_reset	Error	3.36	test_outofbandmanagement.py
test_oobm_issue_power_reset	Error	3.36	test_outofbandmanagement.py
test_oobm_issue_power_soft	Error	4.36	test_outofbandmanagement.py
test_oobm_issue_power_soft	Error	4.36	test_outofbandmanagement.py
test_oobm_issue_power_status	Error	3.31	test_outofbandmanagement.py
test_oobm_issue_power_status	Error	3.31	test_outofbandmanagement.py
test_oobm_multiple_mgmt_server_ownership	Error	1.15	test_outofbandmanagement.py
test_oobm_multiple_mgmt_server_ownership	Error	1.16	test_outofbandmanagement.py
test_oobm_zchange_password	Error	2.23	test_outofbandmanagement.py
test_oobm_zchange_password	Error	2.23	test_outofbandmanagement.py
test_02_edit_primary_storage_tags	Error	0.02	test_primary_storage.py
test_01_primary_storage_scope_change	Error	0.07	test_primary_storage_scope.py
test_01_vpc_privategw_acl	Error	0.03	test_privategw_acl_ovs_gre.py
test_03_vpc_privategw_restart_vpc_cleanup	Error	0.02	test_privategw_acl_ovs_gre.py
test_05_vpc_privategw_check_interface	Error	0.02	test_privategw_acl_ovs_gre.py
test_01_vpc_privategw_acl	Error	52.80	test_privategw_acl.py
test_02_vpc_privategw_static_routes	Error	198.41	test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup	Error	195.52	test_privategw_acl.py
test_04_rvpc_privategw_static_routes	Error	323.69	test_privategw_acl.py
test_01_purge_expunged_api_vm_start_date	Error	51.62	test_purge_expunged_vms.py
test_02_purge_expunged_api_vm_end_date	Error	41.98	test_purge_expunged_vms.py
test_03_purge_expunged_api_vm_start_end_date	Error	42.96	test_purge_expunged_vms.py
test_04_purge_expunged_api_vm_no_date	Error	43.84	test_purge_expunged_vms.py
test_05_purge_expunged_vm_service_offering	Error	265.96	test_purge_expunged_vms.py
test_06_purge_expunged_vm_background_task	Error	330.58	test_purge_expunged_vms.py
test_01_snapshot_root_disk	Error	3.21	test_snapshots.py
test_02_list_snapshots_with_removed_data_store	Error	47.80	test_snapshots.py
ContextSuite context=TestSnapshotStandaloneBackup>:setup	Error	178.10	test_snapshots.py
test_CreateTemplateWithDuplicateName	Error	21.87	test_templates.py
test_01_register_template_direct_download_flag	Error	0.24	test_templates.py
test_01_positive_tests_usage	Error	10.51	test_usage_events.py
test_01_ISO_usage	Error	1.10	test_usage.py
test_01_lb_usage	Error	4.23	test_usage.py
test_01_nat_usage	Error	8.35	test_usage.py
test_01_public_ip_usage	Error	1.08	test_usage.py
test_01_snapshot_usage	Error	2.18	test_usage.py
test_01_template_usage	Error	14.60	test_usage.py
test_01_vm_usage	Error	133.32	test_usage.py
test_01_volume_usage	Error	126.16	test_usage.py
test_01_vpn_usage	Error	9.51	test_usage.py
test_03_secured_to_nonsecured_vm_migration	Error	372.35	test_vm_life_cycle.py
test_12_start_vm_multiple_volumes_allocated	Error	10.56	test_vm_life_cycle.py
test_01_vmschedule_create	Error	0.10	test_vm_schedule.py
test_disable_oobm_ha_state_ineligible	Error	0.05	test_hostha_kvm.py
test_hostha_configure_default_driver	Error	0.05	test_hostha_kvm.py
test_hostha_enable_ha_when_host_disabled	Error	0.05	test_hostha_kvm.py
test_hostha_enable_ha_when_host_disconected	Error	0.04	test_hostha_kvm.py
test_hostha_enable_ha_when_host_in_maintenance	Error	0.04	test_hostha_kvm.py
test_hostha_kvm_host_degraded	Error	0.04	test_hostha_kvm.py
test_hostha_kvm_host_fencing	Error	0.04	test_hostha_kvm.py
test_hostha_kvm_host_recovering	Error	0.05	test_hostha_kvm.py
test_remove_ha_provider_not_possible	Error	0.04	test_hostha_kvm.py

[JoaoJandre]

@DaanHoogland can BO export the agent's logs? There seems to be an error in the snapshot creation, but with only the MGMTs logs I'm unable to trace it properly

[DaanHoogland]

@DaanHoogland can BO export the agent's logs? There seems to be an error in the snapshot creation, but with only the MGMTs logs I'm unable to trace it properly

good feature request @JoaoJandre , not at the moment. But I see your issue. I can run with with keeping the env once to see what I can find, or give you the logs somehow.

[DaanHoogland]

@blueorangutan test keepEnv

[blueorangutan]

@DaanHoogland a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[JoaoJandre]

@DaanHoogland can BO export the agent's logs? There seems to be an error in the snapshot creation, but with only the MGMTs logs I'm unable to trace it properly

good feature request @JoaoJandre , not at the moment. But I see your issue. I can run with with keeping the env once to see what I can find, or give you the logs somehow.

@DaanHoogland I've tracked down the issue, it seems to have been introduced in one of the merges with main. I'm testing the fix and I'll redo all my tests to make sure that everything is ok.

[blueorangutan]

[SF] Trillian test result (tid-10932) Environment: kvm-alma9 (x2), Advanced Networking with Mgmt server a9 Total time taken: 59404 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9270-t10932-kvm-alma9.zip Smoke tests completed. 133 look OK, 4 have errors, 0 did not run Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestListIdsParams>:setup	Error	0.00	test_list_ids_parameter.py
test_06_purge_expunged_vm_background_task	Failure	338.62	test_purge_expunged_vms.py
test_01_snapshot_root_disk	Error	3.23	test_snapshots.py
test_02_list_snapshots_with_removed_data_store	Error	46.84	test_snapshots.py
ContextSuite context=TestSnapshotStandaloneBackup>:setup	Error	188.69	test_snapshots.py
test_01_snapshot_usage	Error	3.22	test_usage.py

[blueorangutan]

[SF] Trillian test result (tid-10940) Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7 Total time taken: 52185 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9270-t10940-kvm-centos7.zip Smoke tests completed. 108 look OK, 29 have errors, 0 did not run Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_role_account_acls_multiple_mgmt_servers	Error	2.35	test_dynamicroles.py
test_DeployVmAntiAffinityGroup_in_project	Error	93.07	test_affinity_groups_projects.py
test_DeployVmAntiAffinityGroup	Error	37.80	test_affinity_groups.py
test_query_async_job_result	Error	105.13	test_async_job.py
test_revoke_certificate	Error	0.01	test_certauthority_root.py
test_configure_ha_provider_invalid	Error	0.01	test_hostha_simulator.py
test_configure_ha_provider_valid	Error	0.01	test_hostha_simulator.py
test_ha_configure_enabledisable_across_clusterzones	Error	0.01	test_hostha_simulator.py
test_ha_disable_feature_invalid	Error	0.01	test_hostha_simulator.py
test_ha_enable_feature_invalid	Error	0.01	test_hostha_simulator.py
test_ha_list_providers	Error	0.01	test_hostha_simulator.py
test_ha_multiple_mgmt_server_ownership	Error	0.01	test_hostha_simulator.py
test_ha_verify_fsm_available	Error	0.01	test_hostha_simulator.py
test_ha_verify_fsm_degraded	Error	0.01	test_hostha_simulator.py
test_ha_verify_fsm_fenced	Error	0.01	test_hostha_simulator.py
test_ha_verify_fsm_recovering	Error	0.01	test_hostha_simulator.py
test_hostha_configure_default_driver	Error	0.01	test_hostha_simulator.py
test_hostha_configure_invalid_provider	Error	0.01	test_hostha_simulator.py
test_hostha_disable_feature_valid	Error	0.01	test_hostha_simulator.py
test_hostha_enable_feature_valid	Error	0.01	test_hostha_simulator.py
test_hostha_enable_feature_without_setting_provider	Error	0.01	test_hostha_simulator.py
test_list_ha_for_host	Error	0.01	test_hostha_simulator.py
test_list_ha_for_host_invalid	Error	0.01	test_hostha_simulator.py
test_list_ha_for_host_valid	Error	0.01	test_hostha_simulator.py
test_01_host_ping_on_alert	Error	0.08	test_host_ping.py
test_01_host_ping_on_alert	Error	0.08	test_host_ping.py
test_01_browser_migrate_template	Error	15.36	test_image_store_object_migration.py
test_01_invalid_upgrade_kubernetes_cluster	Failure	238.79	test_kubernetes_clusters.py
test_02_upgrade_kubernetes_cluster	Failure	242.91	test_kubernetes_clusters.py
test_03_deploy_and_scale_kubernetes_cluster	Failure	243.88	test_kubernetes_clusters.py
test_04_autoscale_kubernetes_cluster	Failure	241.68	test_kubernetes_clusters.py
test_05_basic_lifecycle_kubernetes_cluster	Failure	241.76	test_kubernetes_clusters.py
test_06_delete_kubernetes_cluster	Failure	240.60	test_kubernetes_clusters.py
test_08_upgrade_kubernetes_ha_cluster	Failure	0.07	test_kubernetes_clusters.py
test_10_vpc_tier_kubernetes_cluster	Failure	237.46	test_kubernetes_clusters.py
test_11_test_unmanaged_cluster_lifecycle	Error	81.90	test_kubernetes_clusters.py
test_01_add_delete_kubernetes_supported_version	Error	0.13	test_kubernetes_supported_versions.py
ContextSuite context=TestListIdsParams>:setup	Error	0.00	test_list_ids_parameter.py
login_test_saml_user	Error	3.02	test_login.py
test_01_deployVMInSharedNetwork	Error	76.25	test_network.py
test_03_destroySharedNetwork	Failure	1.08	test_network.py
ContextSuite context=TestSharedNetwork>:teardown	Error	2.18	test_network.py
test_01_non_strict_host_anti_affinity	Failure	124.29	test_nonstrict_affinity_group.py
test_02_non_strict_host_affinity	Error	82.37	test_nonstrict_affinity_group.py
test_oobm_issue_power_cycle	Error	3.31	test_outofbandmanagement_nestedplugin.py
test_oobm_issue_power_off	Error	3.29	test_outofbandmanagement_nestedplugin.py
test_oobm_issue_power_on	Error	3.30	test_outofbandmanagement_nestedplugin.py
test_oobm_issue_power_reset	Error	3.30	test_outofbandmanagement_nestedplugin.py
test_oobm_issue_power_soft	Error	3.29	test_outofbandmanagement_nestedplugin.py
test_oobm_issue_power_status	Error	2.22	test_outofbandmanagement_nestedplugin.py
test_oobm_background_powerstate_sync	Failure	21.67	test_outofbandmanagement.py
test_oobm_background_powerstate_sync	Error	21.67	test_outofbandmanagement.py
test_oobm_configure_default_driver	Error	0.05	test_outofbandmanagement.py
test_oobm_configure_invalid_driver	Error	0.05	test_outofbandmanagement.py
test_oobm_disable_feature_invalid	Error	0.04	test_outofbandmanagement.py
test_oobm_disable_feature_valid	Error	0.13	test_outofbandmanagement.py
test_oobm_enable_feature_invalid	Error	0.04	test_outofbandmanagement.py
test_oobm_enable_feature_valid	Error	0.10	test_outofbandmanagement.py
test_oobm_enabledisable_across_clusterzones	Error	10.88	test_outofbandmanagement.py
test_oobm_enabledisable_across_clusterzones	Error	10.88	test_outofbandmanagement.py
test_oobm_issue_power_cycle	Error	4.33	test_outofbandmanagement.py
test_oobm_issue_power_cycle	Error	4.33	test_outofbandmanagement.py
test_oobm_issue_power_off	Error	4.35	test_outofbandmanagement.py
test_oobm_issue_power_off	Error	4.35	test_outofbandmanagement.py
test_oobm_issue_power_on	Error	4.34	test_outofbandmanagement.py
test_oobm_issue_power_on	Error	4.34	test_outofbandmanagement.py
test_oobm_issue_power_reset	Error	4.40	test_outofbandmanagement.py
test_oobm_issue_power_reset	Error	4.40	test_outofbandmanagement.py
test_oobm_issue_power_soft	Error	4.32	test_outofbandmanagement.py
test_oobm_issue_power_soft	Error	4.33	test_outofbandmanagement.py
test_oobm_issue_power_status	Error	4.34	test_outofbandmanagement.py
test_oobm_issue_power_status	Error	4.34	test_outofbandmanagement.py
test_oobm_multiple_mgmt_server_ownership	Error	0.13	test_outofbandmanagement.py
test_oobm_multiple_mgmt_server_ownership	Error	0.13	test_outofbandmanagement.py
test_oobm_zchange_password	Error	2.20	test_outofbandmanagement.py
test_oobm_zchange_password	Error	2.20	test_outofbandmanagement.py
test_02_edit_primary_storage_tags	Error	0.02	test_primary_storage.py
test_01_primary_storage_scope_change	Error	0.08	test_primary_storage_scope.py
test_01_vpc_privategw_acl	Error	0.03	test_privategw_acl_ovs_gre.py
test_03_vpc_privategw_restart_vpc_cleanup	Error	0.02	test_privategw_acl_ovs_gre.py
test_05_vpc_privategw_check_interface	Error	0.02	test_privategw_acl_ovs_gre.py
test_01_vpc_privategw_acl	Error	51.73	test_privategw_acl.py
test_02_vpc_privategw_static_routes	Error	203.28	test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup	Error	200.23	test_privategw_acl.py
test_04_rvpc_privategw_static_routes	Error	336.51	test_privategw_acl.py
test_01_purge_expunged_api_vm_start_date	Error	49.93	test_purge_expunged_vms.py
test_02_purge_expunged_api_vm_end_date	Error	48.25	test_purge_expunged_vms.py
test_03_purge_expunged_api_vm_start_end_date	Error	46.01	test_purge_expunged_vms.py
test_04_purge_expunged_api_vm_no_date	Error	51.05	test_purge_expunged_vms.py
test_05_purge_expunged_vm_service_offering	Error	274.21	test_purge_expunged_vms.py
test_06_purge_expunged_vm_background_task	Error	340.50	test_purge_expunged_vms.py
test_01_snapshot_root_disk	Error	2.17	test_snapshots.py
test_02_list_snapshots_with_removed_data_store	Error	48.64	test_snapshots.py
ContextSuite context=TestSnapshotStandaloneBackup>:setup	Error	168.49	test_snapshots.py
test_CreateTemplateWithDuplicateName	Error	18.77	test_templates.py
test_01_register_template_direct_download_flag	Error	0.15	test_templates.py
test_01_positive_tests_usage	Error	9.85	test_usage_events.py
test_01_ISO_usage	Error	1.09	test_usage.py
test_01_lb_usage	Error	4.24	test_usage.py
test_01_nat_usage	Error	7.33	test_usage.py
test_01_public_ip_usage	Error	1.08	test_usage.py
test_01_snapshot_usage	Error	2.19	test_usage.py
test_01_template_usage	Error	13.51	test_usage.py
test_01_vm_usage	Error	136.26	test_usage.py
test_01_volume_usage	Error	129.17	test_usage.py
test_01_vpn_usage	Error	8.50	test_usage.py
test_12_start_vm_multiple_volumes_allocated	Error	10.51	test_vm_life_cycle.py
test_01_vmschedule_create	Error	0.09	test_vm_schedule.py
ContextSuite context=TestMigrateVMStrictTags>:setup	Error	0.00	test_vm_strict_host_tags.py
test_disable_oobm_ha_state_ineligible	Error	0.05	test_hostha_kvm.py
test_hostha_configure_default_driver	Error	0.05	test_hostha_kvm.py
test_hostha_enable_ha_when_host_disabled	Error	0.04	test_hostha_kvm.py
test_hostha_enable_ha_when_host_disconected	Error	0.03	test_hostha_kvm.py
test_hostha_enable_ha_when_host_in_maintenance	Error	0.04	test_hostha_kvm.py
test_hostha_kvm_host_degraded	Error	0.04	test_hostha_kvm.py
test_hostha_kvm_host_fencing	Error	0.04	test_hostha_kvm.py
test_hostha_kvm_host_recovering	Error	0.03	test_hostha_kvm.py
test_remove_ha_provider_not_possible	Error	0.03	test_hostha_kvm.py

[JoaoJandre]

@blueorangutan package

[blueorangutan]

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 10491