apache
cleanup some smoke tests
Description
This PR splits the test_vm_life_cycle.py test file and cleans some of the conditional test skipping. This all in an attempt to reduce intermittent integration test failures.
The reason is that a lot of tests are not basic life cycle tests and differnt failures on different platform are mostly pre-conditional instead of genuine positives.
During this work I also noticed that we might gain by eliminating tests as it seems a lot of the same functionalities are tested at different locations. I will not investigate that further or address it in this PR.
I will look at other intermittent failures as well and try to address those as well. This may happen in this PR or in separate PRs.
Types of changes
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [x] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] build/CI
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
- [x] Major
- [ ] Minor
Bug Severity
- [ ] BLOCKER
- [ ] Critical
- [x] Major
- [ ] Minor
- [ ] Trivial
Screenshots (if appropriate):
How Has This Been Tested?
How did you try to break this feature and the system with this change?
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:
Comparison is base (
08749d8) 13.12% compared to head (251a913) 13.13%.
Additional details and impacted files
@@ Coverage Diff @@
## 4.18 #8196 +/- ##
=========================================
Coverage 13.12% 13.13%
- Complexity 9141 9142 +1
=========================================
Files 2720 2720
Lines 257726 257726
Branches 40177 40177
=========================================
+ Hits 33838 33847 +9
+ Misses 219598 219588 -10
- Partials 4290 4291 +1
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
Codecov Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 16.58%. Comparing base (3e3a0c0) to head (bcc129c).
:warning: Report is 276 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@
## main #8196 +/- ##
=========================================
Coverage 16.58% 16.58%
Complexity 13989 13989
=========================================
Files 5743 5743
Lines 510706 510706
Branches 62119 62119
=========================================
Hits 84689 84689
Misses 416543 416543
Partials 9474 9474
| Flag | Coverage Δ | |
|---|---|---|
| uitests | 3.90% <ø> (ø) |
|
| unittests | 17.48% <ø> (ø) |
Flags with carried forward coverage won't be shown. Click here to find out more.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
:rocket: New features to boost your workflow:
- :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
- :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.
Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9791
@DaanHoogland a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
[SF] Trillian test result (tid-10373) Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7 Total time taken: 46547 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8196-t10373-kvm-centos7.zip Smoke tests completed. 110 look OK, 5 have errors, 0 did not run Only failed and skipped tests results shown below:
| Test | Result | Time (s) | Test File |
|---|---|---|---|
| test_01_add_primary_storage_disabled_host | Error |
0.20 | test_primary_storage.py |
| test_03_migration_options_storage_tags | Error |
0.02 | test_primary_storage.py |
| ContextSuite context=TestCpuCapServiceOfferings>:setup | Error |
0.00 | test_service_offerings.py |
| test_02_list_snapshots_with_removed_data_store | Error |
9.49 | test_snapshots.py |
| test_02_list_snapshots_with_removed_data_store | Error |
9.49 | test_snapshots.py |
| test_01_deploy_vm_on_specific_host | Error |
0.08 | test_vm_deployment_planner.py |
| test_04_deploy_vm_on_host_override_pod_and_cluster | Error |
0.10 | test_vm_deployment_planner.py |
| test_08_migrate_vm | Error |
0.06 | test_vm_life_cycle.py |
This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.
@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10779
@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests
[SF] Trillian test result (tid-11173) Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8 Total time taken: 52255 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8196-t11173-kvm-ol8.zip Smoke tests completed. 137 look OK, 7 have errors, 0 did not run Only failed and skipped tests results shown below:
| Test | Result | Time (s) | Test File |
|---|---|---|---|
| test_01_add_primary_storage_disabled_host | Error |
0.23 | test_primary_storage.py |
| test_03_migration_options_storage_tags | Error |
0.02 | test_primary_storage.py |
| test_01_secure_vm_migration | Error |
191.29 | test_secured_vm_migration.py |
| test_02_unsecure_vm_migration | Error |
450.63 | test_secured_vm_migration.py |
| test_04_nonsecured_to_secured_vm_migration | Error |
378.93 | test_secured_vm_migration.py |
| ContextSuite context=TestCpuCapServiceOfferings>:setup | Error |
0.00 | test_service_offerings.py |
| test_02_list_snapshots_with_removed_data_store | Error |
10.64 | test_snapshots.py |
| test_02_list_snapshots_with_removed_data_store | Error |
10.64 | test_snapshots.py |
| test_01_deploy_vm_on_specific_host | Error |
0.10 | test_vm_deployment_planner.py |
| test_04_deploy_vm_on_host_override_pod_and_cluster | Error |
0.15 | test_vm_deployment_planner.py |
| test_08_migrate_vm | Error |
0.07 | test_vm_life_cycle.py |
| test_13_destroy_and_expunge_vm | Error |
0.03 | test_vm_life_cycle.py |
| test_01_migrate_vm_strict_tags_success | Error |
0.30 | test_vm_strict_host_tags.py |
| test_02_migrate_vm_strict_tags_failure | Error |
0.30 | test_vm_strict_host_tags.py |
| test_01_restore_vm_strict_tags_success | Error |
0.30 | test_vm_strict_host_tags.py |
| test_02_restore_vm_strict_tags_failure | Error |
0.30 | test_vm_strict_host_tags.py |
| test_01_scale_vm_strict_tags_success | Error |
0.29 | test_vm_strict_host_tags.py |
| test_02_scale_vm_strict_tags_failure | Error |
0.31 | test_vm_strict_host_tags.py |
| test_01_deploy_vm_on_specific_host_without_strict_tags | Error |
0.27 | test_vm_strict_host_tags.py |
| test_02_deploy_vm_on_any_host_without_strict_tags | Error |
5.72 | test_vm_strict_host_tags.py |
| test_03_deploy_vm_on_specific_host_with_strict_tags_success | Error |
0.24 | test_vm_strict_host_tags.py |
| test_04_deploy_vm_on_any_host_with_strict_tags_success | Error |
5.70 | test_vm_strict_host_tags.py |
| test_05_deploy_vm_on_specific_host_with_strict_tags_failure | Failure |
0.28 | test_vm_strict_host_tags.py |
This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.
@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.
Packaging result [SF]: ✔️ el8 ✖️ el9 ✔️ debian ✖️ suse15. SL-JID 11160
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11165
@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests
[SF] Trillian test result (tid-11534) Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8 Total time taken: 58592 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8196-t11534-kvm-ol8.zip Smoke tests completed. 139 look OK, 7 have errors, 0 did not run Only failed and skipped tests results shown below:
| Test | Result | Time (s) | Test File |
|---|---|---|---|
| test_01_add_primary_storage_disabled_host | Error |
336.07 | test_primary_storage.py |
| test_01_add_primary_storage_disabled_host | Error |
336.09 | test_primary_storage.py |
| test_03_migration_options_storage_tags | Error |
0.02 | test_primary_storage.py |
| test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true | Failure |
203.14 | test_routers_network_ops.py |
| test_03_secured_to_nonsecured_vm_migration | Error |
114.71 | test_secured_vm_migration.py |
| test_03_secured_to_nonsecured_vm_migration | Error |
114.71 | test_secured_vm_migration.py |
| test_04_nonsecured_to_secured_vm_migration | Error |
0.01 | test_secured_vm_migration.py |
| test_02_list_snapshots_with_removed_data_store | Error |
8.51 | test_snapshots.py |
| test_02_list_snapshots_with_removed_data_store | Error |
8.52 | test_snapshots.py |
| ContextSuite context=TestISOUsage>:setup | Error |
0.00 | test_usage.py |
| test_08_migrate_vm | Error |
45.02 | test_vm_life_cycle.py |
| test_13_destroy_and_expunge_vm | Error |
0.02 | test_vm_life_cycle.py |
| test_01_migrate_vm_strict_tags_success | Error |
98.02 | test_vm_strict_host_tags.py |
This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.