cloudstack
cloudstack copied to clipboard
apache
Fix snapshot deletion on template creation failure
Description
This PR addresses an issue as part of #8946
The issue I've observed is not while taking the snapshot but while creating the template from the snapshot (stack trace also refers the same).
I could not reproduce the original issue of failed snapshot showing as backedup state rather than error (it might have already fixed after 4.17.2), but I saw another serious issue.
The issue is whenever a snapshot is used to create a template or volume and if there is failure in backing up the snapshot to the secondary store and as part of handling that failure MS is deleting the snapshot in primary storage itself.
These changes are introduced as part of the PR https://github.com/apache/cloudstack/pull/5297
- Create a snapshot of a volume (set snapshot.backup.to.secondary = False)
- Create a template from that snapshot
- As part of the creation, MS first tries to backup the snapshot to the secondary storage
- I've made it fail
- MS recognized the failure and as part of failure it is deleting the snapshot on the primary storage (also marking the snapshot_store_ref entry for primary store role as "Destroyed")
Here in this PR, the fix is to handle this case of not deleting the snapshot on primary storage.
Types of changes
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] build/CI
Bug Severity
- [ ] BLOCKER
- [x] Critical
- [ ] Major
- [ ] Minor
- [ ] Trivial
Screenshots (if appropriate):
How Has This Been Tested?
First lets check the successful scenario
- Create a snapshot of a volume (set snapshot.backup.to.secondary = False)
- Create a template from that snapshot
- As part of the creation, MS first tries to backup the snapshot to the secondary storage
- One new entry in the snapshot_store_ref table will be seen for store_role "Image" in "creating" state
- After the successful creation of template, this entry will be deleted.
Now the failure scenario
- Create a snapshot of a volume (set snapshot.backup.to.secondary = False)
- Create a template from that snapshot
- As part of the creation, MS first tries to backup the snapshot to the secondary storage
- One new entry in the snapshot_store_ref table will be seen for store_role "Image" in "creating" state
- Make the backup operation fail (I've played with the debugger to fail the operation)
- Observe the snapshot_store_ref table, this entry will be deleted and also keeps the existing entry for store_role "Primary" as "Ready" state. (Previously this row was marked as "Destroyed")
@harikrishna-patnala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.
Codecov Report
Attention: Patch coverage is 45.45455% with 6 lines in your changes missing coverage. Please review.
Project coverage is 14.95%. Comparing base (
e2e46e3) to head (680e3c5). Report is 173 commits behind head on 4.19.
| Files with missing lines | Patch % | Lines |
|---|---|---|
| ...org/apache/cloudstack/snapshot/SnapshotHelper.java | 45.45% | 5 Missing and 1 partial :warning: |
Additional details and impacted files
@@ Coverage Diff @@
## 4.19 #9239 +/- ##
============================================
+ Coverage 4.29% 14.95% +10.65%
- Complexity 0 11011 +11011
============================================
Files 363 5378 +5015
Lines 29347 469941 +440594
Branches 5126 59571 +54445
============================================
+ Hits 1261 70283 +69022
- Misses 27943 391875 +363932
- Partials 143 7783 +7640
| Flag | Coverage Δ | |
|---|---|---|
| uitests | 4.29% <ø> (-0.01%) |
:arrow_down: |
| unittests | 15.66% <45.45%> (?) |
Flags with carried forward coverage won't be shown. Click here to find out more.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
Packaging result [SF]: ✔️ el7 ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 9919
@harikrishna-patnala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.
Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9931
@harikrishna-patnala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.
Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9991
@rohityadavcloud [SL] unsupported parameters provided. Supported mgmt server os are: centos7, centos6, alma9, suse15, alma8, ubuntu18, ubuntu22, ubuntu20, rocky8, ubuntu24. Supported hypervisors are: kvm-centos6, kvm-centos7, kvm-rocky8, kvm-alma8, kvm-alma9, kvm-ubuntu18, kvm-ubuntu20, kvm-ubuntu22, kvm-ubuntu24, kvm-suse15, vmware-55u3, vmware-60u2, vmware-65u2, vmware-67u3, vmware-70u1, vmware-70u2, vmware-70u3, vmware-80, vmware-80u1, xenserver-65sp1, xenserver-71, xenserver-74, xcpng74, xcpng76, xcpng80, xcpng81, xcpng82
@sureshanaparti a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
[SF] Trillian test result (tid-10622) Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7 Total time taken: 42446 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9239-t10622-kvm-centos7.zip Smoke tests completed. 130 look OK, 1 have errors, 0 did not run Only failed and skipped tests results shown below:
| Test | Result | Time (s) | Test File |
|---|---|---|---|
| test_02_trigger_shutdown | Failure |
336.57 | test_safe_shutdown.py |
@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.
Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10157
@harikrishna-patnala Unable to create volume from the snapshot, Not sure if it's related to this or different issue.
2024-06-27 09:56:52,592 DEBUG [o.a.c.s.s.SnapshotObject] (API-Job-Executor-4:ctx-e580ce2e job-54 ctx-5af933d0) (logid:a539fca6) Failed to update state:com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Ready via CreateOnlyRequested
2024-06-27 09:56:52,593 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (API-Job-Executor-4:ctx-e580ce2e job-54 ctx-5af933d0) (logid:a539fca6) Failed to copy snapshot
com.cloud.utils.exception.CloudRuntimeException: Failed to update state: com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Ready via CreateOnlyRequested
at org.apache.cloudstack.storage.snapshot.SnapshotObject.processEvent(SnapshotObject.java:239)
at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:357)
at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:193)
at org.apache.cloudstack.snapshot.SnapshotHelper.backupSnapshotToSecondaryStorageIfNotExists(SnapshotHelper.java:146)
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.createVolumeFromSnapshot(VolumeOrchestrator.java:565)
2024-06-27 09:56:52,598 DEBUG [o.a.c.s.SnapshotHelper] (API-Job-Executor-4:ctx-e580ce2e job-54 ctx-5af933d0) (logid:a539fca6) Expunge template for Snapshot [2] is called for primary storage role. Not expunging it, but we will still expunge the database reference of the snapshot for image storage role if any
2024-06-27 09:56:52,625 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-4:ctx-e580ce2e job-54) (logid:a539fca6) Unexpected exception while executing org.apache.cloudstack.api.command.admin.volume.CreateVolumeCmdByAdmin
com.cloud.utils.exception.CloudRuntimeException: Failed to create volume: be56269c-a6b7-4b80-b3f6-6e921eec79fe
at com.cloud.storage.VolumeApiServiceImpl.createVolume(VolumeApiServiceImpl.java:1060)
at com.cloud.storage.VolumeApiServiceImpl.createVolume(VolumeApiServiceImpl.java:241)
...
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to copy snapshot:com.cloud.utils.exception.CloudRuntimeException: Failed to update state: com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Ready via CreateOnlyRequested
at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:382)
at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:193)
at org.apache.cloudstack.snapshot.SnapshotHelper.backupSnapshotToSecondaryStorageIfNotExists(SnapshotHelper.java:146)
2024-06-27 09:56:52,627 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-4:ctx-e580ce2e job-54) (logid:a539fca6) Complete async job-54, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create volume: be56269c-a6b7-4b80-b3f6-6e921eec79fe"}
@sureshanaparti thanks for testing.
But I could not reproduce the issue with a new environment
I've 2 snapshots from two disks one with backup and one without
I've created 2 volumes and templates from those two snapshots
I've also created 2 templates from those two snapshots
They are successful, can you please try again testing @sureshanaparti
@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.
Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10351
@sureshanaparti a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
@harikrishna-patnala Unable to create template and volume from the backed up snapshot, check whether this is related to the same issue or not (seems root cause for both are the same, calling backupSnapshot() for already backed up snapshot). Able to create template and volume from the non-backed up snapshot - entry created in snapshot_store_ref table during copy and deleted after that.
Create template log =>
2024-07-15 07:17:32,499 DEBUG [o.a.c.s.s.SnapshotObject] (API-Job-Executor-5:ctx-24db41c0 job-51 ctx-b79ad463) (logid:e9288858) Failed to update state:com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Ready via CreateOnlyRequested
2024-07-15 07:17:32,500 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (API-Job-Executor-5:ctx-24db41c0 job-51 ctx-b79ad463) (logid:e9288858) Failed to copy snapshot
com.cloud.utils.exception.CloudRuntimeException: Failed to update state: com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Ready via CreateOnlyRequested
at org.apache.cloudstack.storage.snapshot.SnapshotObject.processEvent(SnapshotObject.java:239)
at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:357)
at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:193)
at org.apache.cloudstack.snapshot.SnapshotHelper.backupSnapshotToSecondaryStorageIfNotExists(SnapshotHelper.java:146)
at com.cloud.template.TemplateManagerImpl.createPrivateTemplate(TemplateManagerImpl.java:1681)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy236.createPrivateTemplate(Unknown Source)
at org.apache.cloudstack.api.command.user.template.CreateTemplateCmd.execute(CreateTemplateCmd.java:297)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2024-07-15 07:17:32,506 DEBUG [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-5:ctx-24db41c0 job-51 ctx-b79ad463) (logid:e9288858) Updating resource Type = template count for Account = 2 Operation = decreasing Amount = 1
2024-07-15 07:17:32,508 DEBUG [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-5:ctx-24db41c0 job-51 ctx-b79ad463) (logid:e9288858) Updating resource Type = secondary_storage count for Account = 2 Operation = decreasing Amount = (8.00 GB) 8589934592
2024-07-15 07:17:32,512 DEBUG [o.a.c.s.SnapshotHelper] (API-Job-Executor-5:ctx-24db41c0 job-51 ctx-b79ad463) (logid:e9288858) Expunge template for Snapshot [1] is called for primary storage role. Not expunging it, but we will still expunge the database reference of the snapshot for image storage role if any
2024-07-15 07:17:32,524 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-5:ctx-24db41c0 job-51) (logid:e9288858) Unexpected exception while executing org.apache.cloudstack.api.command.admin.template.CreateTemplateCmdByAdmin
com.cloud.utils.exception.CloudRuntimeException: Failed to copy snapshot:com.cloud.utils.exception.CloudRuntimeException: Failed to update state: com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Ready via CreateOnlyRequested
at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:382)
at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:193)
at org.apache.cloudstack.snapshot.SnapshotHelper.backupSnapshotToSecondaryStorageIfNotExists(SnapshotHelper.java:146)
at com.cloud.template.TemplateManagerImpl.createPrivateTemplate(TemplateManagerImpl.java:1681)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy236.createPrivateTemplate(Unknown Source)
at org.apache.cloudstack.api.command.user.template.CreateTemplateCmd.execute(CreateTemplateCmd.java:297)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2024-07-15 07:17:32,526 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-5:ctx-24db41c0 job-51) (logid:e9288858) Complete async job-51, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to copy snapshot:com.cloud.utils.exception.CloudRuntimeException: Failed to update state: com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Ready via CreateOnlyRequested"}
Create volume log =>
2024-07-15 07:24:23,234 DEBUG [o.a.c.s.s.SnapshotObject] (API-Job-Executor-6:ctx-479e5ef7 job-52 ctx-d1ee714c) (logid:2227ec93) Failed to update state:com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Ready via CreateOnlyRequested
2024-07-15 07:24:23,235 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (API-Job-Executor-6:ctx-479e5ef7 job-52 ctx-d1ee714c) (logid:2227ec93) Failed to copy snapshot
com.cloud.utils.exception.CloudRuntimeException: Failed to update state: com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Ready via CreateOnlyRequested
at org.apache.cloudstack.storage.snapshot.SnapshotObject.processEvent(SnapshotObject.java:239)
at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:357)
at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:193)
at org.apache.cloudstack.snapshot.SnapshotHelper.backupSnapshotToSecondaryStorageIfNotExists(SnapshotHelper.java:146)
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.createVolumeFromSnapshot(VolumeOrchestrator.java:565)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy241.createVolumeFromSnapshot(Unknown Source)
at com.cloud.storage.VolumeApiServiceImpl.createVolumeFromSnapshot(VolumeApiServiceImpl.java:1082)
at com.cloud.storage.VolumeApiServiceImpl.createVolume(VolumeApiServiceImpl.java:1032)
at com.cloud.storage.VolumeApiServiceImpl.createVolume(VolumeApiServiceImpl.java:241)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy247.createVolume(Unknown Source)
at org.apache.cloudstack.api.command.user.volume.CreateVolumeCmd.execute(CreateVolumeCmd.java:226)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2024-07-15 07:24:23,247 DEBUG [o.a.c.s.SnapshotHelper] (API-Job-Executor-6:ctx-479e5ef7 job-52 ctx-d1ee714c) (logid:2227ec93) Expunge template for Snapshot [1] is called for primary storage role. Not expunging it, but we will still expunge the database reference of the snapshot for image storage role if any
2024-07-15 07:24:23,273 DEBUG [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-6:ctx-479e5ef7 job-52 ctx-d1ee714c) (logid:2227ec93) Updating resource Type = volume count for Account = 2 Operation = decreasing Amount = 1
2024-07-15 07:24:23,280 DEBUG [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-6:ctx-479e5ef7 job-52 ctx-d1ee714c) (logid:2227ec93) Updating resource Type = primary_storage count for Account = 2 Operation = decreasing Amount = (8.00 GB) 8589934592
2024-07-15 07:24:23,295 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-6:ctx-479e5ef7 job-52) (logid:2227ec93) Unexpected exception while executing org.apache.cloudstack.api.command.admin.volume.CreateVolumeCmdByAdmin
com.cloud.utils.exception.CloudRuntimeException: Failed to create volume: 3b6b9ce1-d7cf-45c6-94bf-1d37d70929b4
at com.cloud.storage.VolumeApiServiceImpl.createVolume(VolumeApiServiceImpl.java:1060)
at com.cloud.storage.VolumeApiServiceImpl.createVolume(VolumeApiServiceImpl.java:241)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy247.createVolume(Unknown Source)
at org.apache.cloudstack.api.command.user.volume.CreateVolumeCmd.execute(CreateVolumeCmd.java:226)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to copy snapshot:com.cloud.utils.exception.CloudRuntimeException: Failed to update state: com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Ready via CreateOnlyRequested
at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:382)
at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:193)
at org.apache.cloudstack.snapshot.SnapshotHelper.backupSnapshotToSecondaryStorageIfNotExists(SnapshotHelper.java:146)
... 45 more
2024-07-15 07:24:23,299 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-479e5ef7 job-52) (logid:2227ec93) Complete async job-52, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create volume: 3b6b9ce1-d7cf-45c6-94bf-1d37d70929b4"}
[SF] Trillian test result (tid-10831) Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7 Total time taken: 44824 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9239-t10831-kvm-centos7.zip Smoke tests completed. 132 look OK, 0 have errors, 0 did not run Only failed and skipped tests results shown below:
| Test | Result | Time (s) | Test File |
|---|