one icon indicating copy to clipboard operation
one copied to clipboard
verified publisher icon OpenNebula

Improve kvm live migration for non shared storage

Open dann1 opened this issue 2 years ago • 0 comments

Description When live migrating a KVM VM the process is quite different when dealing with a shared storage deployment (like NFS) or non shared one (like SSH driver). For shared storage the migration is quite simple as the VM assets (disks, snapshots, xmls, etc.) are already present in the destination host. In the case of non shared storage all of these assets need to be copied manually.

Once the migration completes the snapshots need to be redefined. At the moment this only applies to shared storage. It is possible however, to redefine them for non shared storage as well. With this modification enabled, the VM successfully migrates and carries over the snapshots to the new host. However, even though that snapshot is migrated as an asset, it is no longer bound to the VM disk. What happens is that if you try to revert to that snapshot, you'll get an error about libvirt complaining it doesn't exist and then the VM will cease to exist.

For example

ue Oct 24 21:17:07 2023 [Z0][VM][I]: New LCM state is RUNNING
Tue Oct 24 21:17:08 2023 [Z0][VM][I]: New LCM state is MIGRATE
Tue Oct 24 21:17:08 2023 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_premigrate.
Tue Oct 24 21:17:08 2023 [Z0][VMM][I]: ExitCode: 0
Tue Oct 24 21:17:08 2023 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Tue Oct 24 21:17:22 2023 [Z0][VMM][I]: /var/tmp/one/vmm/kvm/migrate: line 35: warning: command substitution: ignored null byte in input
Tue Oct 24 21:17:22 2023 [Z0][VMM][I]: ExitCode: 0
Tue Oct 24 21:17:22 2023 [Z0][VMM][I]: Successfully execute virtualization driver operation: migrate.
Tue Oct 24 21:17:22 2023 [Z0][VMM][I]: ExitCode: 0
Tue Oct 24 21:17:22 2023 [Z0][VMM][I]: Successfully execute network driver operation: clean.
Tue Oct 24 21:17:22 2023 [Z0][VMM][I]: ExitCode: 0
Tue Oct 24 21:17:22 2023 [Z0][VMM][I]: Successfully execute network driver operation: post.
Tue Oct 24 21:17:22 2023 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_postmigrate.
Tue Oct 24 21:17:22 2023 [Z0][VM][I]: New LCM state is RUNNING
Tue Oct 24 21:17:24 2023 [Z0][VM][I]: New LCM state is HOTPLUG_SNAPSHOT
Tue Oct 24 21:17:26 2023 [Z0][VMM][I]: Command execution fail (exit code: 1): cat << 'EOT' | /var/tmp/one/vmm/kvm/snapshot_revert '31ce1f6b-247a-4b33-8aff-2a52364669b2' 'snap-0' 127 ubuntu2204-kvm-ssh-6-99-ipcrc-1.test
Tue Oct 24 21:17:26 2023 [Z0][VMM][E]: snapshot_revert: Command "virsh --connect qemu:///system snapshot-revert --force 31ce1f6b-247a-4b33-8aff-2a52364669b2 snap-0" failed: error: internal error: qemu unexpectedly closed the monitor: 2023-10-24T21:17:25.617780Z qemu-kvm-one: Snapshot 'snap-0' does not exist in one or more devices Could not revert to snapshot snap-0 for domain 31ce1f6b-247a-4b33-8aff-2a52364669b2.
Tue Oct 24 21:17:26 2023 [Z0][VMM][I]: ExitCode: 1
Tue Oct 24 21:17:26 2023 [Z0][VMM][I]: Failed to execute virtualization driver operation: snapshot_revert.
Tue Oct 24 21:17:26 2023 [Z0][VMM][E]: SNAPSHOTREVERT: ERROR: snapshot_revert: Command "virsh --connect qemu:///system snapshot-revert --force 31ce1f6b-247a-4b33-8aff-2a52364669b2 snap-0" failed: error: internal error: qemu unexpectedly closed the monitor: 2023-10-24T21:17:25.617780Z qemu-kvm-one: Snapshot 'snap-0' does not exist in one or more devices Could not revert to snapshot snap-0 for domain 31ce1f6b-247a-4b33-8aff-2a52364669b2. ExitCode: 1
Tue Oct 24 21:17:26 2023 [Z0][VM][I]: New LCM state is RUNNING
Tue Oct 24 21:17:55 2023 [Z0][LCM][I]: VM running but monitor state is POWEROFF
Tue Oct 24 21:17:55 2023 [Z0][VM][I]: New LCM state is SHUTDOWN_POWEROFF

This is how the VM disk looks like prior to the migration.

root@ubuntu2204-kvm-ssh-6-99-ipcrc-1:/var/lib/one/datastores/0# qemu-img info -U --output json 135/disk.0 | jq .
{
  "snapshots": [
    {
      "vm-clock-nsec": 839399858,
      "name": "snap-0",
      "date-sec": 1698243890,
      "date-nsec": 787387000,
      "vm-clock-sec": 1,
      "id": "1",
      "vm-state-size": 66046040
    }
  ],
  "virtual-size": 268435456,
  "filename": "135/disk.0",
  "cluster-size": 65536,
  "format": "qcow2",
  "actual-size": 140455936,
  "format-specific": {
    "type": "qcow2",
    "data": {
      "compat": "1.1",
      "compression-type": "zlib",
      "lazy-refcounts": false,
      "refcount-bits": 16,
      "corrupt": false,
      "extended-l2": false
    }
  },
  "dirty-flag": false
}

then after the VM is migrated, qemu tells a different story

root@ubuntu2204-kvm-ssh-6-99-ipcrc-2:/var/lib/one/datastores/0#  qemu-img info -U --output json 135/disk.0 | jq .
{
  "virtual-size": 268435456,
  "filename": "135/disk.0",
  "cluster-size": 65536,
  "format": "qcow2",
  "actual-size": 50864128,
  "format-specific": {
    "type": "qcow2",
    "data": {
      "compat": "1.1",
      "compression-type": "zlib",
      "lazy-refcounts": false,
      "refcount-bits": 16,
      "corrupt": false,
      "extended-l2": false
    }
  },
  "dirty-flag": false
}
root@ubuntu2204-kvm-ssh-6-99-ipcrc-2:/var/lib/one/datastores/0#

yet libvirt says the snapshot exists

root@ubuntu2204-kvm-ssh-6-99-ipcrc-2:/var/lib/one/datastores/0# virsh snapshot-list one-135
 Name     Creation Time               State
-----------------------------------------------
 snap-0   2023-10-25 14:24:50 +0000   running

root@ubuntu2204-kvm-ssh-6-99-ipcrc-2:/var/lib/one/datastores/0#

If a revert attempt is made the VM will end up like the VM log example shown before.

Use case

  • Improve the snapshot management allowing for this combination of features to be possible
    • non shared storage
    • snapshotting
    • migration
  • Potentially reduce the migration time
    • at the moment for ssh migration the option --copy-storage-all is used + tar copies
    • using rsync could speed up this process after the initial copy when the deltas are synced
    • the fs freeze + thaw could potentially be replaced by a migration with --suspend flag
  • Reduce conditional flow code by treating every disk format the same (qcow2 and raw)

At a high level, the proposal is as follows. Replace the old behavior

  1. Copy disks with tar | ssh
  2. Recreate links in remote host
  3. Copy ancillary files (*.xml)
  4. If there are RAW disks 5. freeze the VM 6. copy the disks again (active VM delta while migration started)
  5. migrate --copy-storage-all
  6. thaw/unfreeze the migrated VM

with

  1. Copy disks with tar | ssh
  2. Recreate links in remote host
  3. Copy ancillary files (*.xml)
  4. migrate --suspend
  5. rsync

for every disk format

Progress Status

  • [ ] Code committed
  • [ ] Testing - QA
  • [ ] Documentation (Release notes - resolved issues, compatibility, known issues)

dann1 avatar Oct 25 '23 20:10 dann1