"Saved-Critical" for Almost-Clustered Virtual Machines in Windows 2008 R2 Hyper-V

While staging a clustered Hyper-V environment this week, I ran into a head-scratching issue. After provisioning a few LUNs, masking them, and adding them as available storage to Failover Clustering, I began setting up the virtual machines on one of my cluster nodes. During maintenance, the physical servers were restarted. When I went back to continue configuring the virtual machines (which were not yet clustered at this point), I connected to the physical host on which I had been configuring them. While my other clustered virtual machines were running fine, the new ones that I had not yet finished configuring were in a "Saved-Critical" state.

I attempted to start them, but received an eror that the saved state could not be restored. I attempted to delete the saved state, but received an "Unable to perform operation" error. Additionally, I could not view the properties of the virtual machines in the Saved-Critical state.

I opened up Failover Cluster manager and noticed that the storage was now owned by the other cluster node. The problem is now obvious--the storage is no longer being presented to the host on which I was configuring the virtual machines. So, how to move these disks? Since the disks are marked "Available" (not having run through the new service Failover Clustering wizard to make the virtual machines highly available), I can't move them by right-clicking on them and selecting a new node.

There is another solution, however, and it involves the cluster.exe command.

1. Open up a command prompt.
2. Run the following command: cluster.exe GROUP "Available Storage" /Move

Voila! The disks moved back to the other node and the virtual machine state in Hyper-V manager changed from "Saved-Critical" to "Off." I was then able to cluster the machines normally through Failover Clustering.

Windows 2008 Hyper-V R2 Background Merge

When you remove a snapshot or snapshot subtree from a Hyper-V VM, there is a merge process that combines the data in the .avhd snapshot file back into the VM's VHD.

In Windows 2008 RTM, the merge process only happened once the virtual machine was shut down. The VM was unavailable for the duration of this process. There was, however, a progress bar displayed in either SCVMM or Hyper-V Manager that would indicate the completion percentage.

In Windows 2008 R2, however, the merge process happens in the background. Any indication of it happening is no longer easily visible in the console.

To view the progress of a merge, run the following PowerShell command:

Get-WmiObject -Namespace "root\virtualization" -Query "select * from Msvm_ConcreteJob" | Where {$_.ElementName -eq 'Merge in Progress'}

This will give you output about all merges happening, the elasped time (in seconds, so be prepared to do some math), as well as the job completion percentage.