Showing posts with label hyper-v. Show all posts
Showing posts with label hyper-v. Show all posts

Wednesday, October 6, 2010

"Saved-Critical" for Almost-Clustered Virtual Machines in Windows 2008 R2 Hyper-V

While staging a clustered Hyper-V environment this week, I ran into a head-scratching issue. After provisioning a few LUNs, masking them, and adding them as available storage to Failover Clustering, I began setting up the virtual machines on one of my cluster nodes. During maintenance, the physical servers were restarted. When I went back to continue configuring the virtual machines (which were not yet clustered at this point), I connected to the physical host on which I had been configuring them. While my other clustered virtual machines were running fine, the new ones that I had not yet finished configuring were in a "Saved-Critical" state.

I attempted to start them, but received an eror that the saved state could not be restored. I attempted to delete the saved state, but received an "Unable to perform operation" error. Additionally, I could not view the properties of the virtual machines in the Saved-Critical state.

I opened up Failover Cluster manager and noticed that the storage was now owned by the other cluster node. The problem is now obvious--the storage is no longer being presented to the host on which I was configuring the virtual machines. So, how to move these disks? Since the disks are marked "Available" (not having run through the new service Failover Clustering wizard to make the virtual machines highly available), I can't move them by right-clicking on them and selecting a new node.

There is another solution, however, and it involves the cluster.exe command.

1. Open up a command prompt.
2. Run the following command: cluster.exe GROUP "Available Storage" /Move

Voila! The disks moved back to the other node and the virtual machine state in Hyper-V manager changed from "Saved-Critical" to "Off." I was then able to cluster the machines normally through Failover Clustering.

Thursday, April 16, 2009

Erratic or Negative Ping Times in Hyper-V Guests

A customer approached me with a some puzzling issues. They noticed a bunch of 1053 and 1054 Userenv errors in their event logs on their virtual machines. 1053 and 1054 error messages have very similar wording:

1053 - Windows cannot determine the user or computer name. (description>). Group Policy processing aborted.
1054 - Windows cannot obtain the domain controller name for your computer network. (). Group Policy processing aborted.

Typically, these are related to DNS. In this instance, however, the customer also presented some other interesting issues--negative ping times or very high ping times (in excess of 5000ms). And, to boot, these erratic ping times were only present on virtual machines with two or more virtual processors.

So, what's the relationship?

It turns out to be a relatively simple explanation.

In order for a Group Policy client to be qualified to process GPOs, AD measures the RTT time between the client and the DC processing the logon and group policy request. If the average RTT is > 10ms for 2048 byte packets, the link is generally considered "slow" by default (another value can be configured). Under "slow" conditions, group policy will not process. I’ve seen this issue before in environments where authentication is happening over a WAN (where times are greater than 10ms) or with routers dropping or fragmenting large ICMP packets (affectionately known as “blackhole router syndrome”).

So, if a machine is reporting a 5000ms ping, it stands to reason that the OS might think that the link is indeed slow.

As previously mentioned, this problem is occurring on Windows 2003 hosts that are configured for multiple virtual processors (VPs). All operating systems use some sort of clock timing mechanism, and frequently they rely on the Time Stamp Counter (TSC), which counts CPU ticks since system start. Each processor has its own TSC, and the TSC for each processor can be different because they’re not necessarily synchronized. What this ends up meaning is that if a VM is reading the TSC from multiple VPs, the date stamps may actually go backwards or be out of order. This does not happen in a single CPU scenario (physical or virtual), since only one TSC is being used.

The three possible workarounds:
1. Upgrade to Windows 2008. Obviously, this won't work for everyone, so for those people, workarounds two or three should provide some relief.
2. Shut down the VM, change the number of CPUs to 1 in Hyper-V manager and then start the VM.
3. Add the /usepmtimer switch to the boot.ini configuration of each Windows 2003 server using multiple processors. In the physical world, this phenomenon only appears to only happen on AMD processors. The VM world is less discriminating against processor type. Windows 2003 SP2 normally is supposed to use the ACPI Power Management Timer (PM Timer), as long as the BIOS check for it succeeds. In the case of Hyper-V, the BIOS check fails, so it falls back to the TSC. Remember, modifying the boot.ini requires a reboot for the change to become effective.

The Win32 API call, QueryPerformanceCounter, uses the TSC by default. Adding the /usepmtimer boot.ini flag tells QueryPerformanceCounter to use the ACPI/PM timer.

Related information:

Wikipedia - Time Stamp Counter

A Windows Server 2003-based server may experience time-stamp counter drift if the server uses dual-core AMD Opteron processors or multiprocessor AMD Opteron processors

Programs that use the QueryPerformanceCounter function may perform poorly in Windows Server 2000, in Windows Server 2003, and in Windows XP

Explanation for the USEPMTIMER switch in the boot.ini

Windows Server Performance Team Blog : Hyper-V and Multiprocessor VMs

Negative ping times in Windows VM's - what's up?

How a slow link is detected for processing user profiles and Group Policy

How to enable user environment debug logging in retail builds of Windows

Available switch options for the Windows XP and the Windows Server 2003 Boot.ini files

Sunday, November 9, 2008

VHDMount for Hyper-V

Even though Microsoft has been my bread and butter for the last 12 years, I can't help but think so many of the things they do aren't fully thought through from the end user perspective.

Take this one, for example. In Virtual Server 2005 R2, Microsoft included an incredible tool called VHDMount.exe. The end result--you can mount a VHD as a volume, and read/write against it. A few quick reg edits (or the reg script provided by the Virtual PC guy) and you could get context mount/dismount from within Windows Explorer. Sweet, right?

One would think that such a useful utility would have followed on to Windows 2008 and Hyper-V, where the hypervisor is built into the OS, right? One would think so, but then they'd apparently be the only one. Yes, this feature is gone. How could it happen?

So, I stumbled upon a couple of scripts that invoke the Hyper-V tools to do just that. Please give some mad props to Ravi Chaganti, who figured it out.

http://www.ravichaganti.com/blog/?p=77 - Right click to Mount & Dismount Hyper-V VHD
http://www.ravichaganti.com/blog/?p=81 - Script Update: Right click to Mount & Dismount Hyper-V VHD

You'll want to visit both blog posts. The first post has a ZIP that includes a REG file for exposing a context menu in Windows Explorer for mounting/dismounting. The second post has an update to the script.