New White Russian on the Block

It's been a while since I've posted a new recipe (or a twist on an old one). Out of one the normal ingredients (whole milk or cream), I substituted and came up with this delightful blend:

New White Russian
2 oz Kahlua Coffee Liqueur
1/2 oz vodka
1 oz Bailey's Irish Cream
2 oz Silk Soy Milk

Combine all ingredients in a shaker with ice. Shake a couple of times, but not too much; it's not supposed to be a frothy drink. Strain into a double old-fashioned glass with a few cracked ice cubes.

It's creamy, coffee goodness. And, with Silk, it's a little less of a calorie hit than cream or whole milk. ;-)

Because you know me. Concerned about health and all.

Using WinDbg for Quick Memory Dump Analysis

Blue screens are no fun. Trying to resolve them without the proper tools can be even less fun.

In my experience, a large percentage of blue screens are the result of some poorly-tested or incompatible third-party device drivers. For the desktop crowd, a round up of the usual suspects includes scanner, printer, and video drivers. On the server end of things, the most likely culprits are usually backup/continuous data protection filter drivers or printer drivers.

All standard troubleshooting questions apply in either case:
- Has any new hardware been installed?
- Has any software recently been installed (either new applications or patches)?
- Have any existing device drivers been updated?
- Can you reproduce the conditions that cause the blue screen (for example, under heavy load conditions or during a backup window)

Take this case. I recently received a dump file from a server that had crashed and recovered overnight. To analyze the dump file, head on over to the microsoft.com site and get the appropriate debugging tools for your platform (x86 or x64/ia64).

In my case, I needed the 32-bit debug package. I downloaded and installed it, and then ran C:\Program Files\Debugging Tools for Windows (x86)\windbg.exe.



Before we can make any progress, we should grab the Windows symbols, which will allow the debugger to go through the crash dump and identify components.

Make a directory on your local computer, such as C:\Symbols. From inside WinDbg, go to File > Symbol File Path.



In the dialog box, type in

SRV*c:\symbols*http://msdl.microsoft.com/download/symbols

and click OK. This will instruct WinDbg to contact the Microsoft symbols server and download the parts that you need and store them in C:\symbols.



Now that the symbols are configured, click File > Open Crash Dump.



Browse to your memory dump file, and select it. WinDbg will process it, and should return something like this:



At this stage, WinDbg has identified vsp.sys as a likely source of the problem. Type !analyze -v in the text box at the bottom and hit enter.

WinDbg will process a bit more and return some (hopefully) useful information.



The key area to look at is the "DEFAULT_BUCKET_ID," which, in this case, says "DRIVER_FAULT_SERVER_MINIDUMP." Browsing through the dump file, you can see that the system ran out of PTEs and subsequently crashed.

Having worked with NetBackup for a number of years, I recognized vsp.sys immediately as part of NetBackup. However, if you want to try to figure out more from the dump file, typing the command lmv will list the loaded modules. After it's done listing the images, press Control-F to and enter the driver that was listed as faulting.



Unfortunately, the dump file didn't have the full path to the loaded driver, so we've hit a little bit of a wall.

In this instance, the faulty driver (vsp.sys) is part of the Advanced Open File option for the Veritas NetBackup client. We upgraded the NetBackup agent to the latest version and all is well again.

Good luck!

Windows 2008 Server Backup

This is not your father's backup. This is not your backup. Frankly, this should be nobody's backup.

Recently, I found my self in need of having to make a quick snapshot of a physical server's configuration before making some complex configuration changes.

So, I hit the Windows Key + R, and typed the tried-and-true ntbackup. No dice. What? Where did this go?!

It's not there anymore. Instead, we now have "Windows Server Backup," a rewrite that adds a lot of new features (backup files are stored as a VHD, which is pretty neat), but they come with a price--an unintuitive and complex command line.

After performing my initial backup without incident, I made my configuration changes, but eventually found myself in need of restoring from backup. I entered the Windows Server Backup console and went to restore, but was dutifully informed that if I wanted to restore the System State, I would have to do it from the command line.

So, I launch the command-line tool, wbadmin.

C:\Users\Administrator>wbadmin
wbadmin 1.0 - Backup command-line tool
(C) Copyright 2004 Microsoft Corp.


ERROR - Command incomplete. See list below.
For more help, type wbadmin -help

---- Commands Supported ----

ENABLE BACKUP -- Enable or modify a scheduled daily backup
DISABLE BACKUP -- Disables running scheduled daily backups
START BACKUP -- Runs a backup
STOP JOB -- Stops the currently running backup or recovery
GET VERSIONS -- List details of backups recoverable from a
specific location
GET ITEMS -- Lists items contained in the backup
START RECOVERY -- Run a recovery
GET STATUS -- Reports the status of the currently running job
GET DISKS -- Lists the disks that are currently online
START SYSTEMSTATERECOVERY -- Run a system state recovery
START SYSTEMSTATEBACKUP -- Run a system state backup
DELETE SYSTEMSTATEBACKUP -- Delete system state backup(s)

Seeing this, I discover what I believe is the appropriate option--START SYSTEMSTATERECOVERY, since the System State is what I'm trying to recover.

So, I run wbadmin START SYSTEMSTATERECOVERY, but ... then I get another screen.

C:\Users\Administrator>wbadmin START SYSTEMSTATERECOVERY
wbadmin 1.0 - Backup command-line tool
(C) Copyright 2004 Microsoft Corp.


ERROR - An option required for command is missing: Version. See usage below.

Usage: WBADMIN START SYSTEMSTATERECOVERY
-version:VersionIdentifier
-showsummary
[-backupTarget:{VolumeName | NetworkSharePath}]
[-machine:BackupMachineName]
[-recoveryTarget:TargetPathForRecovery]
[-authsysvol]
[-quiet]

Runs a system state recovery based on the options specified.

-version Version identifier of the backup in MM/DD/YYYY-HH:MM format,
as listed by WBADMIN GET VERSIONS.

-backupTarget Specifies the storage location that contains the backups for
which you want to do the recovery. Useful when the backups are
stored in a different location than the normal location for
backups of this computer.

-machine Specifies the name of the computer for which you want to do the

recovery. Useful when multiple computers have been backed up to

the same location. Should be used when -backupTarget is
specified.

-recoveryTarget Existing directory path to restore to. Useful if the restore
is to be done to an alternate location.

-authsysvol Perform an authoritative restore of SYSVOL

-quiet Runs the command with no user prompts.

-showsummary Reports the summary of the last run of reboot status of last
successful online system state recovery. This option cannot be
accompanied by any other options.

Examples:
WBADMIN START SYSTEMSTATERECOVERY -version:03/31/2005-09:00
WBADMIN START SYSTEMSTATERECOVERY -version:04/31/2005-09:00 -backupTarget:\\serv
ername\share
-machine:server01

C:\Users\Administrator>

Ug. How do I discover the version that I need to restore? Scrolling back through the list reveals another option--GET VERSIONS.

C:\Users\Administrator>wbadmin GET VERSIONS
wbadmin 1.0 - Backup command-line tool
(C) Copyright 2004 Microsoft Corp.

Backup time: 3/14/2009 6:01 PM
Backup target: Network Share labeled \\server\c$\Software
Version identifier: 03/14/2009-22:01
Can Recover: Volume(s), File(s), Application(s), Bare Metal Recovery, System State

Ahh. The version is 03/14/2009-22:01--the time I made the backup.

So, I now run:

C:\Users\Administrator>wbadmin start systemstaterecovery -version:03/14/2009-22:01
wbadmin 1.0 - Backup command-line tool
(C) Copyright 2004 Microsoft Corp.

Do you want to start the system state recovery operation?
[Y] Yes [N] No

Finally. Progress. I Enter "Y" and then see my progress...

Starting System State Restore [3/15/2009 11:31 PM]
Processing files to restore (This may take a few minutes)...
Processed (1279) files
Processed (5733) files
Processed (11025) files
Processed (21513) files
Processed (34191) files
Processed (47192) files
Processed (60265) files
Processed (62310) files
Processed (62310) files
Processing of files complete
Starting restore of files from backup
Preparing for Restore...
Restore of files reported by 'COM+ REGDB Writer' completed
Overall progress - 1% (Currently restoring files reported by 'System Writer')
Overall progress - 3% (Currently restoring files reported by 'System Writer')
Overall progress - 4% (Currently restoring files reported by 'System Writer')
Overall progress - 6% (Currently restoring files reported by 'System Writer')
Overall progress - 7% (Currently restoring files reported by 'System Writer')
Overall progress - 9% (Currently restoring files reported by 'System Writer')
Overall progress - 11% (Currently restoring files reported by 'System Writer')
Overall progress - 13% (Currently restoring files reported by 'System Writer')
Overall progress - 15% (Currently restoring files reported by 'System Writer')
Overall progress - 18% (Currently restoring files reported by 'System Writer')
Overall progress - 19% (Currently restoring files reported by 'System Writer')
Overall progress - 21% (Currently restoring files reported by 'System Writer')
Overall progress - 24% (Currently restoring files reported by 'System Writer')
Overall progress - 26% (Currently restoring files reported by 'System Writer')
Overall progress - 28% (Currently restoring files reported by 'System Writer')
Overall progress - 30% (Currently restoring files reported by 'System Writer')
Overall progress - 32% (Currently restoring files reported by 'System Writer')
Overall progress - 36% (Currently restoring files reported by 'System Writer')
Overall progress - 40% (Currently restoring files reported by 'System Writer')
Overall progress - 43% (Currently restoring files reported by 'System Writer')
Overall progress - 46% (Currently restoring files reported by 'System Writer')
Overall progress - 49% (Currently restoring files reported by 'System Writer')
Overall progress - 52% (Currently restoring files reported by 'System Writer')
Overall progress - 54% (Currently restoring files reported by 'System Writer')
Overall progress - 57% (Currently restoring files reported by 'System Writer')
Overall progress - 61% (Currently restoring files reported by 'System Writer')
Overall progress - 64% (Currently restoring files reported by 'System Writer')
Overall progress - 68% (Currently restoring files reported by 'System Writer')
Overall progress - 71% (Currently restoring files reported by 'System Writer')
Overall progress - 73% (Currently restoring files reported by 'System Writer')
Overall progress - 76% (Currently restoring files reported by 'System Writer')
Overall progress - 79% (Currently restoring files reported by 'System Writer')
Overall progress - 81% (Currently restoring files reported by 'System Writer')
Overall progress - 84% (Currently restoring files reported by 'System Writer')
Overall progress - 85% (Currently restoring files reported by 'System Writer')
Overall progress - 88% (Currently restoring files reported by 'System Writer')
Overall progress - 91% (Currently restoring files reported by 'System Writer')
Overall progress - 95% (Currently restoring files reported by 'System Writer')
Overall progress - 98% (Currently restoring files reported by 'System Writer')
Cleaning up...

Summary of recovery:
--------------------

Restore of system state completed successfully [3/15/2009 11:40 PM]

Log of files successfully restored
'C:\Windows\Logs\WindowsServerBackup\SystemStateRestore 15-03-2009 23-31-27.log'


Please restart the machine to complete the operation.
NOTE: When you restart your server, System State Recovery will attempt to
recover many system files which may take several minutes to complete depending
on the number of files that are getting replaced. The machine might reboot multi
ple times in the process. Please be patient and do not interrupt the reboot process.

C:\Users\Administrator>

So this is progress? Better than selecting it through the familiar interface? I must be getting old.

Dell DRAC Virtual Media Plugin Problems

Symptoms:
- Trying to access the "Virtual Media" section of a DRAC console
- Client is Windows 2008
- When accessing "Virtual Media," get prompted to accept ActiveX control; accept, and get redirected to login page

Solution:
- Open Internet Explorer, Tools > Internet Options; select Intranet, and select "Automatically detect Intranet network"


- Open %systemroot%\system32\drivers\etc\hosts
- add hosts entry for DRAC (such as 10.1.1.1 server-drac server-drac.domain.com)
- Save and close

Browse to https://server-drac, accept the warning that IE is trying to access a page on your intranet and login. Good to go.

So, what causes this problem?

Internet Explorer's security settings are preventing ActiveX controls from being installed. Once you add the DRAC interface you're trying to work with to what IE considers your local intranet (site able to be browsed by hostname plus local domain suffix), the restrictions are loosened.

Happy DRAC'ing!

If I Could Turn Back Time

For anyone who manages an Active Directory environment, you've undoubtedly come to understand the need for keeping time in sync. In short, you won't be able to log in if the local computer time has more than 5 minutes difference than the logon server.

By default, Windows machines are configured to use NTDS5 or domain hierarchy time synchronization. The idea is that clients and servers synchronize their time with domain controllers, and the domain controllers synchronize their time with the domain controller holding the PDCE FSMO role. The idea is that administrator is supposed to configure the PDCE role holder to synchronize time with an outside NTP source.

However, you may someday find yourself in an environment where "things were changed." Unwitting administrators may have configured their environment differently, and if your luck is anything like mine, you'll eventually find yourself inheiriting or troubleshooting such an environment.

To reset a Windows Server 2003 computer back to default NTDS5 synchronization, run the following command:

w32tm /config /syncfromflags:domhier & w32tm /config /update

"But, Aaron," you say, "I have hundreds of servers!"

No worry, there's help for you as well. Grab the free PSTools from http://www.microsoft.com/sysinternals. Inside you'll find the Windows Administrator's best friend--psexec. With that great tool, you can run any command against a list of servers.

In addition, you'll want to grab a copy of the Gnu32 core utilities, which has Win32 versions of the great *nix utilities. Windows prettymuch sucks at command-line text manipulation, which makes this toolset invaluable.

So, armed with the right tools, how to get that list?

From either a domain controller (or a server/workstation with the AdminPak, PSTools, and Gnu32 Core Utilities installed):

1. ldifde -f temp1.txt -d "dc=domain,dc=com" -r "(&(objectClass=computer)(operatingSystem=Windows Server*))" -l cn
2. findstr /i "cn: " temp1.txt tr -d "cn :" >> temp2.txt

The resulting file (temp2.txt) will contain all of the Windows Server 2003 and 2008 machines in your environment. Once you have that list:

psexec @temp2.txt w32tm /config /syncfromflags:domhier & w32tm /config /update

The last step is to configure the server holding the PDCE FSMO role to use an external NTP time source.

w32tm.exe /config /syncfromflags:manual /manualpeerlist:time.windows.com /reliable:yes & w32tm.exe /config /update

Replace time.windows.com with whatever NTP server you want to use. Make sure you can get to it on UDP 123.

And ... you're done.

Now You See It, Now You Don't

Lots of programs on the internet will claim to "erase your tracks" on your Windows-based computer. Well, you don't need to waste your money. Windows XP and later come with a built-in utility, that when used correctly, will overwrite the free space on your hard drive with ones and zeroes.

To make a long story short, when a file is deleted, the data blocks for the file still exist on the hard drive. The blocks that the deleted file occupy have been marked "available" for use by the filesystem. Recovering deleted data depends on these data blocks not having been overwritten yet.

Every act of creating, copying, or saving data on the hard drive with deleted data increases the chances that those "maked" data blocks will be overwritten, eliminating chances for a full, successful recovery.

If you want to ensure that data you've deleted is not going to be recoverable, you may think that simply formatting a drive will wipe the data. However, several types of "quick" formats may only specify the filesystem type and wipe the MFT or FAT, leaving the data blocks still intact on the drive.

To totally "erase" your data, you'll need to actually write data to every free region of your hard drive.

Now, as I previously mentioned, there are plenty of companies that will sell you programs to wipe your NTFS or FAT16/FAT32 filesystems--but they're not counting on you using a built-in utility to do the task.

What is this marvelous utility, you ask?

The utilitiy is cipher.exe. The main purpose of the utility is for EFS encryption. However, one of the switches can wipe the free space.

Running cipher /? at the command prompt in Windows 7 shows a whole host of options (in fact, too many to display here). The option we're looking for is the /W option:

C:\>cipher /?
Displays or alters the encryption of directories [files] on NTFS partitions.
...
CIPHER /W:directory
...
/W Removes data from available unused disk space on the entire
volume. If this option is chosen, all other options are ignored.
The directory specified can be anywhere in a local volume. If it
is a mount point or points to a directory in another volume, the
data on that volume will be removed.

Cipher with the /W switch will create a temporary file that will write over every block on your filesystem.

The appropriate syntax:

cipher /W:C:\

Will yield a screen like this:



As the screen indicates, close as many programs as possible. To achieve maximum data clearing effect, try grabbing a program like CCleaner to remove temporary internet files, temporary setup files, cookies, and a bunch of other personal data before wiping with cipher.

A Purple Nasty, Snake-Bitin' Good Time

I was introduced to this fine UK collegiate-style drink this weekend by my sister-in-law's most excellent boyfriend, Eddie. It's quite tasty, and definitely sneaks up on you. It's made by mixing a dry cider and a lager.

Depending on your lager choice, you may need to add more cider to gain the desired level of sweetness.

Snake Bite
5 oz Stella Artois Belgian Lager
5 oz Strongbow Cider

Combine ingredients in a Pilsner glass and enjoy.

Adding black currant turns it into a Snake Bite Black.

Snake Bite Black
5 oz Stella Artois Belgian Lager
5 oz Strongbow Cider
2 oz Black Currant

Change the ratio a bit, and it becomes a Purple Nasty.

Purple Nasty
6 1/2 oz Strongbow Cider
3 1/2 oz Stella Artois Belgian Lager
2 oz Black Currant

This is supposedly the drink of Loughborough University in England.

However you mix it and whatever you call it, it's well worth trying.