What you'll learn in this tip: Hyper-V backups bring a whole new troubleshooting learning curve to ensure consistent and reliable recovery points for your virtual machines. Follow these common Hyper-V troubleshooting steps to give you a starting point to quickly solve or head off Hyper-V backup issues when they arise.
The most common and efficient way to back up your virtual machines (VMs) is through host-level backups where the Hyper-V-utilized Volume Shadow Service (VSS) writer on the host coordinates the state of all the VMs, ensuring the data within each VM is in a consistent state before the data is backed up. This requires a VSS-aware backup application like System Center Data Protection Manager (DPM). However, many Hyper-V backup problems pop up due to the number of components that are affected in this process. Below are some of the common problems and solutions at both the VM level and the host level that will make your Hyper-V backups more successful.
Virtual machine troubleshooting
A failure inside the virtual machine can make the entire backup process fail. Below are some of the most common troubleshooting tips from within the virtual machine that will allow for a successful hand off back to the host to complete the virtual machine backup.
Free up disk space: Each VM that is backed up through host-level backups needs to have a minimum of 300 GB of free space on each virtual hard disk (VHD). This space is necessary for the Volume Shadow Copy process that takes place inside the VM to succeed. This call within the VM is requested by the host Hyper-V VSS writer. After the VM has completed its VSS process, ensuring a consistent state of the VM data, the host then proceeds with the backup process by accessing this stateful data, often called quiesced data. The process will fail if this space is not available.
Solution: Free up space on the VHD with low disk space, by either deleting files or expanding the VHD.
Integration components: Integration components are a set of agents installed inside supported VMs that provide peak performance and direct interaction with the host server hypervisor. They assist in timekeeping, OS shutdown, heartbeat, guest to host data exchange, disk I/O, network I/O performance enhancements, and VSS-assisted backups. It's important to have the correct integration components installed on all supported VMs. Having mismatched or down level integration components will cause inconsistent backup operations.
Solution: Utilize the latest integration components. Understand that upgrades or updates to your backup software may require integration component upgrades on all the virtual machines that are on a particular backed up host.
No VSS writers available in the VM: Sometimes backups for a single VM will continue to fail. If it is not a lack of disk space on the VM and you are running the same integration components as other functioning VMs, then the VSS writers on the VM may be in an inconsistent or corrupt state. To see this, run this command line: Vssadmin list writers. If the command returns with no data, you have this issue.
Solution: Restart the “Hyper-V Volume Shadow Copy Requestor” service on the affected VM and run the command again. You should now see all the VSS writers. Check the status of each writer in the output. They should all say OK.
Patches: Above and beyond staying current with Microsoft security patches, there are specific hotfixes addressing non-security issues that correct particular bugs counterproductive to successful backups. For Windows Server 2003 VMs, there are some inconsistent results in using VSS host-level backups that can be solved by a particular hotfix.
Solution: Install VSS Rollup Package (KB940349) and reboot.
With host-level backups the host is responsible for the coordination of the backup process. A failure at this level will cause all VM backups to fail. Luckily, there are steps that can easily be taken to solve most of these difficulties.
Hyper-V VSS writer bad status: Host-level backup failures differentiate themselves by the scale of the failure to backup VMs. What you will usually see is that most, if not all, VMs on a particular host failed during the last backup process. This could mean that you have many VMs with the incorrect integration components, but it most likely points to an issue with the host Hyper-V VSS writer status. To determine this, run the following command from a command prompt: vssadmin list writers status. This will commonly output a status of “Failed” on the Hyper-V VSS writer and will be accompanied by errors in the event log showing the failure.
Solution: Restart the “Hyper-V Virtual Machine Management” (VMMS.exe process) service. Wait a minute or two before rechecking your writer status or trying your backup.
Host-level backups may hang at 0 MB backed up indefinitely: My experience with this issue is with System Center Data Protection Manger. It points to a failed Hyper-V VSS writer on a Hyper-V host. This usually will hang across all VMs on a particular Hyper-V host.
Solution: Track down the offending host and restart the “Hyper-V Virtual Machine Management” service. DPM should recognize the corrected status change in the Hyper-V VSS writer and resume backups of the previously hung backup jobs.
Host hotfix: For Windows Server 2008 R1 and R2 with Hyper-V, there is a bug where after each VSS-based backup, orphaned VMBus devices are left behind. As more backups are run, this causes your registry to grow. As a result both VSS and system stability are affected.
Solution: For Windows Server 2008 R2, install KB982210 or SP1 when it comes out. For Windows Server 2008 R1, you can call Microsoft support for the DevNodeClean utility, go to Byte Solutions and use their compiled DevNodeClean utility, or use DeviceRemover to clean out the orphaned devices. More information on all these utilities and how to use them to solve VSS orphaned devices can be found at VirtuallyAware.com.
Overall, Hyper-V backups and backing up virtual machines is a critical part of protecting your environment from disaster, but like backing up physical servers, it still requires some troubleshooting to ensure your data is protected. Confirming your backups are working correctly on a daily basis is key to a successful restore, and in the end, Hyper-V backups truly shine because they save time vs. physical servers. Have any troubleshooting tips that have worked for you? Email me at Rob@VirtuallyAware.com
About the author: Rob McShinsky is a senior systems engineer at Dartmouth Hitchcock Medical Center in Lebanon, N.H., and has more than 12 years of experience in the industry -- including a focus on server virtualization since 2004. He has been closely involved with Microsoft as an early adopter of Hyper-V and System Center Virtual Machine Manager 2008, as well as a customer reference. In addition, he blogs at VirtuallyAware.com, writing tips and documenting experiences with various virtualization products.