Since the recent release of ESX 3.5 Update 2 and Veeam Backup 2.0, both featuring Microsoft Volume Shadow Copy Service (VSS) support, we’ve been getting many questions from our customers asking why this feature is needed.

It’s true that the whole VSS support issue around VMware disaster recovery solutions created a lot of confusion due to each vendor having different opinions about the usefulness of this feature, as well as different implementation approaches, with some of them being quite questionable. So I decided to perform some testing on real applications to investigate whether VSS support is really required for a disaster recovery solution, and what VSS support implementation approaches are the most correct at this moment.

For my testing, I used one of the most common mission-critical applications, an Active Directory domain controller. To make a long story short, here’s the summary table for my testing results:

For the testing, I used my test lab containing a few clean domain controllers. I’ve chosen one domain controller (DC1) to perform all the testing on, and performed its backup of a live domain controller with the different VMware disaster recovery solutions listed in the table above. For all the solutions supporting VSS integration, I performed the backup with that option enabled.

As soon as I finished creating the backups, I switched to my test DC, created a few test users there to simulate post-backup activity, verified that the test users were replicated over to the other DC successfully, and crashed my test DC. Here’s a short video for this step.

At this point, I shut down the remaining domain controller, and created a copy of the whole lab so that I could test recovery for all solutions in similar conditions. After testing recovery with each solution, I rolled the whole lab backup to this state.

Recovery testing showed that in the case of Veeam Backup 2.0, and the latest VMware Consolidated Backup, the recovered DC was fully functional.

One thing I noted, however, is that with VCB, the domain controller did not start up in the recovery mode during the first boot, as it did with Veeam Backup 2.0. According to Microsoft documentation however, when performing a VSS-integrated domain controller restore, the system must be rebooted in Directory Services Restore mode when Active Directory is running on the server (which is exactly our case). To my understanding, booting in this mode is required so that the NTDS.DIT file is not locked with Active Directory services, antivirus or other applications when the shadow copy restore is performed. So I don’t know whether or not this domain controller restore approach is supported by Microsoft.

This video demonstrates the DC recovery process using the most correct VSS-integrated recovery implementation, as provided by Veeam Backup 2.0.

With all the other solutions I have tested (including vRanger Pro, which was originally the first to claim having VSS support), the recovered DC was not functional and was put into the condition known as an update sequence number rollback, or USN rollback. The only way to recover a DC from rollback is to forcibly demote the domain controller, and reinstall it. Luckily, I had my lab fully preserved, so instead I could simply rollback the entire Active Directory.

This video demonstrates the DC recovery using a solution not featuring correctly implemented VSS support.

As you can see, some applications cannot be restored correctly by simply starting up the VM image, even when VSS is leveraged to perform the backup. Some applications, especially those featuring replication, require a certain sequence of actions to be restored from a backup made by leveraging VSS. Similar to the domain controller that I used to perform my testing, Microsoft Exchange Server is another example of a mission-critical application that must be restored using an application-specific restore technique (refer to the following support KB article for more information about VSS-integrated backup and restore of Microsoft Exchange server).

If you ask me why I am the first one to bring this issue up - I don’t know. Could it be simply because no one ever tried to actually restore VMs to the production environment from their backups? I can understand how this type of issue could be overlooked in a small test lab setting, where typically only one DC is installed. But before you put your VMware backup solution into production – give some serious thought to the recoverability of the backups it produces.

For more detailed information on correctly using VSS in VMware environments, please read the “VMware and VSS: Application Backup and Recovery” white paper available at Veeam Backup product page.