vss

The Great VSS Debate

Jan 7th

Posted by vmdoug in Veeam Backup & Replication

21 comments

UPDATE (January 11, 2010): In an effort to help clarify things, Curtis Preston has blogged about Hyper-V and VMware’s support/use of VSS on his Backup Blog. He also provides a good explanation of VSS. I recommend checking out these resources and you may also want to consider following Curtis on Twitter.

Original Post:

This morning my inbox greeted me with an email from Eric Siebert and my Twitter search folder filled with statements about Veeam’s VSS claims. After my first cup of coffee I started to dig in and see what in the virtual world was going on…this first lead me to a blog post by EMC’s Scott Waterhouse where he states the following:

Therefore, final answer: if you need application consistent backups, you must do a guest level backup. An image level backup is simply not good enough. Even with a MS Windows 2003 or higher guest, even though VMware supports VSS. Yes you may do image level backups too, but they will only complement, no replace, guest level backups.

To be honest Scott was talking about VMware VSS here. I still don’t agree with his statement that for application consistent backups you have to do a guest level backup because there ARE solutions on the market today that provide application consistent backups from an image…of course that would be Veeam Backup & Replication.

What prompted some of the discussion (as far as I can tell) is that people don’t realize that Veeam uses its own VSS driver, not VMware’s. This has been the case since Veeam Backup and Replication 2.0, released in July of 2008. In fact, at the time, we pointed out how our VSS integration was different from the competition through a couple of blog posts: Is your backup really VSS aware? and VSS and VMware ESX: What your VMware backup vendor isn’t telling you. These posts really talk about VSS recovery and making sure that your VSS “backup” is truly able to recover properly. These 2 posts continue to be very popular on this blog…maybe pointing out that there is confusion regarding proper VSS handling when it comes to VMware.

That was all in 2008, what’s going on with VSS in 2010? Well, Veeam has continued to update its VSS integration to keep pace with the new releases of Windows Server 2008, both x86 and x64. Not all vendors have been keeping up with Microsoft though, in fact, looking at VMware’s VCB/VSS support table, it clearly states that they don’t support application consistency on Windows Server 2008, only file-level. So, any virtualization backup vendor that relies on VMware’s SYNC Driver and VSS for file and application consistency is not giving true VSS support for Windows Server 2008 (as Scott pointed out in his blog).

I’d like to finish up by answering Scott’s call for documentation:

Unfortunately, nobody has been able to provide a piece of definitive technical documentation (a white paper, or a support document, or relevant piece of text from an administration guide) that clearly describes the issue.

First, we have real users, using Veeam’s VSS today, that are talking about it in our forums, this is proof that they’re getting application consistent backups from Veeam’s image level process using our VSS:

Postby tsightler » 05 Jan 2010 14:22

Once again, Veeam fully support VSS aware snapshots of both AD and Exchange server when using the Veeam VSS Agent. Veeam doesn’t just “take a VM copy”, the Veeam VSS agent uses Windows VSS services to put these features into a proper, supported VSS backup state prior to taking the VM snapshot. In other words, a Veeam backup is indeed a “backup-aware copy of the Information Store and NTDS database”, and it uses the Windows recommended VSS processes to achieve this.

Postby donikatz » 06 Jan 2010 00:00

Obviously neither Tom nor Anton need my help here, but maybe some real-world testimony would make you feel more comfortable? Not only have I tested this, I’ve performed a *production* restore of a w2k3 DC with Veeam and it worked exactly and as simply as in the video. I’ve also done several *production* SQL restores without issue. Veeam also works well in our Exchange restore tests, although we haven’t had to do any in production (knock on wood). Although agent-based apps like Backup Exec may have more direct hooks for simpler granular restore (we still use BE for Exchange brick-level restores because our admins are more familiar with the process), Veeam is more than capable without the drawbacks of an agent. I hope to move away from BE altogether for Exchange this year; it’s just a matter of updating our runbook and training. Honestly, if there’s one area you certainly don’t need to lose sleep over with Veeam, it’s with Microsoft products. MS has well-proven APIs and Veeam makes great use of them; Veeam VSS is excellent. Heck, if only Oracle on Linux had VSS the way it does on Windows it would make my life a lot easier… ;)

Next, I’ve taking some quotes from Veeam’s own user guide regarding the differences between using VMware tools quiescence (SYNC) and Veeam’s VSS driver:

Transactionally Consistent Backup

Veeam Backup & Replication 4.0 provides two techniques for creating transactionally consistent backup images — the Enable VMware tools quiescence and Enable Veeam VSS integration options. In contrast to restoring a crash-consistent backup, which is essentially equivalent to rebooting a server after a hard reset, restoring transactionally consistent backups ensures safety of data of applications running on VMs.
Please note that when you select both VSS integration and VMware tools quiescence options for a job at the same time, the VSS module will only be used for processing backed up and replicated VMs. However, if you use both VSS and VMware tools quiescence options and select the Continue backup even if Veeam VSS quiescence fails option for backup jobs or the Continue replication even if Veeam VSS quiescence fails option for replication jobs, all your VMs will be processed with VSS first, and in case of VSS failure (e.g., Linux VMs), VMs will be processed with the VMware tools quiescence option enabled.
This can be very useful when you have both Windows- and Linux-based VMs in one job, so all VMs will be processed in a transactionally consistent way using VSS or VMware tools quiescence option.

Additionally, we then go on to explain the VSS process as well as the systems supported by our VSS driver:

Enable VSS Integration

With the Enable VSS integration option selected, Veeam Backup & Replication 4.0 utilizes the Windows Volume Shadow Copy Service (VSS) that ensures consistent backup of VSS-aware application running within your virtual machines (domain controllers, databases and other applications) without shutting them down. The Enable Veeam VSS integration option allows creating a transactionally consistent backup image of a VM, which, in contrast to a crash-consistent backup image, ensures successful VM recovery, as well as proper recovery of all applications installed on the VM without any data loss.
In the process of its work, VSS freezes all I/O at a specific point-in-time by interfacing with all VSS-aware applications and the Windows operating system. Consequently, there remain no unfinished database transactions or incomplete application files. Such backups, when restored correctly, result in fully functional applications.
The VSS works with Windows 2003, Windows XP, Windows 2008, Windows 2008 R2 and Windows 7 guest operating systems. Use VSS to back up 32-bit or 64-bit version of Windows 2003, 32-bit version of Windows XP guest OS, 32-bit and 64-bit versions of Windows 2008. Please note that administrator credentials are required to access the guest OS. Microsoft Windows VSS backup option requires that your guest OS has VMware Tools, and all the latest service packs and patches installed.

If anyone has any questions or further thoughts I’d love to hear from you. Feel free to comment below, hit me up on Twitter or shoot me an email @ doug /dot/ hazelman /at/ Veeam /dot/ com

, , , ,

VSS and VMware ESX: What your VMware backup vendor isn’t telling you

Aug 11th

Posted by Gostev in backup

7 comments

Since the recent release of ESX 3.5 Update 2 and Veeam Backup 2.0, both featuring Microsoft Volume Shadow Copy Service (VSS) support, we’ve been getting many questions from our customers asking why this feature is needed.

It’s true that the whole VSS support issue around VMware disaster recovery solutions created a lot of confusion due to each vendor having different opinions about the usefulness of this feature, as well as different implementation approaches, with some of them being quite questionable. So I decided to perform some testing on real applications to investigate whether VSS support is really required for a disaster recovery solution, and what VSS support implementation approaches are the most correct at this moment.

For my testing, I used one of the most common mission-critical applications, an Active Directory domain controller. To make a long story short, here’s the summary table for my testing results:


For the testing, I used my test lab containing a few clean domain controllers. I’ve chosen one domain controller (DC1) to perform all the testing on, and performed its backup of a live domain controller with the different VMware disaster recovery solutions listed in the table above. For all the solutions supporting VSS integration, I performed the backup with that option enabled.

As soon as I finished creating the backups, I switched to my test DC, created a few test users there to simulate post-backup activity, verified that the test users were replicated over to the other DC successfully, and crashed my test DC. Here’s a short video for this step.


At this point, I shut down the remaining domain controller, and created a copy of the whole lab so that I could test recovery for all solutions in similar conditions. After testing recovery with each solution, I rolled the whole lab backup to this state.

Recovery testing showed that in the case of Veeam Backup 2.0, and the latest VMware Consolidated Backup, the recovered DC was fully functional.

One thing I noted, however, is that with VCB, the domain controller did not start up in the recovery mode during the first boot, as it did with Veeam Backup 2.0. According to Microsoft documentation however, when performing a VSS-integrated domain controller restore, the system must be rebooted in Directory Services Restore mode when Active Directory is running on the server (which is exactly our case). To my understanding, booting in this mode is required so that the NTDS.DIT file is not locked with Active Directory services, antivirus or other applications when the shadow copy restore is performed. So I don’t know whether or not this domain controller restore approach is supported by Microsoft.

This video demonstrates the DC recovery process using the most correct VSS-integrated recovery implementation, as provided by Veeam Backup 2.0.


With all the other solutions I have tested (including vRanger Pro, which was originally the first to claim having VSS support), the recovered DC was not functional and was put into the condition known as an update sequence number rollback, or USN rollback. The only way to recover a DC from rollback is to forcibly demote the domain controller, and reinstall it. Luckily, I had my lab fully preserved, so instead I could simply rollback the entire Active Directory.

This video demonstrates the DC recovery using a solution not featuring correctly implemented VSS support.


As you can see, some applications cannot be restored correctly by simply starting up the VM image, even when VSS is leveraged to perform the backup. Some applications, especially those featuring replication, require a certain sequence of actions to be restored from a backup made by leveraging VSS. Similar to the domain controller that I used to perform my testing, Microsoft Exchange Server is another example of a mission-critical application that must be restored using an application-specific restore technique (refer to the following support KB article for more information about VSS-integrated backup and restore of Microsoft Exchange server).

If you ask me why I am the first one to bring this issue up – I don’t know. Could it be simply because no one ever tried to actually restore VMs to the production environment from their backups? I can understand how this type of issue could be overlooked in a small test lab setting, where typically only one DC is installed. But before you put your VMware backup solution into production – give some serious thought to the recoverability of the backups it produces.

For more detailed information on correctly using VSS in VMware environments, please read the “VMware and VSS: Application Backup and Recovery” white paper available at Veeam Backup product page.