UPDATE (January 11, 2010): In an effort to help clarify things, Curtis Preston has blogged about Hyper-V and VMware’s support/use of VSS on his Backup Blog. He also provides a good explanation of VSS. I recommend checking out these resources and you may also want to consider following Curtis on Twitter.

Original Post:

This morning my inbox greeted me with an email from Eric Siebert and my Twitter search folder filled with statements about Veeam’s VSS claims. After my first cup of coffee I started to dig in and see what in the virtual world was going on…this first lead me to a blog post by EMC’s Scott Waterhouse where he states the following:

Therefore, final answer: if you need application consistent backups, you must do a guest level backup. An image level backup is simply not good enough. Even with a MS Windows 2003 or higher guest, even though VMware supports VSS. Yes you may do image level backups too, but they will only complement, no replace, guest level backups.

To be honest Scott was talking about VMware VSS here. I still don’t agree with his statement that for application consistent backups you have to do a guest level backup because there ARE solutions on the market today that provide application consistent backups from an image…of course that would be Veeam Backup & Replication.

What prompted some of the discussion (as far as I can tell) is that people don’t realize that Veeam uses its own VSS driver, not VMware’s. This has been the case since Veeam Backup and Replication 2.0, released in July of 2008. In fact, at the time, we pointed out how our VSS integration was different from the competition through a couple of blog posts: Is your backup really VSS aware? and VSS and VMware ESX: What your VMware backup vendor isn’t telling you. These posts really talk about VSS recovery and making sure that your VSS “backup” is truly able to recover properly. These 2 posts continue to be very popular on this blog…maybe pointing out that there is confusion regarding proper VSS handling when it comes to VMware.

That was all in 2008, what’s going on with VSS in 2010? Well, Veeam has continued to update its VSS integration to keep pace with the new releases of Windows Server 2008, both x86 and x64. Not all vendors have been keeping up with Microsoft though, in fact, looking at VMware’s VCB/VSS support table, it clearly states that they don’t support application consistency on Windows Server 2008, only file-level. So, any virtualization backup vendor that relies on VMware’s SYNC Driver and VSS for file and application consistency is not giving true VSS support for Windows Server 2008 (as Scott pointed out in his blog).

I’d like to finish up by answering Scott’s call for documentation:

Unfortunately, nobody has been able to provide a piece of definitive technical documentation (a white paper, or a support document, or relevant piece of text from an administration guide) that clearly describes the issue.

First, we have real users, using Veeam’s VSS today, that are talking about it in our forums, this is proof that they’re getting application consistent backups from Veeam’s image level process using our VSS:

Postby tsightler » 05 Jan 2010 14:22

Once again, Veeam fully support VSS aware snapshots of both AD and Exchange server when using the Veeam VSS Agent. Veeam doesn’t just “take a VM copy”, the Veeam VSS agent uses Windows VSS services to put these features into a proper, supported VSS backup state prior to taking the VM snapshot. In other words, a Veeam backup is indeed a “backup-aware copy of the Information Store and NTDS database”, and it uses the Windows recommended VSS processes to achieve this.

Postby donikatz » 06 Jan 2010 00:00

Obviously neither Tom nor Anton need my help here, but maybe some real-world testimony would make you feel more comfortable? Not only have I tested this, I’ve performed a *production* restore of a w2k3 DC with Veeam and it worked exactly and as simply as in the video. I’ve also done several *production* SQL restores without issue. Veeam also works well in our Exchange restore tests, although we haven’t had to do any in production (knock on wood). Although agent-based apps like Backup Exec may have more direct hooks for simpler granular restore (we still use BE for Exchange brick-level restores because our admins are more familiar with the process), Veeam is more than capable without the drawbacks of an agent. I hope to move away from BE altogether for Exchange this year; it’s just a matter of updating our runbook and training. Honestly, if there’s one area you certainly don’t need to lose sleep over with Veeam, it’s with Microsoft products. MS has well-proven APIs and Veeam makes great use of them; Veeam VSS is excellent. Heck, if only Oracle on Linux had VSS the way it does on Windows it would make my life a lot easier… ;)

Next, I’ve taking some quotes from Veeam’s own user guide regarding the differences between using VMware tools quiescence (SYNC) and Veeam’s VSS driver:

Transactionally Consistent Backup

Veeam Backup & Replication 4.0 provides two techniques for creating transactionally consistent backup images — the Enable VMware tools quiescence and Enable Veeam VSS integration options. In contrast to restoring a crash-consistent backup, which is essentially equivalent to rebooting a server after a hard reset, restoring transactionally consistent backups ensures safety of data of applications running on VMs.
Please note that when you select both VSS integration and VMware tools quiescence options for a job at the same time, the VSS module will only be used for processing backed up and replicated VMs. However, if you use both VSS and VMware tools quiescence options and select the Continue backup even if Veeam VSS quiescence fails option for backup jobs or the Continue replication even if Veeam VSS quiescence fails option for replication jobs, all your VMs will be processed with VSS first, and in case of VSS failure (e.g., Linux VMs), VMs will be processed with the VMware tools quiescence option enabled.
This can be very useful when you have both Windows- and Linux-based VMs in one job, so all VMs will be processed in a transactionally consistent way using VSS or VMware tools quiescence option.

Additionally, we then go on to explain the VSS process as well as the systems supported by our VSS driver:

Enable VSS Integration

With the Enable VSS integration option selected, Veeam Backup & Replication 4.0 utilizes the Windows Volume Shadow Copy Service (VSS) that ensures consistent backup of VSS-aware application running within your virtual machines (domain controllers, databases and other applications) without shutting them down. The Enable Veeam VSS integration option allows creating a transactionally consistent backup image of a VM, which, in contrast to a crash-consistent backup image, ensures successful VM recovery, as well as proper recovery of all applications installed on the VM without any data loss.
In the process of its work, VSS freezes all I/O at a specific point-in-time by interfacing with all VSS-aware applications and the Windows operating system. Consequently, there remain no unfinished database transactions or incomplete application files. Such backups, when restored correctly, result in fully functional applications.
The VSS works with Windows 2003, Windows XP, Windows 2008, Windows 2008 R2 and Windows 7 guest operating systems. Use VSS to back up 32-bit or 64-bit version of Windows 2003, 32-bit version of Windows XP guest OS, 32-bit and 64-bit versions of Windows 2008. Please note that administrator credentials are required to access the guest OS. Microsoft Windows VSS backup option requires that your guest OS has VMware Tools, and all the latest service packs and patches installed.

If anyone has any questions or further thoughts I’d love to hear from you. Feel free to comment below, hit me up on Twitter or shoot me an email @ doug /dot/ hazelman /at/ Veeam /dot/ com