VMware Partner Exchange – See you there?

One of my favorite events is just around the corner, VMware Partner Exchange 2010, Feb 8-11 in Las Vegas. I like PEX because it gives me a chance to meet old friends, new friends and new and current Veeam partners. I also get to attend sessions exclusive for VMware partners and get the inside scoop on new VMware technology (all under NDA of course).  Veeam will have a booth again this year and as a special bonus, we’re going to be giving away a Kindle DX Global Edition to one lucky person who registers at our booth.

KindleDX

So if you’re going to be at PEX, be sure to stop by the Veeam booth. Also, feel free to find me on Twitter as I’m sure someone will be organizing a TweetUp or 2.

The Great VSS Debate

UPDATE (January 11, 2010): In an effort to help clarify things, Curtis Preston has blogged about Hyper-V and VMware’s support/use of VSS on his Backup Blog. He also provides a good explanation of VSS. I recommend checking out these resources and you may also want to consider following Curtis on Twitter.

Original Post:

This morning my inbox greeted me with an email from Eric Siebert and my Twitter search folder filled with statements about Veeam’s VSS claims. After my first cup of coffee I started to dig in and see what in the virtual world was going on…this first lead me to a blog post by EMC’s Scott Waterhouse where he states the following:

Therefore, final answer: if you need application consistent backups, you must do a guest level backup. An image level backup is simply not good enough. Even with a MS Windows 2003 or higher guest, even though VMware supports VSS. Yes you may do image level backups too, but they will only complement, no replace, guest level backups.

To be honest Scott was talking about VMware VSS here. I still don’t agree with his statement that for application consistent backups you have to do a guest level backup because there ARE solutions on the market today that provide application consistent backups from an image…of course that would be Veeam Backup & Replication.

What prompted some of the discussion (as far as I can tell) is that people don’t realize that Veeam uses its own VSS driver, not VMware’s. This has been the case since Veeam Backup and Replication 2.0, released in July of 2008. In fact, at the time, we pointed out how our VSS integration was different from the competition through a couple of blog posts: Is your backup really VSS aware? and VSS and VMware ESX: What your VMware backup vendor isn’t telling you. These posts really talk about VSS recovery and making sure that your VSS “backup” is truly able to recover properly. These 2 posts continue to be very popular on this blog…maybe pointing out that there is confusion regarding proper VSS handling when it comes to VMware.

That was all in 2008, what’s going on with VSS in 2010? Well, Veeam has continued to update its VSS integration to keep pace with the new releases of Windows Server 2008, both x86 and x64. Not all vendors have been keeping up with Microsoft though, in fact, looking at VMware’s VCB/VSS support table, it clearly states that they don’t support application consistency on Windows Server 2008, only file-level. So, any virtualization backup vendor that relies on VMware’s SYNC Driver and VSS for file and application consistency is not giving true VSS support for Windows Server 2008 (as Scott pointed out in his blog).

I’d like to finish up by answering Scott’s call for documentation:

Unfortunately, nobody has been able to provide a piece of definitive technical documentation (a white paper, or a support document, or relevant piece of text from an administration guide) that clearly describes the issue.

First, we have real users, using Veeam’s VSS today, that are talking about it in our forums, this is proof that they’re getting application consistent backups from Veeam’s image level process using our VSS:

Postby tsightler » 05 Jan 2010 14:22

Once again, Veeam fully support VSS aware snapshots of both AD and Exchange server when using the Veeam VSS Agent. Veeam doesn’t just “take a VM copy”, the Veeam VSS agent uses Windows VSS services to put these features into a proper, supported VSS backup state prior to taking the VM snapshot. In other words, a Veeam backup is indeed a “backup-aware copy of the Information Store and NTDS database”, and it uses the Windows recommended VSS processes to achieve this.

Postby donikatz » 06 Jan 2010 00:00

Obviously neither Tom nor Anton need my help here, but maybe some real-world testimony would make you feel more comfortable? Not only have I tested this, I’ve performed a *production* restore of a w2k3 DC with Veeam and it worked exactly and as simply as in the video. I’ve also done several *production* SQL restores without issue. Veeam also works well in our Exchange restore tests, although we haven’t had to do any in production (knock on wood). Although agent-based apps like Backup Exec may have more direct hooks for simpler granular restore (we still use BE for Exchange brick-level restores because our admins are more familiar with the process), Veeam is more than capable without the drawbacks of an agent. I hope to move away from BE altogether for Exchange this year; it’s just a matter of updating our runbook and training. Honestly, if there’s one area you certainly don’t need to lose sleep over with Veeam, it’s with Microsoft products. MS has well-proven APIs and Veeam makes great use of them; Veeam VSS is excellent. Heck, if only Oracle on Linux had VSS the way it does on Windows it would make my life a lot easier… ;)

Next, I’ve taking some quotes from Veeam’s own user guide regarding the differences between using VMware tools quiescence (SYNC) and Veeam’s VSS driver:

Transactionally Consistent Backup

Veeam Backup & Replication 4.0 provides two techniques for creating transactionally consistent backup images — the Enable VMware tools quiescence and Enable Veeam VSS integration options. In contrast to restoring a crash-consistent backup, which is essentially equivalent to rebooting a server after a hard reset, restoring transactionally consistent backups ensures safety of data of applications running on VMs.
Please note that when you select both VSS integration and VMware tools quiescence options for a job at the same time, the VSS module will only be used for processing backed up and replicated VMs. However, if you use both VSS and VMware tools quiescence options and select the Continue backup even if Veeam VSS quiescence fails option for backup jobs or the Continue replication even if Veeam VSS quiescence fails option for replication jobs, all your VMs will be processed with VSS first, and in case of VSS failure (e.g., Linux VMs), VMs will be processed with the VMware tools quiescence option enabled.
This can be very useful when you have both Windows- and Linux-based VMs in one job, so all VMs will be processed in a transactionally consistent way using VSS or VMware tools quiescence option.

Additionally, we then go on to explain the VSS process as well as the systems supported by our VSS driver:

Enable VSS Integration

With the Enable VSS integration option selected, Veeam Backup & Replication 4.0 utilizes the Windows Volume Shadow Copy Service (VSS) that ensures consistent backup of VSS-aware application running within your virtual machines (domain controllers, databases and other applications) without shutting them down. The Enable Veeam VSS integration option allows creating a transactionally consistent backup image of a VM, which, in contrast to a crash-consistent backup image, ensures successful VM recovery, as well as proper recovery of all applications installed on the VM without any data loss.
In the process of its work, VSS freezes all I/O at a specific point-in-time by interfacing with all VSS-aware applications and the Windows operating system. Consequently, there remain no unfinished database transactions or incomplete application files. Such backups, when restored correctly, result in fully functional applications.
The VSS works with Windows 2003, Windows XP, Windows 2008, Windows 2008 R2 and Windows 7 guest operating systems. Use VSS to back up 32-bit or 64-bit version of Windows 2003, 32-bit version of Windows XP guest OS, 32-bit and 64-bit versions of Windows 2008. Please note that administrator credentials are required to access the guest OS. Microsoft Windows VSS backup option requires that your guest OS has VMware Tools, and all the latest service packs and patches installed.

If anyone has any questions or further thoughts I’d love to hear from you. Feel free to comment below, hit me up on Twitter or shoot me an email @ doug /dot/ hazelman /at/ Veeam /dot/ com

Product Roadmaps – Is your vendor lying to you?

We’ve recently gone through some exercises here at Veeam as we prepare for 2010. Of course we’re working on our sales goals and marketing plans, but we’re also talking about our product set for 2010. One of the most often questions I get asked by our sales teams at quarterly business reviews (QBR’s) is what is our roadmap and can they share it with customers and partners? This is always a tricky question since if we give too much away we risk our competitors finding out and getting to market first with certain functionality. If we don’t say anything it makes us appear like we don’t know what we’re doing or where we’re going.

Since we’re getting ready for our 2010 kickoff meetings we’ve had a lot of discussion about roadmaps and the Veeam philosophy on sharing product directions, specifically around Veeam Backup & Replication. We need to give something to our sales teams to share with partners and customers but at the same time we need to protect our intellectual property (IP) and also be able to react to an ever changing market.

We’ve also watched as other vendors and competitors have talked about their roadmaps and plans for the future and then continually missed their release dates or promised feature sets. Does this mean they’re lying to their customers? I would tend to say no because at some point they truly believe they’re going to make their release dates but then something happens and they miss it. Customers and partners are now left waiting and wondering when the next version will be coming out…the more times this happens the more and more customers will no longer trust you and start looking at other solutions.

Time, Features and Quality

These are the words our development lives by. Anytime someone asks for more features, it takes more time. Want it faster? Then features will get dropped. Of course development could achieve both but then they end up delivering a poor quality product because there wasn’t enough time to test all the new features (see the relationships here). Some might say just throw more developers at it but then of course there’s the saying: “9 women can’t make a baby in a month”.

The other issue of locking in feature sets too far in advance is what if there’s a great new feature that comes along that more and more customers are asking for and you promise to deliver? Do you ditch your previous plans and deliver the new feature? Do you wait until you deliver what you originally promised and then add the new feature? In either case, someone is going to get called a liar and that’s not good for customer and partner relationships.

It’s the History, not the Future

One of the things I learned early on in the software industry is “sell what you have”. If we don’t have features X, Y and Z today but we do have features A, B, and C then that’s what we’re selling. If I try and sell you X, Y and Z and promise you that we’ll have them at a specific time in the future, I’ve most likely just lied to you. But there is a fine line…if I know development has features X, Y, and Z “feature complete” and we’re now going through testing I’ll let you know. I still won’t tell you exactly when except for maybe something like Q1 or beginning of Q3. Again, I don’t want to lie to you, something may come up in testing that delays the product, something significant enough that it would put the quality at risk and we won’t knowingly release poor quality products to the market.

Rather than trying to sell the future, why not take a look at the history? The truth is, history tells more about a company than any roadmap promise of the future. History is verifiable, history is real. Roadmaps are the future, the future may change, history doesn’t. If a vendor is trying to sell you on the future and not on today, what does that say about the quality of the products or feature set they have today? Are they lying to you?

Others Promise, Veeam Delivers

To Virtualize or Not to Virtualize: A SMB Debate

I was recently reading an Information Week B-Mighty article on SMB Virtualization, where the author mentions that VMware believes SMB customers (defined here as sub-10 physical server) are either jumping in full force, or not virtualizing at all. He also spoke with Dell GM for SMB, Erik Dithmer, who mentioned that customers would save little or no money at all on virtualization. Dithmer must be basing his math on the process of shared storage in a virtualized environment vs. individual server storage on the non-virtualized path. I would actually present an alternate solution to the problem for a user with less than 10 physical servers in their company today.

Physical

On Dell.com today, I can purchase a R710 for $999 at street price. This will be my baseline for a physical server solution. Multiply this by 10, and we have a $10k price tag. To build a proper disaster recovery scenario in a physical world, you need to both backup and replicate your data from one physical location to another. The least expensive backup option I can select is Commvault, at $549 per server. The least expensive replication scenario I can find is a 5-pack of Double-take for $2499. So, at a very minimum, a physical server disaster recovery scenario is going to run you $10k +$5500 +$5000 = $20,500.

Virtual with SAN

Best practice for a disaster recovery scenario including shared storage is for there to be 2 equal SAN arrays on either side of your disaster recovery location. At a minimum, it is required to have 2 servers in your primary location for failover activities, and a single machine in the disaster facility for run-time operations. Once again, I went out to Dell.com to prove out Erik Dithmer’s hypothesis.

Although Dell markets a Virtualization in a Box config listed at $12,999, that configuration contains 3 R710 servers which only have a single socket, rendering most of the features VMware offers like Virtual SMP and CPU affinity fairly useless. So lets config this the old fashioned way. Dell offers the R805 online ($4500) as the “Virtualization” ready choice, so the standard configuration has dual quad-core Opterons and 8GB of memory, when coupled with VMware’s Memory overallocation capabilities, gives it the same performance as 16GB of RAM in physical systems. The shared storage component of this equation brings in the big question mark. Do I go with the MD3000i, iSCSI SAN on both sides for about $5000 each and add the $3500 per ESX server Commvault and $2500 per 5-VM Doubletake ($25,500 total) or do I go with two Dell Equallogic arrays, which have free replication software but would still need the Commvault (well over $25k).

So with the two scenarios side by side, Dithmer is correct that virtualization can in fact be more expensive than business as usual based on some assumptions.

The Third OptionVeeam Backup and Replication

Virtualization is the right option for so many reasons, a few of them including encapsulation, isolation, and portability. The ability to get a VM back up in running in minutes versus hours to rebuild a physical box is definitely the soft dollar value that most people know it is. But how do you accomplish these SLAs without requiring the large capital investment in shared storage? Or how do you leverage something like the Dell MD3000i, which comes in at roughly $5000 a piece in a proper DR strategy? The answer for me is Veeam Backup and Replication 4.1 with its virtualization capabilities. At a retail price point of $599 per physical ESX socket for unlimited VM backup and replication, you have the opportunity to get your disaster recovery scenario under control immediately. Veeam has the capability of backing up local storage, VMFS datastores on iSCSI and deliver those backups to pretty much any data storage site you can see. Veeam’s replication includes the capability of replicating ESXi to ESXi systems as well, so the security of ESXi can also be covered. So in the above scenarios, you can configure your hardware any which way you like for a DR scenario and still get accurate backups and replications.

Let’s take a look at the minimum virtualization configuration capable of accomplishing the DR strategy. Let’s use 3 virtualization capable systems with the R805 ($4500) plus 6 total sockets of Veeam Backup and Replication 4.1 ($3600). The primary location will have two R805’s, each configured to backup to each other’s local storage. The primary systems would also replicate to the DR location in case of site failure, and your data would be contained in two places. Mission accomplished, 10 VMs backed up and replicated for under $14,100. That is roughly $6,000 less than the 10 physical servers in a customers environment. It also takes no consideration for the reduction in energy usage, number of parts that might break, or any of the other environmentals of a data center.

In summary, Veeam Backup and Replication can save your SMB business significant costs when implementing virtualization.

ESXi Anyone?

Veeam Backup & Replication 4.1 Released!

Today Veeam released version 4.1 of our Backup & Replication product. It’s a milestone release for us because it now brings FULL ESXi compatibility to every one of our products. Why is this so important? From the industry view, VMware is pushing very hard for companies to adopt ESXi, it’s even been rumored that vSphere 4 is the last major release of ESX that will have a service console (COS), meaning that ESXi is the future for VMware. Veeam Backup and Replication 4.1 is yet another industry first by Veeam!

But it’s not just Veeam Backup & Replication…

All of Veeam’s products support ESXi, whether it’s reporting, monitoring or disaster recovery. Back at VMworld 2007 in San Francisco (yes, 2007, when ESXi was announced), Veeam R&D heard the message loud and clear that the COS is not the way VMware would like their partners to support ESX. With the VI API developers have a choice when writing software and for Veeam that choice is to move away from the COS.

What about ESXi Free?

Yes, we get the question all the time. While Veeam Backup & Replication did support the Free Version of ESXi for a brief time earlier this year, we no longer support ESXi Free for backup or replication. Some of our other products, however, do support ESXi Free, here’s a quick table of what is/isn’t supported by product:

 

ESX 3.5/4.x

ESXi 3.5/4.x

ESXi Free 3.x/4.x

Veeam Backup & Replication

Fully supported

Fully supported

Not supported

Veeam Monitor

Fully supported

Fully supported

Support for read operations only

Veeam/nworks MP and SPI

Fully supported

Fully supported

Support for read operations only

Veeam Reporter Enterprise

Fully supported

Fully supported

Support for read operations only

Happy Holidays from Veeam

snowglobe
2009 Holiday Giveaway Extravaganza

This year for the holidays we’ve teamed up with Rick Scherer over at VMwaretips.com and his 2009 Holiday Giveaway Extravaganza to give away some Veeam Software to some lucky winners. The contest is only open to US residents and the grand prize is an Iomega StorCenter Pro ix4-100 NAS Server. The contest is open from now until 11:59 PM PST January 3rd so be sure and enter to win.

vCalendars (2010 Edition)

VMUGEVENT2009-63
If you didn’t get a vCalendar, written by Jason Boche, from Veeam at either VMworld San Francisco or the Dutch VMUG, have no fear, there are more on the way! While you can still order one from the Printer Owl, Veeam has placed another special order for both the US and Europe to be distributed at various events (VMUG’s, etc). Please read your Veeam newsletters for announcements on when and where we’ll be with more vCalendars!

Veeam Communities!

Veeam is lucky to have great community support for our products. Recently Veeam awarded our top forum members with special recognition as MVP’s. Besides getting a special MVP logo in the forums, we also sent them special edition Veeam t-shirts. Congratulations to our first Veeam Forum MVP’s

tsightler
TrevorBell
glemmestad
mdornfeld

VeeamMVP

UPDATE – The t-shirt design!

MVPt-Shirt

Want to know how else to connect with Veeam? Check out our new Communities page with links to all the ways you can stay connected with all things Veeam!

Happy Holidays From Veeam!

Backing up MySQL on a Linux VM

A question commonly asked is how can you quiesce MySQL databases when running on a Linux VM? Since Linux doesn’t have a VSS mechanism like Windows does, it’s not as straight forward as a Microsoft SQL quiescence. On of Veeam’s Systems Engineers, Ricky El-Qasem, has written a white paper on the subject and it’s available now for download from the Veeam website. The white paper describes 2 different options/methods for running pre and post snapshot scripts using VMware tools installed on the Linux VM.

Here’s a quick preview:

Option 1. Suspend MySQL service

The MySQL service is stopped for a few seconds while the snapshot is created then
started again and is described in Method 1.

Advantage: It’s quick and easy and will allow you to take a transaction of
all databases without additional disk usage local to the MySQL server.

Disadvantage: For a brief period of time no databases running on the
MySQL server will be available which may not be suitable for applications
that need 100% uptime.

Option 2. Use Online Dump

An online dump of each database is taken using the mysqldump command which
copies a database to storage accessible from the MySQL server and is described in
Method 2.
Advantage: The dumped databases are in a transaction consistent state
and this is achieved without stopping the MySQL service and allows for
100% uptime.
Disadvantage: Extra storage space is required to maintain a second copy
of the database and process may take a considerable amount of time to
achieve depending on the size of your databases.

Veeam Backup & Replication 4

A few weeks ago Veeam released Veeam Backup & Replication 4.0. I should have written a post the week it was released but I’ve been pretty busy and I also wanted to wait and get some of our customer’s reactions before writing about it. Our 4.0 release is a major milestone for us since it provides full support for the vStorage APIs for Data Protection. There appears to be a bit of confusion with some on what the vStorage APIs for Data Protection really do, especially when compared with VMware Consolidated Backup (VCB). As VMware states on the vStorage page, the vStorage APIs for Data Protection are the next generation of VCB. For a good explanation on the differences, check out my friend @Gostev’s blog over at vNotion “What is VMware vStorage API?

So why do we still have support for VCB? The simple answer is that we have a number of current customers that are using VCB and have not switched over to vSphere yet. We could have dropped support for VCB but then that would require us to maintain both 3.x and 4.x releases. Since 4.0 is an built upon 3.x and not a complete re-write of the software, we left VCB support in for our customers. The benefit is that our customers have a choice of how to process their backups: vStorage APIs, VCB or Veeam’s Service Console or Network modes. Of course I do recommend that customers use the vStorage API methods, it gives you the best performance and is also built specifically for vSphere.

Up to 10x Faster

Yes, it’s really true and our customers have posted as much on our forums. We’ve consolidated this feedback into a handy PDF document INSANE BACKUP SPEED, it’s not us saying this, it’s real people using our 4.0 product.

How is it so fast? One of the biggest reasons is Changed Block Tracking (CBT) which is included in the Virtual Disk Development Kit (VDDK). With CBT enabled for VM’s running on ESX or ESXi 4.x, VMware actually tracks the block level changes made to the VMDK. This way, when software like Veeam Backup & Replication takes a Snapshot, VMware returns a list of what blocks have changed. This significantly increases backup speed as we no longer have to try and determine the changes ourselves, VMware tells us in a matter of seconds.

The Critics

Of course anytime a vendor releases software that’s cutting edge and ahead of the rest of the field, others will try and knock it down. This recently happened on a blog sponsored by one of our competitors, claiming Veeam Backup & Replication was corrupting data. Of course when I first saw this I was a bit alarmed, but since they included a link to our forum where a user was reporting the problem, I followed the link and realized they were just sensationalizing something for their own apparent gain. Since this particular blog decides to heavily moderate comments, I thought I would post the facts here, lest any other competitors decide they want to sensationalize this non-issue.

The Facts

Taken from the Veeam Forum post:

  1. Backups are NOT corrupted.
  2. You can only run into this issue with NON-DEFAULT restore mode, in 1 restore mode of 3 existing modes.
  3. Despite what competition may be claiming, there is no actual user data loss or corruption – VM will still boot and work.

The only real issue is OS and file system check tools complaining about unexpected content of the unused disk blocks. Linux ext3 file system and disk test tools merely suspect a problem seeing unused blocks being non-zeroed, and warn about this. This is specific to certain file systems only, for example, Windows NTFS considers this situation absolutely normal.

Of course every software has bugs, the example above is a bug in our software when using 1 particular recovery method and we have a fix available and it will be included in our next release. The important piece of information is that NO DATA WAS EVER CORRUPTED, just an issue on recovery and zero byte blocks. We are very thankful to our active forum  community and for bringing this to our attention. Veeam’s motto after all is “Listening to You, Building the Tools You Need”, it’s on the back of all of our business cards. I’d like to point out the below points about our user forum:

  • Our forum is one of our greatest resources where people share their experience, best practices and getting support from Veeam and the community
  • We understand that our competitors are so desperate that they will continue using our own forum to try and fool the community
  • We will continue to be honest and direct with our customers in our forum vs. ending all the threads with ‘please contact our support’
  • User feedback: The quality and effectiveness of this Forum alone would be enough to justify switching to Veeam, even if the product wasn’t superior to its competition (as it is at the moment). It seems just too good to be true… I hope it will continue this way!

Conclusion

Sorry to waste so much space on this post answering the competition. If they would allow comments on their own blog I would have written it there. In closing I just want to say that we encourage everyone to evaluate our software and make decisions for themselves. Put us in the lab and you will see, others promise, Veeam delivers!

Put us in the lab!

If you’re on the Veeam distribution list you’ll be getting some of the text below today as part of an ecard. I wanted to go ahead and blog it here as well because I think it’s an important message from Veeam. Sorry if this seems to be a bit of a marketing message (but it’s important to me):

“We’ve received numerous questions from our partners and customers on why a competing product won the Gold award in the Business Continuity & Data Protection category. We know that judges are put in a difficult position when they have to choose from almost 200 products in a short period of time. For this award, the judges cannot, and do not, install, test or evaluate the products in a lab. Instead, they mostly rely on vendor statements in their judging process. We encourage all customers to “Put Veeam in the Lab” for a head-to-head comparison with competitive products, and make the right decision for your environment. We are confident you will find out why Veeam Backup & Replication is #1 for VMware backup.”

So, if you’re interested in seeing why Veeam is #1, fill out the form here and we’ll send you some information as well as help you get started on evaluating Veeam Backup & Replication in YOUR environment so you can makeup your own mind on who the winner really is.

The Power of Veeam Business View and Monitor 4.5

I’m very excited for the upcoming release of Veeam Monitor 4.5 with Veeam Business View integration. While we released Veeam Business View a few months ago, we are now getting our products aligned with how it works and how it enables you to start taking a “business view” of your virtual infrastructure rather that just an “infrastructure view”. Monitor 4.5 will show the Business View tree from within it’s interface, making it very easy to set alerts, create reports and in general just see how things are performing based on the business category or group. This includes even setting alerts by service level agreement (SLA) or by application. Virtual Machines are not limited to 1 dimension, they can be in multiple categories/groups. The video below (featuring the voice of yours truly) goes into more detail of what’s coming up and shows the power of Veeam Business View along with Veeam Monitor 4.5.

On YouTube:

The best part in my opinion? Veeam Business View is free and Veeam Monitor has a free edition. You’ll get all this capability without spending a penny! Of course there are some limitations on Veeam Monitor Free Edition, here’s a summary:

  • real-time monitoring only (24 hours of history)
  • real-time reporting only (24 hours of history in report)
  • 10 alarms
  • no alarm modeling
  • no VM/ESX/VC drill-down (process monitoring/management)
  • no access to datastore load monitoring (other datastore graphs are available)

Veeam Monitor 4.5 should be available in early Q4, sign up here to be notified.