Disappointing Review.

It's not very often I'm provoked into making a criticism of a respected publication, however there's nothing like being grumpy!
I've published a number of posts about baselines, trending and performance analysis and I've been quick to remark when published examples don't really match real world, so I was surprised to read a review of compressed backup software for SQL Server and the conclusions drawn in a leading publication.

I'm a keen user of such a product, which wasn't part of the review, and although I've only used the one, but at several client sites, I suspect that the choice of the Quest or Redgate product is more a personal one, however if one is to write a review of such a product then it would make sense to provide accurate comparisions and test in an enterprise environment.

From the description of the test "server" I'd figure the tests were carried out on a laptop with an external drive. A "large", 540Gb, database was used to be fair but because the database was "so large" the tester could not actually perform a native backup or restore as there wasn't sufficient disk space, only a single TB disk was availble, so estimated times were published as the base to compare against. Hmmm - pick a number, add 5, divide by the number you first thought of, add 15 seconds to make it look good - you get my feeling on this.
A single core 1.8Ghz processor also doesn't really cut the mustard either.

Now I may be wrong but I can't really think that the average enterprise user of Lite Speed or Redgate SQL Backup will be using a low spec single processor with two sata drives - but feel free to comment if indeed your production databases do run on such a system.

So my point is that the average Production Server is likely to have multiple cores and at least half way decent scsi/fc storage, the product I use can be configured to use threads, buffers and compression ratio's - I spent some time testing which combination of these settings gave me the best compression vs time performance, and I didn't even touch the use of multiple backup files, all the types of things you'd investigate if you were looking to back up "large" databases or you want really fast backups and restores. And if you were going to write a review aimed at Database Professionals who'd be looking into deployment in the enterprise.

Such is the same with performance testing or trending, you must have a valid baseline and you must have a test environment which at least gets somewhere near to the actual production environment you intend to deploy to.

My personal view is such reviews do not add any value and also give a misguided view of the technology and products, I obviously also disagree with the recommendations of the review, but that's another matter.

Published 17 April 2008 12:26 by GrumpyOldDBA

Comments

# re: Disappointing Review.

17 April 2008 13:47 by BrentO

I read that exact same review and I came away with that exact same conclusion.  I was so disappointed because DBAs rely on review results like this, and this is nowhere near enough information to make a good decision.  

Forget who won or lost the race - they weren't even going around a track.  If the test server doesn't have enough CPU or storage speed, then it's not really going to showcase any of the compression methods, and it's not going to pinpoint bottlenecks inside the backup software.

I immediately thought back to other product comparisons I've seen where the reviewer said, "This column is how each product performed out of the box, and this column is how it performed after I'd tried all of the parameters and picked the fastest one for my hardware."  When I evaluated & purchased backup software a couple of years ago, that was exactly what I did: I grabbed demos of everything out there and slapped together a spreadsheet of metrics with the fastest hardware in the shop.

When I read a comparison review like that of products I know really, really well, then it makes me wonder about their reviews of products I don't know so well, and makes me think they're not credible either.