Technical Support Services Blog
Discover the latest trends in technology, and the technical issues customers are overcoming with the aid of HP Technology Services.

DPTIPS: 1 way to ensure uninspired StoreOnce VTL performance and 7 ways not to

It's a secret I reveal to precious few. I dare not share it carelessly. But if you promise not to run around all willy nilly shouting my mystic formula hither, thither, and yon, I will impart to you the complex series of steps necessary to extract mediocre performance from your StoreOnce appliance.

 

This is a little hard to digest all at once, so be prepared to read it through again.

 

Are you ready?

 

1. Nothing.

 

That's right, nothing. Don't follow best practices for integrating Data Protector with StoreOnce VTL. Don't create more than one VTL based upon data types. Don't disable multiplexing. Don't use enough streams. Don't use the correct cartridge size. Don't make your media pools non-appendable. Don't force Data Protector to overwrite expired cartridges before touching blank ones. Don't license your StoreOnce appliance by capacity rather than drives and slots. Don't do any of that. Just do nothing -- a single, easy step. One and done. Presto! Crumby performance and no effort.

 

Or not.

 

If you would prefer to enjoy maximum benefit from your StoreOnce investment, let's take a look at what you SHOULD do and how easy it is.

 

Multiple VTLs
Let's create one gigantic VTL and send everything to it. Simple, yes, but it's wrong. Here's why.

 

StoreOnce tracks duplicates within each VTL but not across VTLs. That's on purpose, and you should use it to your advantage. Why spend resources trying to find duplicates in data that holds very little promise of actually being duplicate? Your odds of finding a 4 KB chunk of Exchange data that matches a 4 KB chunk of Oracle data? Not likely. The odds of finding a chunk of Fileserver data this week that matches a chunk from last week? Much better.

 

Create one VTL per data type. If you are backing up Exchange, Oracle, Fileserver, Windows OS, and Unix OS data, I would create five VTLs with crazy names like EXCH_VTL, ORA_VTL, FILE_VTL, WINOS_VTL, and UNIXOS_VTL. Segregate your backup data accordingly. Each VTL represents a smaller dedupe domain so the hash comparisons are faster, and you'll likely find more duplicates which means the 4 KB backend write is avoided saving time and space.

 

And don't be afraid to create each VTL with 12 drives for reasons that will become clear in the following sections.

 

No Multiplexing
Also known as device concurrency, this is a metric that describes the maximum number of disk agents that will be allowed to send data to a target device at once. The default device concurrency in DP is 4 which is too low for physical tape and about 3 too high for virtual tape. To disable multiplexing, simply set the device concurrency to one (1) for every virtual tape drive. Now increase the number of virtual drives you have available and put plenty of objects in a backup. You've just replaced multiplexing with multistreaming.

 

Why is this important? Because multiplexing wrecks inline dedupe performance and diminishes the ingest rate. When you let DP slice and dice, commingling data blocks from multiple objects to one virtual tape, the StoreOnce appliance chunking is far less likely to find an inbound block that matches exactly a previously stored block. Now you have paid the price for inline dedupe processing without realizing any benefit in terms of space savings or ingest performance. In other words, a net loss. So don't multiplex with virtual drives. Device concurrency = 1. Always.

 

Sufficient Streams
Each virtual tape drive that DP sees is actually a small collection of Linux processes within the StoreOnce appliance pretending to be a tape drive. Each set of processes is capable of a finite number of I/O operations per second. Don't be surprised when a single virtual drive cannot outperform a single physical drive. The physical drive is not doing inline dedupe, but the virtual drive is. Fortunately, the StoreOnce appliance as a whole has enormous resources and can host many, many virtual drives running at the same time.

 

You should leverage that fact by ensuring that you have a sufficient number of streams (active drive connections) inbound to the appliance concurrently to realize its rated ingest. These streams can be coming from one or several backup specifications, and the objects involved are likely to be sourced from numerous clients.

 

Using our StoreOnce 4430 appliance as an example, you will need at least twenty (20) concurrent streams with no more than twelve (12) going to any one VTL before you can achieve the appliance's maximum ingest. But once you replace multiplexing with multistreaming, it's a cinch. Plus, you will achieve greater ingest (with dedupe!) than you had formerly with a handful of physical drives.

 

Cartridges Sizing
There is no penalty for not filling every virtual cartridge. Additionally, StoreOnce does not preallocate space based upon the size and number of cartridges you create. Finally, there are reasons (beyond the scope of this discussion) why it would be best not to span dozens of cartridges per backup.

 

Assess the size of every object to be written to a given VTL. You can see object sizes in the properties of previous backups. Based upon the size of the single largest object, select a cartridge size at the next increment. If the largest object is 650 GB for instance, select 800 GB as your cartridge size.

 

Max_Cart_Size.jpg

 

One object per cartridge per backup. Nice and clean.

 

Non-Appendable Media
StoreOnce VTLs are happier -- especially where low-bandwidth replication is in use -- when you start each backup with fresh tapes rather than appending to partially filled cartridges. Space used is space used. It doesn't matter to the appliance or to DP whether 800 GB of data is on 8 cartridges or 80. Plus, there is no incremental cost for virtual cartridges as there would be in the physical world.

 

Naturally, you will want to calculate how much media is consumed by backups utilizing a particular VTL, multiply that figure by the number of iterations you plan to retain, then provision an adequate number of slots in the VTL. Adding a few slots for margin wouldn't hurt, either.

 

Let's say you are creating a VTL for MS-Exchange backups. The weekly full consumes 6 cartridges and is protected for 10 weeks. The daily incremental consumes 2 cartridges and is protected for 7 days. How does the math work with that?

 

Weekly: 6 tapes x 11 weeks = 66 tapes
Daily: 2 tapes x 8 days = 16 tapes
Total: 82 tapes + 8 margin = 90 tapes

 

Oh, go ahead and call it 100. I like nice round numbers.

 

You might wonder why I bumped up the weeks and days in those calculations. It's because the protection begins at the end of the backup, but media is needed at the beginning of the backup. Thus the oldest media would not be quite expired when it came time to be used at the beginning of the next cycle.

 

Expired vs. Scratch
Given 100 freshly formatted tapes, DP will think it's doing a good thing by preferring unused media over cartridges that have been used before and are now expired. This will continue until all 100 cartridges have been written once, then DP starts from the top with overwrite #1 for each (expired) cartridge. In the physical world, this behavior is desirable because it guards against reusing a small subset of cartridges while others sit gathering dust. In the virtual world, this is the very last thing you want DP to do because it prematurely fills up your StoreOnce appliance.

 

Why? Easy. Expired cartridges still consume space on the StoreOnce appliance until they are reformatted or overwritten with new backup data. So unless you enjoy reformatting tapes each morning as a hobby, use one of these two simple methods for giving DP an attitude adjustment.

 

Starting from ... scratch? (No pun intended. Sort of.) Then don't format any of your virtual media. Seriously, don't. Instead, ensure that InitOnLoosePolicy=1 is set in your global options file. Scan all of your new virtual media so DP registers it as blank rather than unknown. Create the media pool to be used by that VTL, and start running backups using that pool. If insufficient expired media are already in the pool, DP will format enough blank media to satisfy current demand. If DP has enough expired media already in the pool, it will be used (overwritten) keeping StoreOnce appliance space consumption to a minimum.

 

Already have all your virtual media formatted? Don't worry, we can achieve the same behavior in a different manner. Create a free pool for each VTL. Reformat all of your expired media, placing them in the relevant free pool. Now modify the properties of your regular VTL pool so that it draws scratch media from the free pool but does NOT put it back there.

 

Free_One_Way.jpg

 

Voila! DP prefers to overwrite expired media already in the pool before drawing any scratch media from the free pool thus keeping StoreOnce appliance used space to a minimum.

 

Capacity Licensing
In the physical world, we license tape libraries by the number of drives and storage slots. Although you could continue to do that with virtual libraries, why would you want to? With DP's Advanced Backup to Disk licensing, you cover the physical usable space in your StoreOnce appliance -- 72 TB for instance. Then you are free to create as many VTLs with as many drives and slots as your needs dictate without purchase of any further licensing.

 

For each of your VTLs, be sure to let DP know that it's really a virtual library by checking the box indicated. Doing so switches license checking for that library from drives & slots to Advanced Backup to Disk capacity.

 

Virtual_Library.jpg

 

And we don't care if you cram 500 or 600 TB of user data into that 72 TB of physical space. The DP licensing is the same regardless of how much benefit you derive from inline deduplication. You're welcome. :-)

 

Summary

Though I've only hit the highlights, these are principles that I share onsite with customers nearly every week.  It's the same thing I taught in my Technical Online Seminar (TOS) sessions two years ago.  They are in many respects a summation of guidelines documented in our best practices white paper for Gen 1 & Gen 2 D2D appliances as well as Gen 3 StoreOnce appliances.  I have proven the accuracy of these principles time and time again in numerous customer environments, and I can do the same for you.

 

If you would like an indepth assessment of your backup and recovery environment and a week of intense, customized knowledge transfer, contact your HP Services Sales Representative and ask about HP's Integration & Technical Services team.

 

Tell them Mr_T sent you.  :-)

 

 

Labels: Data Protector
Comments
Michael Pegg(anon) | ‎09-17-2013 02:55 PM

I will refer back to this, it's a great summary. Cheers!

Jenni ‎09-18-2013 11:08 AM - edited ‎09-18-2013 11:14 AM

Hi Mr T

 

Great Article and long overdue! (Yes I know there is a white paper but getting customers to ingest these can be as hard as getting them to read the product announcements....)

 

I disagree a bit with your comment on not mixing data types in the same VTL. Totally agree that this is important for large businesses where backup & restore performance must be maximised but for smaller businesses where long-term data retention is key and the size of the backup device was dictated by the required usable storage to hold the backup data (e.g. length of time they want the backup data available online) sometimes having everything in one big VTL can improve this.

 

For example, I have a customer that needs to retain all his weekly backup data for 2 years online if possible. Most of his restore requests are for data minimum 6 months old. He had best practice config where each data type was in its own VTL and he ran out of space on the D2D4324 (G2 D2D). The cost to upgrade (have a 2nd device) was pretty high and as the full backup of the env is only approx 20 TB this a relatively small customer. We decided to forgo performance and combine data types into a single VTL and the device is only 52% full now, the dedupe ratio is 27:1 and growing and he is still totally happy with the backup & restore peformance, although a bit slower than before it is still well within his SLAs and he can now keep everything online for 2 years to meet his online data retention requirements.

 

I see a lot of smaller businesses buying these devices to suit online data retention requirements / to negate physical media management; and separating the data types doesn't always get them the dedupe ratio they need, a lot of them are investing so they can do away with physical tape, not for backup & restore performance.

 

So while I agree with your article in principle I do think that this isn't always the best way to go for smaller businesses and you'd be surprised at the actual overlap in identical blocks between data types provided your backup & restore SLA can accomodate the drop in performance?

 

I guess there is a lot to be said for simplicity, a lot of SMB customers just wanna replace their physical tape library with a virtual one, they don't care about the performance they just want it simple. I spend a lot of time trying to convince them that the performance is key but all they see is complications and additional config. I'll certainly be showing them your article as its a lot more succinct than the best practices whitepapers so thank you very much for taking the time to write this for us :smileyhappy:

 

Thanks,
Jenni

 

P.S. Are there plans to write a similar post for catalyst store & usage in DP? *Lots* of head scratching with my customers on the whole gateway can open multiple MAs but MA<->DA conccurency remains 1, so something around the useage of min/max devices to dicate the number of objects backed up concurrently in the session would be really useful..... And maybe DP object copy with replication enabled? Or am I asking too much now :smileywink:

Mr_T | ‎09-18-2013 01:23 PM

Good morning Jenni,

 

Thank you so much for taking the time to post your wonderful comments.  It is the ability to correlate emperical data from different practices that helps us all.  It looks like you've found a usage case where one size does not fit all.

 

If they are seeing a 27:1 dedupe ratio, that would imply that on average only 1 in 27 chunks coming in are unique and being passed through for a backend write.  The other 26 are being tossed and replaced with pointers since they are duplicates.  That being the case, the performance should be pretty good.  In fact, it would only be diminished if they were pushing less than 20 concurrent streams at the 4324.  Well that plus the larger dedupe domain and its overhead.

 

I will definitely keep in mind your experience and factor that in to future scenarios.

 

Thanks also for your suggestion on future topics.  They are definitely worthy.

 

Best regards,

Jim

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the community guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
About the Author
Mr_T? Yes, but in name only. No mohawk or gold chains. In real life I'm Jim Turner, a Master Technologist with HP's Integration and Techn...


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation