DPTIPS: Device Block Size Challenges

[#1]

Alrighty then, it's time to integrate your shiny new HP D2D StoreOnce appliance into your Data Protector environment.  You're an old hand at DP, so that's no issue.  For the new D2D, you've read the Concepts Guide, Install & Config Guide, User Guide, and finally the all-important HP D2D Backup Systems Best practices for VTL, NAS and Replication implementations document.  Among all the data swimming in your head, you remember reading on page 25 of that last tome ...

Use a block size of 256KB or greater. For HP Data Protector and EMC Networker software a block size of 512 KB has been found to provide the best deduplication ratio and performance balance.

Not a problem.  You know the default for LTO/Ultrium is 64k, and you've typically increased that to 256k which has historically been the sweet spot for backup to physical tape.  So now you're going to set your VTL tape device block size to 512k.  What could possibly go wrong?

Well, a couple of things, specifically.  If your Media Agent (MA) host is running MS-Windows or HP-UX, you may encounter a message similar to the following.

Media Agent host is HP-UX

If you are running HP-UX 11i v2 (11.23) or earlier, the stape driver is limited to a maximum block size of 256k.  Attempting the use of a larger block size until recently would work for the DP write operation but fail miserably on restores.  This was caused by the stape driver quietly "chunking" larger block writes down to multiple 256k block writes.  Details are included in Document ID KM915678 and Document ID c02256887.

Don't fret if you're stuck at 11i v2 which dictates a max block size of 256k.  It's infinitely better than the default of 64k where you will see diminished D2D performance and virtually no deduplication.  Sure, 512k may be a tad better, but 256k is still good.

For those running 11i v3 (11.31), it's somewhat of a mixed bag.  To guard against unpleasant surprises that may result from the quiet "chunking" noted above, the latest MA patch for DP has imposed an artificial block size limit of 256k for HP-UX.  That's good except that the stape driver in 11.31 can in fact handle larger block sizes like 512k.  Your options then if your media host is HP-UX 11i v3 are:

1. Stay at a 256k block size with tape devices (D2D and physical).
2. Remove the latest MA patch (PHSS_41963) and revert to the next-to-latest (PHSS_41460).
3. Keep the latest MA patch, open a support call, and request the SSP for QCCR2A31441.

Options 2 and 3 will allow you to successfully use a 512k device block size with DP and HP-UX 11i v3.  From reading the lab case notes, it looks like the SSP provides updated BMA and MMA binaries that will not impose the 256k limit on HP-UX 11.31 where such restriction is not required.

Media Agent host is MS-WIndows

Here's where things may get a little cloudy.  If you experience an error in DP attempting to use a tape device configured with a 512k block size, the cause is most likely a configurable parameter called MaximumSGList which is used by the service for your Fiber Channel (FC) Host Bus Adapter (HBA).  A little sleuthing is required to track down the proper service to modify in the registry.  A quick trip to Device Manager will give us a start.  Expand SCSI and RAID controllers then locate the FC controller(s).  In my example, we have a pair of QLogic HBAs.

 Check the properties of the HBA and look specifically for the Service property on the Details tab.  Here we see a value of ql2300.

That tells me I need to dive into the registry at \HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ql2300\Parameters\Device

Note that there is no current key named MaximumSGList.  In the absence of this key, the service will be operating at a default Scatter/Gather value.  On this particular host, I happen to know that the default value will support up to a 256k block size.  If we wanted to allow operation at 512k, it would be necessary to add a new DWORD "MaximumSGList" with a decimal value of 129.  The formula is (BlockSize/4096) + 1.

Mention is also made of MaximumSGList in DP's help facility.  For a direct hit, search on example of changing block size on windows media agent.

Note that a restart is necessary to implement the new value before testing.

With our addition made and the reboot complete, I reconfigured one of the D2D VTL drives (LTO4 emulation) to use a block size of 512k.  The format of a virtual cartridge was successful at 512k and so was the subsequent backup of data to the cartridge.

Conclusions

It is possible to achieve a 512k block size with MS-Windows and HP-UX 11i v3.  I haven't addressed Linux as a Media Agent host primarily because it wasn't possible in the test environment to which I am so graciously allowed access.  It's also important to note that you shouldn't go crazy with MaximumSGList and arbitrarily set it to some really high value.  As noted in Microsoft Support Article ID 280793:

MaximumSGList is a value between 0 and 255. Use this adjustment sparingly as scatter/gather lists consume nonpaged memory.

My profound gratitude goes to a number of people who responded to my internal queries:  Christian, Paul, Andreas, Jaime, Joerg, Mike, Nikhil, Harold, and anyone I might have missed.  It is indeed a pleasure to work with such knowledgeable and helpful folks.

Have we missed anything?  Do you have experience that can augment or clarify what is documented here?  If so, please share with us using the comments section so that we may all benefit!

[#2]

Mr T,

Great blog and thanks for all the screenshots! Convincing customers to change variables on their production servers (that are also MA's) via the registry can be challenging, especially if they decide to consult with the Storage Team first - your blog made this easy for me and the customer was also a lot more confident about what he needed to change and why.

jen

[#3]

Thank you for taking the time to comment, Jen.  I'm very pleased to know that the information was beneficial.  Please share the URL hp.com/go/dptips with your colleages and customers.  The more we work together, the more we all win!

[#4]

how about BS of 1024 in a environment with windows MA ?

And also can you provide recommendation for segment size ?

thanks

[#5]

I tried to fight larger block size then 256kb on Linux Media Agent host on DP 6.11, maybe you can provide me some info about that? First try was setting ST_MAX_SG      256 do 512 value in "st" tape driver , but it didn't work...

[#6]

Hello there!

Thank you for a such a great article... I try to fight the same on Linux Media Agent hosts... and don't have luck, still stuck to 256KB on VLS system. I tried to look on st driver sources and give a try to set up:  #define ST_MAX_SG      256 to 512 in st_options.h and "max_sg_segs:Maximum number of scatter/gather segments to use (256) (int)" setting via insmod also, no luck either. Can you provide me some directions how to fight the issue? Maybe some documents somewhere?

[#7]

Hi, I'm setting up a fully blown D2D4324 and dual site 3PAR SAN.

I was windering if it's possible to edit the blocksite via a script or something as otherwise I need to go through 60 odd drives? Have looked a bit at "omnidownload" but it doesn't seem any easier?

Great blog by the way!

[#8]

Hello,

I have a mixed set of tapes (mixed in terms of block-sizes) in my media-pool. Some are 64KB and some are 256KB. Is there any way to identify which tape is of which block size without affecting the data it contains?

Regards,

Kota :-)

[#9]

Hello all.  Sorry for my absence.  Let me do my best to answer recent queries.

Kota, I reckon you could try scanning each one first in a 64k drive then in a 256k drive to see which one doesn't throw an error.  Of course, if the tapes are scratch, why not just reformat all of them at the desired block size.  Another method is with TapeAnalyser.exe.  I did a bit of experimenting, and here's the command I honed to look at the first block of the first segment on a tape loaded in a drive:

E:\Program Files\OmniBack\bin\utilns>tapeanalyser.exe -device Tape0:0:1:0C -recdmp -range 1 1 1 1
======================================================

BLKSIZE = 65536

 You can use TapeAnalyser.exe -help for a listing of syntax and usage.

Rasser, I've faced a similar situation before.  Though the details were different, I found myself needing to update one parameter on dozens of drives.  I wound up writing a small shell script to omnidownload all of the device configs, another script using sed to make the updates, and then a third script to do the omniuploads.  I don't have them handy, or I'd list them here.  Thanks for your kind comments.

Dominik, I'm afraid I do not have a Linux box in the lab right now with D2D and physical tape, so I'm at a bit of a lost for testing.  You might pose your question on the Data Protector discussion forum:  http://h30499.www3.hp.com/t5/Data-Protector-Support-and-News/bd-p/itrc-251

sc_2111, larger block sizes are generally desired if you can achieve them.  The larger your block size, the fewer headers you will have for the same amount of data.  That's less backup app metadata for inline dedupe to sort through.

Thank you all for taking the time to comment!

[#10]

Hi

I had the same problem recently with a DP6.2 Windows CM and Linux MA clients writing to a D2D.

The Windows clients were happy with the 512K block size but the Linux clients gave the error you describe in the Blog.

I got around it for any SAN attached Linux clients by adding the following entry in .omnirc on each client:

OB2LIMITBLKSIZE=0

Regards

Barry

[#11]

Hi -

I too was able to overcome the 256k block size limitation on Linux and DP6.2 with OB2LIMITBLKSIZE=0 in .omnirc. But on HP-UX 11.31, adding OB2LIMITBLKSIZE=0 to .omnirc only changed the nature of the error message

From this:

[Warning] From: MMA@bkup02.vcd.hp.com "HP:Ultrium 5-SCSI_3_bkup02"  Time: 05/01/12 11:38:44

[90:253]        Block Size setting of 512KB is too big. Maximum allowed block size for

        this device type is 256KB.

To this:

[Critical] From: MMA@bkup02.vcd.hp.com "HP:Ultrium 5-SCSI_1_bkup02"  Time: 05/01/12 11:40:11

[90:47]         Invalid device block size for HP-UX, supports only <= (256) KB => aborting 

[Major] From: MMA@bkup02.vcd.hp.com "HP:Ultrium 5-SCSI_1_bkup02"  Time: 05/01/12 11:40:11

        Initialization of medium failed.

(Incidentally, OB2LIMITBLKSIZE is not, by default, documented in .omnirc on DP6.2!)

I have doubts that the 256k MA block size limitation has been removed in DP6.2 for HP-UX 11.31.  

The only recourse at this point - to be able to run with 512k or greater block size and fully maximize performance and dedup on our D2D4324 - is to bring up a new Linux cell server to host the virtual tape libs for our hp-ux hosts.

Any ideas?