Technical Support Services Blog
Discover the latest trends in technology, and the technical issues customers are overcoming with the aid of HP Technology Services.

Performance – HPUX, Qlogic, ZIO, Interrupt

Performance – HPUX, Qlogic, ZIO, Interrupt

By: Greg Tinker

 

One of the more dreaded statements in technology support services concerning performance is “Performance is not meeting expectations.…” Technology based performance issues are among the most difficult due to the complexities with the application layer (user space), kernel and physical hardware layers.  Identifying the cause of performance degradation takes expertise in the affected technology areas, expert execution of action plans in order to obtain the “right” data, and most of the expert level experience that is found in HP Technology Services. 

 

Throughout the life of the Technology Support Services blog, we will have many performance based discussions.  To that end, today’s discussion draws attention to a single entity that can cause considerable performance impact and presents itself as a storage array bottleneck.

 

Environment:

  • Qlogic HBA (exact model does not matter)
  • HPUX:  11.XX

Any O.S. that uses Qlogic HBA & Driver with ZIO parameter can have this issue…   Typically only seen in enterprise UNIX shops that drive thousands of IOPS

 

  • Storage: Any Storage, but usually seen on Enterprise High end Arrays (Hitachi {HDS}, HP XP/P9000 &> , EMC DMX/V-MAX &>)

 

Case Study:

In environments that have heavy sequential I/O or streaming I/O (batch jobs etc…) throughput appears to be “starved” on disk I/O yet disk latency is low.

 

Though this condition has many possible causes such as IO profile, IO access type, cache usage, etc, the problem may be as simple as an interruption.

 

Single Item to focus on:

Qlogic ZIO (Zero Interrupt delay) is a feature that was introduced many years ago to start collating I/O interrupts to reduce the number of interrupts to a given CPU and allow multiple SCSI commands to be completed during a single system interrupt. 

Quick background on ZIO:

 

    To view the ZIO parameter:

    /opt/fcms/bin/fcmsutil device_file get_int_delay

 

    To change the ZIO Parameter

    /opt/fcms/bin/fcmsutil device_file [-f] set_int_delay { off | [-z {5|6}] value }

 

(Other O.S.’s refer to Qlogic.com user manual…)

 

 

ZIO disabled: Not Recommended.

  •   Syntax: fcdutil /dev/fcdx set_int_delay off

 

The ZIO is disabled.  The system will process one SCSI command per Interrupt. This behavior, though once the norm, is not recommended today as CPU utilization/optimization is a huge concern.

 

ZIO mode 5: (NEW Default as of March 2010 driver for HPUX)

  • Syntax: fcdutil /dev/fcdx set_int_delay -z 5 1

ZIO mode 5 is enabled.  SCSI command are transferred by DMA into the response queue with minimal interrupts.

 

Note: in order to be persistent over reboots the driver containing the new default parameter needs to be loaded. example 11.31.1003.01

 

ZIO mode 6: (default with older driver than 11.31.1003.01)

  • fcdutil /dev/fcdx set_int_delay -z 6 1

ZIO mode 6 is enabled. Mode is similar to mode 5, except that an interrupt is generated when the firmware has no active exchanges.

 

Additional note: This behavior is true in other versions of HPUX 11.XX, but only one is provided above for discussion sake.

 

In the above case, using a ZIO setting of 5, or SCSI commands transferred by DMA into the response queue you may find a substantial increase in overall performance verses using 6 or disabling ZIO.

 

If you find yourself having performance concerns on HPUX with no smoking gun, please remember to check under the hood (ZIO) and make sure your not bond on interrupts.

Labels: Qlogic ZIO
Comments
Michael Leu | ‎09-26-2010 12:29 AM

# fcmsutil /dev/fcd0 | grep 'Driver Version'
                         Driver Version = @(#) fcd B.11.31.1009 May  9 2010
# fcmsutil /dev/fcd0 get_int_delay | grep ZIO
ZIO Mode (Default: 6)                    : 6

 

It looks to me as if the default ist still 6 with FibrChanl-01 B.11.31.1009...

ChrisTinker | ‎09-28-2010 12:18 AM

Thank you for submitting your response.

 

This evening, I placed some calls to the lab and it turns out that this setting has once again changed. The default, as of the latest driver, is to have ZIO set to 6. 

 

As a result of having to use ZIO of 5 on some heavily loaded IO systems, timing enhancements were made to the interrupt routines in the driver to make better use of the Interrupt delay sequences.  These enhancements are reflected in the latest drivers and allow for the most optimum setting of 6 to be used.

 

Thanks again for your post.

 

GregTinker | ‎09-28-2010 12:36 AM

Chris, Thanks for researching...  I was not aware of it changing back.  :smileyhappy: 

 

Michael, Welcome to our blog...

Michael Leu | ‎09-28-2010 10:38 AM

Thanks for the update! As long as the default is the optimum I'm happy.. ignorance is bliss for this clueless sysadmin ;-)

akhil | ‎05-27-2012 02:56 AM

thanx 

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the community guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
Showing results for 
Search instead for 
Do you mean 
About the Author
I am an identical twin. My brother’s name is Chris Tinker and we have been extremely fortunate working similar careers within HP, known to ...
Featured


Follow Us
Top Kudoed Posts
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.