Technical Support Services Blog
Discover the latest trends in technology, and the technical issues customers are overcoming with the aid of HP Technology Services.

Performance – HPUX, Qlogic, ZIO, Interrupt

Performance – HPUX, Qlogic, ZIO, Interrupt

By: Greg Tinker

 

One of the more dreaded statements in technology support services concerning performance is “Performance is not meeting expectations.…” Technology based performance issues are among the most difficult due to the complexities with the application layer (user space), kernel and physical hardware layers.  Identifying the cause of performance degradation takes expertise in the affected technology areas, expert execution of action plans in order to obtain the “right” data, and most of the expert level experience that is found in HP Technology Services. 

 

Throughout the life of the Technology Support Services blog, we will have many performance based discussions.  To that end, today’s discussion draws attention to a single entity that can cause considerable performance impact and presents itself as a storage array bottleneck.

 

Environment:

  • Qlogic HBA (exact model does not matter)
  • HPUX:  11.XX

Any O.S. that uses Qlogic HBA & Driver with ZIO parameter can have this issue…   Typically only seen in enterprise UNIX shops that drive thousands of IOPS

 

  • Storage: Any Storage, but usually seen on Enterprise High end Arrays (Hitachi {HDS}, HP XP/P9000 &> , EMC DMX/V-MAX &>)

 

Case Study:

In environments that have heavy sequential I/O or streaming I/O (batch jobs etc…) throughput appears to be “starved” on disk I/O yet disk latency is low.

 

Though this condition has many possible causes such as IO profile, IO access type, cache usage, etc, the problem may be as simple as an interruption.

 

Single Item to focus on:

Qlogic ZIO (Zero Interrupt delay) is a feature that was introduced many years ago to start collating I/O interrupts to reduce the number of interrupts to a given CPU and allow multiple SCSI commands to be completed during a single system interrupt. 

Quick background on ZIO:

 

    To view the ZIO parameter:

    /opt/fcms/bin/fcmsutil device_file get_int_delay

 

    To change the ZIO Parameter

    /opt/fcms/bin/fcmsutil device_file [-f] set_int_delay { off | [-z {5|6}] value }

 

(Other O.S.’s refer to Qlogic.com user manual…)

 

 

ZIO disabled: Not Recommended.

  •   Syntax: fcdutil /dev/fcdx set_int_delay off

 

The ZIO is disabled.  The system will process one SCSI command per Interrupt. This behavior, though once the norm, is not recommended today as CPU utilization/optimization is a huge concern.

 

ZIO mode 5: (NEW Default as of March 2010 driver for HPUX)

  • Syntax: fcdutil /dev/fcdx set_int_delay -z 5 1

ZIO mode 5 is enabled.  SCSI command are transferred by DMA into the response queue with minimal interrupts.

 

Note: in order to be persistent over reboots the driver containing the new default parameter needs to be loaded. example 11.31.1003.01

 

ZIO mode 6: (default with older driver than 11.31.1003.01)

  • fcdutil /dev/fcdx set_int_delay -z 6 1

ZIO mode 6 is enabled. Mode is similar to mode 5, except that an interrupt is generated when the firmware has no active exchanges.

 

Additional note: This behavior is true in other versions of HPUX 11.XX, but only one is provided above for discussion sake.

 

In the above case, using a ZIO setting of 5, or SCSI commands transferred by DMA into the response queue you may find a substantial increase in overall performance verses using 6 or disabling ZIO.

 

If you find yourself having performance concerns on HPUX with no smoking gun, please remember to check under the hood (ZIO) and make sure your not bond on interrupts.

Labels: Qlogic ZIO
Comments
Michael Leu(anon) | ‎09-26-2010 12:29 AM

# fcmsutil /dev/fcd0 | grep 'Driver Version'
                         Driver Version = @(#) fcd B.11.31.1009 May  9 2010
# fcmsutil /dev/fcd0 get_int_delay | grep ZIO
ZIO Mode (Default: 6)                    : 6

 

It looks to me as if the default ist still 6 with FibrChanl-01 B.11.31.1009...

ChrisTinker | ‎09-28-2010 12:18 AM

Thank you for submitting your response.

 

This evening, I placed some calls to the lab and it turns out that this setting has once again changed. The default, as of the latest driver, is to have ZIO set to 6. 

 

As a result of having to use ZIO of 5 on some heavily loaded IO systems, timing enhancements were made to the interrupt routines in the driver to make better use of the Interrupt delay sequences.  These enhancements are reflected in the latest drivers and allow for the most optimum setting of 6 to be used.

 

Thanks again for your post.

 

GregTinker | ‎09-28-2010 12:36 AM

Chris, Thanks for researching...  I was not aware of it changing back.  :smileyhappy: 

 

Michael, Welcome to our blog...

Michael Leu(anon) | ‎09-28-2010 10:38 AM

Thanks for the update! As long as the default is the optimum I'm happy.. ignorance is bliss for this clueless sysadmin ;-)

akhil(anon) | ‎05-27-2012 02:56 AM

thanx 

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the community guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
About the Author
I am an identical twin. My brother’s name is Chris Tinker and we have been extremely fortunate working similar careers within HP, known to ...
About the Author(s)
  • I graduated in Software Engineering. Joined HP family five years ago, I deliver Insight Remote Support technical consulting for HP customers, in North America, Canada and Latin America. Assist setting up, installing and configuring the solution in customers' IT environments.
  • I am an identical twin. My brother’s name is Greg Tinker and we have been extremely fortunate working similar careers within HP, known to our HP colleagues and many of our customers as "The Tinkers". Our job is to be the technical lead on major business operational outages with millions of Dollars/Euros hanging in the balance. We both have a complete background in architectural, Infrastructure and application environments from both the proactive and reactive side of HP Enterprise Service (HP ES), and HP Enterprise Business (HP EB).
  • I am an identical twin. My brother’s name is Chris Tinker and we have been extremely fortunate working similar careers within HP, known to our HP colleagues and many of our customers as "The Tinkers". Our job is to be the technical lead on major business operational outages with millions of Dollars/Euros hanging in the balance. We both have a complete background in architectural, Infrastructure and application environments from both the proactive and reactive side of HP Enterprise Service (HP ES), and HP Enterprise Business (HP EB). We have always attended the same schools, studied the same material (big surprise, as we are identical twins), and have always worked as a close team and strive to demonstrate our teaming ability’s to others. We each have more than 11 years experience supporting mission-critical enterprise customers on a broad range of technologies. We’ve both won the HP MVP award multiple times as well as coauthored books, programs, and whitepapers in our spare time.
  • More than 25 years in the IT industry, managing ITSM, service development and delivery projects in Technology Services. Specialized in end2end support for ISV based business solutions. Certified ITIL and project management expert.
  • Eduardo Zepeda, WW TS Social Media Program Manager & Internal Communications for WW Technology Services Blogging on behalf of HP Technology Services (TS_Guest)
  • I have been with HP for 13 years, always in Services - first as a Services Channel Sales rep, then a Channel Services Segment Manager, and now, in WW Technology Services Marketing. These may be my formal job titles, but I'm really a Cheerleader for HP Services! I feel that HP has great services, exceptional Technical Experts and Delivery teams, and so many cool things are going on at HP Services. So, stay tuned...
  • Mr_T? Yes, but in name only. No mohawk or gold chains. In real life I'm Jim Turner, a Master Technologist with HP's Integration and Technical Services team. I leverage my 27 years of system, storage, and networking experience to ensure optimal performance with Data Protector and StoreOnce technologies. Many times the source of a problem is not where it appears to be. A broader view of the landscape and a deeper knowledge of system internals are frequently required, and that's precisely what I deliver.
  • MrCollaboration (aka Jim Evans) is an HP Global Services Alliance Manager. He has worked in the IT industry for more than 30 years, 22 of which were spent with Digital Equipment Corporation, Compaq and HP. He works with many third party vendors and partners to develop processes to facilitate excellent support and service for mutual customers. Jim is also HP’s representative to the Technical Support Alliance Network (TSANet).
  • I've been working in Customer Service for over 20 years. During my career I've provided support services for Languages, Programming Libraries and Operating Systems. During the last 10 years I've provide support for Linux and more recently VMware. My current role is as a Technical Account Manager working in the HP Custom Mission Critical Services Industry Standard Operating Systems team. I provide both reactive and proactive operating system support for proLiant servers and blades. Our services in the Custom teams are built on statement of work contracts for large HP customers who need a customized mission critical support offering.
  • I like to listen as much as I like to talk. Why? My 25+ years in the technology industry has taught me that the key to delivering value to customers is to understand what they value in the first place! I developed this passion for customers and consultative selling during my 12 years with Accenture, and I have continued to approach customers in a consultative way during my 12+ year tenure with HP. I also have a passion for HP given my knowledge of our Product and Service Portfolio and the differentiators we possess that position us as a leader in the areas our customers are telling us they want to go. Converged Infrastructure, Converged Cloud, Big Data – and the associated Service and Support implications – all such exciting technology trends where our success will hinge upon our ability to differentiate ourselves versus others in the areas that matter most to our customers. Right up my alley, and I am proud to be part of the great HP team where I know we have the best solutions in the industry!
  • Tom Clement has over 30 years experience in the areas of adult learning, secondary education, and leadership development. During this time Tom has been a consistent champion of “non-traditional” training delivery methods, including blended learning, virtual delivery (self paced and instructor led), the use of training games and simulations, and experiential learning. Tom has spent the past 25 years of his career at Hewlett Packard, focused most recently on HP’s global Virtualization, Cloud, and Converged Infrastructure customer training programs. Tom manages the strategic direction and overall performance of these training programs, ensuring these worldwide programs help HP’s customers capitalize on the business opportunities made available by IT advancements in each of these subject areas. Tom and his global teammates utilize best in class instructors, course content and supporting equipment infrastructure to deliver these training programs to HP’s customers. The team prides itself on providing the Virtualization, Cloud, and Converged Infrastructure content customers need when and where they need it, anywhere in the world. Tom is based in the Washington, DC suburbs and can be reached at tom.clement@hp.com.


Follow Us
Top Kudoed Posts