Performance – HPUX, Qlogic, ZIO, Interrupt

[#1]

Performance – HPUX, Qlogic, ZIO, Interrupt

By: Greg Tinker

One of the more dreaded statements in technology support services concerning performance is “Performance is not meeting expectations.…” Technology based performance issues are among the most difficult due to the complexities with the application layer (user space), kernel and physical hardware layers.  Identifying the cause of performance degradation takes expertise in the affected technology areas, expert execution of action plans in order to obtain the “right” data, and most of the expert level experience that is found in HP Technology Services. 

Throughout the life of the Technology Support Services blog, we will have many performance based discussions.  To that end, today’s discussion draws attention to a single entity that can cause considerable performance impact and presents itself as a storage array bottleneck.

Environment:

• Qlogic HBA (exact model does not matter)
• HPUX:  11.XX

Any O.S. that uses Qlogic HBA & Driver with ZIO parameter can have this issue…   Typically only seen in enterprise UNIX shops that drive thousands of IOPS

• Storage: Any Storage, but usually seen on Enterprise High end Arrays (Hitachi {HDS}, HP XP/P9000 &> , EMC DMX/V-MAX &>)

Case Study:

In environments that have heavy sequential I/O or streaming I/O (batch jobs etc…) throughput appears to be “starved” on disk I/O yet disk latency is low.

Though this condition has many possible causes such as IO profile, IO access type, cache usage, etc, the problem may be as simple as an interruption.

Single Item to focus on:

Qlogic ZIO (Zero Interrupt delay) is a feature that was introduced many years ago to start collating I/O interrupts to reduce the number of interrupts to a given CPU and allow multiple SCSI commands to be completed during a single system interrupt. 

Quick background on ZIO:

    To view the ZIO parameter:

    /opt/fcms/bin/fcmsutil device_file get_int_delay

    To change the ZIO Parameter

    /opt/fcms/bin/fcmsutil device_file [-f] set_int_delay { off | [-z {5|6}] value }

(Other O.S.’s refer to Qlogic.com user manual…)

ZIO disabled: Not Recommended.

•   Syntax: fcdutil /dev/fcdx set_int_delay off

The ZIO is disabled.  The system will process one SCSI command per Interrupt. This behavior, though once the norm, is not recommended today as CPU utilization/optimization is a huge concern.

ZIO mode 5: (NEW Default as of March 2010 driver for HPUX)

• Syntax: fcdutil /dev/fcdx set_int_delay -z 5 1

ZIO mode 5 is enabled.  SCSI command are transferred by DMA into the response queue with minimal interrupts.

Note: in order to be persistent over reboots the driver containing the new default parameter needs to be loaded. example 11.31.1003.01

ZIO mode 6: (default with older driver than 11.31.1003.01)

• fcdutil /dev/fcdx set_int_delay -z 6 1

ZIO mode 6 is enabled. Mode is similar to mode 5, except that an interrupt is generated when the firmware has no active exchanges.

Additional note: This behavior is true in other versions of HPUX 11.XX, but only one is provided above for discussion sake.

In the above case, using a ZIO setting of 5, or SCSI commands transferred by DMA into the response queue you may find a substantial increase in overall performance verses using 6 or disabling ZIO.

If you find yourself having performance concerns on HPUX with no smoking gun, please remember to check under the hood (ZIO) and make sure your not bond on interrupts.

[#2]

# fcmsutil /dev/fcd0 | grep 'Driver Version'
                         Driver Version = @(#) fcd B.11.31.1009 May  9 2010
# fcmsutil /dev/fcd0 get_int_delay | grep ZIO
ZIO Mode (Default: 6)                    : 6

It looks to me as if the default ist still 6 with FibrChanl-01 B.11.31.1009...

[#3]

Thank you for submitting your response.

This evening, I placed some calls to the lab and it turns out that this setting has once again changed. The default, as of the latest driver, is to have ZIO set to 6. 

As a result of having to use ZIO of 5 on some heavily loaded IO systems, timing enhancements were made to the interrupt routines in the driver to make better use of the Interrupt delay sequences.  These enhancements are reflected in the latest drivers and allow for the most optimum setting of 6 to be used.

Thanks again for your post.

[#4]

Chris, Thanks for researching...  I was not aware of it changing back.   

Michael, Welcome to our blog...

[#5]

Thanks for the update! As long as the default is the optimum I'm happy.. ignorance is bliss for this clueless sysadmin ;-)

[#6]

thanx