Article Options
Technical Support Services Blog
Discover the latest trends in technology, and the technical issues customers are overcoming with the aid of HP Technology Services.

Converting a vmkernel-zdump to Text and Making Sense of it

pat-lampert| August 10, 2012
2 Comments
0

Well I’m back from a wonderful trip to Korea with my wife to visit my daughter and her husband who are teaching English in a small town called Haenam at southern tip of the Korean peninsula.  When the local High School, where my son in law teaches found out I work for HP, I even enjoyed an opportunity to speak to his High School class on career opportunities in the IT industry.

 

Anyway, as promised in my last blog entry Examining the logs from a vm-support diagnostic here is some information on what you can do if your ESX/ESXi host has crashed with what is called a Purple Screen Of Death or PSOD for short.

 

One of the pieces of information captured by the vCenter host (vm-support) diagnostic is a copy of the binary dump file when the host has crashed. This will be located in the “root” directory once you have unpacked the diagnostic. See my earlier post Unpacking and Making Sense of a VMware vm-support Diagnostic Bundle for information on unpacking the diagnostic bundle.  

 

If your vm-support contains a dump, it will have an unreadable binary file with a name in the format of vmkernel-zdump-<timestamp>. For example:  vmkernel-zdump-052111.03.12.1

 

On ESX 4 and above you can convert this file to readable text using the command esxcfg-dumppart –L. The file produced will be named vmkernel-log.1 by default.

 

Example:

# esxcfg-dumppart -L vmkernel-zdump-052111.03.12.1

Created file vmkernel-log.1

Log wrapped

#

 

The resulting vmkernel-log can be quite lengthy, thousands of lines in some cases, because it contains all recent VMkernel log messages leading up to the PSOD.  The actual crash trace-back will always be at the end of this document.

 

Example:

36:17:19:45.957 cpu7:4200)ALERT: IDT: 590: VMK DF handler: ip=0x41802e463165 sp=41802e8e75b7^[[0m

36:17:19:45.957 cpu7:4200)ALERT: Bluescreen: 375: PSOD level 1: #DF Exception(8) in world 4200:helper20-13 @ 0x41802e463165

LBR: from 0x41802e463160 to 0x41802e463164

 ra=0x41802e422c6e^[[0m

VMware ESX [Releasebuild-208167 X86_64]

#PF Exception(14) in world 4100:idle4 ip 0xff addr 0xff

LBR: from 0x41802e7bbf90 to 0xff

cr2=0xff cr3=0xcfe27000 cr4=0x16c

frame=0x4100c00275d8 ip=0xff err=16 rflags=0x10002

rax=0x4100c03479d0 rbx=0x4100c0347a10 rcx=0x0

rdx=0x0 rbp=0x4100c00276c8 rsi=0x3

rdi=0x4100c03479b8 r8=0x0 r9=0x1a4d23a7c00000

r10=0xbadc0ded r11=0xbadc0ded r12=0xc0347b00

r13=0x1 r14=0x4100c0347990 r15=0x0

0:4096/console 1:4515/vmm1:CDSV 2:4940/vmm1:CDSV 3:4388/vmm0:CDSV

*4:4100/idle4 5:4995/vmm0:CDSV 6:4939/vmm0:CDSV 7:4200/helper20-

@BlueScreen: #PF Exception(14) in world 4100:idle4 ip 0xff addr 0xff

LBR: from 0x41802e7bbf90 to 0xff

Code starts at  0x41802e400000

0x4100c00276c8:[0xff]Unknown stack: 0x417fef2343f0

0x4100c0027718:[0x41802e7bc12e]__wake_up+0x6d stack: 0x1

0x4100c0027758:[0x41802e8e75af]lpfc_sli_wake_iocb_wait+0x9e stack: 0x4100c0027778

0x4100c0027978:[0x41802e8e29c9]lpfc_sli_handle_fast_ring_event+0x4ac stack: 0x4

0x4100c0027a08:[0x41802e8e7b76]lpfc_intr_handler+0x145 stack: 0x410000000022

0x4100c0027a48:[0x41802e7b8208]Linux_IRQHandler+0x77 stack: 0x4100c0027a68

0x4100c0027ab8:[0x41802e42e0e5]IDTDoInterrupt+0x310 stack: 0x4100c0027ba0

0x4100c0027ae8:[0x41802e42e484]IDT_HandleInterrupt+0x8b stack: 0x417fee65c700

0x4100c0027b08:[0x41802e42e9f2]IDT_IntrHandler+0x91 stack: 0x4018

0x4100c0027c18:[0x41802e4a9b16]gate_entry+0x25 stack: 0x4100c0027d00

0x4100c0027c48:[0x41802e42e524]IDT_HandleInterrupt+0x12b stack: 0x418040000000

0x4100c0027c68:[0x41802e42e9f2]IDT_IntrHandler+0x91 stack: 0x4018

0x4100c0027d48:[0x41802e4a9b16]gate_entry+0x25 stack: 0x410006409040

0x4100c0027e58:[0x41802e5a09ce]CpuSchedIdleLoopInt+0x8f9 stack: 0x4100c0027e88

0x4100c0027e68:[0x41802e5a2847]CpuSched_IdleLoop+0x16 stack: 0x2

0x4100c0027e88:[0x41802e431753]Init_SlaveIdle+0x9e stack: 0x0

0x4100c0027fe8:[0x41802e646a56]SMPSlaveIdle+0x3d1 stack: 0x0

VMK uptime: 36:17:19:45.964 TSC: 7403164751824695

FSbase (0x0) GSbase (0x0) kernelGSbase (0x0)

36:17:19:45.957 cpu7:4200)IDT: 590: VMK DF handler: ip=0x41802e463165 sp=41802e8e75b7

36:17:19:45.957 cpu7:4200)Bluescreen: 375: PSOD level 1: #DF Exception(8) in world 4200:helper20-13 @ 0x41802e463165

LBR: from 0x41802e463160 to 0x41802e463164

Starting coredump to disk.

 

This will reflect the same information that appears on the screen during the crash, but viewing it in this format has several benefits.

 

1.     You’ll have the complete trace-back in case some of it scrolled off your console display.

2.     It contains additional messages leading to the panic.

3.     Rather than just taking an image of the console it’s in a much more portable text format, from which you can cut and paste for searches etc.

 

Most VMware ESX PSODs tend to be hardware related. In some cases you may be able to identify the source of the problem without even needing to elevate to VMware. In the above example even without VMware sources we can already tell that the problem was related to the Emulex Light Pulse Fibre Channel HBA so something was going on with the Storage Network.

 

For additional information:

 

Here is an excellent KB article from VMware on interpreting a PSOD:

http://kb.vmware.com/kb/1004250

 

And another KB article showing the basics of how to extract information from a dump:

http://kb.vmware.com/kb/1006796

 

 

Comments
it consulting nj(anon) | ‎09-04-2012 08:56 AM
Options

Thanks a lot for your great posting,I love this post very much,please keep sharing of knowledges with us.

0
Local It Support(anon) | ‎11-03-2012 01:50 PM
Options

Thanks for sharing this it related information. I didn't know it.....

0
Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the community guidelines.
Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
 
Search
turn on suggestions Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
About the Author
  • More than 30 years in Sales and Marketing in IT services business. Currently managing global campaigns for Datacenter Care.
  • I graduated in Software Engineering. Joined HP family five years ago, I deliver Insight Remote Support technical consulting for HP customers, in North America, Canada and Latin America. Assist setting up, installing and configuring the solution in customers' IT environments.
  • I am an identical twin. My brother’s name is Greg Tinker and we have been extremely fortunate working similar careers within HP, known to our HP colleagues and many of our customers as "The Tinkers". Our job is to be the technical lead on major business operational outages with millions of Dollars/Euros hanging in the balance. We both have a complete background in architectural, Infrastructure and application environments from both the proactive and reactive side of HP Enterprise Service (HP ES), and HP Enterprise Business (HP EB).
  • I am an identical twin. My brother’s name is Chris Tinker and we have been extremely fortunate working similar careers within HP, known to our HP colleagues and many of our customers as "The Tinkers". Our job is to be the technical lead on major business operational outages with millions of Dollars/Euros hanging in the balance. We both have a complete background in architectural, Infrastructure and application environments from both the proactive and reactive side of HP Enterprise Service (HP ES), and HP Enterprise Business (HP EB). We have always attended the same schools, studied the same material (big surprise, as we are identical twins), and have always worked as a close team and strive to demonstrate our teaming ability’s to others. We each have more than 11 years experience supporting mission-critical enterprise customers on a broad range of technologies. We’ve both won the HP MVP award multiple times as well as coauthored books, programs, and whitepapers in our spare time.
  • More than 25 years in the IT industry, managing ITSM, service development and delivery projects in Technology Services. Specialized in end2end support for ISV based business solutions. Certified ITIL and project management expert.
  • Eduardo Zepeda, WW TS Social Media Program Manager & Internal Communications for WW Technology Services Blogging on behalf of HP Technology Services (TS_Guest)
  • I have been with HP for 13 years, always in Services - first as a Services Channel Sales rep, then a Channel Services Segment Manager, and now, in WW Technology Services Marketing. These may be my formal job titles, but I'm really a Cheerleader for HP Services! I feel that HP has great services, exceptional Technical Experts and Delivery teams, and so many cool things are going on at HP Services. So, stay tuned...
  • I have 27 years of system, storage, and networking experience including detailed work with Data Protector (formerly Omniback II) for the past 14 years. My expertise includes StoreOnce deduplication technology, D2D appliances, performance tuning, complex remediation, and online backup integration with applications like Oracle and infrastructure like VMware. Traveling across the United States and Canada as a Sr. Technical Consultant, I deliver specialized consulting for a broad variety of HP customers.
  • MrCollaboration (aka Jim Evans) is an HP Global Services Alliance Manager. He has worked in the IT industry for more than 30 years, 22 of which were spent with Digital Equipment Corporation, Compaq and HP. He works with many third party vendors and partners to develop processes to facilitate excellent support and service for mutual customers. Jim is also HP’s representative to the Technical Support Alliance Network (TSANet).
  • I've been working in Customer Service for over 20 years. During my career I've provided support services for Languages, Programming Libraries and Operating Systems. During the last 10 years I've provide support for Linux and more recently VMware. My current role is as a Technical Account Manager working in the HP Custom Mission Critical Services Industry Standard Operating Systems team. I provide both reactive and proactive operating system support for proLiant servers and blades. Our services in the Custom teams are built on statement of work contracts for large HP customers who need a customized mission critical support offering.
  • I've been working in HP since 2007 like IT agent, developer, Web designer and then like Web Project Manager
  • I like to listen as much as I like to talk. Why? My 25+ years in the technology industry has taught me that the key to delivering value to customers is to understand what they value in the first place! I developed this passion for customers and consultative selling during my 12 years with Accenture, and I have continued to approach customers in a consultative way during my 12+ year tenure with HP. I also have a passion for HP given my knowledge of our Product and Service Portfolio and the differentiators we possess that position us as a leader in the areas our customers are telling us they want to go. Converged Infrastructure, Converged Cloud, Big Data – and the associated Service and Support implications – all such exciting technology trends where our success will hinge upon our ability to differentiate ourselves versus others in the areas that matter most to our customers. Right up my alley, and I am proud to be part of the great HP team where I know we have the best solutions in the industry!
  • Tom Clement has over 30 years experience in the areas of adult learning, secondary education, and leadership development. During this time Tom has been a consistent champion of “non-traditional” training delivery methods, including blended learning, virtual delivery (self paced and instructor led), the use of training games and simulations, and experiential learning. Tom has spent the past 25 years of his career at Hewlett Packard, focused most recently on HP’s global Virtualization, Cloud, and Converged Infrastructure customer training programs. Tom manages the strategic direction and overall performance of these training programs, ensuring these worldwide programs help HP’s customers capitalize on the business opportunities made available by IT advancements in each of these subject areas. Tom and his global teammates utilize best in class instructors, course content and supporting equipment infrastructure to deliver these training programs to HP’s customers. The team prides itself on providing the Virtualization, Cloud, and Converged Infrastructure content customers need when and where they need it, anywhere in the world. Tom is based in the Washington, DC suburbs and can be reached at tom.clement@hp.com.
Follow Us