Article Options
Technical Support Services Blog
Discover the latest trends in technology, and the technical issues customers are overcoming with the aid of HP Technology Services.

Examining the logs from a vm-support diagnostic

pat-lampert| June 22, 2012
Post a Comment
0

Virtual Machine Logs

 

As I described in my last post; after unpacking a vm-support you’ll have several subdirectories.

 

bootbank  etc  locker  proc  tmp  usr  var  vmfs

 

Just as on a real ESXi host the VM logs are contained under the vmfs directory. One key difference is that the vm-support only captures logs for VMs that are registered on the host the vm-support was taken from. If you need the logs from a specific VM you’ll need to first identify which host currently is running that VM then capture a vm-support from that host. Under the vmfs directory you’ll see one or more lun ids.  You can translate these lun IDs to real names by looking in the tmp information file volume_list.vpa.txt.

 

Typically the directories under each volume are named after the VM they contain. For example:

 

vmfs/volumes/4d3deaf5-fdcd98ab-5d96-001f29eb6984:

VM1 VM2

 

vmfs/volumes/4d3deb1b-5ab31db5-133b-001f29eb6984:

PatsLinuxVM3 TerrysWindowsVM4

 

Let’s burrow down further into one of these VM directories.The vm-support captures the VM definition file and files describing the Virtual disks along with logs and a file named filelist.txt which contain the names of all the files that were in the directory.

 

VM1.vmdk  VM1.vmx   filelist.127677508.txt  vmware-1.log  vmware-3.log  vmware.log

VM1.vmsd  VM1.vmxf  vmdk-types.txt          vmware-2.log  vmware-4.log

 

The logs periodically roll. The most current VM log is vmware.log vmware-1.log is the next oldest and so on.

 

You can get a good picture of how the VM is configured by looking at the .vmx file. This is where all the answers are stored when you create a VM and this file basically defines the VM for VMware. The logs contain messages specific to that VM from a VMware perspective. You’ll be able to determine when the VM has been powered on, off, migrated, etc. You can also examine this log for any VM specific errors. This log should not be confused with OS logs within the VM itself; rather they should be viewed as virtual hardware logs.

 

 

ESX/ESXi host logs

 

The logs for the actual physical host are located under the var/log directory.

 

Like the VM logs, these logs also rotate, with the current log having no number and older logs labeled .1 .2 .3 etc. On an ESX host there are two sets of logs. The messages log contains the console VM log information and the vmkernel log contains the ESX host hypervisor information. On an ESXi host, since there is no separate console VM, all messages are logged to the messages file.

 

Like the VM logs, if you are looking for a specific event all of the entries are time stamped, however VMware is not exactly consistent with their time stamp format. The VM logs list milliseconds, where the host logs don’t. It gets even worse when we look at the agent logs later. The format is completely different. Anyway, with these stamps it’s possible to correlate events from VMs recorded in the VM logs with host events recorded in /var/log/messages.

 

A typical approach to looking at a problem is to get a rough time frame for an event, such as a host locking up or a VM locking up, then use this to search the logs for time stamps just prior to that event. Even without a strong knowledge of the error logging messages you can often locate a solution by picking out suspicious events or items specifically labeled as errors, then searching for KB articles and other web resources to explain these messages.

 

Sometimes the error message will even contain an explanation and a pointer to a KB article such as this example:

 

May 06 20:06:47.186: vcpu-0| [msg.hbacommon.locklost] The lock protecting /vmfs/volumes/

4d260c91-0ea6ed28-5af0-0017a4770008/VMname/VMname.vmdk has been lost.

May 06 20:06:47.186: vcpu-0| This is most likely due to underlying storage having

May 06 20:06:47.186: vcpu-0| problems,resulting in this virtual machine getting powered-on

May 06 20:06:47.186: vcpu-0|  on another ESX host as well.This virtual machine needs

May 06 20:06:47.186: vcpu-0| to be powered off on this host now. Kindly confirm that

May 06 20:06:47.186: vcpu-0| the virtual machine is running successfully on another host

May 06 20:06:47.186: vcpu-0| before clicking the OK button

May 06 20:06:47.186: vcpu-0|

May 06 20:06:47.186: vcpu-0| For more information regarding this problem kindly refer to

May 06 20:06:47.186: vcpu-0| http://kb.vmware.com/kb/1006936 

May 07 05:53:26.144: vcpu-0| Msg_Question: msg.hbacommon.locklost reply=0

May 07 05:53:26.145: vcpu-0| Exiting because of failed disk operation.

 

 

VMware Agent Logs

 

Besides the VM and Host logs there are also important logs from the VMware agents that are captured by vm-support. These logs are located under var/log/vmware. They record communication between vCenter and the vSphere client and the hosts. They also record any errors these agents encounter.

 

The hostd logs, located under /var/log/vmware, contain information on the agent that manages and configures the ESX host and virtual machines. Depending on the version of ESX/ESXi you may find the older logs have been gzipped. You’ll need to uncompress these with gunzip before examining them. The time stamp on these logs is also formatted differently than the messages and VM logs.

 

[2012-05-02 15:09:34.836 10FA1B90 verbose 'ha-license-manager'] Load: Loading existing file: /etc/vmware/license.cfg

[2012-05-02 15:09:51.936 112A8B90 verbose 'DvsManager'] PersistAllDvsInfo called

[2012-05-02 15:09:52.682 10FA1B90 verbose 'DvsTracker'] FetchSwitches: added 1 items

[2012-05-02 15:09:52.682 10FA1B90 verbose 'DvsTracker'] FetchDVPortgroups: added 10 items

 

Like the other VM logs, you’ll want to identify the time of an event and look at the messages surrounding it. Once again, you can search key words such as “error” or “warning” or “failed” and without having a complete understanding of all messages you can search for matches in KB articles and on the web.

 

Another important set of agent logs are the vpxa logs located under /var/log/vmware/vpx. These logs record the activity of the agent that communicates with vCenter. They have a format similar to the hostd logs. You can examine these logs to obtain more detail from the host side about any warnings or messages you see in vCenter.

 

Section for VMware VirtualCenter Agent, pid=126218680, version=5.0.0, build=build-623373, option=Release

2012-05-02T17:42:36.910Z [17A78B90 verbose 'Default' opID=SWI-a4932b8] [VpxaMoVm::CheckMoVm] did not find a VM with ID 39 in the vmList

2012-05-02T17:42:36.922Z [17A78B90 verbose 'Default' opID=SWI-a4932b8] [VpxaAlarm] VM with vmid = 39 not found

 

Almost all VMware administrators have encountered problems where the agents, for one reason or another, have had to be restarted. Examination of the hostd or vpxa logs might explain what led up to the problem.

 

There are other logs and files contained in the vmware-support diagnostic bundle. You could almost write a book about just the diagnostics. If your host has crashed, vm-support will also capture a dump file.  I’ll be away for a few weeks on vacation, but when I return my next post will show how to convert this file to text and make some sense of it.

Tags: VMware
Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the community guidelines.
Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
 
Search
turn on suggestions Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
About the Author
  • More than 30 years in Sales and Marketing in IT services business. Currently managing global campaigns for Datacenter Care.
  • I graduated in Software Engineering. Joined HP family five years ago, I deliver Insight Remote Support technical consulting for HP customers, in North America, Canada and Latin America. Assist setting up, installing and configuring the solution in customers' IT environments.
  • I am an identical twin. My brother’s name is Greg Tinker and we have been extremely fortunate working similar careers within HP, known to our HP colleagues and many of our customers as "The Tinkers". Our job is to be the technical lead on major business operational outages with millions of Dollars/Euros hanging in the balance. We both have a complete background in architectural, Infrastructure and application environments from both the proactive and reactive side of HP Enterprise Service (HP ES), and HP Enterprise Business (HP EB).
  • I am an identical twin. My brother’s name is Chris Tinker and we have been extremely fortunate working similar careers within HP, known to our HP colleagues and many of our customers as "The Tinkers". Our job is to be the technical lead on major business operational outages with millions of Dollars/Euros hanging in the balance. We both have a complete background in architectural, Infrastructure and application environments from both the proactive and reactive side of HP Enterprise Service (HP ES), and HP Enterprise Business (HP EB). We have always attended the same schools, studied the same material (big surprise, as we are identical twins), and have always worked as a close team and strive to demonstrate our teaming ability’s to others. We each have more than 11 years experience supporting mission-critical enterprise customers on a broad range of technologies. We’ve both won the HP MVP award multiple times as well as coauthored books, programs, and whitepapers in our spare time.
  • More than 25 years in the IT industry, managing ITSM, service development and delivery projects in Technology Services. Specialized in end2end support for ISV based business solutions. Certified ITIL and project management expert.
  • Eduardo Zepeda, WW TS Social Media Program Manager & Internal Communications for WW Technology Services Blogging on behalf of HP Technology Services (TS_Guest)
  • I have been with HP for 13 years, always in Services - first as a Services Channel Sales rep, then a Channel Services Segment Manager, and now, in WW Technology Services Marketing. These may be my formal job titles, but I'm really a Cheerleader for HP Services! I feel that HP has great services, exceptional Technical Experts and Delivery teams, and so many cool things are going on at HP Services. So, stay tuned...
  • I have 27 years of system, storage, and networking experience including detailed work with Data Protector (formerly Omniback II) for the past 14 years. My expertise includes StoreOnce deduplication technology, D2D appliances, performance tuning, complex remediation, and online backup integration with applications like Oracle and infrastructure like VMware. Traveling across the United States and Canada as a Sr. Technical Consultant, I deliver specialized consulting for a broad variety of HP customers.
  • MrCollaboration (aka Jim Evans) is an HP Global Services Alliance Manager. He has worked in the IT industry for more than 30 years, 22 of which were spent with Digital Equipment Corporation, Compaq and HP. He works with many third party vendors and partners to develop processes to facilitate excellent support and service for mutual customers. Jim is also HP’s representative to the Technical Support Alliance Network (TSANet).
  • I've been working in Customer Service for over 20 years. During my career I've provided support services for Languages, Programming Libraries and Operating Systems. During the last 10 years I've provide support for Linux and more recently VMware. My current role is as a Technical Account Manager working in the HP Custom Mission Critical Services Industry Standard Operating Systems team. I provide both reactive and proactive operating system support for proLiant servers and blades. Our services in the Custom teams are built on statement of work contracts for large HP customers who need a customized mission critical support offering.
  • I've been working in HP since 2007 like IT agent, developer, Web designer and then like Web Project Manager
  • I like to listen as much as I like to talk. Why? My 25+ years in the technology industry has taught me that the key to delivering value to customers is to understand what they value in the first place! I developed this passion for customers and consultative selling during my 12 years with Accenture, and I have continued to approach customers in a consultative way during my 12+ year tenure with HP. I also have a passion for HP given my knowledge of our Product and Service Portfolio and the differentiators we possess that position us as a leader in the areas our customers are telling us they want to go. Converged Infrastructure, Converged Cloud, Big Data – and the associated Service and Support implications – all such exciting technology trends where our success will hinge upon our ability to differentiate ourselves versus others in the areas that matter most to our customers. Right up my alley, and I am proud to be part of the great HP team where I know we have the best solutions in the industry!
  • Tom Clement has over 30 years experience in the areas of adult learning, secondary education, and leadership development. During this time Tom has been a consistent champion of “non-traditional” training delivery methods, including blended learning, virtual delivery (self paced and instructor led), the use of training games and simulations, and experiential learning. Tom has spent the past 25 years of his career at Hewlett Packard, focused most recently on HP’s global Virtualization, Cloud, and Converged Infrastructure customer training programs. Tom manages the strategic direction and overall performance of these training programs, ensuring these worldwide programs help HP’s customers capitalize on the business opportunities made available by IT advancements in each of these subject areas. Tom and his global teammates utilize best in class instructors, course content and supporting equipment infrastructure to deliver these training programs to HP’s customers. The team prides itself on providing the Virtualization, Cloud, and Converged Infrastructure content customers need when and where they need it, anywhere in the world. Tom is based in the Washington, DC suburbs and can be reached at tom.clement@hp.com.
Follow Us