Knowledge

• Identify vCenter Server log file names and locations
• Identify ESXi log files names and locations
• Identify tools used to view vSphere log files

Skills and Abilities

• Generate vCenter Server and ESXi log bundles
• Use esxcli system syslog to configure centralized logging on ESXi hosts
• Test centralized logging configuration
• Analyze log entries to obtain configuration information
• Analyze log entries to identify and resolve issues
• Install and configure VMware syslog Collector and ESXi Dump Collector

 

Generate vCenter Server and ESXi log bundles

Official Documentation:
vCenter Server Host Management Guide, Chapter 8, “System Log Files”, page 91.
Export System Log Files

When the vSphere Client is connected to vCenter Server, you can select hosts from which to download system log files.

To save diagnostic data for ESXi hosts and vCenter Server, the vSphere Client must be connected to the vCenter Server system. If you are connected directly to an ESXi host, you can save diagnostic data only for that specific ESXi host. 

Required privileges:

• To view diagnostic data: Read-Only User
• To manage diagnostic data: Global.Licenses 

Procedure

1. Select File > Export > Export System Logs.
2. If you are connected to vCenter Server, select the object for which you want to export data.
   Selecting an object selects all of its child objects.
3. If you are connected to vCenter Server, select Include information from vCenter Server and vSphere Client to download vCenter Server and vSphere Client log files and host log files, and click Next.
4. If the selected host supports manifest driven exports of system log files, select the system log files to collect.
   Select the specific system log files to download.
   If the host does not support manifest exports of log files, all system log files are exported.
5. Select Gather performance data to include performance data information in the log files. Click Next.
   You can update the duration and interval time you want the data collected.
6. Click Next.
7. Click Browse and specify the location to which to save the log files.
   The host or vCenter Server generates .zip bundles containing the log files. The Recent Tasks panel shows the Generate diagnostic bundles task in progress.
8. Click Next.
   The Downloading Log Bundles dialog box appears when the Generating Diagnostic Bundle task is finished. The download status of each bundle appears in the dialog box.
   Some network errors can cause download failures. When you select an individual download in the dialog box, the error message for that operation appears under the name and location of the log bundle file.
9. Verify the information in the Summary and click Finish to download the log files.
10. If the download fails, click Retry to attempt to download the generated bundles again.

Diagnostic bundles containing log files for the specified objects are downloaded to the location specified.
Collecting Log Files

VMware technical support might request several files to help resolve technical issues. The following sections describe script processes for generating and collecting some of these files.
Set Verbose Logging

You can specify how verbose log files will be.
Procedure

1. Select Administration > vCenter Server Settings.
2. Select Logging Options.
3. Select Verbose from the pop-up menu.
4. Click OK.

Collect vSphere Log Files

You can collect vSphere log files into a single location.
Procedure

• View the log file using one of the following methods.

Collect ESXi Log Files

You can collect and package all relevant ESXi system and configuration information, a well as ESXi log files.

This information can be used to analyze the problems.
Procedure

• Run the following script on the ESXi Shell: /usr/bin/vm-support
  The resulting file has the following format: esx-date-unique-xnumber.tgz

Use esxcli system syslog to configure centralized logging on ESXi hosts

Official Documentation:

VMware KB “Configuring syslog on ESXi 5.0“
Configuring Local and Remote Logging using the esxcli command

Local and Remote syslog functionality can be configured for a host using the esxcli command line utility, which can be used at the console of an ESXi host, in the vCLI, or in the vMA.

For more information regarding the use of esxcli, see Configuring ESXi Syslog Services in the vSphere Command-Line Interface Documentation.

1. Open a ESXi Shell console session where the esxcli command is available, such as the vCLI or on the ESXi host directly.
2. Display the existing five configuration options on the host using the command:
   esxcli system syslog config get
3. Set new host configuration, specifying options to change, using a command similar to:
   esxcli system syslog config set –logdir=/path/to/vmfs/directory/ –loghost=RemoteHostname –logdir-unique=true|false –default-rotate=NNN –default-size=NNN
   For example, to configure remote syslog using TCP on port 514:
   esxcli system syslog config set –loghost=’tcp://10.11.12.13:514′
4. After making configuration changes, load the new configuration using the command:
   esxcli system syslog reload

Note: This command can be used to restart the syslog service, if the service has stopped due to some reason.
To open outbound traffic via the ESXi Firewall on UDP port 514, TCP port 514 and 1514, use the following commands:

esxcli network firewall ruleset set –ruleset-id=syslog –enabled=true

esxcli network firewall refresh

Test centralized logging configuration

Official Documentation:

When everything has been configured correctly, log files should show up in the Syslog server.

In my case, using the “Network Syslog Collector”, the actual log files can be retrieved.

Notice that:

• A folder has been created for every ESXi host, identified by the management IP address;
• In each folder a single file, named syslog.log, containing entries from the Hostd.log and the Vpxa.log

In case, logging does not show up, try the following:

• Check the configuration of the ESXi host, especially the syntax of the loghost;
• Check the configuration of the ESXi firewall, outgoing syslog allowed;
• On the ESXi host, try restarting the Managent Agent. From the DCUI or
  # /sbin/services.sh restart
• On the Syslog server, also check the firewall settings, is incoming traffic allowed?
• Try to connect to the Syslo server using the telnet command, e.g.:
  > telnet <IP Syslog server> 514
• In case you use the “Network Syslog Collector”, review the settings

Analyze log entries to obtain configuration information

Official Documentation:

VMware KB2004201 “Location of ESXi 5.0 log files“
ESXi 5.0 Host Log Files

Logs for an ESXi 5.0 host are grouped according to the source component:

• /var/log/auth.log: ESXi Shell authentication success and failure.
• /var/log/dhclient.log: DHCP client service, including discovery, address lease requests and renewals.
• /var/log/esxupdate.log: ESXi patch and update installation logs.
• /var/log/hostd.log: Host management service logs, including virtual machine and host Task and Events, communication with the vSphere Client and vCenter Server vpxa agent, and SDK connections.
• /var/log/shell.log: ESXi Shell usage logs, including enable/disable and every command entered. For more information, see the Managing vSphere with Command-Line Interfaces section of the vSphere 5 Command Line documentation and Auditing ESXi Shell logins and commands in ESXi 5.x (2004810).
• /var/log/sysboot.log: Early VMkernel startup and module loading.
• /var/log/boot.gz: A compressed file that contains boot log information and can be read using zcat /var/log/boot.gz|more.
• /var/log/syslog.log: Management service initialization, watchdogs, scheduled tasks and DCUI use.
• /var/log/usb.log: USB device arbitration events, such as discovery and pass-through to virtual machines.
• /var/log/vob.log: VMkernel Observation events, similar to vob.component.event.
• /var/log/vmkernel.log: Core VMkernel logs, including device discovery, storage and networking device and driver events, and virtual machine startup.
• /var/log/vmkwarning.log: A summary of Warning and Alert log messages excerpted from the VMkernel logs.
• /var/log/vmksummary.log: A summary of ESXi host startup and shutdown, and an hourly heartbeat with uptime, number of virtual machines running, and service resource consumption. For more information, see Format of the ESXi 5.0 vmksummary log file (2004566).

Note: For information on sending logs to another location (such as a datastore or remote syslog server), see Configuring syslog on ESXi 5.0 (2003322)
Logs from vCenter Server Components on ESXi 5.0

When an ESXi 5.0 host is managed by vCenter Server 5.0, two components are installed, each with its own logs:

• /var/log/vpxa.log: vCenter Server vpxa agent logs, including communication with vCenter Server and the Host Management hostd agent.
• /var/log/fdm.log: vSphere High Availability logs, produced by the fdm service. For more information, see the vSphere HA Security section of the vSphere 5.0 Availability Guide.

Note: If persistent scratch space is configured, many of these logs are located on the scratch volume and the /var/log/ directory contains symlinks to the persistent storage location. Rotated logs are compressed at the persistent location and/or at /var/run/log/. For more information, see Creating a persistent scratch location for ESXi (1033696).

Analyze log entries to identify and resolve issues

Official Documentation:

See also previous topic. While investigating an issue, it is a good idea to analyze log files, like the hostd.log or vmkernel.log for specific messages. Those messages can help you finding a VMware KB that can solve your issue or contacting a colleague or VMware Support. 

Install and configure VMware syslog Collector and ESXi Dump Collector

Official Documentation:

vSphere Installation and Setup Guide, Chapter 12 “After You Install vCenter Server”, Section “Install vSphere ESXi Dump Collector”, page 212. Also section “Install vSphere Syslog Collector”, page 213
Install vSphere ESXi Dump Collector

You can configure ESXi to dump the vmkernel memory to a network server, rather than to a disk, when the system has encountered a critical failure. Install vSphere ESXi Dump Collector to collect such memory dumps over the network.

NOTE In the vCenter Server Appliance, the ESXi Dump Collector is installed and enabled by default. These instructions apply to Windows-based deployments.

You can install the Dump Collector on the same machine as the associated vCenter Server, or on a different machine that has network connection to the vCenter Server. ESXi Dump Collector does not support vSphere distributed switches in ESXi 5.0.

The Dump Collector service binds to an IPv4 address for communication with vCenter Server, and does not support IPv6. The vCenter Server can be on a host machine in an IPv4-only, IPv4/IPv6 mixed-mode, or IPv6-only network environment, but the machine that connects to the vCenter Server through the vSphere Client must have an IPv4 address for the Dump Collector service to work.
Prerequisites

• Verify that you have administrator privileges
• Verify that the host machine has Windows Installer 3.0 or later.
• Verify that the host machine has a supported processor and operating system. The Dump Collector supports the same processors and operating systems as vCenter Server. See “vCenter Server Software Requirements,” on page 36 and “vCenter Server and vSphere Client Hardware Requirements,” on page 33.
• Verify that the host machine has a valid IPv4 address. You can install the Dump Collector on a machine in an IPv4-only or IPv4/IPv6 mixed-mode network environment, but you cannot install the Dump Collector on a machine in an IPv6-only environment.
• If you are using a network location for the Dump Collector repository, make sure the network location is mounted.

Gather the following information to complete the installation:

• The location to install the Dump Collector to, if you are not using the default location.
• The location for the Dump Collector repository where the dump files will be stored.
• (Optional) The maximum size for the Dump Collector repository. The specified network location must have at least that much free space.
• Whether to install the Dump Collector as a standalone instance or to integrate the Dump Collector with a vCenter Server. The Dump Collector is not supported for integration with vCenter Server versions earlier than version 5.0. 
• If the Dump Collector is integrated with a vCenter Server, the address and credentials for the vCenter Server: IP address or name, HTTP port, user name, and password.
• The Dump Collector server port, if you are not using the default setting.
• The host name or IP address to identify the Dump Collector on the network.

Procedure

1. In the software installer directory, double-click the autorun.exe file to start the installer.
2. Select VMware ESXi™Dump Collector and click Install.
3. Follow the wizard prompts to complete the installation.

Install vSphere Syslog Collector

Install the vSphere Syslog Collector to enable ESXi system logs to be directed to a server on the network, rather than to a local disk.

You can install the Syslog Collector on the same machine as the associated vCenter Server, or on a different machine that has network connection to the vCenter Server. The Syslog Collector service binds to an IPv4 address for communication with vCenter Server, and does not support IPv6. The vCenter Server can be on a host machine in an IPv4-only, IPv4/IPv6 mixed-mode, or IPv6-only network environment, but the machine that connects to the vCenter Server through the vSphere Client must have an IPv4 address for the Syslog Collector service to work.
Prerequisites

• Verify that you have administrator privileges.
• Verify that the host machine has Windows Installer 3.0 or later.
• Verify that the host machine has a supported processor and operating system. The Syslog Collector supports the same processors and operating systems as vCenter Server. See “vCenter Server Software Requirements,” on page 36 and “vCenter Server and vSphere Client Hardware Requirements,” on page 33.
• Determine whether to install the Syslog Collector as a standalone instance or to integrate the Syslog Collector with a vCenter Server. The Syslog Collector is not supported for integration with vCenter Server versions earlier than version 5.0.
• Verify that the host machine has a valid IPv4 address. You can install the Syslog Collector on a machine in an IPv4-only or IPv4/IPv6 mixed-mode network environment, but you cannot install the Syslog Collector on a machine in an IPv6-only environment.

Gather the following information to complete the installation:

• The location to install the Syslog Collector to, if you are not using the default location.
• The location for the Syslog Collector repository where the syslog files will be stored.
• (Optional) The maximum size for the Syslog Collector repository. The specified network location must have at least that much free space.
• (Optional) The maximum number of Syslog Collector log rotations to keep.
• If the Syslog Collector is integrated with a vCenter Server, the address and credentials for the vCenter Server: IP address or name, HTTP port, user name, and password.
• The Syslog Collector server port, if you are not using the default setting, and whether to use TCP and UDP protocols for this port.
• The Syslog Collector server SSL port, if you are not using the default setting, and whether to use secure connection (SSL) for this port.
• The host name or IP address to identify the Syslog Collector on the network.

Procedure

1. In the software installer directory, double-click the autorun.exe file to start the installer.
2. Select VMware® Syslog Collector and click Install.
3. Follow the wizard prompts to complete the installation.

Other exam notes

• The Saffageek VCAP5-DCA Objectives http://thesaffageek.co.uk/vcap5-dca-objectives/
• Paul Grevink The VCAP5-DCA diaries http://paulgrevink.wordpress.com/the-vcap5-dca-diaries/
• Edward Grigson VCAP5-DCA notes http://www.vexperienced.co.uk/vcap5-dca/
• Jason Langer VCAP5-DCA notes http://www.virtuallanger.com/vcap-dca-5/
• The Foglite VCAP5-DCA notes http://thefoglite.com/vcap-dca5-objective/

VMware vSphere official documentation

Disclaimer.
The information in this article is provided “AS IS” with no warranties, and confers no rights. This article does not represent the thoughts, intentions, plans or strategies of my employer. It is solely my opinion.