Knowledge

• Identify hot keys and fields used with resxtop/esxtop
  
• Identify fields used with vscsiStats
  

Skills and Abilities

• Configure esxtop/resxtop custom profiles
  
• Determine use cases for and apply esxtop/resxtop Interactive, Batch and Replay modes
  
• Use vscsiStats to gather storage performance data
  
• Use esxtop/resxtop to collect performance data
  
• Given esxtop/resxtop output, identify relative performance data for capacity planning purposes
  

Configure esxtop/resxtop custom profiles

Official Documentation:

vSphere Monitoring and Performance Guide, Chapter 7 “Performance Monitoring Utilities: resxtop and esxtop”, page 45.

The resxtop and esxtop command-line utilities provide a detailed look at how ESXi uses resources in real time.

You can start either utility in one of three modes: interactive (default), batch, or replay.

The fundamental difference between resxtop and esxtop is that you can use resxtop remotely, whereas you can start esxtop only through the ESXi Shell of a local ESXi host.

Using the esxtop Utility

You can run the esxtop utility using the ESXi Shell to communicate with the ESXi host’s management interface.

You must have root user privileges.

Type the command, using the options you want:

esxtop [-] [h] [v] [b] [s] [a]

[R vm-support_dir_path] [d delay] [n iter]

The esxtop utility reads its default configuration from .esxtop50rc on the ESXi system. This configuration file consists of nine lines.

The first eight lines contain lowercase and uppercase letters to specify which fields appear in which order on the CPU, memory, storage adapter, storage device, virtual machine storage, network, interrupt, and CPU power panels. The letters correspond to the letters in the Fields or Order panels for the respective esxtop panel.

The ninth line contains information on the other options. Most important, if you saved a configuration in secure mode, you do not get an insecure esxtop without removing the s from the seventh line of your .esxtop50rc file. A number specifies the delay time between updates. As in interactive mode, typing c, m, d, u, v, n, I, or p determines the panel with which esxtop starts.

NOTE Do not edit the .esxtop50rc file. Instead, select the fields and the order in a running esxtop process, make changes, and save this file using the W interactive command.

Using the resxtop Utility

The resxtop utility is a vSphere CLI command.

Before you can use any vSphere CLI commands, you must either download and install a vSphere CLI package or deploy the vSphere Management Assistant (vMA) to your ESXi host or vCenter Server system.

After it is set up, start resxtop from the command line. For remote connections, you can connect to a host either directly or through vCenter Server.

The command-line options listed in the following table are the same as for esxtop (except for the R option) with additional connection options.

NOTE resxtop does not use all the options shared by other vSphere CLI commands.

resxtop Command-Line Options

You can also use resxtop on a local ESXi host by omitting the server option on the command line. The command defaults to localhost.

Using esxtop or resxtop in Interactive Mode

By default, resxtop and esxtop run in interactive mode. Interactive mode displays statistics in different panels.

A help menu is available for each panel.

Interactive Mode Command-Line Options

You can use various command-line options with esxtop and resxtop in interactive mode.

Interactive Mode Command-Line Options

Common Statistics Description

Several statistics appear on the different panels while resxtop (or esxtop) is running in interactive mode. These statistics are common across all four panels.

The Uptime line, found at the top of each of the four resxtop (or esxtop) panels, displays the current time, time since last reboot, number of currently running worlds and load averages. A world is an ESXi Vmkernel schedulable entity, similar to a process or thread in other operating systems.

Below that the load averages over the past one, five, and fifteen minutes appear. Load averages take into account both running and ready-to-run worlds. A load average of 1.00 means that there is full utilization of all physical CPUs. A load average of 2.00 means that the ESXi system might need twice as many physical CPUs as are currently available. Similarly, a load average of 0.50 means that the physical CPUs on the ESXi system are half utilized.

Statistics Columns and Order Pages

You can define the order of fields displayed in interactive mode.

If you press f, F, o, or O, the system displays a page that specifies the field order on the top line and short descriptions of the field contents. If the letter in the field string corresponding to a field is uppercase, the field is displayed. An asterisk in front of the field description indicates whether a field is displayed.

The order of the fields corresponds to the order of the letters in the string.

From the Field Select panel, you can:

• Toggle the display of a field by pressing the corresponding letter.
  
• Move a field to the left by pressing the corresponding uppercase letter.
  
• Move a field to the right by pressing the corresponding lowercase letter.
  

Interactive Mode Single-Key Commands

When running in interactive mode, resxtop (or esxtop) recognizes several single-key commands.

All interactive mode panels recognize the commands listed in the following table. The command to specify the delay between updates is disabled if the s option is given on the command line. All sorting interactive commands sort in descending order.

Interactive Mode Single-Key Commands

CPU Panel

The CPU panel displays server-wide statistics as well as statistics for individual world, resource pool, and virtual machine CPU utilization.

Resource pools, virtual machines that are running, or other worlds are at times called groups. For worlds belonging to a virtual machine, statistics for the virtual machine that is running are displayed. All other worlds are logically aggregated into the resource pools that contain them.

CPU Panel Statistics

You can change the display using single-key commands.

CPU Panel Single-Key Commands

CPU Power Panel

The CPU Power panel displays CPU Power utilization statistics.

On the CPU Power panel, statistics are arranged per PCPU. A PCPU is a physical hardware execution context — a physical CPU core if hyper-threading is unavailable or disabled, or a logical CPU (LCPU or SMT thread) if hyper-threading is enabled.

CPU Power Panel Statistics

Memory Panel

The Memory panel displays server-wide and group memory utilization statistics. As on the CPU panel, groups correspond to resource pools, running virtual machines, or other worlds that are consuming memory.

The first line, found at the top of the Memory panel displays the current time, time since last reboot, number of currently running worlds, and memory overcommitment averages. The memory overcommitment averages over the past one, five, and fifteen minutes appear. Memory overcommitment of 1.00 means a memory overcommitment of 100 percent.

Memory Panel Statistics

Memory Panel Interactive Commands

Storage Adapter Panel

Statistics in the Storage Adapter panel are aggregated per storage adapter by default. Statistics can also be viewed per storage path.

Storage Adapter Panel Statistics

The following table displays the interactive commands you can use with the storage adapter panel.

Storage Adapter Panel Interactive Commands

Storage Device Panel

The storage device panel displays server-wide storage utilization statistics.

By default, the information is grouped per storage device. You can also group the statistics per path, per world, or per partition.

Storage Device Panel Statistics

The following table displays the interactive commands you can use with the storage device panel.

Storage Device Panel Interactive Commands

Virtual Machine Storage Panel

This panel displays virtual machine-centric storage statistics.

By default, statistics are aggregated on a per-resource-pool basis. One virtual machine has one corresponding resource pool, so the panel displays statistics on a per-virtual-machine basis. You can also view statistics on per-VSCSI-device basis.

Virtual Machine Storage Panel Statistics

The following table lists the interactive commands you can use with the virtual machine storage panel.

Virtual Machine Storage Panel Interactive Commands

Network Panel

The Network panel displays server-wide network utilization statistics.

Statistics are arranged by port for each virtual network device configured. For physical network adapter statistics, see the row in the table that corresponds to the port to which the physical network adapter is connected. For statistics on a virtual network adapter configured in a particular virtual machine, see the row corresponding to the port to which the virtual network adapter is connected.

Network Panel Statistics

The following table displays the interactive commands you can use with the network panel.

Network Panel Interactive Commands

Interrupt Panel

The interrupt panel displays information about the use of interrupt vectors.

Interrupt Panel Statistics

Determine use cases for and apply esxtop/resxtop Interactive, Batch and Replay modes

Official Documentation:

vSphere Monitoring and Performance Guide, Chapter 7 “Performance Monitoring Utilities: resxtop and esxtop”, page 59.

See previous objective.

Using Batch Mode

Batch mode allows you to collect and save resource utilization statistics in a file.

After you prepare for batch mode, you can use esxtop or resxtop in this mode.

Prepare for Batch Mode

To run in batch mode, you must first prepare for batch mode.

Procedure

1. Run resxtop (or esxtop) in interactive mode.
   
2. In each of the panels, select the columns you want.
   
3. Save this configuration to a file (by default ~/.esxtop50rc) using the W interactive command.
   

You can now use resxtop (or esxtop) in batch mode.

Use esxtop or resxtop in Batch Mode

After you have prepared for batch mode, you can use esxtop or resxtop in this mode.

Procedure

1. Start resxtop (or esxtop) to redirect the output to a file.
   For example:
   esxtop -b > my_file.csv
   The filename must have a .csv extension. The utility does not enforce this, but the post-processing tools require it.
   
2. Process statistics collected in batch mode using tools such as Microsoft Excel and Perfmon.
   

In batch mode, resxtop (or esxtop) does not accept interactive commands. In batch mode, the utility runs until it produces the number of iterations requested (see command-line option n, below, for more details), or until you end the process by pressing Ctrl+c.

Batch Mode Command-Line Options

You can use batch mode with command-line options.

Command-Line Options in Batch Mode

Using Replay Mode

In replay mode, esxtop replays resource utilization statistics collected using vm-support.

After you prepare for replay mode, you can use esxtop in this mode. See the vm-support man page.

In replay mode, esxtop accepts the same set of interactive commands as in interactive mode and runs until no more snapshots are collected by vm-support to be read or until the requested number of iterations are completed.

Prepare for Replay Mode

To run in replay mode, you must prepare for replay mode.

Procedure

1. Run vm-support in snapshot mode in the ESXi Shell.
   Use the following command.
   vm-support -S -d duration -I interval
   
2. Unzip and untar the resulting tar file so that esxtop can use it in replay mode.
   

You can now use esxtop in replay mode.

Use esxtop in Replay Mode

You can use esxtop in replay mode.

Replay mode can be run to produce output in the same style as batch mode (see the command-line option b, below).

NOTE Batch output from esxtop cannot be played back by resxtop.

Snapshots collected by vm-supported can be replayed by esxtop. However, vm-support output generated by ESXi can be replayed only by esxtop running on the same version of ESXi.

Procedure

• To activate replay mode, enter the following at the command-line prompt.
  esxtop -R vm-support_dir_path
  

Replay Mode Command-Line Options

You can use replay mode with command-line options.

The following table lists the command-line options available for esxtop replay mode.

Command-Line Options in Replay Mode

Use vscsiStats to gather storage performance data

Official Documentation:

“Using vscsiStats for Storage Performance Analysis“, from the VMware Communities seems to be the official documentation  on this subject. Unofficial are a lot of excellent Blog posts. I will mention a few in the “Other references” section.

esxtop is a great tool for performance analysis of all types. However, with only latency and throughput statistics, esxtop will not provide the full picture of the storage profile. Furthermore, esxtop only provides latency numbers for Fibre Channel and iSCSI storage. Latency analysis of NFS traffic is not possible with esxtop.

Since ESX 3.5, VMware has provided a tool specifically for profiling storage: vscsiStats. vscsiStats collects and reports counters on storage activity. Its data is collected at the virtual SCSI device level in the kernel. This means that results are reported per VMDK (or RDM) irrespective of the underlying storage protocol. The following data are reported in histogram form:

• IO size
  
• Seek distance
  
• Outstanding IOs
  
• Latency (in microseconds)
  
• More!
  

Running vscsiStats

vscsiStats collection and analysis requires two steps:

1. Start statistics collection.
   
2. View accrued statistics.
   

Documentation on command-line parameters are available when running ‘/usr/lib/vmware/bin/vscsiStats -h’.

Starting and Stopping vscsiStats Collection

The tool is started with the following command:

/usr/lib/vmware/bin/vscsiStats -s -w <world_group_id>

This command starts the process that will accrue statistics. The world group ID must be set to a running virtual machine. The running VMs’ IDs can be obtained by running ‘/usr/lib/vmware/bin/vscsiStats -l’.

After about 30 minutes vscsiStats will stop running. If the analysis is needed for a longer period, the start command should be repeated above in this window. That will defer the timeout and termination by another 30 minutes.

Since results are accrued and reported out in summary, the histograms will include data since collection was started. To reset all counters to zero, run ‘/usr/lib/vmware/bin/vscsiStats -r’.

Viewing Statistics

Counters are displayed by using the following command:

/usr/lib/vmware/bin/vscsiStats -p <histo_type> [-c]

The histogram type is used to specify either all of the statistics or one group of them. Options include all, ioLength, seekDistance, outstandingIOs, latency, interarrival.

Results can be produced in a more compact comma-delimited list by adding the optional “-c” above.

Using vscsiStats Results

Use Case 1: Identifying Sequential IO

Storage arrays can process sequential IO much faster than random IO. You can therefore improve the performance of a sequential workload by placing it on a dedicated LUN to allow the array to optimize access. vscsiStats can help you identify your sequential workloads even if you don’t understand anything about the application in the VM.

Take the following graph as example, which I generated by running ‘/usr/lib/vmware/bin/vscsiStats -p seekDistance’:



This graph shows that most of the commands are being issued a great distance from the previous command. It looks like all of the commands were 50,000 or more logical blocks away from the previous command. When I looked at the raw data, I saw that over 99% of the commands were more than 128 blocks away from the previous command. That’s random access if I’ve ever seen it. Here’s the opposite example:



In this case the logical block number (LBN) of each command is most frequently exactly one larger than the previous command. That’s the signature of a heavily sequential workload. It shouldn’t surprise you to learn that both of these profiles were generated by Iometer using random and sequential writes, respectively.

Use Case 2: Optimizing for IO Sizes

The IO size is an important characteristic of storage profiles. A variety of best practices have been provided by storage vendors to enable customers to tune their storage to a particular IO size. As an example, it may make sense to optimize an array’s stripe size to its average IO size. vscsiStats can provide a histogram of IO sizes to help this process. The following graph was generated by ‘/usr/lib/vmware/bin/vscsiStats -p ioLength’:



From these results I can see that about a quarter of the commands came in IOs smaller than 4k. About half of the commands were sized to 4k commands. The minute number of remaining IOs were larger than 4k. This signature is common of a VMDK formatted to 4k blocks and supporting OS and application execution. The storage array should be optimized for 4k blocks if this disk’s performance is a priority.

Use Case 3: Storage Latency Analysis (Including NFS!)

esxtop is a terrific tool for latency-based storage analysis. Fibre Channel and iSCSI HBAs have device and kernel latencies in esxtop’s storage panel. Software iSCSI initiators will show up as vmhba32 (ESX 3.5 and earlier) and vmhba33 (ESX 4.0 and later.) But esxtop does not provide latency statistics for NFS stores.

Because vscsiStats collects its results where the guest interacts with the hypervisor, it is unaware of the storage implementation. Latency statistics can be collected for all storage configurations with this tool.



The above graph shows that the server in my office with a single direct-attached SCSI disk is performing as I would expect. About half of all the operations are completing in under 5 ms. The other half take 5-15 ms to complete. A few commands took longer than 15 ms, but the number is so small that it doesn’t concern me. Similar results can be seen with NFS arrays.

vscsiStats on ESXi

vscsiStats can be installed on ESXi hosts after putting the host into tech support mode. More information on this process is availalble on Scott’s blog on the subject on vPivot.

More information

• Gabe’s Virtual World: http://www.gabesvirtualworld.com/using-vscsistats-the-full-how-to/
  
• Duncan Epping Blog Yellow Bricks:  http://www.yellow-bricks.com/2009/12/17/vscsistats/
  

Use esxtop/resxtop to collect performance data

Official Documentation:

vSphere Monitoring and Performance Guide, Chapter 7 “Performance Monitoring Utilities: resxtop and esxtop”, page 45.

See previous objectives.

More information

• Duncan Epping Blog Yellow Bricks: http://www.yellow-bricks.com/esxtop/
  

Given esxtop/resxtop output, identify relative performance data for capacity planning purposes

Official Documentation:

None.

More information

• You want to know more about the various statistics and how to interpret these? Read VMware Communities “Interpreting esxtop Statistics”
  
• Now you know everything about the statisctics, but what are the thresholds? Read this  excellent post on esxtop by Duncan Epping.
  
• A recent nice example by Duncan Epping “Why is %WAIT so high in esxtop?”
  
•  VMware KB 1017926  “Troubleshooting a virtual machine that has stopped responding: VMM and Guest CPU usage comparison”
  

Other exam notes

• The Saffageek VCAP5-DCA Objectives http://thesaffageek.co.uk/vcap5-dca-objectives/
  
• Paul Grevink The VCAP5-DCA diaries http://paulgrevink.wordpress.com/the-vcap5-dca-diaries/
  
• Edward Grigson VCAP5-DCA notes http://www.vexperienced.co.uk/vcap5-dca/
  
• Jason Langer VCAP5-DCA notes http://www.virtuallanger.com/vcap-dca-5/
  
• The Foglite VCAP5-DCA notes http://thefoglite.com/vcap-dca5-objective/
  

VMware vSphere official documentation

Disclaimer.
The information in this article is provided “AS IS” with no warranties, and confers no rights. This article does not represent the thoughts, intentions, plans or strategies of my employer. It is solely my opinion.