Utilising vSphere Performance Monitoring Tools – Part Two: vscsiStats

The vscsiStats command allows for the troubleshooting of performance issues for virtual machine storage. It can collect data at the virtual SCSI device level and report this performance data as a histogram by collecting information on each I/O operation and reports on I/O metrics such as length, seek distance, number of outstanding I/O, I/O latency and inter-arrival time.

This provides more performance data than the ESXTOP which in comparison only provides latency and throughput statistics. Also, as vscsiStats targets the virtual SCSI device level it can report on virtual machine hard disks of all types. This can be useful for determining the behaviour of a workload in order to determine the placement of storage for a virtual machine.

In order to retrieve performance data for a virtual machine we will be required to obtain the virtual machine worldGroupID and if you require filter the retrieval based on a specific virtual machine hard disk the handleID. This information can be retrieved as below:

vscsiStats -l

In this example, we will initially retrieve performance data for the virtual machine ‘vm1’ wherer the worldGroupID is ‘2718498’ as described in the below output.

Virtual Machine worldGroupID: 2718498, Virtual Machine Display Name: vm1, Virtual Machine Config File: /vmfs/volumes/54afe658-9ab37544-54a8-0026b9746656/vm1/vm1.vmx, {
Virtual SCSI Disk handleID: 11547 (scsi0:0)
Virtual SCSI Disk handleID: 11548 (scsi0:1)
Virtual SCSI Disk handleID: 11549 (scsi0:2)
}

Now we will start the retrieval of performance data for all the virtual machine hard disk for the specific virtual machine.

vscsiStats -s -w 2718498

The retrieval of performance data will now occur in the background, whilst data is being retrieved which will taake 30 minutes by default, you may print the histogram for a specific statistic by specifying the statistic name (iolength, seekDistance, outstandingIOs, latency and interarrival)

vscsiStats -p latency

This will show the current data retrieved for latency of IOs, Read IOs and Write IOS in microseconds. In the below example we may see that no Write I/Os took longer than 15000 microseconds where the peak Write I/O demand was 7916 microseconds.

Histogram: latency of Write IOs in Microseconds (us) for virtual machine worldGroupID : 3595341, virtual disk handleID : 11916 (scsi0:0) {
 min : 207
 max : 7916
 mean : 371
 count : 13421
 {
 0 (<= 1)
 0 (<= 10)
 0 (<= 100)
 12000 (<= 500)
 1198 (<= 1000)
 222 (<= 5000)
 1 (<= 15000)
 0 (<= 30000)
 0 (<= 50000)
 0 (<= 100000)
 0 (> 100000) 
 }

If you require to stop the retrieval of performance data before this completes, you may so with the following command.

vscsiStats -x

In the above example, we invoked the vscsiStats command to retrieve performance data for all virtual machine hard disks for a sepcific virtual machine. If we required only to retrieve performance data for the Hard Disk (0:2) we can invoke the command and specify the handleID.