Once you have have assessed the hardware requirements for ESXi host systems (requires a minimum of 2GB to install) to ensure sufficient memory resources are provided to satisfy the demand of the virtual machines, system overhead and the level of failure protection required, we shall look at various options for optimising ESXi host system memory.
Hardware Assisted Memory Management Unit (MMU) Virtualization
Hardware Assisted Memory Management Unit (MMU) Virtualization allows for enhanced virtual machine performance as the host system hardware provide an additional layer of page tables in the hardware mapping guest operating system memory to host system physical memory and therefore there is no requirement to maintain shadow page tables, reducing overhead memory consumption.
- Implemented by Intel using extended page tables (EPTs).
- Implemented by AMD with rapid virtualization index (RVI).
In some instances the performance benefit may be negated if the virtual machines workload results in a high frequency of misses in the hardware lookaside buffer (TLB) as the time required for the host system to service a TLB miss is increased due to the absence of shadow page tables. However, the cost of TLB misses can be negated from the guest operating system by utilising large pages in the virtual machines workload.
From the below performance evaluations we can see the following performance gains:
Intel EPT – results in performance gains of up to 48% for MMU-intensive benchmarks and up to 600% for MMU-intensive microbenchmarks.
AMD RVI – results in performance gains of up to 42% for MMU-intensive benchmarks and up to 500% for MMU-intensive microbenchmarks.
In both cases, the use of large pages are recommended for TLB-intensive workloads.
http://www.vmware.com/pdf/Perf_ESX_Intel-EPT-eval.pdf
http://www.vmware.com/pdf/RVI_performance.pdf
Memory Scrub Rate
The host system can where possible utilize ECC memory to set the memory scrub rate in the BIOS, this process onsists of reading from each computer memory location, correcting bit errors (if any) with an error-correcting code (ECC), and writing the corrected data back to the same location. It is recommended to configure the memory scrub rate to be that of the manufacturers recommendations, which is most likely to be the default BIOS setting.
Non-uniform memory access (NUMA)
Non-uniform memory access (NUMA) allows for a processor to access its own local memory faster than non-local memory (memory local to another processor or memory shared between processors). The benefits of NUMA are limited to particular workloads, notably on servers where the data are often associated strongly with certain tasks or users. Therefore, in most cases VMware recommends disabling node interleaving, which will enable NUMA and allow for the host system to optimally place each page of the virtual machines virtual memory
Their is also a design consideration to take into account when sizing memory for virtual machines as when NUMA is enabled as the CPU scheduler utilises NUMA optimisation to assign each virtual machine to a NUMA node to keep vCPU and memory in the same location with the support memory locality. If a virtual machine cannot be assigned within a single NUMA node the CPU scheduler spreads load across all sockets in a round robin manner, similar to a where NUMA is not enabled or supported.
The impact of the CPU scheduler spreading the load across all sockets will result in the virtual machine not achieving NUMA optimizations as the virtual machine load will be spread across the host system.
For more information on NUMA see the below:
Memory Overhead
Memory overhead is required for the VMKernal an host agents on each host system, to which this can be configured with the use of a system swap file which allows for 1GB of memory to be reclaimed when the host system is under pressure. By default this option is disabled but the host system can be configured to use a swap file by invoking the below from the vSphere Command Line interface, where the example creates and enables a system swap file on datastore named ‘Datastore-01’.
esxcli sched swap system set -d true -n Datastore-01
As well as the memory overhead on the host system each virtual machine will require a memory overhead to support the following:
- VM executable (VMX) process – required to bootstrap and support the guest operating system.
- VM monitor (VMM) – contains the virtual machine hardware data structures, such as memory mappings and CPU state.
- Virtual machine hardware devices
- Subsystems, such as kernel and management agents
Once a virtual machine is powered on the memory requirements for VMM and virtual hardware devices are reserved. By utilising a VMX swap file you can reduce the VMX memory reservation from 50MB and greater to approximately 10MB per virtual machine, the total size of the VMX swap file is approximately 100MB.
By default the host system will create a VMX swap file in the virtual machines working directory. However, the behaviour of a VMX swap file can be controlled by the virtual machines configuration file.
As discussed, the VMX swap file is created by default when a virtual machine is powered on, if you require to disable this behaviour you will be required to set the ‘sched.swap.vmxSwapEnabled’ parameter value to be ‘False’.
The VMX swap file is not related to the virtual machine swap file (VSWP) which allow for a virtual machine guest operating system to consume less physical memory than configured.
By default, the swap file location for virtual machines specified by the host system is to store the swap files in the same directory as the virtual machine. However, the swap file location can be configured on the host system to store the swap file in a specific datastore.
This can be modified by selecting the host system in the vSphere Web Client and browsing to Manage > Settings > Virtual Machines > Swap file location and enable ‘Use a specific datastore’ and selecting the datastore to be used to store the virtual machine files.
Alternatively you can configure the swap file location using PowerCLI by invoking the ‘Set-VMHost’ cmdlet and specifying the VMSwapFileDatastore parameter, as below:
Get-VMHost esxihost1.domain.local | Set-VMHost -VMSwapfileDatastore Datastore-01 -Confirm:$False
In order for the virtual machine swap files to be created in the specific datastore either invoke a vMotion task or apply the above in a powered off state and power on. One condition for creating the virtual machine swap files is that the virtual machine must be inheriting the swap file policy from the host system. In order to retrieve virtual machines that may not be configured with this behaviour , we can invoke the following from PowerCLI:
(Get-VMHost esxihost1.domain.local | Get-VM | Where-Object {$_.VMSwapFilePolicy -ne "Inherit"}).Name
It is important to ensure that the datastore selected to store the virtual machine swap file is attached to each host system in the cluster, as this can impact vMotion performance for the affected virtual machine(s). If the virtual machine is unable to store the swap files in the specified datastore, the behaviour is to revert to the default and store the swap files in the same directory as the virtual machine .
You may also, specify a location for the VMX swap file by configuring the virtual machine parameter ‘sched.swap.vmxSwapDir’ in the vSphere Web Client by selecting a virtual machine and browsing to Manage > Settings > VM Options > Advanced > Configuration Paramaters > Edit Configuration > Add Row , add the parameter name ‘sched.swap.vmxSwapDir’ and datastore location to create the VMX swap file as the value.
Again, you can use PowerCLI to make the above change by invoking the ‘New-AdvancedSetting’ cmdlet:
New-AdvancedSetting -Entity vm1 -Name sched.swap.vmxSwapDir -Value "/vmfs/volume/Datastore-01"
The below article provides a sample of virtual machine memory overhead values required to power on a virtual machine:
Deans Blog 
One thought on “VMware vSphere Performance – Part One: Optimizing ESXi Host Memory”