Microsoft has published their Performance And Tuning Guide For Windows Server 2012 R2 on TechNet. This guide includes information for different workloads and roles, including Hyper-V. In this article, I will summarize and discuss some of the included suggestions on enhancing performance in Windows Server 2012 R2 Hyper-V, as well as add a few of my own.
The host management OS should be the management OS and nothing but the management OS of that host. It should not be a domain controller, it should not be a file server, it should not be a database server… you get the idea. If you need additional services to run on a physical server, then install them into virtual machines that will run on the host.
You can install management features such as monitoring agents, virtualization management, job management, backup agents, and so on, as long as those agents will not be heavy consumers of processor, storage IO, or memory.
Limit who can log into a host. Local administrator rights on virtualization hosts (any vendor) should be limited to a subset of the IT department, just as the domain admin right it. If you need to grant rights then use System Center Virtual Machine Manager and delegate the subset of rights that are required to an Active Directory group that contains the selected people.
You should also try to avoid logging directly into a host. Use tools that are installed onto your PC, such as the free Remote Server Administration Toolkit or the System Center consoles.
When installing the paid-for version of Windows Server you can choose to follow Microsoft’s recommendation that you use a core installation with no GUI. They say it has:
I prefer to use a full installation of Windows Server for the management OS. This means that if something goes wrong on the host I can still use the OEM’s tools to determine the cause and I don’t need to search the web for command prompt or PowerShell commands.
Microsoft does allow us to switch between core and full GUIs, but this requires a reboot, and reboots of physical servers take a long time.
Try to purchase a processor with a large L2/L3 cache. Haswell processors will include an L4 or Last Level Cache (LLC). These caches can make a large difference in workload performance, offering more than pure gigahertz horsepower will add.
Each host processor has a number of logical processors (LPs). For example, an Intel Xeon with 8 cores and Hyperthreading enabled will have 16 LPs. A 16 core AMD processor will have 16 LPs (no Hyperthreading). WS2012 R2 Hyper-V allows for a total of 320 logical processors in a host. Note that just 64 will be used by and be visible in the management OS, but the others are actually used by virtual machines, despite not being visible in the host. The old LP to virtual processor (VP) ratio limits of 8:1 (server workloads) or 12:1 (VDI workloads) are no longer applicable since WS2012; instead you can place as many virtual machines onto a host up to the maximum number of virtual machines (1024) or the host’s maximum performance capacity (dependent on virtual machines’ loads).
RAM is usually the first bottleneck for adding virtual machines to a host. You can have up to 4 TB RAM in a single host. That is theoretical for the vast majority of us! Enabling Dynamic Memory in appropriate virtual machines will improve the utilization of RAM and reduce host capital and operational costs. You might consider low-power RAM if electrical costs are a driving force for your organization. However, most will opt for faster RAM to get the most from their hosts.
It makes sense from a cost perspective to add as much RAM into a host as possible. However, these can mean using more expensive DIMMs that are larger. It can also mean that you will put too many eggs into one basket. Try to balance capital costs with the placement of redundant VMs in too few hosts.
Some tricks from older versions of Hyper-V should not be applied since WS2012:
The disks (RAID1) that you need for the management OS and paging file of the host do not need to be fast. Their only purpose is to boot up the management OS and page for the RAM that is used by the management OS.
On the other hand, the disks used to store the virtual machines should be fast. RAID10 is strongly recommended for performance. Please don’t be the person who would install database log files onto RAID5 arrays. In fact, in Storage Spaces, you should use 2-way mirrored or 3-way mirrored virtual disks (storage spaces or LUNs); parity disks are only supported for archive workloads.
On non-clustered hosts, the management OS should be on a different array (RAID1) to the virtual machines (ideally RAID10). This gives you flexibility (for when Murphy’s Law comes a-calling) and performance.
This is a huge topic that I have covered numerous times here on the Petri IT Knowledgebase. 1 Gbps networking is fine for many. 10 Gbps is better again. And RDMA-capable networking is the best.
If you want to aggregate NICs and have network path fault tolerance then you should use NIC teaming. Third-party NIC teaming is not supported by Microsoft, so take all your issues (and expect them) to your hardware vendor if you use it instead of the completely supported NIC teaming that has been included since WS2012. In WS2012 R2 you should use the Dynamic load-balancing mode because it offers the best of Hyper-V Port and Address Hashing.
My tips for best performance and stability:
Doing the above prevents the majority of network related issues in all versions of Hyper-V.
Always get the latest drivers and firmware from the manufacturer.
You should not add lots of virtual processors (vCPUs) to virtual machines just because you can. A virtual machine with 16 vCPUs will occupy 16 LPs on the host even if it only has one thread to execute. Assign a small number of vCPUs and monitor the performance of the VM. A good metric to watch in virtual machines with multi-threaded services (such as SQL Server) is CPU Run Queue Length. High values will indicate that the guest OS is having processor latency while trying to queue up processes to execute. Adding more vCPUs will resolve this issue.
Only assign the required number of virtual processors to a virtual machine.
You should try to use Dynamic Memory as often as possible, but make sure that the services and guest OS will support it. For example, older versions of SQL Server Standard and all versions of the Exchange mailbox role do not support Dynamic Memory.
I try to set the Startup RAM in my templates to 1024 MB. This is typically the minimum amount of RAM required to install software. Starting VMs with this amount of RAM will avoid needless calls to helpdesk when delegated users (such as in a cloud) try to install SQL Server and the setup only detects 728 MB RAM (preventing the install). Set the maximum RAM to be a realistic value; the default of 1 TB is crazy; applications with memory leaks will eat up that memory (up to the maximum the host can offer), and this will cause issues in clouds where cross-charging is enabled.
WS2012 (and later) hosts support virtual machines with up to 1 TB RAM. When you have large memory workloads you need to consider how the guest (or physical) OS will schedule processes and assign RAM. Physical hardware has hard boundary Non-Uniform Memory Access (NUMA) boundaries. Spanning those boundaries degrades performance. Hyper-V has the ability to do virtual NUMA where the guest OS (Linux or Windows) is made aware of the NUMA hardware that it is occupying. This allows the guest OS to schedule processes efficiently, placing them in the same NUMA node as their assigned RAM. This is important for large memory workloads that require great performance. Virtual NUMA is disabled if you enable Dynamic Memory in a virtual machine. Therefore you should use static memory in large memory virtual machines if NUMA node spanning will unacceptably degrade the performance of those virtual machines.
Always use VHDX files instead of VHD files. There is one exception to this: Windows Azure IaaS does not support VHDX files yet, so you will need to stick with legacy VHD if you plan on moving virtual machines to Microsoft’s public cloud.
If you are using Hyper-V Replica then move the paging file of Windows guest OSs to a dedicated virtual hard disk. This will allow you to avoid needless replication of this constantly changing file.
Be sure to place all data in a virtual machine into virtual hard disks that are attached to the virtual machine’s virtual SCSI controller. This will allow the best possible performance and manageability. For example, live VHDX resizing is only possible with VHDX files that are connected to SCSI controllers.
There is a split in the Hyper-V community regarding the type of virtual hard disk to use: fixed size vs. dynamic. Fixed consumes more space but is less prone to fragmentation. Performing a defrag on a CSV requires redirect IO. I always prefer to use fixed size virtual hard disks for production server workloads.
You can configure Storage QoS to limit the number of IOPS any virtual hard disks are starving others of storage bandwidth (IOPS).
There are some tweaks you can do to your virtual machine templates to reduce the amount of hardware that is included when a new machine is deployed. This will have a small beneficial effect on the host, but for some, every little counts. There are also some tips for improving the performance of the guest OS and the virtual device drivers. According to Microsoft you should:
In the case of VDI workloads, Microsoft also says that you should:
The choice of full versus core GUI is more relevant when it comes to the virtual machine guest OS. The sum total of your virtual machines consume lots of RAM. They also consume lots of disk space, and that’s probably expensive disk space in the case of a Hyper-V cluster. If you have the skills and your services support it, then using a core installation for the guest OS can offer great savings. Just be sure that you know or have documented how to resolve issues if they happen.
Just like with the host’s management OS, you should limit your direct console activity with the guest OS, and use tools that are installed on your PC such as the Remote Server Administration Toolkit or System Center. It only takes a few minutes to install the tools and it makes day-to-day administration a lot easier.
The Hyper-V section of the Windows Server 2012 R2 Performance And Tuning Guide includes more content such as very advanced configurations (which you should only do if you understand them or under Microsoft’s supervision) and performance diagnostics using Performance Monitor.