Patching Hyper-V Clusters

In this article we will look at how you can patch Hyper-V in a failover cluster using Windows Server 2012 Cluster Aware Updating and System Center 2012 SP1 – Virtual Machine Manager.

To Patch or Not to Patch

By now we hope that all administrators should understand the need to regularly run Windows Update to secure their Windows desktop and server computers. There should be a certain Darwinian element to this, but it seems that the weak in our industry continue to survive, despite the best efforts of malware such as Conficker. Any reasons to not deploy patches on a regular basis should have been done away with the presence of patching solutions from Microsoft and many others.
But what about patching a cluster? It’s messy because, by definition, a cluster hosts mission critical services such as virtual machines. You can’t just patch the entire set of hosts all at once and hope for the best. You need some kind of orchestration. There are also solutions to help us patch our Hyper-V clusters including System Center 2012 SP1 – Virtual Machine Manager Baselines and Windows Server 2012 Cluster Aware Updating.

Virtual Machine Manager (VMM) Baselines

System Center 2012 introduced a new feature in VMM called Baselines. Virtual machine manager can synchronize the Windows Catalog using Windows Server Update Services (WSUS). These updates are then managed by VMM to update the VMM managed Hyper-V hosts and the servers on which the VMM infrastructure is installed.
A baseline is created and you manually add the updates that you require to this baseline. This baseline is then assigned (targeted) to host groups and VMM infrastructure servers. You can perform a compliance scan on your infrastructure to determine compliance with the updates that are contained within the assigned baseline(s). If the infrastructure is found to be noncompliant then you can remediate the noncompliance (forcing updates to be deployed by VMM) or choose to skip the noncompliant updates.

Adding updates to a baseline in VMM 2012 SP1

Adding updates to a baseline in VMM 2012 SP1

VMM will orchestrate the patching of all VMM-supported versions of Hyper-V. In the case of Hyper-V clusters, VMM will use a maintenance mode feature to drain hosts one-by-one of virtual machines, and patch the hosts in order, until all hosts in a non-compliant cluster are patched. This process of moving virtual machines using Live Migration eliminates downtime to services provided by these virtual machines. Non-clustered hosts cannot offer this feature. However, by deploying fault tolerant services (such as guest clusters with virtual machines on different hosts – can be automated by using VMM Availability Sets in Service Templates) then you can prevent downtime to services.

Cluster Aware Updating

Windows Server 2012 Failover Clustering introduced a new feature called Cluster Aware Updating (CAU), which is similar to VMM baseline compliance in that it will orchestrate patching of clustered hosts:

  • Drain each host one-by-one of virtual machines using Live Migration. Speed up using concurrent Live Migration. Windows Server 2012 accelerates this further using Live Migration Compression or Live Migration using SMB 3.0 (2 or more 10 GbE NICs).
  • Patch each emptied host in turn until all hosts have been patched.

CAU will use Windows Update to download Windows Updates from your normal patching source (such as WSUS), and it has no dependency on the presence of VMM. Note that you can also customize the CAU job to run scripts and configure host fault tolerance. You can also deploy non-Microsoft updates from a file share, for example, installing Dell updates to your hosts.
CAU can be manually invoked from the Failover Cluster Manager console from a non-cluster machine; this is to allow the non-cluster machine to orchestrate the movement of virtual machines and the patching/reboots of hosts.

Manually invoked Cluster Aware Updating

Manually invoked Cluster Aware Updating

Once you have confidence in CAU you can enable a special HA role on the cluster to allow automated scheduled updates of the cluster. One might consider running these updates during the workday, since there is no downtime to services provided by the virtual machines thanks to Live Migration. Also, you’re around in case something goes wrong, instead of being an hour’s commute away at 3 a.m. on a Sunday.

Choosing a Patching Solution

VMM baselines involve a lot of manual work, and there are those who believe that manual patching never happens at all. If you have versions of Hyper-V that precede Windows Server 2012 then you have no choice – you must either patch manually or use VMM baselines. You can use PowerShell and/or System Center Orchestrator to automate the VMM process as much as possible.
Those who are running Windows Server 2012 Hyper-V or later can make use of CAU. Not only can you patch Windows Server, but you can also download other Microsoft hotfixes and updates for your server hardware and use CAU to deploy them. Most will choose to invoke CAU manually at first to gain confidence in the solution, but after a while you can switch to a completely automated solution. Don’t rule VMM baselines completely out, though, as many organizations have a division between patch approval (IT security) and server management (infrastructure). VMM baselines can still be used by the fabric administrators to double check patch compliance of the hosts and VMM servers to ensure that everything is patched sufficiently.