CPU incompatible adding host to cluster after patching for Spectre/Meltdown

Interesting problem here. Customer was adding some new hosts to an existing cluster, but got this error:

Move host into cluster

The host’s CPU hardware should support the cluster’s current Enhanced vMotion Compatibility mode, but some of the necessary CPU features are missing from the host. Check the host’s BIOS configuration to ensure that no necessary features are disabled (such as XD, VT, AES, or PCLMULQDQ for Intel, or NX for AMD).

Untitled picture

Usually, this would be due to different CPU hardware, or CPU features in the UEFI not being enabled to match the existing hosts.

In this case, the hardware and UEFI settings were the same. It was discovered as part of QA testing the new hosts were updated with the current patch level which includes CPU microcode updates for Spectre/Meltdown.

This changes the available CPU features and causes a problem. While you can have hosts with differing patch levels coexist within the same cluster for the purposes of a rolling upgrade (and vcenter will only enable the fixes once all hosts have been updated). You cannot add NEW hosts to a cluster that has this microcode installed until the existing hosts have been updated with it.

SOLUTION

In this instance it was a simple solution: use the host rollback option to revert to the previous build level, which matched the other hosts in the cluster and did not display differing CPU features due to the spectre/meltdown microcode.

Reboot the host, and at the ESXi boot screen, press SHIFT+R

You will be presented with this warning:

Current hypervisor will permanently be replaced
with build: X.X.X-XXXXXX. Are you sure? [y/n]

Press Y to revert to the previous build

You can read more about this process at vmware KB1033604

 

Alternatively, you can fully patch the cluster before adding in the new hosts.

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s