VMware announces remote exam testing!

VMware have announced that remote exam testing will be available from 20th April via Pearson Vue. This is for all exams except for lab based VCAP Deploy exams which will come at a later date.

Remote exam testing involves all the same security checks as being at a test center. You show ID, take pictures of your room, cannot have materials to hand and are recorded via webcam the entire time.

Having taken remotely proctored exams for other technologies, the convenience of it is hugely valuable. Being able to now sit these for VMware is a huge plus.

Check out the announcement at vmware.com

Consolidating disks – unable to access file since it is locked

Had a vm which had flagged up as requiring disk consolidation
Attempting consolidation failed with the error ‘ Unable to access file since it is locked’

The error stack showed the following : msg.fileio.lock


Storage vMotion the disk to another data store and reattempt consolidation, this time it has cleared the locks and works. Nice and quick one, although not obvious.
Alternatively you could find the host that has a lock on the file and restart hostd, but depending on the environment this method can be a lot faster.

Module ‘MonitorLoop’ power on failed error – unable to power on vm

Came across this amazingly obtuse error today trying to deploy a new VM
Module ‘MonitorLoop’ power on failed
And was unable to power on the vm. I was surprised to find that despite the data store having some free space, this error actually indicates there is not enough space to inflate the Swap file (being 0k when powered off, and it can’t then extend to the correct size).
You can see this by going to the details of the task and digging into the error stack.

Move the VM to a data store with more space, or increase the free space on the data store and the VM will successfully power on.

vSphere 6.7 Update 3 released!

vSphere 6.7 Update 3 has now been released. Among the new features:

Ability to change vCenter Server PNID/Hostname

PNID (Primary Network IDentifier of vCenter Server) is the host name of a vCenter and 6.7 Update 3 now supports changing this post-deployment. This will be great news to anyone who has had to rename or change the domain of a vCenter on prior releases and found it required a whole rebuild.

Read more about this feature at this blog @ vmware.com

Support for multiple NVIDIA® vGPUs

VMware vSphere 6.7 Update 3 will introduce support for multiple NVIDIA GRID virtual GPUs (vGPU) per virtual machine to enable more graphics and compute intensive workloads running on vSphere. You will be able to configure up to four NVIDIA vGPUs connected to one virtual machine.

AMD EPYC™ Generation 2 support

The vSphere 6.7 U3 is compatible with the 2nd Generation of AMD EPYC™ processors.

Dynamic DNS support

With vSphere 6.7 Update 3, the usage of Dynamic DNS will be supported! vCenter will now support dynamically updating IP information in DNS, another manual job saved.

Driver Enhancements

Enhancements to VMXNET3: Guest encapsulation offload and UDP, and ESP RSS support to the Enhanced Networking Stack (ENS). Checksum calculations are offloaded from encapsulated packets to the virtual device emulation and you can run RSS on UDP and ESP packets on demand. The feature requires a corresponding VMXNET3 v4 driver.

Various driver updates will be shipped with 6.7 Update 3. The ixgben driver adds queue pairing to optimize CPU efficiency. The bnxtnet driver will support Broadcom 100 GbE network adapters and multi-RSS feeds. These are just some highlights, the following drivers are updated:

  • VMware nvme
  • Microchip smartpqi
  • Marvell qlnativefc
  • Broadcom lpfc/brcmfcoe
  • Broadcom lsi_msgpt2
  • Broadcom lsi_msgpt35
  • Broadcom lsi_msgpt3
  • Broadcom lsi_mr3
  • Intel i40en
  • Intel ixgben
  • Cisco nenic
  • Broadcom bnxtnet

More information about this update @vmware.com

Release Notes:
ESXi 6.7 Update 3
vCenter 6.7 Update 3
vSAN 6.7 Update 3

Enable CDP advertising – help the network team help themselves!

A customer of mine requested help in documenting which switch ports were connected to ESXi hosts. Rather than simply documenting this which may get out of date if not maintained, I instead suggested we enable CDP advertising on the vSwitch level, in order for the network team to be able to obtain this information themselves on an ongoing basis.

By default vSwitches come with CDP enabled in listen mode only, being able to detect information about the switches they are connected to but not relaying info about themselves to the switches.


To configure advertising on a standard vSwitch, you SSH onto the host and run the following, changing the vSwitch name for the relevant one:

# esxcli network vswitch standard set -v vSwitch0 -c both

If running distributed switches, you can do this in the GUI of the web console. Select your distributed vSwitch and select Manage > Settings > Properties and click Edit.

Under Discovery Protocol change Operation to Both, and it will both listen for CDP info from the switch and Advertise its own CDP info also.

Unable to delete datastore – filesystem is busy

I had noticed that a customer had been building ESXi hosts but the local datastore on the host was being created as vmfs5 instead of vmfs6.

No problem – just delete the local datastore, assuming no VMs have been built on it, and recreate as vmfs6 right? not so simple. Attempting to delete the datastore threw up this error:

Cannot remove datastore ‘Datastore Name: because file system is busy. Correct the problem and retry the operation.

So whilst this occured due to local datastore being created on the older vmfs version, it could apply to any datastore you need to delete. Here we needed to try and find out what could be writing to the datastore that would affect the ability to delete it. Some things to check:


There is likely a dumpfile set up on the host. Run the following command to check

 # esxcli system coredump file list 

If it lists there is a dumpfile configured on the local disk, run this command to turn it off:

 # esxcli system coredump file remove --force

If the datastore being deleted is a shared datastore, run the following command to find the owner of the file:

# vmkfstools -D /vmfs/volumes/Datastore/vmkdump/684938663845.dumpfilevmkfstools -D /vmfs/volumes/Datastore/vmkdump/123456789101.dumpfile

The output will look like:

 Lock [type 10c00001 offset 200392704 v 10, hb offset 3875328  gen 3, mode 1, owner 52ebd042-43b191f0-0173-012345678910 mtime 250 

The last part of that id relates to the mac address of vnic0 of the owning host, e.g 01:23:45:67:89:10

Run the above vmkfstools command to delete.

Once you have deleted the datastore, run the following command to reenable the dump file elsewhere

 esxcli system coredump file add -d datastore_name

Scratch Location

Browse ESXi > Configure > System > Advanced System Settings and find setting ScratchConfig.CurrentScratchLocation  (). If the ESxi host is used as Scratch Location, edit to something like /tmp and reboot the host.

You can then delete the problematic datastore. Remember to go back and change the scratch location to the new datastore.

Unable to update VCSA to 6.5 u2d

 If you are using any version of VCSA 6.5 U1d or below, you may come across this error whilst attempting to update to 6.5 U2d:

Latest updates already installed on vCSA, Nothing to stage/install

This is due to the way the VCSA updater processes build numbers on older versions. Fortunately, there is a workaround

Visit KB59659 at vmware.com and download changebuild.sh from the attechments on the article

  1. Log in as root via SSH to VCSA
  2. Run the command to enable the shell. shell
  3. Download the 59659_changebuild.sh script attached to the article.
  4. Using a supported tool, sftp/scp the 59659_changebuild.sh to the VCSA.
  5. Change the file name to changebuild.sh and copy it to /root on the vCenter Server Appliance. cp 59659_changebuild.sh /root/changebuild.sh
  6. Run the command to execute permission: chmod +x /root/changebuild.sh
  7. Run the command to execute the script: /root/changebuild.sh
  8. Proceed with the update.

vCenter PSC converge tool now available for vCenter 6.5 with U2d

VMware is spreading some holiday cheer in the form of the latest update for vCenter 6.5 U2d. Buried in the release notes is:

  • vCenter Server 6.5 Update 2d adds a CLI tool to convert instances of vCenter Server Appliance with an external Platform Services Controller into vCenter Server Appliance with an embedded Platform Services Controller connected in Embedded Linked Mode. For more information, see the 6.7 vCenter Server Installation and Setup guide.
  • With vCenter Server 6.5 Update 2d, you can add VMware Platform Services Controller appliances to Active Directory 2016 domains.
  • With vCenter Server 6.5 Update 2d, you can use the new vRealize Operations Manager plug-in that provides specific metrics and high-level information about data centers, datastores, virtual machines, and ESXi hosts, to vCenter Server and vSAN. The plug-in is supported only in the vSphere Client.
  • With vCenter Server 6.5 Update 2d, the new vRealize Operations Manager plug-in adds by default the Patch method, supported by the HTTP protocol, to facilitate the online installation stage.
  • With vCenter Server 6.5 Update 2d, you can configure the property config.vpxd.macAllocScheme.method in the vCenter Server configuration file, vpxd.cfg, to allow sequential selection of MAC addresses from MAC address pools. The default option for random selection does not change. Modifying the MAC address allocation policy does not affect MAC addresses for existing virtual machines.

This feature was an eagerly awaited core feature of vCenter 6.7 U1 which has now been back-ported to 6.5.
While vCenter 6.5 Update 2 included support for enhanced linked mode using embedded PSCs, this was only good for new deployments leaving everyone with existing installations stuck with external PSCS. This feature allows us to migrate from a setup with external PSCs to embedded PSCs in a few easy steps.


  1. Edit the converge.json and decommission_psc.json templates to include information about the managing vCenter Server Appliance. See Preparing JSON Configuration Files for Reconfiguring External to Embedded Nodes for information on preparing the converge.json template.
    1. Confirm the appliances have been backed up
    1. Accept the thumb print
    1. Run the vcsa-util converge converge.json command on the client machine running on a Windows, Linux, or Mac OS operating system to begin the convergence process to install and configure the new embedded Platform Services Controller. See Syntax of the Converge Command for a list of available arguments for the vsca-util converge command.
    1. You can log into the vCenter Server Appliance appliance management interface (https://appliance-IP-address-or-FQDN:5480) and see that it is now a vCenter Server with embedded Platform Services Controller.
    1. Reconfigure any products that use the external PSC such as vRealize suite, NSX Manager, etc to use the new embedded Platform Services Controller.
    1. Run the vcsa-util decommission decomission.json command to decommission the original Platform Services Controller. This operation removes the external Platform Services Controller from the SSO domain.
    1. Shut down and delete the old PSC VMs.

Worth noting:

  • You still need to update all nodes, including existing external PSCs to 6.5 U2d before running this tool
  • This tool is only for vCenter Appliance (VCSA) deployments, not windows. If you have not already migrated from windows to vcsa, take this as a sign to do so sooner rather than later.
  • The external PSC configuration is being deprecated by VMware, so it is worth taking the time to migrate


vCenter 6.5 Release Notes at vmware.com
vCenter converge process at vmware.com

vSphere 6.7 U1 GA released! Release notes & downloads

VMware have finally bestowed upon us their latest release of vSphere 6.7 – Update 1!
This brings with it some rather welcome new festures and quality-of-life tweaks:

  • Migrate vCenter with Embedded PSC *between* vSphere domains, retaining data such as tags & licences.
  • vCenter can provide relevant links to KB articles
  • Burst filter to protect vCenter from identical alert flooding
  • HTML5 client now fully featured including new simplified workflows for VCHA
  • vCenter converge tool to migrate from external to easier to manage embedded PSCs
  • Provides upgrade path from 6.5 U2 to 6.7

And many more
vCenter Server 6.7 U1 :
Release Notes
ESXi 6.7 U1:
Release Notes
PowerCLI 11:
What’s New

vCheck – Could not establish trust relationship for the SSL/TLS secure channel

Firstly, if you are not using vCheck in your vSphere environment, hop on over to Alan Renouf’s blog where you will find vCheck – a community driven free set of PowerCLI vSphere environment checks that will generate a nice report (now with the new Clarity html5 client theme!) to be emailed out to you before you reach the office each morning. It is not a replacement for proper monitoring but provides a level of insight you won’t get elsewhere and I swear by it to get a head’s up on potential issues before they have a chance to become catastrophies.
I have recently been finding when running vCheck that several checks would return the error

The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.

After much troubleshooting I discovered this was down to certificate trust.  In order to resolve this issue:

Download the vCenter certificates

Browse to your vcenter address over https:  Https://vcenter.domain.com and on the bottom right, click Download Trusted Root CA Certificates
You’ll get a .zip file with the certificates. Unzip, and if on windows browse to /certs/win and grab the CRT that has a corresponding CRL.
Import this certificate via whatever method is used on your OS of choice. On Windows for example Internet Options > Content > Certificates > Trusted Root Certification Authorities.
Click Import, the CA should be named CA. If it is named ssoserver, you’ve grabbed the wrong cert.
Once imported, vCheck should now be running without errors!