Site Tools


VMware Troubleshooting

Cant apply or remediate a host profile

A host profile is attached by the cluster and there is no way to apply or remediate.

It seems like you are using the HTML webclient which doesnt have this option (yet).

Either use the flash version (if you have vCenter 6.0 the software client will also work)
apply it via PowerCLI

#remove everything after <vcenter> to use your windows authentication
Connect-VIServer <vcenter> -User <user> -Password <password>
Apply-VMHostProfile -Entity <esxhostname.your.domain>
#if you are not sure what name to use, just list all your esx hosts

Recheck the compliance of the host profile. If some issues still exist, execute “Apply-VMHostProfile …” again.

Cant configure vSphere HA for host

Trying to activate vSphere HA fails with the following messages:

vSphere HA agent cannot be correctly installed or configured

Reconfigure vSphere HA host
Status: Operation timed out.

There could be different reasons…

At first, if you dont want to get VMs migrated right away, activate maintenance before executing something from below.

Check if your service “vSphere High Availability Agent” is running on your host. (its stopped by default if you are in maintenance mode)
If it isnt running, start it and execute “Reconfigure for vSphere HA”.

Still not working? alright then…

Go to your cluster configuration and select “Services → vSphere Availability”
On the right side of “vSphere HA is Turned ON” click edit and turn it off.
Apply and then turn it back on.

Your vSphere should now work again.

If it still doesnt,

Search in vSphere Client doesnt work

Trying to search a system fails with the following message:

Login to the query service failed.

The server could not interpret the communication from the client. (The remote server returned an error: (500) Internal Server Error.)

VMware says that this is “an expected behavior

Searching for Inventory objects when logged in to the vSphere Client using the Use Windows session credentials option is not supported.

For sure this is expected… why would you let lazy people search in your vCenter? m(

At the login screen deselect “Use Windows session credentials” and log in with your username and password.

Cant connect a VMDK backup because of duplicated UUIDs

Connecting a backed up VMDK to the same machine results in a duplicated UUID

msg.disk.duplicateUUID:Virtual disks

"/vmfs/volumes/DATASTORE/VM_FOLDER/VM_NAME.vmdk" and "/vmfs/volumes/DATASTORE/VM_FOLDER/VM_NAME2.vmdk" have the same UUID. Virtual disks with the same UUID should not be assigned to a virtual machine because duplications can lead to problems when the guest boots. Make sure the virtual disks have unique UUIDs.

Well… Duplicated UUIDs…

Login on the ESX host to change the UUID of the VMDK.
Check the current and set a new one afterwards.

vmkfstools -J getuuid <vmname>.vmdk
vmkfstools -J setuuid <vmname>.vmdk

Snapshot Manager fails with the error: Operation not allowed in current state

Creating / removing a snapshot fails with the following message

Cannot create snapshot
Operation not allowed in current state

On the ESX itself you may find the following error in /var/log/hostd.log

2019-10-08T07:16:52.016Z error hostd[14B81B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/acde0a05-cd36531b/VMNAME/VMNAME.vmx] Invalid transition requested (VM_STATE_ON_SHUTTING_DOWN -> VM_STATE_CREATE_SCREENSHOT): Invalid state
[07:43:03.404 F64A8B90 error 'vm:/vmfs/volumes/4acc704a-2b59540f-116b-0017a477fc18/VMNAME/VMNAME.vmx'] Invalid transition requested (VM_STATE_ON_SHUTTING_DOWN -> VM_STATE_CREATE_SNAPSHOT): Invalid state


This issue occurs if the vCenter Server management agents have stopped causing the snapshot attempt to move in to the ON_SHUTTING_DOWN state.


Connect to your ESX host via SSH and restart the management agents

/etc/init.d/hostd restart
/etc/init.d/vpxa restart

You should be able to create or remove the snapshot afterwards

VMware KB 1027707 - Invalid state
VMware KB 1003490 - Restart management agents
VMware KB 2004746 - Activate SSH

No coredump target has been configured

You see a yellow warning on your ESX hosts with the following message

No coredump target has been configured. Host core dumps cannot be saved


Just like the warning says, there is no coredump target configured.


Solution 1:
Create the target → VMware KB 2004299

Solution 2:
Suppress the warning (WARNING: this will “disable” core dumps when you run into purple screens!)

Go to your “Advanced Settings” and set “SuppressCoredumpWarning” to “1”

Host > Configuration > Advanced Settings > UserVars > SuppressCoredumpWarning > 1

Lost connectivity to the device naa.60xxxx backing the boot filesystem

You see a yellow warning on your ESX hosts with the following message

Error: Lost connectivity to the device naa.60xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx backing the boot filesystem /vmfs/devices/disks/naa.60xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx. 


Your ESX lost connection to its bootdevice, maybe because of a network outage (switch reboot, cable disconnect,…)


Solution 1:
Put the host into maintenance mode to migrate all VMs to another host and reboot.

Solution 2:
Access your ESX via SSH and restart the managementagent

/etc/init.d/hostd restart 

VMware KB 50441

No space left on device (vCenter)

  • vCenter AD Login doesnt work
  • “shell.set –enabled True” shows the error “Unknown command: `shell.set'“
  • creating files will show “No space left on device”

You can also check this by connecting via SSH and entering the following commands



>Host to connect to (user@host): root@localhost
>Are you sure you want to continue connecting (y/n)? y

Warning: Permanently added 'localhost' (vim) to the list of known hosts
Failed to connect to localhost: No space left on device @ dir_s_mkdir - /root/.rvc

You have no space left on your device, mostly /var/log/ and mostly because of audit.log


Because you cant use PuTTY to operate the shell, you need to open the shell with a kernelparameter in the GRUB bootloader. You need to reboot your vCenter (DOWNTIME!). Before starting your VM, set the “Power On Boot Delay” to 10 seconds.



After starting the VM, check the VMware console and wait for the bootloader. You can stop the autostart by hitting “spacebar”

Select the vCenter appliance and hit “e”
Then select the right kernel and hit “e” again
Add “init=/bin/bash” to the parameters (seperated by space “ARG1 ARG2”) and boot with “b”.
You should now be in the shell. Check the drives with “df -h”
If audit.log is the problem, empty it with

echo "" > audit.log

To fix the logrotate/cron, check VMware KB 2149278

Restore from different datastore

A VM is broken and needs to be restored. To do so you need to copy the VM from the backup datastore to your productive one

You dont want to run VMs from your backup datastore

Connect to your esx host via ssh and copy the files with “cp” or “vmkfstools”
Make sure you either rename your old .vmdk + flatfile to .old or your restore to something different (rename flat in .vmdk)

cp -a myvm.vmdk /vmfs/volumes/prod_datastore/myvm/myvm.vmdk
cp -a myvm-flat.vmdk /vmfs/volumes/prod_datastore/myvm/myvm-flat.vmdk

With vmkfstools you dont need to edit the flat name inside .vmdk, the tool does this for you if you choose another name

vmkfstools -i myvm.vmdk /vmfs/volumes/prod_datastore/myvm/myvm.vmdk

vmware/troubleshooting.txt · Last modified: 2022/09/03 16:27 by lunetikk