Secret Commands for Emergency Maintenance from the ESXi4 Console

Anyone who has installed VMware's ESXi4 hypervisor (if you haven't then get an idea of what you are missing with my article on how to setup your own ESXi4 server will be aware that the options for maintenance at the server itself are somewhat limited. In fact its just a one page menu:

For setting up your ESXi4 server you don't even need to go that far in fact, since you can just point your vSphere Client at the IP assigned to the server by DHCP and do everything afterwards from there. So why should you even bother to attach a monitor and keyboard to your ESXi server?

To be honest, if everything is working as it should then you can happily leave your server humming away in its place and never need to touch it again, but when in IT has everything worked happily ever after? During my day job supporting a multitude of ESXi servers I have encountered several scenarios where the vSphere Client has failed me and more direct access was required. Power cycling the ESXi Server is usually an effective solution but in a production environment it poses additional risks and should only be treated as a last resort, so here I will look at the other options available.
Once you are at the server itself there are basically two interfaces available to you, the "official" ESXi4 console with its system configuration menu, and then the "unsupported" command line Linux console. Yes, according to the official documentation one of the fundamental differences between ESX and ESXi is the latter's lack of a console, but there is a very reduced version hidden in there. Its "unsupported" because VMware doesn't want you to go there unless directed by their support personnel, so bear that in mind and stay away from it unless you are confident you know what you are doing. Now you have been warned let's have a look at the two interfaces:

The (official) ESXi4 Console

Whether you've got a keyboard and mouse plugged in or are fortunate enough to have some sort of KVM over IP solution then the first thing you should see on your ESXi server is a screen like this:

Its most likely you have got this far because your vSphere Client isn't working, so the first thing to do is press F2 to access the System Customization menu. On here you can check the management network settings to make sure you have the correct IP address for the host, and "Test the Management Network" to see if there is connectivity to the rest of your network. Finally select the "Restart Management Agents" option, these are the services which manage the connection from the vSphere Client and control the VMs, so restarting them may enable you to connect.
Should that not work then its tempting to just go for the F12 option to restart the host, but first consider the Virtual Machines running on it, and what will happen to them. ESXi4 is remarkably resilient and often you can find that while your vSphere Client thinks the host and all its VMs are dead to the world they are in fact running fine.
Now, when everything is working properly, issuing a shutdown or restart from the console will tell ESXi to instruct the VMware Tools service on each VM to perform a safe shutdown before it does so to itself. Since you're at this console though its likely everything isn't fine, or maybe you haven't installed VMtools on every VM, which means you can't be sure of a safe shutdown so don't want to press F12 yet. Provided you know which VMs are running on this host you may be able to logon to them via RDP/SSH/etc and issue a shutdown command that way, then once you are confident they have all indeed shutdown then it will be safe to restart the host.

You may be reluctant to issue the restart command yet though, perhaps you aren't sure that all the VMs have properly shut down, or even exactly what VMs are running on that host (if you are running a fully automated vSphere cluster with several ESXi hosts this can happen quite easily!).  In this case there is another option available to you, the "unsupported" console.

The Unsupported ESXi4 Console

A word of warning again, VMware call this "unsupported" with good reason - if you make a mess of your server while in this mode they won't help you out. All the same if you follow instructions and don't get adventurous then you should be pretty safe, and after all this will usually be your last resort when all else has failed.
How to access the unsupported console has already been well documented so I won't repeat it here, instead you can take a look at either David Davis' article here or my help4IT blog.  Now you have a command prompt on your ESXi server let's have a look at what commands you might find useful:
services.sh restart
Its worth trying this as it will issue a restart command to all the management agents and services on the host, as it processes you should see each one stopping and then being started. However it is effectively the same as selecting the "restart management agents" option from the standard console so if that didn't help you its unlikely to be much different here.
vim-cmd vmsvc
This is the most useful command for emergency maintenance and takes several extensions, first of all you need to run vim-cmd vmsvc/getallvms . This will return a list of the virtual machines registered on the server, and most importantly, their unique ID numbers which you will require for subsequent commands.

As you can see the command returns a list of VMs, with the ID number first, followed by the VM name and other useful information such as the location of the .vmx file and any notes you have appended to the VM properties in the vSphere Client.
Following that you can try vim-cmd vmsvc/power.getstate #### , where #### is the ID number of the VM to query. This will show you the actual power status of the VM in question, in the example below I am querying my "Win2003Test" server, which is suspended:

Forcing a power off of a VM is possible using vim-cmd vmsvc/power.off #### but that rather defeats the object of what we are trying to achieve (avoiding a dirty shutdown of our virtual servers in case you have forgotten!). Instead you could try vim-cmd vmsvc/power.shutdown #### which should issue a shutdown command to the VMware Tools agent on the VM, wait a few minutes and then try another power.getstate to see if it has been successful. Should that not work then you can issue a vim-cmd vmscv/power.suspend #### which will simply suspend your VM to disk so it will not be affected by a restart of the host.
There are several other commands available with vim-cmd vmsvc/ - power.hibernate, power.reset and power.reboot for example. To see a full list of all the commands available just type vim-cmd vmsvc without any extension:

As you can see there are a lot more commands available, which replicate the actions available in the vSphere Client, and as such there is little need for you to use them.

Conclusion

Its obvious from this exercise that despite what the documentation says there is still a lot of functionality left in ESXi's stripped down console, albeit well hidden and not user-friendly. In this article we have covered those commands which could come in useful in some problem situations when you need to get direct control over a host's VMs to avoid a dirty shutdown. Should you have a need (or desire) to do more management and configuration tasks of ESXi via the command line then VMware provide a Remote Command Line Interface (RCLI) appliance for that purpose. For further information please refer to this section of the VMware website.