ESXi Autostart Start Delay, Stop Delay & Stop Action

I spent a lot longer than I would have liked today googling for more information on the ESXi Autostart options of Start Delay, Stop Delay and Stop Action. In the end my friend Keith’s Google-Fu was stronger than mine and he pointed me in the right direction. Because the articles we found (listed below) don’t use exactly the same terminology (hence me not finding the answers I wanted) I decided to summarise them here with the “correct” terminology of the latest ESXi 7.0 UI.

Start Delay – After you start the ESXi host, it starts powering on the virtual machines that are configured for autostart. After the ESXi host powers on the first virtual machine, the host waits for the specified delay time and then powers on the next virtual machine. The virtual machines are powered on in the startup order specified in the Per-VM Overrides pane.

Why would you want to do this? Why not just start all VM’s at the same time? A couple of reasons.

1. Dependencies. You might need to wait for some slower storage arrays to boot before you bring up file sharing servers. Or you might need to bring up some sort of domain controllers before bringing up applications like SQL.
2. Resources. When VM’s start their hardware requirements often spike. Starting them all together might could either put a huge stress on the Host or even exhaust the host resources if you VM’s are oversubscribed.

Stop Delay – is the maximum time the ESXi host will wait for a shutdown command to complete. If the Virtual Machine does not shutdown within the delay time specified a hard Power Off is issued to the Virtual Machine. When an ESX host is shut down, the host follows this process:

1. Autostart Manager initiates the automatic shutdown of the first Virtual Machine. The shut down is one of the power operations (Power Off, Shutdown or Suspend) depending on the configuration.
2. Autostart Manager waits the amount of time specified in the Stop Delay parameter for the Virtual Machine to complete the power operation.
   • If the power operation is completed in less time than specified in the Stop Delay, the ESXi host begins to shut down its next Virtual Machine.
   • If the power operation is not completed within the time specified in the Stop Delay, the ESXi host sends a power off command then begins to shut down its next Virtual Machine.

Stop Action – Power Off, Suspend, Shutdown, System Default

• Power Off: This is the equivalent of you physically pulling the power cord out of a real life server. It immediately terminates the VM with no graceful saving of files/cache.
• Suspend: Put the VM into a sleep state that can be resumed at exactly the same point later on. This sounds great but lots of VM’s won’t like this due to the clock differences from suspension to resumption.
• Shutdown: This instructs the VM to perform a graceful shutdown of itself. For a lot of servers, like Windows, you must have VMWare Tools installed on the VM for this action to be honoured by the operating system. However, I’ve seen many VM’s that are “appliances” running a custom operating system, like Cisco or HPE virtual networking equipment to honour this command even though they don’t support installing any software (e.g. VMWare Tools) on them.
• System Default: the VM will do whatever is set as the global Stop Action under Host>Manage>Autostart

Note: under ESXi 7.0 I found that the Stop Action setting under Host>Manage>Autostart also included the option of “System Default”. This appears to be a UI bug as this IS the place to set System Defaults, not defer to them. You can see a question I logged on this on the VMWare forum and the moderators response in the links below.

Wait for heartbeat – if Wait For Heartbeat is true, then the power-on sequence continues after the the first heartbeat has been received from the VM. If it is false, the system waits for the specified delay and then continues the power-on sequence. If all your VM’s have VMWare tools installed then you can speed up the boot process as it’ll immediately move on to the next VM once the tools send a heartbeat to say its up (instead of waiting the Start Delay interval), but the VM could still be resource hungry at this point so you may prefer just to let the Start Delay run its full course every time.

If you have documented evidence that any of the above is incorrect then please drop me a comment so I can correct the above. This is my interpretation of various articles and discussions I’ve read through today, that I thought I’d pull into a single place for others.

Links:

Understanding the Shutdown Delay setting for ESX hosts (1004421) (vmware.com)

Edit Virtual Machine Startup and Shutdown Settings in the vSphere Web Client (vmware.com)

ESXi Autostart system default option wrong? – VMware Technology Network VMTN