How I Relearned the Consequences of Improper Monitoring
I had just lost the RAID array that hosts my ESXi data store. I didn’t yet know that’s what had happened, but with some investigation, some embarrassment, and a bit of swearing, I would find out that an oversight on my part three years ago would lead to this happening.
I first realized there was trouble when every VM on the host became unresponsive. Most notably, the Plex Media Server fell off the network which caused the episode of Modern Family that we were watching to immediately freeze. What was odd to me is that while the VMs were unreachable, the ESXi host itself was fine. I could ping it, ssh to it and load it up with the vSphere client. The first wave of panic hit me when I found messages like this in the host’s event log:

This was quickly confirmed from the ssh shell by looking for the data store and finding that a) the symlink for the volume (RAID1) pointed to a non-existent directory and b) the reported size of the volume was a paltry 450MB compared to the 930GB I expected.

Since I knew from prior experience Continue reading