Be very careful what you click

No, this not about clicking on silly click bait articles.

Against everything I know, I decided to quickly make a few changes and clean up the file stores on a vmware cluster. I was thinking maybe an hour or so tops and I can enjoy the rest of my Saturday.

Using vmotion I moved a few virtual machines around to try balance everything nicely on the storage. Vmotion often leaves old directories lying around, so I decided to delete them to prevent confusion and keep things tidy for a change. The experienced sysadmins among you already know what’s coming.

I opened the datastore browser for vmstore1 and right clicked on the menu to delete directory proxy1 because the proxy1 virtual machine now lives on vmstore2. Suddenly I see a pop-up “Cannot delete file mail-flat.vmdk”. What? Yes! I clicked mail instead of proxy1 and when the system asked me I said yes I do want to delete it.

All the mail virtual machine’s file were gone except the virtual disk. No more virtual machine config or anything else. Yes the virtual machine was gone and would not start again. I deleted the dead machine from the vsphere inventory and decided to create a new one that will use the old virtual disk.

When I tried to select the existing disk while creating the new virtual machine, it didn’t appear in the selection. Not good at all, and the old saying, “When you are up to your ass in alligators it’s pretty easy to forget you originally wanted to drain the swamp”, comes to mind.

Panic slowly sets in. If I cannot recover the deleted virtual machine I get to spend 3 hours recovering the backup over the network. So I Google frantically and find a promising vmware knowledge base article.

The article assumes that you still have your virtual machine config available but if you follow the instructions you will we able to recover the disk and use it in another virtual machine. Just remember this is an article that doesn’t say it out loud but implies “You screwed up, Here is a way that may fix it, Don’t blame us if it doesn’t work”.

Rather just restore your backup if you are not sure of your command line skills. You may even find that it doesn’t work out although you did everything correctly. If you have space available make a copy of your virtual disk file, if the steps in the article fail, you may still find another way to recover it on the net. I was lucky and had a fully running virtual machine in 15 minutes instead of 3 hours without any data loss. It could have been very,very different.