I have the same Motherboard and CPU combo - my way around that error was to hit F11 during the MSI splash screen to enter the boot menu and manually select the USB drive I wanted to boot from. (NOT the UEFI option, that's the one causing the errors)
I used the info here to create a bootable usb from Windows:
http://www.vm-help.com/esx40i/ESXi_USB_install.php
When running syslinux.exe make sure you use version 3.86 specifically and make sure you are running the command prompt as administrator.
Also I changed isolinux.cfg to syslinux.cfg and (possibly unnecessarily) isolinux.bin to syslinux.bin
Trying to install at this point you will see the installer freeze halfway through loading. The solution is in the ESXi manual:
In the file syslinux.cfg change the line "Append -c boot.cfg" to "APPEND -c boot.cfg -p 1"
The unfortunate next hurdle is that once the installer loads it is telling me there is no driver for the onboard network adapter for our MSI motherboard, and won't let you install without one. So you'll be stuck here unless there's a driver we can use somewhere.
===========UPDATE==========
My network adapter issue is due to using version 5.5, this issue was apparently not present with 5.1, 5.0
Our MSI motherboard's NIC is a Realtek RTL8111E and though I haven't tried it yet, this looks like a promising workaround: TinkerTry IT @ home | Install ESXi 5.5 with Realtek 8111/8168 NIC