Did you figure this out?
I am in the same boat. Have a cluster of esx 3.5 with three hosts and trying to upgrade to esxi 5.0. Recently upgrade vcenter server to version 5.0 (needed to upgrade esx 3.5 U5 to build 317866 in order to enable HA in cluster) . VMware recommends performing clean install of esxi 5.0 instead of trying to upgrade to esx 4 first. My hardware is listed in the HCL and vcenter 5.0 supports a cluster of mixed hosts.
VMware support suggested putting the host in maintenance mode before performing clean install and that everything would be taken care of after you bring the host out of maintenance mode. However, the support guy had not done this in the past and I am not comfortable with his suggestions.
This was what i was thinking of doing:
1. Migrate VMs from the host to other two hosts.
2. Disconnect and remove the host from the cluster.
3. Shut down the host. Detach the iSCSI network cables from the host before the clean install to avoid any accidental writing to the LUNs during installation (read this somewhere)
4. perform clean install. (the part i am not sure is what to choose when the installer asks "Migrate ESX, preserve vmfs datastore", "install esxi, preserve vmfs datastore", and "overwrite vmfs datastore".
5. Configure network on the host and then connect it to the cluster containing the other ESX 3.5 hosts.
6. Rescan storage
But someone who has done it needs to verify the steps before I proceed with the upgrade.
If anyone should shed light on what to choose for step 4 and also point out the steps so that the host with a clean install sees the storage that ESX 3.5 hosts are attached to (vmfs 3.x), it would be much appreciated.