The first step is to upgrade vCenter to vSphere 5 -
Please help me outline the procedure correctly. I plan to reuse the hostname and the IP. There are three ESX 3.5 hosts sharing vmfs 3.x datastores. For a clean install on one of the hosts, do i enter it into maintenance mode, shut it down once in maintenance mode, and then clean install esxi 5.0? Or, do I remove the host from the cluster first and then shut it down?
Yes you will want to enter into maintenance mode which will force you to move or shut down all the VMs at which point you shoudl remove it form the cluster and then shut it down.
Also, during clean install I would choose "overwrite vmfs datastores" since I don't have any local VMs? Once the host is up and network configured I would add it to the cluster that has ESX 3.5 hosts and then rescan the storage (I am assuming even if it is in maintenance mode, taking it out of maintaining mode would require reconnecting the host first, similar to adding host to the cluster)?
You will want select the overwrite VMFS datastores you will want to make sure that is not connected to the iSCSI network. Once you have ESXi 5 installed before restarting reattach the iSCSI network and reboot the host - when it is up readd it to the cluster -
Also I am sure you have reviewed this but here is alink to the upgrade guide - http://pubs.vmware.com/vsphere-51/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-511-upgrade-guide.pdf