I just had this same issue in one of our clusters when I upgraded from 5.0 to 5.1.U1. After working for several hours, we realized cause of the issue to be following:
- On the same vSwitch as our Management Network and our VMOTION ports, we had a third port for NFS access. As this cluster pre-dated both my and my co-workers start of employment, we didn't know why this was created this way.
- At a SSH session for the affected hosts, we used vmkping to successfully ping out to multiple networks. However, when used with the "-i" parameter to force the ping out the management network, nothing beyond the local subnet could be pinged (i.e. DNS servers, other gateways, etc.)
- Despite this, all VM traffic and functionality continued, as well as vCenter.
- We removed the NFS port from the vSwitch and everything began to work.
It seems that in locking down ICMP to only the management interface, ESXi may get confused after an upgrade as to which interface this actually is. I would recommend taking a close look at the management network vSwitch before an upgrade to 5.1 to avoid any related issues.
Thanks.
Matt