Evening,
Lots of questions. Allow me to ask some to help narrow it down.
I would suggest the following course of action:
1. Install ESXi and setup management network using a single nic (native or vlan tagged)
2.Direct connect to the ESXi host configure shared storage and make sure all node have the storage
3.Install vcenter appliance and configure to use same network as ESXi Hosts (you will need host entries if no DNS is available you may want to reference the hosts in vcenter by ip)
4.Point the whole pile to public ntp servers
5.Setup your dvswitch and migrate management off
Now to a larger question with 8 nic's to use how do you divide traffic.
First real question to ask is how much bandwidth does NFS require? I assume these are all 1GB nic's and your NFS has a single target that is a VIP that moves between controllers. As such you really can only one one nic to use for NFS. I would suggest the following:
2 X Nic for NFS (active/standby) Running tagged VLAN2 or Native depends on switch config
4 X Nic for Server networks using load balanced teaming (active/active/active/active) tagged vlans
2 x nic for Management / vmotion - Use Them active/standby management and standby/active for vmotion with NIOC to control flows in case they end up on same connection.
It depends thou... if you plan on having a lot of vmotion you may want to reduce the amount of nic's for vm's and add to vmotion.
Also look to keeping loads off a single nic (For NFS use one port on one nic and another nic for the second port.
I hope this helps let me know if you have additional questions.
Thanks,
J