Sorry please ignore this now! Basically I forgot that you cant have 2 x default gateways so instead you need to add static routes so thats what I did:
/bin # esxcfg-route -a 10.81.0.0/16 10.81.253.1
Adding static route 10.81.0.0/16 to VMkernel
/bin # esxcfg-route -a 10.96.0.0/16 10.81.253.1
Adding static route 10.96.0.0/16 to VMkernel
/bin # esxcfg-route -a 10.97.0.0/16 10.81.253.1
Adding static route 10.97.0.0/16 to VMkernel
/bin # esxcfg-route -a 10.254.240.0/20 10.254.129.1
Adding static route 10.254.240.0/20 to VMkernel
/bin # esxcfg-route -a 10.254.224.0/20 10.254.129.1
Adding static route 10.254.224.0/20 to VMkernel
/bin # esxcfg-route -a 10.254.128.0/20 10.254.129.1
Adding static route 10.254.128.0/20 to VMkernel
Cheers
Dave