Refer the product matrix VMware Product Interoperability Matrixes , vCenter server 5.1 is not compatible to work with ESX3.5 . So its recommend to upgrade ESX 3.5 to latest or at least next supported version ie ESX4.x.
For vmotion, make sure you have shared storage and VMFS volume created from older ESX, so that it would be compatible with both new and old ESX.
Also make sure you will continue to work with older Virtual machine hardware to make sure its compatible with old and new ESX.
In summary, both VMFS and Virtual machine hardware cannot be upgraded to latest that is supported by ESX5.1 to make sure the compatibility to work on Old and new ESX.
-Nithin