The discussion can be closed since it seems to be not related to VMware.
I have tried installing a Linux distro as well as Windows W2K12R2 on the machine and in both cases, the installer fails as soon as the machine has more than 32 Gb of RAM.
With 32 Gb, VMware, windows and linux can all run fine and get installed properly, with 64, all of them fail.
I have opend a ticket with Dell for faulty hardware.
Thanks to both of you for trying to help!
Kind regards,
Maarten