I did some digging in the vmvcsr.log file and noticed that it was changing the SetupDir and SETUPEXEDIR to Z:\vCenter-Server instead of Z:\vCenter-Server\
I added the SETUPEXEDIR= and SetupSource= commandline options and the installer still changed them back. So, I removed the /v`" and put the directories all in escaped quotes. This resolved the issue.
I still think that the installer should put the correct path on it's own installation source dir. It should be smart enough for that!
Here's the Arguments line that worked:
$processInfo.arguments = "/S /qr SourceDir=`"$sourceDir`" SETUPEXEDIR=`"$sourceDir`" DB_EXISTING_VERSION=550 DB_USERNAME=`"$AdminUser`" DB_PASSWORD=`"$AdminPass`" LAUNCHWIZARD=1 VCS_DB_BACKUP=1 SSO_ADMIN_USER=`"$AdminUser`" SSO_ADMIN_PASSWORD=`"$AdminPass`" /L*v $env:temp\vmvcsvr.log"