There is definitely something weird going on. The parameter seems to have reset itself:
/vmfs/volumes/5370a9e1-56c8f49a-d807-000c29dce64f/TCLVM-58 # esxcli system settings kernel list -o maxVMs Name Type Description Configured Runtime Default ------ ------ -------------------------------------------------------------------- ---------- ------- ------- maxVMs uint32 Max number of VMs VMKernel should support. 0 == determine at runtime 0 0 0