I continued work on my project creating the 100 VMs I was talking about to achieve that "enterprise-scale" environment.
Well to be honest it's not really enterprise-level, more like medium, but I think the possibility of running hundreds of VMs a desktop PC is close to nil.
Anyway, having at least 100 VMs has an impact on the way vSphere handles things. The various algorithms (DRS, HA etc) behave quite differently!
It will be interesting to see how these algorithms behave and be able to test things. One can't do this type of tests with a small number of modern VMs, for sure.
Also, one gets the change to learn a lot of interesting things. Such as mass deploying/cloning VMs..
I manually created my first VM. It's called TCLVM1 (which stands for Tiny Core Linux VM #1). This is its configuration:
OS: TinyCore Linux 'CorePlus' 5.2 32bit
CPU: 1x vCPU single-core
RAM: 64MB vRAM
DISK: 64MB vDisk
NET: 1x vNIC
OTHER: -
To have 100 total VMs I had originally planed to make a template based on the original VM and then use a script to deploy 99 VMs using said template.
Then I realized it might be easier to make 99 clones through a script.
Searching on the Internet, I found the following script prepapred by VCDX Magnus Andersson:
# PowerCLI to create VMs from existing vSphere VM
# Version 1.0
# Magnus Andersson RTS
#
# Specify vCenter Server, vCenter Server username and vCenter Server user password
$vCenter=”192.168.0.100“
$vCenterUser=administrator@vsphere.local“
$vCenterUserPassword=Ganxter1.“
#
# Specify number of VMs you want to create
$vm_count = “99“
#
# Specify the VM you want to clone
$clone = “TCLVM1“
#
# Specify the Customization Specification to use
$customspecification=”TinyCoreLinux-VM“
#
# Specify the datastore or datastore cluster placement
$ds = “Host1DS1-12.5GB-VMFS5.6-SSD-AGIL“
#
# Specify vCenter Server Virtual Machine & Templates folder
$Folder = “TCLVM-clones“
#
# Specify the vSphere Cluster
$Cluster = “X-Systems-DRS-HA-cluster“
#
# Specify the VM name to the left of the – sign
$VM_prefix = “TCLVM-“
#
# End of user input parameters
#_______________________________________________________
#
write-host “Connecting to vCenter Server $vCenter” -foreground green
Connect-viserver $vCenter -user $vCenterUser -password $vCenterUserPassword -WarningAction 0
1..$vm_count | foreach {
$y=”{0:D2}” -f $_
$VM_name= $VM_prefix + $y
$ESXi=Get-Cluster $Cluster | Get-VMHost -state connected | Get-Random
write-host “Creation of VM $VM_name initiated” -foreground green
New-VM -Name $VM_Name -VM $clone -VMHost $ESXi -Datastore $ds -Location $Folder -OSCustomizationSpec $customspecification -RunAsync
}
Which satisfies the following criteria:
• Specify the number of VMs to be created.
• Specify a customization specification to be used.
• The VMs must be placed on all ESXi hosts in the vSphere cluster.
• Specify a vCenter Server Folder in the Virtual Machine & Templates view where the VMs will be placed.
• Specify VM datastore/datastore cluster placement.
• Specify the VM prefix.
• Use numeric incremental number after the VM prefix.
I modified it a bit to fit my needs, and this is the final version:
$vm_count = “99“
$clone = “TCLVM1“
$customspecification=”TinyCoreLinux-VM“
$ds = “Host1DS1-12.5GB-VMFS5.6-SSD-AGIL“
$Folder = “TCLVM-clones“
$Cluster = “X-Systems-DRS-HA-cluster“
$VM_prefix = “TCLVM-“
1..$vm_count | foreach {
$y=”{0:D2}” -f $_
$VM_name= $VM_prefix + $y
$ESXi=Get-Cluster $Cluster | Get-VMHost -state connected | Get-Random
write-host “Creation of VM $VM_name initiated” -foreground green
New-VM -Name $VM_Name -VM $clone -VMHost $ESXi -Datastore $ds -Location $Folder -OSCustomizationSpec $customspecification -RunAsync
}
The script creates N clones (where N is defined by the $vm_count variable) placing them inside the TCLVM-clones folder ($Folder) on the cluster X-Systems-DRS-HA-cluster ($Cluster) (comprised of 2 ESXi hosts virtualized with VMware Workstation).
Each time a clone gets created, the script selects a random host of the cluster.
The datastore is Host1DS1-12.5GB-VMFS5.6-SSD-AGIL, a local and virtual disk (vmdk) of ESXi host #1 which is located on my PC's 60GB partially damaged SSD (Windows won't boot off it, but I can still store files).
The host has 3 more disks, each with a corresponding datastore in vSphere and the other host of the cluster also has 4 disks.
I want Storage DRS to automatically move the clones around and to spread them on all disks.
I ran the script but I immediately got a bunch of errors:
It seems that vSphere is not able to create all the clones. Apparently the problem is that the datastore i defined in the script is not accessible from ESXi host #2.
Consequently few clones have been successfully created:
I was expecting this. Both ESX servers have only local disks, there is no shared storage. ESXi host #1 can't access the second host's disks and viceversa.
As I mentioned in my previous posts it makes little sense to have a vSphere environment with no shared storage, but I have a few ideas on how to rectify the situation.
For now, as a workaround, it was sufficient for me to rerun the script a few times, redefining the datastore from time to time in order to have some VMs on the second host as well.
Having done that, everything seemed OK from the command line. The clones were being successfully created.
However, taking a look with the vSphere web client, some clones were still not being created...
"Snapshot hierarchy is too deep." What a weird error! I selected my original VM and launched the snapshot manager:
Indeed as vSphere said there are numerous snapshots, one for each clone! vSphere makes a snapshot to create each clone, that's clear, but I don't understand why said snapshots don't get automatically deleted!
Anyway I decided to manually delete them, then I ran the script a few more times.
Slowly, the VMs were all starting to appear in the inventory..
Et voilà, 100 VMs!!!
Now on the VMs I placed a script that I wrote in Ash (a stripped-down bash shell). The script gets automatically executed when the machines boot and creates fake workloads for the CPU and memory (for now just the memory).
As you can see in the screenshot below, some VMs have a low memory usage, while other VMs have a medium memory usage:
I need to modify the script to have it consume up to 100% of RAM. It will be interesting to see machines that end up non responsive and HA has to intervene and restart them, while DRS attempts to dinamically balance the load across the cluster (I will set it on aggressive).
As you might have noticed not all VMs are running in the screenshots. At the moment vSphere complains that there are insufficient resources to power on all VMs.
I find this weird because I have more than enough physical resources, but I guess the next step is to figure out how to reconfigure the cluster in order to be able to power them all on.
As regards storage, I want, if possible, to dedicate the OCZ RevoDrive PCI-E SSD entirely to vSphere as a single LUN visible to all nodes in the cluster. At that point it should be interesting to see the graphs and stats of vCenter in terms of IOPS!