ESXi on ARM Fling 2.0 - Challenges with the vSphere 8.0 update

TL;DR - if you're using an Orange Pi 5 Plus or Raspberry Pi 4/5, you might want to stick with 1.15/7.0.

Apologies for the lack of updates. I've run into more issues trying to get VCF running on the embedded AMD EPYC build. With ESXi on ARM Fling 2.0 released, the base hypervisor has been upgraded to ESXi 8.0 Update 3b. I haven't had much luck with it as the release does not include the USB NIC community driver that the 7.0/1.15 release had. This limits us to exactly one USB NIC, that will only work at 100Mb/s. The onboard network adapter for the Raspberry Pi 5 is a different model than the 4, and as such is not compatible with the uether driver.

Tested BOMs:

Orange Pi 5 Plus

• ADT-LINK Riser PCIe X4 3.0 PCI-E 4X to M.2 NGFF NVMe M Key 2280 Riser Card
• Dell Broadcom 5719
• Intel PRO 1000
• Intel i210-T1 (generic brand ordered off of Amazon)
• Intel i225-V
• UEFI version 0.11.2 commit d668228 from the edk2-rk3588 project

Raspberry Pi 5

• GeeekPi P02 PCIe Slot for Raspberry Pi 5
• Same NICs as above
• UEFI version 0.3 commit c1ca184 from the rpi5-uefi project

With the Orange Pi 5 Plus, the issue first appeared to be related to MSI-X. In previous blog posts, I had to disable MSI interrupts to get NVMe drives to work. The behavior witnessed with Arm Fling 2.0 is that if MSI is disabled with a NIC, it simply disabled the device. In UEFI release 0.9.1, there were some messages indicating that MSIX interrupts were still being allocated:

VMK_PCI: 599: 0000:01:00.0: allocated 3 MSIX interrupts

These messages were also accompanied by TX and RX hangs. While the link light remained up, I could not pull a DHCP address on any card that I tried. 

I'm not sure about everything that changed between 0.9.1 and 0.11.2, but when I attempted it on the latest version, there were no messages indicating MSIX interrupts, yet the behavior remained the same; link light, no DHCP, tx/rx hangs.

With the Broadcom 5719, the ntg3 module has an advanced parameter to enable legacy interrupts. I disabled MSI interrupts, enabled legacy with the driver, only for it to purple screen on boot. None of the Intel modules have this option. I've documented all of these issues on the Broadcom forum, in this topic.

With the Raspberry Pi 5, I ran into similar issues, with messaging indicating "Failed to allocate MSI interrupts". There has been significant development for Linux on RPi5, but the UEFI build has stagnated, as the primary contributors have shifted their focus to RK3588 based SBCs.

I believe the path forward will be with the Orange Pi 5 Plus. The only thing left to test is with an enterprise-grade i210 network adapter. I ordered a used Dell-based i210-T1 which should be here soon. This adapter has been proven to work with Ampere based servers, and the hope is that something with the device firmware will prevent the tx/rx hang issues. If it doesn't resolve the problem, I'll have to hurry up and wait on the USB NIC fling to be re-introduced to the ARM fling. For now, I'm going to stay on the 7.0 release.

Using Intel Optane for NVMe Tiering

 A series of unfortunate events occurred shortly after posting the previous blog post:

• DIMM H1 decided to fail
• Replacement was ordered
• Post office lost the replacement

The joys of homelabbing :) These things happen. While I wait for the replacement for the replacement, a new feature that was introduced in vSphere 8.0 Update 3 was brought to my attention. William Lam posted a blog covering NVMe tiering, currently in tech preview, that allows NVMe devices to act as RAM for inactive pages.

This does mean that I'll need to disrupt my current vSAN deployment. This can be utilized without vCenter. My plan is to utilize one of the NVMe devices to provide effectively triple the RAM that is currently in the system.

As it stands, I have 128GB installed, 112GB usable as H1 left the lobby. By following William Lam's blog, I configured the host with the following commands:

esxcli system settings kernel set -s MemoryTiering -v TRUE

esxcli system tierdevice create -d /vmfs/devices/disks/t10.NVMe____SSDPE21D280GAD_NVME_INTEL_280GB_________0001B33DC1E4D25C

esxcli system settings advanced set -o /Mem/TierNvmePct -i 200

Then rebooted the host. On boot, the increased RAM is realized:

This should provide enough breathing room to attempt a single node VCF deployment. This does change my original plan a little bit, but hopefully I can perform an embedded 5.2 install with supported principal storage. Stay tuned!

Revisiting the vExpert Intel Optane drives - single node vSAN ESA the "hard" way

Hello, internet! Long time no see, how you been? It's been a pretty interesting year so far, and the homelab has not been spared from the chaos; hardware failures and upgrades have caused my projects to come to a standstill. Fortunately, I've made headway by consolidating some of the hardware into a project I've been trying to get online for some time now.

I hit a stroke of luck by winning an AMD based Supermicro motherboard off of my favorite auction site, which came with a processor and some memory for about $300. The large number of PCI-e lanes opens up a number of expansion options in a standard mid-tower case. In this blog, I'm going to discuss consolidating all ten of the Intel Optane disks that I received last year into one compute node, and detail the process of getting the latest versions of ESXi and vCenter Server installed.

My BOM:

H11SSL-i - each PCI-e slot configured for x4x4 or x4x4x4x4 bifurcation

AMD EPYC 7551

128GB (8x16GB) 2133 DDR4 RAM

10x Intel Optane 280GB NVMe SSDs (vSAN pool)

5x 10Gtek PCI-e x8 to 2x U.2 NVMe adapters

Solidigm P41 Plus 2TB M.2 NVMe SSD (boot disk)

Corsair RM1000x PSU

Silverstone CS380 8 bay mid-tower case

Noctua NH-U9 TR4-SP3 heatsink

This system will consume the same Optane drives that I used in my Supermicro BigTwin SuperServer, which comprised of two X11DPT-B boards, each containing 2x Xeon Platinum 8160's and 768GB of RAM. The previous vSAN ESA build gave 5 of the Optane drives to each node, a vSAN Witness VM on a third node, and 100Gbe direct connect to share bandwidth. 

Consolidating down to the tower will be considerably quieter and draw less power, while allowing us to benchmark all ten drives without networking overhead. Downsides include less processing power, far less memory, and a little more work to do under the hood to get it working. Unlike the two node cluster, this will have no redundancy.

I'm going to detail how to accomplish all of this without a vSphere license of any kind; this will utilize the 60 day trial license, and a copy of ESXi that was acquired through supported means. Some of the old tricks of standing up a vSAN node still work with ESA, and can be deployed without vCenter.

As most of my blog posts go, this is strictly for lab use - I would not suggest running a single node vSAN cluster in production, nor would I suggest running a vSAN cluster without a proper vCenter server. We will install vCenter in a later blog post.

The first step is to download the ESXi-Customizer-PS script. This can be found here: https://github.com/VFrontDe-Org/ESXi-Customizer-PS/tree/master

PowerCLI is required to use the script. Full documentation on the script can be found here: https://www.v-front.de/p/esxi-customizer-ps.html

Simply running the script without any options will seek out the VMware online depot and create an ISO based on the latest patch version. As of this writing, I can confirm that the script works to download ESXi 8.0 Update 3 build 24022510.

Install the OS to the boot disk, then reboot.

Once booted, clear any partitions that may be on the Optane disks.

Prior to creating the vSAN cluster, we'll want to get a list of the disks that we want to use in the cluster. For my use case, I was able to run the command "esxcli storage core device list | grep t10" to list out all NVMe drives. I removed my 2TB boot disk from that output.

Since we're using a single node vSAN cluster, we can create a vSwitch with no uplinks for the purpose of vSAN networking:

• Create vSwitch
• Add vmkernel port
• From an SSH session, mark the vmkernel port for vSAN traffic: esxcli vsan network ip add -i vmk1

Previously, we would use the command "esxcli vsan cluster new" to create a vSAN OSA cluster. With 8.0, we have more options:

[root@localhost:~] esxcli vsan cluster new --help

Usage: esxcli vsan cluster new [cmd options]

Description:

  new                   Create a vSAN cluster with current host joined. A random

                        sub-cluster UUID will be generated.

Cmd options:

  -c|--client-mode      vSAN client mode allows mount of vSAN datastore from the

                        server cluster without enabling vSAN.

  -s|--storage-mode     vSAN storage mode allows to create a vSAN Max cluster.

  -x|--vsanesa          vSAN ESA mode allows to create a vSAN ESA cluster.

By using the -x option, we can create the ESA cluster: esxcli vsan cluster new -x

We can then add disks to the storage pool with: "esxcli vsan storagepool add" then specify the disks that we want to use by adding the -d option to each device we listed previously. 

For example, the command should read: "esxcli vsan storagepool add -d t10.longnumbergoeshere1 -d t10.longnumbergoeshere2", repeating for each device you wish to add.

This will take a few minutes, but once it is complete, we should have a vSAN datastore:

We won't be able to use this datastore quite yet; a little more housekeeping is in order. Because the default storage policy can only be changed by vCenter, we need to update the policy to tolerate 0 failures. To do so, run the following commands:

esxcli vsan policy setdefault -c cluster -p "((\"hostFailures

ToTolerate\" i0) (\"forceProvisioning\" i1))"

esxcli vsan policy setdefault -c vdisk -p "((\"hostFailuresTo

Tolerate\" i0) (\"forceProvisioning\" i1))"

esxcli vsan policy setdefault -c vmnamespace -p "((\"hostFail

uresToTolerate\" i0) (\"forceProvisioning\" i1))"

esxcli vsan policy setdefault -c vmswap -p "((\"hostFailuresT

oTolerate\" i0) (\"forceProvisioning\" i1))"

esxcli vsan policy setdefault -c vmem -p "((\"hostFailuresToT

olerate\" i0) (\"forceProvisioning\" i1))"

Once these commands have been run, we should be able to create virtual machines on the vSAN datastore. We are now ready for a vCenter install and HCIBench testing. This also primes a VMware Cloud Foundation single node lab deployment. For now, we'll call this entry done and cover HCIBench in the next one.

Evacuate ESXi host without DRS

One of the biggest draws to vSphere Enterprise Plus licensing is the Distributed Resource Scheduler feature. DRS allows for recommendations and automated actions to help balance virtual machine workloads across hosts, as well as affinity rules to keep VMs on or off of specific hosts. 

One of the more common functions is the ability to automatically migrate virtual machines off hosts when they are placed in maintenance mode to perform firmware or hardware upgrades. I set out to create a script that would do this for me on a vSphere Standard license. That script can be found here: https://github.com/ThisGuyFuchs/Evacuate-ESXi-Host-without-DRS

The script is pretty straightforward:

# Connect to vCenter Server
Connect-VIServer -Server "Your-vCenter-Server" -User Your-Username -Password Your-Password

Replace "Your-vCenter-Server" with the IP address or FQDN of your vCenter, as well as the administrator account (mine for example is administrator@vsphere.local) and the password for that account. You can remove -Password if you want it to prompt for it instead.

# Specify the ESXi host to evacuate
$esxiHost = "ESXi-Host-Name"

Replace "ESXi-Host-Name" with the IP address or the FQDN of the host you wish to evacuate.

From there, the script will generate a list of VMs, regardless of power state, and then migrate those VMs to any powered on host in the cluster. Once the script finishes, you are free to put the host into maintenance mode manually, or you can add this step to the script with:

# Put the ESXi host into maintenance mode
Set-VMHost -VMHost $esxiHost -State Maintenance -Confirm:$false

Keep in mind that this will migrate ALL virtual machines, whether they are powered on or off. While this isn't a true replacement to DRS, I find this useful to facilitate firmware updates and add hardware to hosts when needed. Hopefully this can provide additional value to vSphere Standard license holders.

ESXi on Arm: NVMe working on Orange Pi 5 Plus

Recently, a few awesome things have happened. First, the VMware Flings page returned after a short absence. Then, the ESXi on Arm team released an update that brings NVMe support to the Raspberry Pi CM4. This update got me thinking about the issues I had trying to get NVMe working on the Orange Pi 5 Plus. I checked on the edk2-rk3588 project to see if there were any updates, and there were a few that addressed PCI-e. So I flashed my eMMC module to 0.9.1, updated ESXi on Arm to 1.15 and... no dice.

It was at this point that I decided to start actually reading about the problem. Turns out, the issue is occurring because of how the RK3588 chip handles MSI. Erratum 3588001 goes into detail, but the point is that for the edk2-rk3588 project, it was easier to disable MSI than it was to try to fix it. This is what causes the problem that I was experiencing; ESXi will load modules, but will hang at a certain point and fail to boot.

After a bit more research, I found that there's a rather easy work around: kernel options! William Lam has an awesome list of advanced kernel options and lo and behold, there is an option to disable MSI. 

DISCLAIMER: I have no idea what the implications of disabling MSI are in a VMware environment. I offer no warranty. Any change to advanced settings in ESXi has a non-zero chance of wreaking havoc. I would not put any data I care about on the device we're about to configure!

With that out of the way...

I rebooted the Orange Pi and hit shift+O during boot to type:

disableMSI=TRUE

Then hit enter, and let ESXi continue to boot. It was able to get through module load without hanging, and was rewarded with a storage device in the vSphere client:

Created a datastore, made a VM and installed Ubuntu all with no consequence. Great! So now that it works, all we need to do is make the change persistent. To do so, let's check the status of the disableMSI kernel option:

[root@localhost:~] esxcli system settings kernel list -o "disableMSI"
Name        Type  Configured  Runtime  Default  Description
----------  ----  ----------  -------  -------  -----------
disableMSI  Bool  FALSE       TRUE     FALSE    Disable use of MSI/MSI-X

This is what we'd expect to see; runtime is TRUE, but configured is FALSE. Let's fix that with:

esxcli system settings kernel set -s "disableMSI" -v "TRUE"

And then double check with the list command from before:

[root@localhost:~] esxcli system settings kernel list -o "disableMSI"
Name        Type  Configured  Runtime  Default  Description
----------  ----  ----------  -------  -------  -----------
disableMSI  Bool  TRUE        TRUE     FALSE    Disable use of MSI/MSI-X

Now, the kernel option we set should persist through reboots. Good luck and happy homelabbing!

ESXi on Lenovo ThinkCentre M75q Gen 2 part 2 - Adding supported network cards

To follow up on my previous blog post, I've made a few changes to the M75q Gen 2. It can run vSphere 7.0 and 8.0, but is unable to use the onboard NIC as it is a Realtek 8168, which doesn't have a supported driver. With the recent removal of the VMware Flings webpage (archive still exists), the move to supported network adapters is becoming more prudent. While 10Gbe networking is nice, I'd like to expand to perhaps a quad port 2.5Gbe card if possible. If unable to do so, I could perhaps utilize the onboard A+E M.2 slot for a supported gigabit card.

I've optimized the 3D print to allow for a more open air approach to the previous "box" design. This should allow for optimal airflow, at the expense of being able to stack boxes on top of each other. 

Tested and working

As previously mentioned, using the M.2 to PCI-e adapter enables the use of a PCI-e 3.0 x4 slot, with some limitations. The AQC107 based SYBA network card works without issue, establishing a full 10Gbe connection. This card is natively supported as of ESXi 7.0 Update 2.

Tested, limited or uncertain capability

I tested the zimaboard 4x 2.5Gbe i225 network card on the same slot. I was only able to get three of the four ports to work. The fourth port showed link lights even when a cable wasn't connected. I suspected that the issue was with the card on it's own, as the power consumption should be similar to the 10Gbe card. Testing the card in a proper PCI-e slot yielded the same result. DOA parts happen, but I haven't bothered purchasing another to test. If I work up the nerve to try again, I'll order another and post an update, or will try a different brand altogether.

I then tested a Lenovo OEM Intel X550-T2. This is a PCI-e 3.0 x4 card with dual 10Gbe ports. It isn't detected during POST, but works in other systems, which leads me to draw two conclusions:

1. The power draw of the card is too much, which could be worked around with a different adapter that allows more power (such as a M.2 to Oculink converter with external PSU).
2. The card has OEM pins that prevent it from being detected during POST, which could be worked around with a tape mod. I'm less inclined to believe this is the case, as it's a Lenovo card going into a Lenovo system, and works in non-OEM machines seamlessly.

As of right now, the card is being used in another system. I might test using the Oculink converter at a later time. The benefit of the Oculink card would mean more compatibility with GPUs, as well as PCI-e 4.0 support for Ryzen 5000 series processors. The downside is that I'd need an external PSU for that purpose.

Tested, not working

The last card I tested was a sketchy looking M.2 A+E to gigabit network adapter. This was a best effort attempt as I have several machines that could use this card. Unfortunately, the M75q Gen 2 did not detect this card at all. If you're going to test this in a compact system, I would suggest getting a A+E extension cable, as the card is wider than most WiFi adapters and may not fit. If/when the Orange Pi 5 Plus supports PCI-e for the ARM fling, I plan on testing this adapter with the extension cable for that purpose.

Overall, I like the Ryzen based, 8 core, 64GB of RAM system for ESXi 6.7 as I can still use the community Realtek driver for it. Nothing in my homelab is essential, but if you wish to learn the capabilities of the latest versions of ESXi, I would suggest against trying to hack a machine to do so. The USB network card fling can be utilized for gigabit to 2.5Gbe connectivity with some success, but for general reliability, stick to Intel based onboard networking with real PCI-e slots or Thunderbolt capabilities. 

Three changes you should be prepared for when upgrading from 6.x

This post outlines some of the impactful changes that come with new versions of vSphere. Sometimes, these changes aren't evident until they're pushed to prod, other times, they creep up and cause problems. As updates are released, I'll publish more topics. For now, these are the main things you should be aware of when upgrading from an ESXi 6.x environment.

The SD card thing

Starting with vSphere 7.0, SD cards and USB boot devices are no longer supported. These devices cannot handle the new partitioning scheme. Unlike 6.x and previous versions, the boot device is now used for logs, which can cause issues when said devices aren't rated for frequent writes. 

The solution is to use a write-intensive local boot disk, or boot from SAN.

VMware KB: https://kb.vmware.com/s/article/85685 

vVol minimum size change

In vSphere 8.0 and below, the minimum size for Virtual Volumes is 4GB. Starting with 8.0 Update 1, the size has been updated to 255GB. Any vVols that existed prior to upgrade may disappear, or other errors may occur if you try to make a vVol smaller than 255GB.

The solution is to increase vVols to 255GB or larger, and create new vVols at the same or larger capacity. A workaround exists, which allows the smaller vVols to be used, by using the following command:

esxcli system settings advanced set -o /VVOL/vvolUseVMFS6AndLargeConfigVVols -i 0

VMware KB: https://kb.vmware.com/s/article/90791

CPU Support

An unpleasant surprise awaited me in upgrading to vSphere 8.0 Update 2. The workhorse servers in my homelab are all first generation Xeon Scalable (Skylake), and have been put on notice that they may not be supported in future releases of vSphere. Granted, the processors are 5 years old, but Naples-based AMD EPYC are 6 years and do not have this warning, and Broadwell processors (E5-26xx v4) are still supported as well. 

The solution is to append a boot option when loading ESXi, although this is a band-aid that probably shouldn't be used in production as it may cause instability. Using SHIFT+O prior to module load, use the following: allowLegacyCPU=true

VMware KB: https://kb.vmware.com/s/article/82794