ESXi on Lenovo ThinkCentre M75q Gen 2 part 2 - Adding supported network cards

To follow up on my previous blog post, I've made a few changes to the M75q Gen 2. It can run vSphere 7.0 and 8.0, but is unable to use the onboard NIC as it is a Realtek 8168, which doesn't have a supported driver. With the recent removal of the VMware Flings webpage (archive still exists), the move to supported network adapters is becoming more prudent. While 10Gbe networking is nice, I'd like to expand to perhaps a quad port 2.5Gbe card if possible. If unable to do so, I could perhaps utilize the onboard A+E M.2 slot for a supported gigabit card.

I've optimized the 3D print to allow for a more open air approach to the previous "box" design. This should allow for optimal airflow, at the expense of being able to stack boxes on top of each other. 

Tested and working

As previously mentioned, using the M.2 to PCI-e adapter enables the use of a PCI-e 3.0 x4 slot, with some limitations. The AQC107 based SYBA network card works without issue, establishing a full 10Gbe connection. This card is natively supported as of ESXi 7.0 Update 2.

Tested, limited or uncertain capability

I tested the zimaboard 4x 2.5Gbe i225 network card on the same slot. I was only able to get three of the four ports to work. The fourth port showed link lights even when a cable wasn't connected. I suspected that the issue was with the card on it's own, as the power consumption should be similar to the 10Gbe card. Testing the card in a proper PCI-e slot yielded the same result. DOA parts happen, but I haven't bothered purchasing another to test. If I work up the nerve to try again, I'll order another and post an update, or will try a different brand altogether.

I then tested a Lenovo OEM Intel X550-T2. This is a PCI-e 3.0 x4 card with dual 10Gbe ports. It isn't detected during POST, but works in other systems, which leads me to draw two conclusions:

1. The power draw of the card is too much, which could be worked around with a different adapter that allows more power (such as a M.2 to Oculink converter with external PSU).
2. The card has OEM pins that prevent it from being detected during POST, which could be worked around with a tape mod. I'm less inclined to believe this is the case, as it's a Lenovo card going into a Lenovo system, and works in non-OEM machines seamlessly.

As of right now, the card is being used in another system. I might test using the Oculink converter at a later time. The benefit of the Oculink card would mean more compatibility with GPUs, as well as PCI-e 4.0 support for Ryzen 5000 series processors. The downside is that I'd need an external PSU for that purpose.

Tested, not working

The last card I tested was a sketchy looking M.2 A+E to gigabit network adapter. This was a best effort attempt as I have several machines that could use this card. Unfortunately, the M75q Gen 2 did not detect this card at all. If you're going to test this in a compact system, I would suggest getting a A+E extension cable, as the card is wider than most WiFi adapters and may not fit. If/when the Orange Pi 5 Plus supports PCI-e for the ARM fling, I plan on testing this adapter with the extension cable for that purpose.

Overall, I like the Ryzen based, 8 core, 64GB of RAM system for ESXi 6.7 as I can still use the community Realtek driver for it. Nothing in my homelab is essential, but if you wish to learn the capabilities of the latest versions of ESXi, I would suggest against trying to hack a machine to do so. The USB network card fling can be utilized for gigabit to 2.5Gbe connectivity with some success, but for general reliability, stick to Intel based onboard networking with real PCI-e slots or Thunderbolt capabilities. 

Three changes you should be prepared for when upgrading from 6.x

This post outlines some of the impactful changes that come with new versions of vSphere. Sometimes, these changes aren't evident until they're pushed to prod, other times, they creep up and cause problems. As updates are released, I'll publish more topics. For now, these are the main things you should be aware of when upgrading from an ESXi 6.x environment.

The SD card thing

Starting with vSphere 7.0, SD cards and USB boot devices are no longer supported. These devices cannot handle the new partitioning scheme. Unlike 6.x and previous versions, the boot device is now used for logs, which can cause issues when said devices aren't rated for frequent writes. 

The solution is to use a write-intensive local boot disk, or boot from SAN.

VMware KB: https://kb.vmware.com/s/article/85685 

vVol minimum size change

In vSphere 8.0 and below, the minimum size for Virtual Volumes is 4GB. Starting with 8.0 Update 1, the size has been updated to 255GB. Any vVols that existed prior to upgrade may disappear, or other errors may occur if you try to make a vVol smaller than 255GB.

The solution is to increase vVols to 255GB or larger, and create new vVols at the same or larger capacity. A workaround exists, which allows the smaller vVols to be used, by using the following command:

esxcli system settings advanced set -o /VVOL/vvolUseVMFS6AndLargeConfigVVols -i 0

VMware KB: https://kb.vmware.com/s/article/90791

CPU Support

An unpleasant surprise awaited me in upgrading to vSphere 8.0 Update 2. The workhorse servers in my homelab are all first generation Xeon Scalable (Skylake), and have been put on notice that they may not be supported in future releases of vSphere. Granted, the processors are 5 years old, but Naples-based AMD EPYC are 6 years and do not have this warning, and Broadwell processors (E5-26xx v4) are still supported as well. 

The solution is to append a boot option when loading ESXi, although this is a band-aid that probably shouldn't be used in production as it may cause instability. Using SHIFT+O prior to module load, use the following: allowLegacyCPU=true

VMware KB: https://kb.vmware.com/s/article/82794

Orange Pi 5 Plus ESXi on ARM Fling 1.14 update

 With VMware Fling 1.14 and the latest commits of the edk2-3588 firmware, the Orange Pi 5 Plus can now utilize some USB NICs, and make use of all USB ports within an ESXi environment.

I can hot add USB NICs as well. It seems to run into the same problems that I had on the ThinkCentre M75q Gen 2 and USB NICs, in that the vmnic needs to be assigned to the management vSwitch on boot. The requirement to comment the line out of the BIOS doesn't seem necessary, either; you can do so by entering system setup and disabling PCI-e 3.0 in the menu. My BOM now consists of:

Orange Pi 5 Plus

3D printed case

Noctua NF-A4x20 PWM

USB fan adapter (USB 2.0 port)

UGREEN USB hub + network adapter Model 60544 (Managment network + additional USB ports) 

Cable Matters USB NIC Model 202013 (dedicated VM traffic)

Sabrent USB to SATA adapter Model EC-SSHD

Crucial MX500 1TB SATA SSD

This leaves the option of adding additional USB 3.0 devices if needed, thanks to the UGREEN hub. The onboard Realtek 8125's are not of much use as there is no driver available, but getting dual gigabit NICs makes this much more useful to run some basic web servers. 

I was able to port my Jira instance off of my Raspberry Pi 4 onto a Ubuntu 22.04 VM successfully, which opens the door to back up not only at the application level, but full VM backup as well. The only remaining caveat is that the cores run at their minimum speed (800 Mhz), but that bodes well for me given that I want to keep power consumption as low as possible. It'll be worthwhile to have a virtual platform where the power sipping Orange Pi 5 Plus can remain on, without having to power on a rack to deliver some basic web applications. Other ideas would be to install Tautulli for my Plex server, or Wiki.js for lab documentation. 

Booting the Orange Pi 5 Plus into ESXi on ARM

Disclaimer: Orange Pi products are not officially supported for ESXi on ARM. Even if you do get it to work, functionality will be limited at the time of this writing. I would not recommend purchasing an Orange Pi for the expressed purpose of ESXi on ARM. The folks working on the edk2-3588 UEFI project are doing amazing work, and I can't wait to see what comes next.

With that out of the way...

Booting and installing the ESXi on ARM Fling is possible on the Orange Pi 5 Plus. There are some pretty sizable caveats in doing so:

• On-board NICs are Realtek 8125, which do not have a compatible driver
• The only usable network adapter is the Realtek 8153 USB, which is capped at 100Mbps
• M.2 NVMe slot is not supported - having a device in the M.2 slot will cause drivers to hang when trying to boot
• Modification and creation of the edk2-rk3588 UEFI firmware project is necessary
• Hardware BOM needs to be used in specific ports in order to work
• Once installed, ESXi only recognizes the lone USB-C port, a hub is necessary
• As of this writing, the USB NIC Fling driver does not work with the ESXi on ARM Fling (no Fling-ception allowed)

This blog post will serve as a guide to get this installed, the bulk of which is modifying the firmware to work properly. My BOM is as follows:

• Orange Pi 5 Plus with 3D printed case and 40x40mm fan
• 16GB USB 3.0 thumb drive for ESXi installer (connected to the top USB 3.0 port)
• USB-C to USB A adapter
• Cable Matters 4 port USB hub, connected to the USB-C to USB A adapter 
• Cable Matters USB Network Adapter model 202013 (connected to hub)
• USB keyboard (connected to hub)
• USB 3.0 to SATA adapter with 1TB SSD as an install target/ESXi boot (connected to hub)
• 8GB Micro SD Card for edk2-3588 UEFI firmware (can use eMMC if your model comes with it, this guide will cover SD card)

The first step is to build out the UEFI firmware. The folks working on this project have been hard at work, and have successfully built firmware that allows things like the WoR Project (Windows on ARM) to gain some functionality. The base release works pretty well for Windows, but for ESXi, we'll have to build a custom version of it.

To do so, we'll need to use a Linux environment. I chose Ubuntu 23.04. The install instructions on the github page linked above include almost all of the packages needed. For my version of Ubuntu, I installed the following:

sudo apt install git gcc g++ build-essential gcc-aarch64-linux-gnu iasl python3-pyelftools uuid-dev device-tree-compiler

Then, clone the repository and change directory:

git clone https://github.com/edk2-porting/edk2-rk35xx.git --recursive

cd /edk2-rk35xx

Prior to building the firmware, we need to modify one of the files to disable PCI-e 3.0. If this step is not followed, the installation media will purple screen during boot. Use vim to modify this file:

sudo vi /edk2-rockchip/Platform/OrangePi/OrangePi5Plus/AcpiTables/AcpiTables.inf

Once you're in the file, comment the following line:

$(RK_COMMON_ACPI_DIR)/Pcie2x1l0.asl

Confirm that the line looks like this, then write changes:

# $(RK_COMMON_ACPI_DIR)/Pcie2x1l0.asl

Now we can build it (change release number to match the latest release found on the github page. As of this writing, 0.7.1 is the latest:

sudo ./build.sh --device orangepi-5plus --release 0.7.1

After a few minutes, the script should finish, and create a file named "RK3588_NOR_FLASH.img" in the edk2-rk35xx directory. You can then either use dd to copy the .img file onto an SD card, or SCP the file to a Windows machine and use the Raspberry Pi imager tool to write the file instead.

With the freshly imaged SD card inserted into the Orange Pi, we can now plug everything else into the machine. Remember, the device order is as follows:

• USB drive containing ESXi installer in the top USB 3.0 port
• USB hub plugged into the USB-C port
• Keyboard, SSD and Realtek 8153 adapter plugged into the hub

If all goes well, ESXi should boot without issue, and will see the network adapter and SSD. Install ESXi as you normally would. I made the mistake of thinking that I could put the USB SSD on the 3.0 ports after installing - this will work, meaning ESXi will still boot, but any additional capacity used for the datastore will be inaccessible, and ESXi will be read only. You must keep the SSD and network adapter on the hub. 

The promise of an 8 core, 16GB RAM, ARM based ESXi machine looks great. I'm hoping that in the not too distant future, we can see more improvements, such as additional USB port and NVMe disk utilization. A workaround for the lack of networking would be to pass another USB network adapter to a VM, which could act as a DHCP server and present gigabit connectivity for other guests, but this would only work if the other USB ports were recognized. 

Time will tell what the future holds for the Orange Pi 5 Plus, but to have UEFI functioning at this level is fantastic, and I can't say enough good things about the people working on this project. Also, a special thanks to the ESXi on Arm team, who have made this endeavor possible.

HCIBench analysis part 2: vSAN OSA vs. vSAN ESA with all Optane drives

 In my previous post, I compared what a huge difference having proper caching drives can make with vSAN OSA, replacing my read intensive disks with Intel Optane. One question remains, however: since Optane are high performance, mixed use drives, how would they get along with a vSAN ESA deployment? To find out, I reconfigured my vSAN cluster by deleting the witness VM, deploying a vSAN ESA witness, and loaded each server with 5 Intel Optane disks each. Let's see how it got along.

Caveats: 

• ESA has compression only mode on automatically, and cannot be disabled (OSA was tested without dedupe or compression enabled)
• ESA best practices call for a minimum of three nodes. It can absolutely work with two nodes but would see true performance benefits in a right sized cluster

100% read, 4K random

Interesting result, as the vSAN OSA with 2x Optane for caching and 2 read intensive capacity disks actually managed 24K higher IOPS on this test. There are a few likely reasons that I think this would happen:

1. The benchmark likely kept everything in the hot tier throughout the benchmark
2. As mentioned previously, ESA is running compression while OSA is not. I plan on re-running this benchmark specifically with compression only enabled on the OSA configuration.
3. ESA works best at the recommended configuration (3+ nodes).

70% read, 4K random

Where Optane improved write performance in the OSA model, ESA with 5 Optane disks per node improved further. We see an improvement of 53K IOPS, as well as higher throughput, and generally better latency across the board.

50% read/write, 8K random

We get a decent bump in performance in comparison to the OSA build. Read latency remains low, write latency remains about the same.

100% write, 256KB sequential

This is perhaps the biggest difference between all the tests, and highlights the key advantage of vSAN ESA especially for a 2 node cluster. Where some of the other tests showed percentage bumps in performance, ESA with write intensive disks managed to get over twice the throughput of vSAN OSA. 5.69GB/s represents 45.52 Gb/s over the network cards. Latency also improved dramatically.

Overall, the 2 node vSAN ESA with 5 Intel Optane disk per server configuration performs generally as expected, comparatively outperforming the OSA with capacity disks. While OSA went blow for blow with ESA, it should be reiterated that deduplication and compression were disabled; IOPS performance tends to be lower in favor of capacity. One thing that I've omitted from these is the usable capacity, of which I would defer to the vSAN calculators to get final numbers, but keep in mind: vSAN OSA accomplished it's numbers with a datastore size of ~13TB, whereas the 280GB Intel Optane disks granted a capacity just north of 2TB.

So what can we take away from this? Is OSA dead? Not by a long shot. In a real world, 2-node ROBO scenario, several questions should be asked:

• What is the workload?
• How much performance do you need?
• Are mixed use drives going to be able to meet capacity demands?
• What's the best bang for the buck hardware to meet workload requirements?

In a workload that favors capacity and read intensive workloads, I would favor vSAN OSA. For performance, ESA should absolutely be a consideration. And if you have the chance to get more nodes, the capacity and performance improvements tend to favor ESA.

HCIBench analysis part 1: OSA vs. OSA with Intel Optane

In my previous post, I shared my current vSAN setup details. In this post, we'll take a look at the performance of the original vSAN deployment, and see how much of a performance difference Optane can make when we replace sub-optimal cache disks with Optane for OSA, followed by a full Optane and ESA redeployment.

As previously stated, the cache disks that I have in my current vSAN OSA 2 node cluster are not optimized for caching as they are read intensive disks, and they are mismatched capacity. When running vSAN in a production environment, it is best to adhere to the vSAN HCL as well as the appropriate disks for cache and capacity. While my 3.84TB read intensive drives will be great for capacity, I should be using a write intensive cache disk. Fortunately, the Intel Optane 905's that were sent to me through the vExpert program and Intel should do the trick.

For benchmarking comparisons, I'm using HCIBench 2.8.1. This free utility is provided by VMware as a "Fling". It is an OVA template that, once deployed, allows us to automatically create Linux based VMs that run preconfigured Flexible I/O (FIO) benchmarks. Throughout the benchmarking process, I have selected "Easy Run", which will automatically deploy the number of VMs it feels is correct based on my hardware configuration.

The first thing I did was select the default benchmarks and deploy each on the original vSAN OSA configuration. The "Easy Run" mode determined it would be best to deploy 4 VMs, as I suspect the cache wasn't up to par. 

Of note, all OSA benchmarks were run with deduplication and compression disabled.  

Here's how it went:

100% read, 4KB random

This is decent, but expected considering all of the disks are read intensive. 

70% read, 4KB random

This is more of a realistic bench, with some write performance metrics. Read latency is pretty high.

50% read/write, 8KB random

This simulates database workloads with an equal mix of reads and writes. With the larger block size, performance improves a bit, but latency remains a concern.

100% write, 256KB sequential

The throughput bench, best for video and media. We come close to saturating the 10Gbe link between the two servers on this one. 

For the next tests, I'll remove the 1.2TB and 1.92TB cache disks and replace them with two of the Intel Optane 905's. These are 280GB each, and have drastically better read and write performance capabilities. I could, in theory, use four cache disks, but vSAN prefers a 1:1 cache to capacity disk ratio. This was also the point where I swapped out the 10Gbe 82599 network card and replaced it with a ConnectX-4 CX455 100Gbe card to ensure we don't run into a bandwidth bottleneck (although I doubt I'll be able to saturate 100Gb). We should see a measurable difference across the board in terms of vSAN OSA performance. 

Of note, when selecting "Easy Run" on the following tests, HCIBench deployed 8 benchmark VMs.

100% read, 4KB random

Here we can see an increase of 200K IOPS. Despite the read intensive nature of the original disks, the Optane disks are rated higher for reads and substantially higher for writes. I'm not sure why the results failed to capture the 95th percentile read latency, but I would expect it to be similar to the original OSA result

70% read, 4KB random

Unsurprisingly, the write performance improved dramatically, as well as read latency. 188K IOPS at 70/30 for a 2 node cluster is impressive.

50% read/write, 8KB random

Once again, we see the difference in performance thanks to the Optane drives. 172K IOPS and much lower latency.

100% write, 256KB sequential

This was surprising for me. Just by replacing the cache disks, we can see a near 3.5x bandwidth increase. We see that it would've more than saturated the 10Gbe link, so switching to the 100Gbe card gave us a better idea of what to expect. Average write latency also improved considerably!

Overall, we can see a clear advantage in using the Intel Optane disks. It is critical to choose a cache disk that excels in write intensive workloads, as "hot tier" data gets pushed to capacity over time. By contrast, ESA excels with uniform, mixed use disks. We've seen that Optane has impressive write capabilities, but it also has great read performance. Optane can do it all, but what will the HCIBench numbers reflect? Stay tuned for my next blog post, where we'll compare vSAN OSA with 4 Optane cache/4 capacity disks to the ESA with 10 Optane disks test. 

My current setup: vSAN OSA 2 node cluster

I'm excited to announce that I was selected for the vExpert Intel Optane giveaway. To prepare for the incoming drives, I've started a series of blog posts focusing on benchmarking my two-node vSAN cluster. This will serve as a baseline of what to expect in terms of differences between using best practices for OSA as well as the fundamental performance differences of ESA.

My BOM consists of:

Supermicro BigTwin 6029BT-DNC0R

2x X11DPT-B compute servers (OEM branded), each containing:

• 2x Xeon Silver 4116
• 768GB DDR4 (24x32GB)
• Intel X550 RJ-45 10Gbe dual port SIOM network card
• RSC riser to break up x16 slot into 2 x8
• Intel 82599 SFP+ 10Gbe dual-port network card in slot 1 of the riser
• 10Gtek PCI-e x8 to 2x U.2 NVMe drive adapter in slot 2 of the riser
• Open PCI-e x16 low-profile slot
• Backplane supports 2x U.2 NVMe with 4 SAS/SATA HDD/SSD per server

Each node contains a single NVMe cache disk along with two NVMe drives for capacity. The drive list is as follows:

• Node 1: Intel DC P3500 1.2TB for cache, 2x SanDisk Skyhawk 3.84TB for capacity
• Node 2: SanDisk Skyhawk 1.92TB for cache, 2x SanDisk Skyhawk 3.84TB for capacity

The cache disks are not recommended for two reasons: They are mismatched in size, and they are not write-intensive disks. Nevertheless, they are what I have on hand and should be enough to establish a baseline of vSAN OSA performance. 

The first port of the X550 will be used for management and VM traffic (green lines), and the second will be used for vSAN witness traffic (blue lines). I didn't have a switch capable of handling VLANs, so this will run over two "dumb" 8-port switches. The servers will be direct-connected over the 82599 network cards to pass vSAN storage traffic (red line).

Once we have benchmarked the original build, the plan is to swap the cache disks with Optane drives for OSA, then use them all for ESA. With the 10Gtek cards, each server can hold a maximum of 6 NVMe disks. vSAN OSA prefers to have a 1:1 cache-to-capacity disk ratio, so we will test with 2x Optane and 2x capacity disks, followed by 5x Optane for ESA. To make room for the second NVMe adapter, I'm going to use a ConnectX-4 100Gbe adapter in the open x16 slot for testing, while it is overkill, I don't want there to be a bottleneck (and the original build won't likely saturate the 10Gbe link currently in place). The latest HCIBench utility as of this writing is 2.8.1 and will be utilized in "easy run" mode. Stay tuned!