With VMware Fling 1.14 and the latest commits of the edk2-3588 firmware, the Orange Pi 5 Plus can now utilize some USB NICs, and make use of all USB ports within an ESXi environment.
vShoestring - Homelabbing on a budget
I am the warranty.
Tuesday, September 5, 2023
Orange Pi 5 Plus ESXi on ARM Fling 1.14 update
Thursday, July 20, 2023
Booting the Orange Pi 5 Plus into ESXi on ARM
- On-board NICs are Realtek 8125, which do not have a compatible driver
- The only usable network adapter is the Realtek 8153 USB, which is capped at 100Mbps
- M.2 NVMe slot is not supported - having a device in the M.2 slot will cause drivers to hang when trying to boot
- Modification and creation of the edk2-rk3588 UEFI firmware project is necessary
- Hardware BOM needs to be used in specific ports in order to work
- Once installed, ESXi only recognizes the lone USB-C port, a hub is necessary
- As of this writing, the USB NIC Fling driver does not work with the ESXi on ARM Fling (no Fling-ception allowed)
- Orange Pi 5 Plus with 3D printed case and 40x40mm fan
- 16GB USB 3.0 thumb drive for ESXi installer (connected to the top USB 3.0 port)
- USB-C to USB A adapter
- Cable Matters 4 port USB hub, connected to the USB-C to USB A adapter
- Cable Matters USB Network Adapter model 202013 (connected to hub)
- USB keyboard (connected to hub)
- USB 3.0 to SATA adapter with 1TB SSD as an install target/ESXi boot (connected to hub)
- 8GB Micro SD Card for edk2-3588 UEFI firmware (can use eMMC if your model comes with it, this guide will cover SD card)
sudo apt install git gcc g++ build-essential gcc-aarch64-linux-gnu iasl python3-pyelftools uuid-dev device-tree-compiler
git clone https://github.com/edk2-porting/edk2-rk35xx.git --recursive
cd /edk2-rk35xx
sudo vi /edk2-rockchip/Platform/OrangePi/OrangePi5Plus/AcpiTables/AcpiTables.inf
$(RK_COMMON_ACPI_DIR)/Pcie2x1l0.asl
# $(RK_COMMON_ACPI_DIR)/Pcie2x1l0.asl
sudo ./build.sh --device orangepi-5plus --release 0.7.1
- USB drive containing ESXi installer in the top USB 3.0 port
- USB hub plugged into the USB-C port
- Keyboard, SSD and Realtek 8153 adapter plugged into the hub
Thursday, May 11, 2023
HCIBench analysis part 2: vSAN OSA vs. vSAN ESA with all Optane drives
In my previous post, I compared what a huge difference having proper caching drives can make with vSAN OSA, replacing my read intensive disks with Intel Optane. One question remains, however: since Optane are high performance, mixed use drives, how would they get along with a vSAN ESA deployment? To find out, I reconfigured my vSAN cluster by deleting the witness VM, deploying a vSAN ESA witness, and loaded each server with 5 Intel Optane disks each. Let's see how it got along.
Caveats:
- ESA has compression only mode on automatically, and cannot be disabled (OSA was tested without dedupe or compression enabled)
- ESA best practices call for a minimum of three nodes. It can absolutely work with two nodes but would see true performance benefits in a right sized cluster
100% read, 4K random
- The benchmark likely kept everything in the hot tier throughout the benchmark
- As mentioned previously, ESA is running compression while OSA is not. I plan on re-running this benchmark specifically with compression only enabled on the OSA configuration.
- ESA works best at the recommended configuration (3+ nodes).
- What is the workload?
- How much performance do you need?
- Are mixed use drives going to be able to meet capacity demands?
- What's the best bang for the buck hardware to meet workload requirements?
Monday, May 8, 2023
HCIBench analysis part 1: OSA vs. OSA with Intel Optane
In my previous post, I shared my current vSAN setup details. In this post, we'll take a look at the performance of the original vSAN deployment, and see how much of a performance difference Optane can make when we replace sub-optimal cache disks with Optane for OSA, followed by a full Optane and ESA redeployment.
As previously stated, the cache disks that I have in my current vSAN OSA 2 node cluster are not optimized for caching as they are read intensive disks, and they are mismatched capacity. When running vSAN in a production environment, it is best to adhere to the vSAN HCL as well as the appropriate disks for cache and capacity. While my 3.84TB read intensive drives will be great for capacity, I should be using a write intensive cache disk. Fortunately, the Intel Optane 905's that were sent to me through the vExpert program and Intel should do the trick.
For benchmarking comparisons, I'm using HCIBench 2.8.1. This free utility is provided by VMware as a "Fling". It is an OVA template that, once deployed, allows us to automatically create Linux based VMs that run preconfigured Flexible I/O (FIO) benchmarks. Throughout the benchmarking process, I have selected "Easy Run", which will automatically deploy the number of VMs it feels is correct based on my hardware configuration.
The first thing I did was select the default benchmarks and deploy each on the original vSAN OSA configuration. The "Easy Run" mode determined it would be best to deploy 4 VMs, as I suspect the cache wasn't up to par.
Of note, all OSA benchmarks were run with deduplication and compression disabled.
Here's how it went:
100% read, 4KB random
This is decent, but expected considering all of the disks are read intensive.
70% read, 4KB random
This is more of a realistic bench, with some write performance metrics. Read latency is pretty high.
50% read/write, 8KB random
This simulates database workloads with an equal mix of reads and writes. With the larger block size, performance improves a bit, but latency remains a concern.
100% write, 256KB sequential
The throughput bench, best for video and media. We come close to saturating the 10Gbe link between the two servers on this one.
For the next tests, I'll remove the 1.2TB and 1.92TB cache disks and replace them with two of the Intel Optane 905's. These are 280GB each, and have drastically better read and write performance capabilities. I could, in theory, use four cache disks, but vSAN prefers a 1:1 cache to capacity disk ratio. This was also the point where I swapped out the 10Gbe 82599 network card and replaced it with a ConnectX-4 CX455 100Gbe card to ensure we don't run into a bandwidth bottleneck (although I doubt I'll be able to saturate 100Gb). We should see a measurable difference across the board in terms of vSAN OSA performance.
Of note, when selecting "Easy Run" on the following tests, HCIBench deployed 8 benchmark VMs.
100% read, 4KB random
Here we can see an increase of 200K IOPS. Despite the read intensive nature of the original disks, the Optane disks are rated higher for reads and substantially higher for writes. I'm not sure why the results failed to capture the 95th percentile read latency, but I would expect it to be similar to the original OSA result
70% read, 4KB random
Unsurprisingly, the write performance improved dramatically, as well as read latency. 188K IOPS at 70/30 for a 2 node cluster is impressive.
50% read/write, 8KB random
Once again, we see the difference in performance thanks to the Optane drives. 172K IOPS and much lower latency.
100% write, 256KB sequential
This was surprising for me. Just by replacing the cache disks, we can see a near 3.5x bandwidth increase. We see that it would've more than saturated the 10Gbe link, so switching to the 100Gbe card gave us a better idea of what to expect. Average write latency also improved considerably!
Overall, we can see a clear advantage in using the Intel Optane disks. It is critical to choose a cache disk that excels in write intensive workloads, as "hot tier" data gets pushed to capacity over time. By contrast, ESA excels with uniform, mixed use disks. We've seen that Optane has impressive write capabilities, but it also has great read performance. Optane can do it all, but what will the HCIBench numbers reflect? Stay tuned for my next blog post, where we'll compare vSAN OSA with 4 Optane cache/4 capacity disks to the ESA with 10 Optane disks test.
Tuesday, May 2, 2023
My current setup: vSAN OSA 2 node cluster
- 2x Xeon Silver 4116
- 768GB DDR4 (24x32GB)
- Intel X550 RJ-45 10Gbe dual port SIOM network card
- RSC riser to break up x16 slot into 2 x8
- Intel 82599 SFP+ 10Gbe dual-port network card in slot 1 of the riser
- 10Gtek PCI-e x8 to 2x U.2 NVMe drive adapter in slot 2 of the riser
- Open PCI-e x16 low-profile slot
- Backplane supports 2x U.2 NVMe with 4 SAS/SATA HDD/SSD per server
- Node 1: Intel DC P3500 1.2TB for cache, 2x SanDisk Skyhawk 3.84TB for capacity
- Node 2: SanDisk Skyhawk 1.92TB for cache, 2x SanDisk Skyhawk 3.84TB for capacity
Monday, March 20, 2023
Adding 10Gbe networking to the Lenovo ThinkCentre M75q Gen 2
This post has been a process, but finally, I'm happy to report that 10Gbe works on the M75q Gen 2 and also with ESXi 7.0. There were several challenges that needed to be addressed:
- No PCI-e slot
- No expansion chassis
- Only 1x SATA and 1x NVMe
- Realtek NIC onboard
- An NVMe to PCI-e x4 adapter
- A USB to SATA converter to provide power
- A supported 10Gbe adapter
- A 3D printed case extender (optional)
Tuesday, February 21, 2023
VMware Cloud Professional certification - thoughts and tips to pass
I recently passed the VCP-VMC 2023 exam, which was made possible by the free VMware course that allowed me to check the prerequisite for the certification. For those looking to take on the exam, I'll include what I can remember in terms of general concepts.
For starters, and as per usual with any VMware exam, start with the exam guide.
Like anything to do with cloud, it is network heavy. I'm not a networking engineer but have a long-since lapsed CCENT certification. This exam is going to grill you on CIDR, subnets, and network overlaps, and assumes that you have a general knowledge of the OSI model. Focus as well on the different connection and VPN types for each cloud provider. While it primarily focuses on AWS, GCP and Azure questions were in there as well, so be sure to know what each provider supports for configuration minimums and maximums regarding management networking configuration.
Speaking of minimums and maximums, you'll want to read up on cluster sizes and hardware configurations. What are the specs of an i3.metal instance vs. an i3en.metal? What kind of nodes can you get with Azure and GCP? And how many can you throw into a cluster? All of these may appear on the exam.
Managed services, such as VMware Cloud on Dell EMC and AWS Outposts should be studied as well. What physical requirements do these carry? What are their responsibilities?
HCX... woo boy. Several of these questions showed up, and did nothing but generate anxiety. Get to know HCX. Get to know the deployment models, and read up on how to troubleshoot different scenarios.
Containers, Kubernetes and Tanzu all showed up on my exam. Know what TKG does, how to deploy it, what value Kubernetes brings to containers in general, and what Tanzu services do what function.
That's all I can remember as of this moment. I'm still kind of pumped from getting through it. The only feedback that I have is that I don't know if some of my answers were right as they may have changed after the exam was written. For instance, Google has updated their networking requirements as of November 2022, so I'm not sure if I got the question wrong by answering based on current requirements, or if I should've answered based on previous specs. Perhaps a higher-level question that isn't dependent on something that can change with relative frequency would be better.
I hope you found this helpful. Feel free to comment below or ping me on Twitter if you have any questions!
Orange Pi 5 Plus ESXi on ARM Fling 1.14 update
With VMware Fling 1.14 and the latest commits of the edk2-3588 firmware, the Orange Pi 5 Plus can now utilize some USB NICs, and make use o...