My current setup: vSAN OSA 2 node cluster

I'm excited to announce that I was selected for the vExpert Intel Optane giveaway. To prepare for the incoming drives, I've started a series of blog posts focusing on benchmarking my two-node vSAN cluster. This will serve as a baseline of what to expect in terms of differences between using best practices for OSA as well as the fundamental performance differences of ESA.

My BOM consists of:

Supermicro BigTwin 6029BT-DNC0R

2x X11DPT-B compute servers (OEM branded), each containing:

• 2x Xeon Silver 4116
• 768GB DDR4 (24x32GB)
• Intel X550 RJ-45 10Gbe dual port SIOM network card
• RSC riser to break up x16 slot into 2 x8
• Intel 82599 SFP+ 10Gbe dual-port network card in slot 1 of the riser
• 10Gtek PCI-e x8 to 2x U.2 NVMe drive adapter in slot 2 of the riser
• Open PCI-e x16 low-profile slot
• Backplane supports 2x U.2 NVMe with 4 SAS/SATA HDD/SSD per server

Each node contains a single NVMe cache disk along with two NVMe drives for capacity. The drive list is as follows:

• Node 1: Intel DC P3500 1.2TB for cache, 2x SanDisk Skyhawk 3.84TB for capacity
• Node 2: SanDisk Skyhawk 1.92TB for cache, 2x SanDisk Skyhawk 3.84TB for capacity

The cache disks are not recommended for two reasons: They are mismatched in size, and they are not write-intensive disks. Nevertheless, they are what I have on hand and should be enough to establish a baseline of vSAN OSA performance. 

The first port of the X550 will be used for management and VM traffic (green lines), and the second will be used for vSAN witness traffic (blue lines). I didn't have a switch capable of handling VLANs, so this will run over two "dumb" 8-port switches. The servers will be direct-connected over the 82599 network cards to pass vSAN storage traffic (red line).

Once we have benchmarked the original build, the plan is to swap the cache disks with Optane drives for OSA, then use them all for ESA. With the 10Gtek cards, each server can hold a maximum of 6 NVMe disks. vSAN OSA prefers to have a 1:1 cache-to-capacity disk ratio, so we will test with 2x Optane and 2x capacity disks, followed by 5x Optane for ESA. To make room for the second NVMe adapter, I'm going to use a ConnectX-4 100Gbe adapter in the open x16 slot for testing, while it is overkill, I don't want there to be a bottleneck (and the original build won't likely saturate the 10Gbe link currently in place). The latest HCIBench utility as of this writing is 2.8.1 and will be utilized in "easy run" mode. Stay tuned!