This post covers how to enable your Dell server hardware for Single Root I/O Virtualization (SR-IOV) capability, and how to configure SR-IOV for a VMware Virtual Machine.
The main reason for this post is that I was looking into SR-IOV after seeing how it was showing as Disabled when I was Adding a vSphere infrastructure to VMware Telco Cloud Automation.
SR-IOV is a common feature required in the Telco space, specifically for use with Virtualised Network Functions (VNFs). An SR-IOV capable NIC is required in order to provide VNFs with direct access to the physical network card, bypassing the network functionality in the hypervisor.
As per Dell KB000144029: “The Single Root I/O Virtualization (SR-IOV) specification details how a single PCIe device can be shared between various guest operating systems. Devices capable of SR-IOV functionality support multiple virtual functions on top of the physical function. The virtual function is enabled in the hardware as a light-weight PCIe function. The virtual function of an SR-IOV capable card can be directly assigned to the guest operating system. A virtual function driver running in the guest operating system manages this device.“
Configuring SR-IOV in vSphere means configuring a passthrough device (VMDirectPath I/O) for a Virtual Machine, which in turn means a trade-off in other vSphere features for that VM. More here.
This post contains the following sections:
- Check Hardware Compatibility
- Configure server device hardware for SR-IOV
- Configure vSphere ESXi hosts for SR-IOV
- Configure virtual machine for SR-IOV
Check hardware compatibility
Check your Dell server to confirm that it is SR-IOV capable, by looking in the iDRAC, under Hardware > Network Devices. The system in use here, is a 13G Dell PowerEdge R730, using an Intel 10G 2P X520 Adapter for the SR-IOV functionality.
Also confirm VMware NIC support for SR-IOV @ VMware Compatibility Matrix
I used’ x520‘ as a Keyword to filter the search results a bit quicker!
Configure server device hardware for SR-IOV
Reboot the host, and on boot hit F2 to enter System Setup.
As shown in the slideshow below:
- Select Device Settings and from there select the physical NIC to be configured for SR-IOV, which, in this example, is in Slot 5 – Intel 10G 2P X520 Adapter.
- Select Device Level Configuration
- Set Virtualization Mode to SR-IOV
- This will apply the setting to both ports on this 2-port NIC
- The change need to be saved, and will be implemented on next reboot of host
Configure vSphere ESXi hosts for SR-IOV
The next layer to configure for SR-IOV is the VMware layer, so via the vSphere client, we select and configure each ESXi host as follows:
- Navigate to Configure > Networking > Physical Adapters and select the appropriate vmnic, which in this example if vmnic4 (one of a two-port 10Gb NIC). As per the adapter details summary, and the SDR-IOV Status column, SR-IOV is currently Disabled.
- Click the Edit button for the vmnic
- Set the Status of SR-IOV to Enabled
- For this example we are just leaving the Number of virtual functions at the default value of ‘1’
- We can repeat this for the second vmnic (if also required), so that both vmnics will be SR-IOV enabled.
Configure virtual machine for SR-IOV
The remaining step is to provide a virtual machine with an SR-IOV capable pass-through device. For a RHEL 6+ or Windows VM, the following steps can be followed:
- Add a new Network Adapter
- Configure the relevant PortGroup
- Change Adapter Type to SR-IOV Passthrough
- Confirm the correct vmnic for the Physical Function
- Reserve all guest memory
Note also the warnings around features that are not supported or available when SR-IOV is in use (for example, vMotion, HA, FT, DRS, DPM, Snapshots, and more). Please refer to VMware’s SR-IOV Support documentation for more details.
In terms of functional performance of SR-IOV, Faisal Khan over on TelcoCloudBridge has a great breakdown of how SR-IOV works in comparison to DPDK (Data Plane Development Kit).
And that’s it for this post, that’s how to enable SR-IOV on a Dell PowerEdge server for VMware vSphere, hope that helps!
Resources
- Configure a Virtual Machine to use SR-IOV
- DPDK vs SR-IOV for NFV? – Why a wrong decision can impact performance!
- Dell Solutions leveraging SR-IOV:
- Building an open vRAN ecosystem @ https://infohub.delltechnologies.com/section-assets/ra-altiostar-rh-nec-dell-technologies with Altiostar, RedHat, NEC
- MEC (Multi Access Edge Computing) v1.1 Ref Arch @ https://infohub.delltechnologies.com/t/reference-architecture-guide-dell-technologies-multi-access-edge-computing-mec-1-1/
- Intel/Affirmed/Dell Whitepaper @ https://infohub.delltechnologies.com/t/dell-technologies-5g-core-solution-with-affirmed-and-red-hat-openshift-container-platform-ref-arch-guide/
Scamallach 
Pingback: Adding a vSphere infrastructure to VMware Telco Cloud Automation | Scamallach