Vmware Direct Path IO

Introduction 

Direct I/O is available from vsphere 4 and later versions that leverage Intel  VT-D and AMD V CPU hardware feature . With Direct I/O VM can directly access physical network card by passing  Virtualized NIC ( E1000 , Valance ) and para virtualized NIC ( Vm Next ) . With Direct I/O VM can sustain high bandwidth beyond 10 GIG additional to saving CPU cycles . VMware recommends to use only when VM has high IO load and Saving CPU benefit the overall infra.

Para virtualized NICs can provide the throughput 9 GIG +, But the vsphere handles all the network related tasks  like physical NIC interrupts, processes packets, determines the recipient of the packet and copies them into the destination VM, if needed. The vSphere host also mediates packet transmissions over the physical NIC.This will consume lot of CPU.

Direct IO bypass the virtualized network layer while saving the CPU cycle , but this feature trades the virtualization features like physical NIC sharing , Vmotion and Network IO control . The VM need to have memory reserved to avoid the  Memory swap while physical NIC is processing.

vSphere Features 

Features that are not available for the VMs which had Direct IO control .

• Hot adding and removing of virtual devices
• Suspend and resume
• Record and replay
• Fault tolerance
• High availability
• DRS (limited availability. The virtual machine can be part of a cluster, but cannot migrate across hosts)
• Snapshots

Cisco UCS  and Direct IO 

Following features are available while using Direct IO with Cisco UCS 

• Vmotion 
• Hot Adding and removing of virtual hardware
• Suspend and resume
• High availability
• DRS
• Snapshots

Configure Passthrough Devices on a Host 

vsphere Client 

1. Select a host from the inventory panel of the vSphere Client.
2. On the Configuration tab, click Advanced Settings.
3. The Passthrough Configuration page appears, listing all available passthrough devices. A green icon indicates that a device is enabled and active. An orange icon indicates that the state of the device has changed and the host must be rebooted before the device can be used.
4. Click Edit.Select the devices to be used for passthrough and click OK

Web Client 

1. Browse to a host in the vSphere Web Client navigator.
2. Click the Manage tab, click Settings.
3. In the Hardware section, click PCI Devices.
4. To add a PCI device to the host, click Edit.

Key Points : 

1. An adapter can only be used by a single virtual machine when using DirectPath I/O.
2. Only two devices can be passed through to a virtual machine.
3. Virtual machine hardware version 7 must be used.
4. Host requires a restart once the device has been added for passthrough.
5. Check the VMware HCL (Hardware Compatibility Guide) to make sure the device is supported.
6. It is typically used on virtual machines that have very high I/O requirements such as database servers that need direct access to a storage HBA (host bus adapter).
7. It relies on Intel VT-d (Virtual Technology for Directed I/O) or AMD IOMMU (IO Memory Management Unit), although the latter is experimental. Remember to enable this option in the BIOS!