Skip to main content
K80 GPU

Setup GPU Passthrough in OpenStack (Liberty) with Tesla K80

As part of the work we are doing with the OpenStack Community / Scientific Working Group (https://wiki.openstack.org/wiki/Scientific_working_group), the vScaler team are investigating the implementation of GPUs within an OpenStack environment for the purpose of HPC computing.

 

This blog post will be added to added to as we progress our investigation. For now it covers the basic setup and configuration of GPUs (Tesla K80s) within an OpenStack environment (Liberty/Centos 7)

 

Contents

1 Enable Memory Management
2 Get the GPU IDs for the Nova configuration
3 Configure the Controller(s) to be GPU aware
4 Setup the GPU Flavor in OpenStack
5 Spin up an Instance with your new GPU flavor
6 Next Steps

 

Enable Memory Management

To ensure the devices perform in a virtualised / passthrough environment we need to enable IOMMU within the GPU server. IOMMU (I/O Memory Management Unit) is a feature supported by motherboard chipsets that provides enhanced virtual-to-physical memory mapping capabilities, including the ability to map large portions of non-contiguous memory. IOMMU can be enabled in the motherboard's BIOS, please refer to your server provider for instructions on how to ensure this is set. Once this is applied you can pass a boot parameter (intel_iommu=on) to the kernel at boot time to ensure its enabled and turned on within the OS.

 

# Ensure Grub is configured
[root@gpu ~]# grep intel_iommu=on /boot/grub2/grub.cfg | head -n 1
linux16 /vmlinuz-3.10.0-327.18.2.el7.x86_64 root=UUID=33011dab-c75a-45d0-b7a2-ae23545c850f ro quiet rdblacklist=nouveau intel_iommu=on

 

Then verify IOMMU is enabled when the system is booted up in dmesg

 

[root@gpu ~]# dmesg | grep -iE "dmar|iommu" | grep -i enabled 
[ 0.000000] Intel-IOMMU: enabled 

 

Get the GPU IDs for the Nova configuration

The first thing we will need to capture is the vendor ID and device ID from the host system of the GPUs we want to pass through. In the example below we are using 1x NVIDIA Tesla K80 and 1x NVIDIA GRID K2

 

# check for the nvidia GPUs 
[root@gpu ~]# lspci | grep NVIDIA 
04:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1) 
05:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1) 
83:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K2] (rev a1) 
84:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K2] (rev a1) 
# get their IDs with the -n flag 
[root@gpu ~]# lspci -nn | grep NVIDIA 
04:00.0 3D controller [0302]: NVIDIA Corporation GK210GL [Tesla K80] [10de:102d] (rev a1) 05:00.0 3D controller [0302]: NVIDIA Corporation GK210GL [Tesla K80] [10de:102d] (rev a1) 83:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104GL [GRID K2] [10de:11bf] (rev a1) 
84:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104GL [GRID K2] [10de:11bf] (rev a1) 

 

Based on the above we now have:

  1. The vendor id is: 10de (Which corresponds to NVIDIA)
  2. And the product id is: 102d and 11bf (Where 102d is the K80 and 11bf is the GRID K2)

 

Configure the GPU Nova Service

The next step is to configure the GPU server Nova Service with the appropriate passthrough flags:

 

# file: /etc/nova/nova.conf / On the GPU server 
[DEFAULT] 
... 
pci_passthrough_whitelist={"vendor_id":"10de","product_id":"102d"}

 

Restart the nova service

 

[root@gpu ~]# systemctl restart openstack-nova-compute 
[root@gpu ~]#

 

Configure the Controller(s) to be GPU aware

We then need to add the PCI configuration and resource scheduling parameters to the nova.conf on the controller(s)

 

# In the file: /etc/nova/nova.conf on the controller
[DEFAULT]
... 
pci_alias={"name":"K80_Tesla","vendor_id":"10de","product_id":"102d"} 
scheduler_available_filters=nova.scheduler.filters.all_filters 
scheduler_available_filters=nova.scheduler.filters.pci_passthrough_filter.PciPassthroughFilter 
scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,PciPassthroughFilter 
scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler scheduler_max_attempts=5 

 

Restart all the nova services on the controller(s) to enable the configuration.

Setup the GPU Flavor in OpenStack

 

# Replace with whatever CPU/RAM requirements are appropriate for your environment
openstack flavor create --public --ram 2048 --disk 20 --vcpus 2 m1.large.2xK80 

# Add a passthrough property for the flavor 
openstack flavor set m1.large.2xK80 --property pci_passthrough:alias='K80_Tesla:2' 

# Then once created you should see something like this: 
openstack flavor show m1.large.1xK80   

Field Value
OS-FLV-DISABLED:disabled False
OS-FLV-EXT-DATA:ephemeral 0
disk 20
id 481b0dd4-1148-4714-8f0b-fceb8aed884f
name m1.large.2xK80
os-flavor-access:is_public True
properties pci_passthrough:alias='K80_Tesla:2'
ram 2048
rxtx_factor 1.0
swap  
vcpus 2

Spin up an Instance with your new GPU flavor

Spin up an instance with CLI/GUI and ssh in, you'll hopefully see

 

# yum install pciutils if a barebone centos 7 
[root@gpupass1 ~]# lspci 
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01) 
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03) 
00:02.0 VGA compatible controller: Cirrus Logic GD 5446 
00:03.0 Ethernet controller: Red Hat, Inc Virtio network device 
00:04.0 SCSI storage controller: Red Hat, Inc Virtio block device 
00:05.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1) # <----- BOOM! 
00:06.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon 

 

Next Steps

* Lots still to do
** Performance Analysis
** GPU to GPU performance within a VM
** GPU to GPU performance across nodes (SR-IOV on Mellanox Fabric)
** P100 cards to be added to lab environment shortly.
** Mail david.power@vscaler.com for anything else you'd like to see!