Setting up DPDK with a Mellanox ConnectX-3 Pro 10/40Gbps NIC

RDMAI want to set up some point to point links using DPDK to give the advantages of kernel bypass and had lots of “fun” setting things up for this. Probably it’s because I am incapable of following instructions properly that I need to document the process properly so I don’t waste so much time next time. One major issue I discovered (at the time of writing) is that Ubuntu 16.04 isn’t yet supported – 15.10 is the highest version that seems to be supported by the Mellanox OFED. It might be possible to get it working with 16.04 but it seemed easier just to go back to 15.10. Instructions after the jump…

First thing is to install the Mellanox OFED from here. The simplest thing is to select the .iso file for download once the appropriate OS and version has been selected. The user manual explains how to install everything from the .iso but I have repeated it here just to save time. The .iso just needs to be mounted using:

cd ~/Downloads
sudo mount -o ro,loop MLNX_OFED_LINUX-3.2- /mnt

This is for the current version – change as required. Then run the install:

sudo /mnt/mlnxofedinstall
sudo umount /mnt

That does pretty much everything to install the card drivers.

Next step is to set up huge pages. I just used 2M pages but 1G is also an option for 64 bit processors. Full instructions are here. Edit /etc/default/grub so that the kernel config line looks like:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash hugepages=1024"

This will generate 1024 2M huge pages. To use this configuration, run:

sudo update-grub

To make the memory available for DPDK, it must be mounted. Enter:

sudo mkdir /mnt/huge

Then edit /etc/fstab and add the line:

nodev /mnt/huge hugetlbfs defaults 0 0

Restart the machine to kick in the new configuration.

DPDK can be downloaded from here. The download can be expanded to somewhere convenient. Before compilation, support for the Mellanox ConnectX-3 Pro cards must be enabled. Full details are here but, to get started, edit <dpdkdir>/config/common_base so that the relevant line looks like this:


Then DPDK can be compiled with:

cd <dpdkdir>
make -j8 install T=x86_64-native-linuxapp-gcc

The “-j8” part should be changed to reflect how many cores are to be used for compilation. I get an error complaining about the lack of DESTDIR but it doesn’t seem to matter. A parameter DESTDIR=<something> can be added if required.

To test the build, one or more of the examples can be tried. I used l2fwd which can be found at <dpdkdir>/examples/l2fwd. To try this out, enter:

cd <dpdkdir>/examples/l2fwd
export RTE_SDK=<dpdkdir>
cd build/app
sudo ./l2fwd -c f -- -p f

This should find the card and start forwarding frames between the ports on the card (does rather assume a dual port card!). My card was connected to another card that was sending pings and I could see these being received and sent out of the other port.

One thought on “Setting up DPDK with a Mellanox ConnectX-3 Pro 10/40Gbps NIC”

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.