From Linux NFS

Soft ROCE is a software implementation of RoCE that allows RDMA to be used on any Ethernet adapter. It is available in Linux kernels after 4.8.

Contents 1 Client and Server Common Setup 1.1 Ping Test 2 NFS Setup 2.1 Server 2.2 Client 3 See Also

Client and Server Common Setup

• Check for the rdma_rxe kernel module. If you don't have it, then enable the following Kconfig options and rebuild your kernel

CONFIG_INFINIBAND
CONFIG_INFINIBAND_RDMAVT
CONFIG_RDMA_RXE

• Install the rdma-core package and use the rxe_cfg script to load the module and start an RDMA interface

anna@gouda ~ % sudo rxe_cfg status
  rdma_rxe module not loaded
  Name  Link  Driver      Speed  NMTU  IPv4_addr  RDEV  RMTU  
  eth0  yes   virtio_net
anna@gouda ~ % sudo rxe_cfg start
  Name  Link  Driver      Speed  NMTU  IPv4_addr  RDEV  RMTU  
  eth0  yes   virtio_net 
anna@gouda ~ % sudo rxe_cfg add eth0
anna@gouda ~ % sudo rxe_cfg status  
  Name  Link  Driver      Speed  NMTU  IPv4_addr  RDEV  RMTU          
  eth0  yes   virtio_net                          rxe0  1024  (3)

Ping Test

• Start an rping server on one machine

anna@server ~ % sudo rping -s -v -C 3
 server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
 server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
 server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
 server DISCONNECT EVENT...
 wait for RDMA_READ_ADV state 10

• Now check that the connection works from another machine

anna@client ~ % sudo rping -c -a 255.255.255.255 -v -C 3
  ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
  ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
  ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
  client DISCONNECT EVENT...

NFS Setup

Server

• Set the insecure mount option in /etc/exports
• Install the nfs-utils-rdma-server package and enable the nfs-utils-rdma-server.service service through systemd

Client

• Install the nfs-utils-rdma-client package and enable the nfs-utils-rdma-client.service service through systemd
• Mount the server to check that it worked

anna@client ~ % sudo mount -o rdma,port=20049,vers=4.1 192.168.100.215:/export
anna@client ~ % mount | grep proto=rdma
  192.168.100.215:/home/anna/tests/tests/anna/cthon on /nfs type nfs4 (rw,relatime,vers=4.1,rsize=524288,wsize=524288,proto=rdma,port=20049)

See Also

• Mellanox's softRoCE setup guide