NFS over SoftRoCE setup

From Linux NFS

(Difference between revisions)
Jump to: navigation, search
Amschuma (Talk | contribs)
(Created page with "Soft ROCE is a software implementation of [https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet RoCE] that allows RDMA to be used on any Ethernet adapter. It is available ...")
Newer edit →

Revision as of 19:53, 26 November 2018

Soft ROCE is a software implementation of RoCE that allows RDMA to be used on any Ethernet adapter. It is available in Linux kernels after 4.8.

Contents

Client and Server Common Setup

  • Check for the rdma_rxe kernel module. If you don't have it, then enable the following Kconfig options and rebuild your kernel
CONFIG_INFINIBAND
CONFIG_INFINIBAND_RDMAVT
CONFIG_RDMA_RXE
  • Install the rdma-core package and use the rxe_cfg script to load the module and start an RDMA interface
anna@gouda ~ % sudo rxe_cfg status
  rdma_rxe module not loaded
  Name  Link  Driver      Speed  NMTU  IPv4_addr  RDEV  RMTU  
  eth0  yes   virtio_net
anna@gouda ~ % sudo rxe_cfg start
  Name  Link  Driver      Speed  NMTU  IPv4_addr  RDEV  RMTU  
  eth0  yes   virtio_net 
anna@gouda ~ % sudo rxe_cfg add eth0
anna@gouda ~ % sudo rxe_cfg status  
  Name  Link  Driver      Speed  NMTU  IPv4_addr  RDEV  RMTU          
  eth0  yes   virtio_net                          rxe0  1024  (3)

Ping Test

  • Start an rping server on one machine
anna@server ~ % sudo rping -s -v -C 3
 server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
 server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
 server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
 server DISCONNECT EVENT...
 wait for RDMA_READ_ADV state 10
  • Now check that the connection works from another machine
anna@client ~ % sudo rping -c -a 255.255.255.255 -v -C 3
  ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
  ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
  ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
  client DISCONNECT EVENT...

NFS Setup

Server

  • Set the insecure mount option in /etc/exports
  • Install the nfs-utils-rdma-server package and enable the nfs-utils-rdma-server.service service through systemd

Client

  • Install the nfs-utils-rdma-client package and enable the nfs-utils-rdma-client.service service through systemd
  • Mount the server to check that it worked
anna@client ~ % sudo mount -o rdma,port=20049,vers=4.1 192.168.100.215:/export
anna@client ~ % mount | grep proto=rdma
  192.168.100.215:/home/anna/tests/tests/anna/cthon on /nfs type nfs4 (rw,relatime,vers=4.1,rsize=524288,wsize=524288,proto=rdma,port=20049)

See Also

Personal tools