NFS over SoftRoCE setup

From Linux NFS

(Difference between revisions)
Jump to: navigation, search
(Created page with "Soft ROCE is a software implementation of [https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet RoCE] that allows RDMA to be used on any Ethernet adapter. It is available ...")
 
Line 1: Line 1:
-
Soft ROCE is a software implementation of [https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet RoCE] that allows RDMA to be used on any Ethernet adapter.  It is available in Linux kernels after 4.8.
+
Soft ROCE is a software implementation of [https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet RoCE] that allows RDMA to be used on any Ethernet adapter.  It is available in Linux kernels after 4.8. The following instructions are based on [https://www.archlinux.org/ Arch Linux]
= Client and Server Common Setup =
= Client and Server Common Setup =
Line 6: Line 6:
  CONFIG_INFINIBAND_RDMAVT
  CONFIG_INFINIBAND_RDMAVT
  CONFIG_RDMA_RXE
  CONFIG_RDMA_RXE
-
* Install the <code>rdma-core</code> package and use the <code>rxe_cfg</code> script to load the module and start an RDMA interface
+
* Install the <code>rdma-core</code> package and use the <code>rxe_cfg</code> script to load the module and start an RDMA interface.  Note that this does not survive reboots.
  anna@gouda ~ % sudo rxe_cfg status
  anna@gouda ~ % sudo rxe_cfg status
   rdma_rxe module not loaded
   rdma_rxe module not loaded
Line 36: Line 36:
= NFS Setup =
= NFS Setup =
== Server ==
== Server ==
-
* Set the ''insecure'' mount option in <code>/etc/exports</code>
 
* Install the <code>nfs-utils-rdma-server</code> package and enable the <code>nfs-utils-rdma-server.service</code> service through systemd
* Install the <code>nfs-utils-rdma-server</code> package and enable the <code>nfs-utils-rdma-server.service</code> service through systemd
== Client ==
== Client ==
Line 47: Line 46:
= See Also =
= See Also =
* [https://community.mellanox.com/docs/DOC-2184 Mellanox's softRoCE setup guide]
* [https://community.mellanox.com/docs/DOC-2184 Mellanox's softRoCE setup guide]
 +
* [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-configuring_soft-_roce Red Hat's softRoCE setup guide]
 +
* [https://github.com/linux-rdma/rdma-core/blob/master/Documentation/rxe.md Rdma-core's documentation]
 +
* [https://www.systutorials.com/docs/linux/man/8-rxe_cfg/ The rxe_cfg man page]

Latest revision as of 20:55, 26 November 2018

Soft ROCE is a software implementation of RoCE that allows RDMA to be used on any Ethernet adapter. It is available in Linux kernels after 4.8. The following instructions are based on Arch Linux

Contents

Client and Server Common Setup

  • Check for the rdma_rxe kernel module. If you don't have it, then enable the following Kconfig options and rebuild your kernel
CONFIG_INFINIBAND
CONFIG_INFINIBAND_RDMAVT
CONFIG_RDMA_RXE
  • Install the rdma-core package and use the rxe_cfg script to load the module and start an RDMA interface. Note that this does not survive reboots.
anna@gouda ~ % sudo rxe_cfg status
  rdma_rxe module not loaded
  Name  Link  Driver      Speed  NMTU  IPv4_addr  RDEV  RMTU  
  eth0  yes   virtio_net
anna@gouda ~ % sudo rxe_cfg start
  Name  Link  Driver      Speed  NMTU  IPv4_addr  RDEV  RMTU  
  eth0  yes   virtio_net 
anna@gouda ~ % sudo rxe_cfg add eth0
anna@gouda ~ % sudo rxe_cfg status  
  Name  Link  Driver      Speed  NMTU  IPv4_addr  RDEV  RMTU          
  eth0  yes   virtio_net                          rxe0  1024  (3)

Ping Test

  • Start an rping server on one machine
anna@server ~ % sudo rping -s -v -C 3
 server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
 server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
 server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
 server DISCONNECT EVENT...
 wait for RDMA_READ_ADV state 10
  • Now check that the connection works from another machine
anna@client ~ % sudo rping -c -a 255.255.255.255 -v -C 3
  ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
  ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
  ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
  client DISCONNECT EVENT...

NFS Setup

Server

  • Install the nfs-utils-rdma-server package and enable the nfs-utils-rdma-server.service service through systemd

Client

  • Install the nfs-utils-rdma-client package and enable the nfs-utils-rdma-client.service service through systemd
  • Mount the server to check that it worked
anna@client ~ % sudo mount -o rdma,port=20049,vers=4.1 192.168.100.215:/export
anna@client ~ % mount | grep proto=rdma
  192.168.100.215:/home/anna/tests/tests/anna/cthon on /nfs type nfs4 (rw,relatime,vers=4.1,rsize=524288,wsize=524288,proto=rdma,port=20049)

See Also

Personal tools