NFS over SoftRoCE setup

From Linux NFS

(Difference between revisions)
Jump to: navigation, search
(Created page with "Soft ROCE is a software implementation of [https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet RoCE] that allows RDMA to be used on any Ethernet adapter. It is available ...")
 
(5 intermediate revisions not shown)
Line 1: Line 1:
-
Soft ROCE is a software implementation of [https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet RoCE] that allows RDMA to be used on any Ethernet adapter.  It is available in Linux kernels after 4.8.
+
Soft ROCE is a software implementation of [https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet RoCE] that allows RDMA to be used on any Ethernet adapter.  It is available in Linux kernels after 4.8.  The following instructions are suitable for most of the latest linux distributions.
= Client and Server Common Setup =
= Client and Server Common Setup =
Line 6: Line 6:
  CONFIG_INFINIBAND_RDMAVT
  CONFIG_INFINIBAND_RDMAVT
  CONFIG_RDMA_RXE
  CONFIG_RDMA_RXE
-
* Install the <code>rdma-core</code> package and use the <code>rxe_cfg</code> script to load the module and start an RDMA interface
+
* Install the <code>iproute2</code> package and use the <code>rdma link add</code> command to load the module and start an RDMA interface. Note that this does not survive reboots.
-
  anna@gouda ~ % sudo rxe_cfg status
+
* Comment: <code>rxe_cfg</code> has been deprecated from 01/2020, use <code>rdma</code> command instead.
-
  rdma_rxe module not loaded
+
  [foo@linux-bar ~]$ sudo rdma link add rxe0 type rxe netdev eth0
-
  Name  Link  Driver      Speed  NMTU  IPv4_addr  RDEV  RMTU 
+
  [foo@linux-bar ~]$ sudo rdma link
-
  eth0  yes  virtio_net
+
  link rxe0/1 state ACTIVE physical_state LINK_UP netdev eth0
-
anna@gouda ~ % sudo rxe_cfg start
+
-
  Name Link  Driver      Speed  NMTU  IPv4_addr  RDEV  RMTU 
+
-
  eth0  yes  virtio_net
+
-
anna@gouda ~ % sudo rxe_cfg add eth0
+
-
  anna@gouda ~ % sudo rxe_cfg status 
+
-
  Name Link  Driver      Speed  NMTU  IPv4_addr  RDEV  RMTU         
+
-
  eth0 yes  virtio_net                          rxe0  1024  (3)
+
== Ping Test ==
== Ping Test ==
* Start an <code>rping</code> server on one machine
* Start an <code>rping</code> server on one machine
-
  anna@server ~ % sudo rping -s -v -C 3
+
  [foo@linux-bar ~]$ sudo rping -s -v -C 3
-
  server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
+
server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
-
  server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
+
server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
-
  server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
+
server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
-
  server DISCONNECT EVENT...
+
server DISCONNECT EVENT...
-
  wait for RDMA_READ_ADV state 10
+
wait for RDMA_READ_ADV state 10
* Now check that the connection works from another machine
* Now check that the connection works from another machine
-
  anna@client ~ % sudo rping -c -a 255.255.255.255 -v -C 3
+
  [foo@linux-bor ~]$ sudo rping -c -a 192.168.122.40 -v -C 3
-
  ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
+
ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
-
  ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
+
ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
-
  ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
+
ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
-
  client DISCONNECT EVENT...
+
client DISCONNECT EVENT...
= NFS Setup =
= NFS Setup =
== Server ==
== Server ==
-
* Set the ''insecure'' mount option in <code>/etc/exports</code>
+
* Install the <code>nfs-utils</code> package and enable rdma from <code>/etc/nfs.conf</code> and restart service nfs-server
-
* Install the <code>nfs-utils-rdma-server</code> package and enable the <code>nfs-utils-rdma-server.service</code> service through systemd
+
[foo@linux-bar]$ sudo sed -i '/rdma/{s/^#//; s/rdma=n/rdma=y/}' /etc/nfs.conf
 +
[foo@linux-bar]$ grep -v ^# /etc/nfs.conf
 +
[general]
 +
[exportfs]
 +
[gssd]
 +
  use-gss-proxy=1
 +
[lockd]
 +
[mountd]
 +
[nfsdcld]
 +
[nfsdcltrack]
 +
[nfsd]
 +
  rdma=y
 +
  rdma-port=20049
 +
[statd]
 +
[sm-notify]
 +
[foo@linux-bar]$ sudo mkdir -p /expdir
 +
[foo@linux-bar]$ sudo bash -c 'echo "/expdir *(rw,no_root_squash)" >/etc/exports'
 +
[foo@linux-bar]$ sudo systemctl restart nfs-server
== Client ==
== Client ==
-
* Install the <code>nfs-utils-rdma-client</code> package and enable the <code>nfs-utils-rdma-client.service</code> service through systemd
+
* Install the <code>nfs-utils</code> package and do nfs mount
* Mount the server to check that it worked
* Mount the server to check that it worked
-
  anna@client ~ % sudo mount -o rdma,port=20049,vers=4.1 192.168.100.215:/export
+
  [foo@linux-bor]$ sudo mount -o rdma,port=20049,vers=4.2 192.168.122.40:/expdir /mnt/nfsmp
-
  anna@client ~ % mount | grep proto=rdma
+
  [foo@linux-bor]$ mount | grep proto=rdma
-
  192.168.100.215:/home/anna/tests/tests/anna/cthon on /nfs type nfs4 (rw,relatime,vers=4.1,rsize=524288,wsize=524288,proto=rdma,port=20049)
+
192.168.122.40:/expdir on /mnt/nfsmp type nfs4 (rw,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=192.168.122.161,local_lock=none,addr=192.168.122.40)
= See Also =
= See Also =
* [https://community.mellanox.com/docs/DOC-2184 Mellanox's softRoCE setup guide]
* [https://community.mellanox.com/docs/DOC-2184 Mellanox's softRoCE setup guide]
 +
* [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_infiniband_and_rdma_networks/configuring-roce_configuring-and-managing-networking#configuring-soft-roce_configuring-roce Red Hat's softRoCE setup guide]
 +
* [https://github.com/linux-rdma/rdma-core/blob/master/Documentation/rxe.md Rdma-core's documentation]
 +
* [https://man7.org/linux/man-pages/man8/rdma.8.html The rdma man page]
 +
* [https://www.systutorials.com/docs/linux/man/8-rxe_cfg/ The rxe_cfg man page]
 +
* [https://manpages.debian.org/testing/rdma-core/rxe_cfg.8.en.html rxe_cfg deprecated]

Latest revision as of 05:27, 11 August 2021

Soft ROCE is a software implementation of RoCE that allows RDMA to be used on any Ethernet adapter. It is available in Linux kernels after 4.8. The following instructions are suitable for most of the latest linux distributions.

Contents

Client and Server Common Setup

  • Check for the rdma_rxe kernel module. If you don't have it, then enable the following Kconfig options and rebuild your kernel 
CONFIG_INFINIBAND
CONFIG_INFINIBAND_RDMAVT
CONFIG_RDMA_RXE
  • Install the iproute2 package and use the rdma link add command to load the module and start an RDMA interface. Note that this does not survive reboots.
  • Comment: rxe_cfg has been deprecated from 01/2020, use rdma command instead. 
[foo@linux-bar ~]$ sudo rdma link add rxe0 type rxe netdev eth0
[foo@linux-bar ~]$ sudo rdma link
link rxe0/1 state ACTIVE physical_state LINK_UP netdev eth0

Ping Test

  • Start an rping server on one machine 
[foo@linux-bar ~]$ sudo rping -s -v -C 3
server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
server DISCONNECT EVENT...
wait for RDMA_READ_ADV state 10
  • Now check that the connection works from another machine 
[foo@linux-bor ~]$ sudo rping -c -a 192.168.122.40 -v -C 3
ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
client DISCONNECT EVENT...

NFS Setup

Server

  • Install the nfs-utils package and enable rdma from /etc/nfs.conf and restart service nfs-server 
[foo@linux-bar]$ sudo sed -i '/rdma/{s/^#//; s/rdma=n/rdma=y/}' /etc/nfs.conf
[foo@linux-bar]$ grep -v ^# /etc/nfs.conf
[general]
[exportfs]
[gssd]
 use-gss-proxy=1
[lockd]
[mountd]
[nfsdcld]
[nfsdcltrack]
[nfsd]
 rdma=y
 rdma-port=20049
[statd]
[sm-notify]
[foo@linux-bar]$ sudo mkdir -p /expdir
[foo@linux-bar]$ sudo bash -c 'echo "/expdir *(rw,no_root_squash)" >/etc/exports'
[foo@linux-bar]$ sudo systemctl restart nfs-server

Client

  • Install the nfs-utils package and do nfs mount
  • Mount the server to check that it worked 
[foo@linux-bor]$ sudo mount -o rdma,port=20049,vers=4.2 192.168.122.40:/expdir /mnt/nfsmp
[foo@linux-bor]$ mount | grep proto=rdma
192.168.122.40:/expdir on /mnt/nfsmp type nfs4 (rw,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=192.168.122.161,local_lock=none,addr=192.168.122.40)

See Also

Retrieved from "http://wiki.linux-nfs.org/wiki/index.php/NFS_over_SoftRoCE_setup"
Personal tools