NFS over SoftRoCE setup
From Linux NFS
(Difference between revisions)
Jianhong Yin (Talk | contribs) (→Client and Server Common Setup) |
Jianhong Yin (Talk | contribs) |
||
| (3 intermediate revisions not shown) | |||
| Line 1: | Line 1: | ||
| - | Soft ROCE is a software implementation of [https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet RoCE] that allows RDMA to be used on any Ethernet adapter. It is available in Linux kernels after 4.8. The following instructions are | + | Soft ROCE is a software implementation of [https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet RoCE] that allows RDMA to be used on any Ethernet adapter. It is available in Linux kernels after 4.8. The following instructions are suitable for most of the latest linux distributions. |
= Client and Server Common Setup = | = Client and Server Common Setup = | ||
| Line 29: | Line 29: | ||
= NFS Setup = | = NFS Setup = | ||
== Server == | == Server == | ||
| - | * Install the <code>nfs-utils | + | * Install the <code>nfs-utils</code> package and enable rdma from <code>/etc/nfs.conf</code> and restart service nfs-server |
| + | [foo@linux-bar]$ sudo sed -i '/rdma/{s/^#//; s/rdma=n/rdma=y/}' /etc/nfs.conf | ||
| + | [foo@linux-bar]$ grep -v ^# /etc/nfs.conf | ||
| + | [general] | ||
| + | [exportfs] | ||
| + | [gssd] | ||
| + | use-gss-proxy=1 | ||
| + | [lockd] | ||
| + | [mountd] | ||
| + | [nfsdcld] | ||
| + | [nfsdcltrack] | ||
| + | [nfsd] | ||
| + | rdma=y | ||
| + | rdma-port=20049 | ||
| + | [statd] | ||
| + | [sm-notify] | ||
| + | [foo@linux-bar]$ sudo mkdir -p /expdir | ||
| + | [foo@linux-bar]$ sudo bash -c 'echo "/expdir *(rw,no_root_squash)" >/etc/exports' | ||
| + | [foo@linux-bar]$ sudo systemctl restart nfs-server | ||
== Client == | == Client == | ||
| - | * Install the <code>nfs-utils | + | * Install the <code>nfs-utils</code> package and do nfs mount |
* Mount the server to check that it worked | * Mount the server to check that it worked | ||
| - | + | [foo@linux-bor]$ sudo mount -o rdma,port=20049,vers=4.2 192.168.122.40:/expdir /mnt/nfsmp | |
| - | + | [foo@linux-bor]$ mount | grep proto=rdma | |
| - | + | 192.168.122.40:/expdir on /mnt/nfsmp type nfs4 (rw,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=192.168.122.161,local_lock=none,addr=192.168.122.40) | |
= See Also = | = See Also = | ||
* [https://community.mellanox.com/docs/DOC-2184 Mellanox's softRoCE setup guide] | * [https://community.mellanox.com/docs/DOC-2184 Mellanox's softRoCE setup guide] | ||
| - | * [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/ | + | * [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_infiniband_and_rdma_networks/configuring-roce_configuring-and-managing-networking#configuring-soft-roce_configuring-roce Red Hat's softRoCE setup guide] |
* [https://github.com/linux-rdma/rdma-core/blob/master/Documentation/rxe.md Rdma-core's documentation] | * [https://github.com/linux-rdma/rdma-core/blob/master/Documentation/rxe.md Rdma-core's documentation] | ||
| + | * [https://man7.org/linux/man-pages/man8/rdma.8.html The rdma man page] | ||
* [https://www.systutorials.com/docs/linux/man/8-rxe_cfg/ The rxe_cfg man page] | * [https://www.systutorials.com/docs/linux/man/8-rxe_cfg/ The rxe_cfg man page] | ||
| + | * [https://manpages.debian.org/testing/rdma-core/rxe_cfg.8.en.html rxe_cfg deprecated] | ||
Latest revision as of 05:27, 11 August 2021
Soft ROCE is a software implementation of RoCE that allows RDMA to be used on any Ethernet adapter. It is available in Linux kernels after 4.8. The following instructions are suitable for most of the latest linux distributions.
Contents |
Client and Server Common Setup
- Check for the
rdma_rxekernel module. If you don't have it, then enable the following Kconfig options and rebuild your kernel
CONFIG_INFINIBAND CONFIG_INFINIBAND_RDMAVT CONFIG_RDMA_RXE
- Install the
iproute2package and use therdma link addcommand to load the module and start an RDMA interface. Note that this does not survive reboots. - Comment:
rxe_cfghas been deprecated from 01/2020, userdmacommand instead.
[foo@linux-bar ~]$ sudo rdma link add rxe0 type rxe netdev eth0 [foo@linux-bar ~]$ sudo rdma link link rxe0/1 state ACTIVE physical_state LINK_UP netdev eth0
Ping Test
- Start an
rpingserver on one machine
[foo@linux-bar ~]$ sudo rping -s -v -C 3 server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst server DISCONNECT EVENT... wait for RDMA_READ_ADV state 10
- Now check that the connection works from another machine
[foo@linux-bor ~]$ sudo rping -c -a 192.168.122.40 -v -C 3 ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst client DISCONNECT EVENT...
NFS Setup
Server
- Install the
nfs-utilspackage and enable rdma from/etc/nfs.confand restart service nfs-server
[foo@linux-bar]$ sudo sed -i '/rdma/{s/^#//; s/rdma=n/rdma=y/}' /etc/nfs.conf
[foo@linux-bar]$ grep -v ^# /etc/nfs.conf
[general]
[exportfs]
[gssd]
use-gss-proxy=1
[lockd]
[mountd]
[nfsdcld]
[nfsdcltrack]
[nfsd]
rdma=y
rdma-port=20049
[statd]
[sm-notify]
[foo@linux-bar]$ sudo mkdir -p /expdir
[foo@linux-bar]$ sudo bash -c 'echo "/expdir *(rw,no_root_squash)" >/etc/exports'
[foo@linux-bar]$ sudo systemctl restart nfs-server
Client
- Install the
nfs-utilspackage and do nfs mount - Mount the server to check that it worked
[foo@linux-bor]$ sudo mount -o rdma,port=20049,vers=4.2 192.168.122.40:/expdir /mnt/nfsmp [foo@linux-bor]$ mount | grep proto=rdma 192.168.122.40:/expdir on /mnt/nfsmp type nfs4 (rw,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=192.168.122.161,local_lock=none,addr=192.168.122.40)
