NFS over SoftRoCE setup
From Linux NFS
(Difference between revisions)
(Created page with "Soft ROCE is a software implementation of [https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet RoCE] that allows RDMA to be used on any Ethernet adapter. It is available ...") |
|||
| Line 1: | Line 1: | ||
| - | Soft ROCE is a software implementation of [https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet RoCE] that allows RDMA to be used on any Ethernet adapter. It is available in Linux kernels after 4.8. | + | Soft ROCE is a software implementation of [https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet RoCE] that allows RDMA to be used on any Ethernet adapter. It is available in Linux kernels after 4.8. The following instructions are based on [https://www.archlinux.org/ Arch Linux] |
= Client and Server Common Setup = | = Client and Server Common Setup = | ||
| Line 6: | Line 6: | ||
CONFIG_INFINIBAND_RDMAVT | CONFIG_INFINIBAND_RDMAVT | ||
CONFIG_RDMA_RXE | CONFIG_RDMA_RXE | ||
| - | * Install the <code>rdma-core</code> package and use the <code>rxe_cfg</code> script to load the module and start an RDMA interface | + | * Install the <code>rdma-core</code> package and use the <code>rxe_cfg</code> script to load the module and start an RDMA interface. Note that this does not survive reboots. |
anna@gouda ~ % sudo rxe_cfg status | anna@gouda ~ % sudo rxe_cfg status | ||
rdma_rxe module not loaded | rdma_rxe module not loaded | ||
| Line 36: | Line 36: | ||
= NFS Setup = | = NFS Setup = | ||
== Server == | == Server == | ||
| - | |||
* Install the <code>nfs-utils-rdma-server</code> package and enable the <code>nfs-utils-rdma-server.service</code> service through systemd | * Install the <code>nfs-utils-rdma-server</code> package and enable the <code>nfs-utils-rdma-server.service</code> service through systemd | ||
== Client == | == Client == | ||
| Line 47: | Line 46: | ||
= See Also = | = See Also = | ||
* [https://community.mellanox.com/docs/DOC-2184 Mellanox's softRoCE setup guide] | * [https://community.mellanox.com/docs/DOC-2184 Mellanox's softRoCE setup guide] | ||
| + | * [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-configuring_soft-_roce Red Hat's softRoCE setup guide] | ||
| + | * [https://github.com/linux-rdma/rdma-core/blob/master/Documentation/rxe.md Rdma-core's documentation] | ||
| + | * [https://www.systutorials.com/docs/linux/man/8-rxe_cfg/ The rxe_cfg man page] | ||
Revision as of 20:55, 26 November 2018
Soft ROCE is a software implementation of RoCE that allows RDMA to be used on any Ethernet adapter. It is available in Linux kernels after 4.8. The following instructions are based on Arch Linux
Contents |
Client and Server Common Setup
- Check for the
rdma_rxekernel module. If you don't have it, then enable the following Kconfig options and rebuild your kernel
CONFIG_INFINIBAND CONFIG_INFINIBAND_RDMAVT CONFIG_RDMA_RXE
- Install the
rdma-corepackage and use therxe_cfgscript to load the module and start an RDMA interface. Note that this does not survive reboots.
anna@gouda ~ % sudo rxe_cfg status rdma_rxe module not loaded Name Link Driver Speed NMTU IPv4_addr RDEV RMTU eth0 yes virtio_net anna@gouda ~ % sudo rxe_cfg start Name Link Driver Speed NMTU IPv4_addr RDEV RMTU eth0 yes virtio_net anna@gouda ~ % sudo rxe_cfg add eth0 anna@gouda ~ % sudo rxe_cfg status Name Link Driver Speed NMTU IPv4_addr RDEV RMTU eth0 yes virtio_net rxe0 1024 (3)
Ping Test
- Start an
rpingserver on one machine
anna@server ~ % sudo rping -s -v -C 3 server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst server DISCONNECT EVENT... wait for RDMA_READ_ADV state 10
- Now check that the connection works from another machine
anna@client ~ % sudo rping -c -a 255.255.255.255 -v -C 3 ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst client DISCONNECT EVENT...
NFS Setup
Server
- Install the
nfs-utils-rdma-serverpackage and enable thenfs-utils-rdma-server.serviceservice through systemd
Client
- Install the
nfs-utils-rdma-clientpackage and enable thenfs-utils-rdma-client.serviceservice through systemd - Mount the server to check that it worked
anna@client ~ % sudo mount -o rdma,port=20049,vers=4.1 192.168.100.215:/export anna@client ~ % mount | grep proto=rdma 192.168.100.215:/home/anna/tests/tests/anna/cthon on /nfs type nfs4 (rw,relatime,vers=4.1,rsize=524288,wsize=524288,proto=rdma,port=20049)
