NfsRdmaClient/MemRegModes
From Linux NFS
Chucklever (Talk | contribs) (→Long-term Plans) |
Chucklever (Talk | contribs) (→Long-term Plans) |
||
Line 243: | Line 243: | ||
| yes | | yes | ||
| no | | no | ||
- | | | + | | Remove |
| Never intended for production | | Never intended for production | ||
|- | |- | ||
Line 249: | Line 249: | ||
| yes | | yes | ||
| no | | no | ||
- | | | + | | Remove |
| Safe, but not fast | | Safe, but not fast | ||
|- | |- | ||
Line 255: | Line 255: | ||
| yes | | yes | ||
| no | | no | ||
- | | | + | | Remove |
| No value to keeping this | | No value to keeping this | ||
|- | |- | ||
Line 261: | Line 261: | ||
| no | | no | ||
| yes | | yes | ||
- | | | + | | Remove |
| Generally unsafe | | Generally unsafe | ||
|- | |- |
Latest revision as of 13:57, 30 May 2014
Contents |
Simplifying NFS/RDMA Client Memory Registration Modes
The RPC/RDMA transport uses a variety of memory registration strategies to mark areas of the NFS client's memory that are eligible for RDMA. At mount time, one registration mode is chosen based on the provider and HCA used to connect to the NFS server. By default, RPC/RDMA attempts to use FRMR because it is fast and generally safe.
However, not all providers and HCAs support FRMR. Thus the RPC/RDMA client transport code maintains 7 different memory registration modes, which introduces a lot of complexity to the code base and our test matrix. Over time we would like to remove some of these modes to simplify the code base and testing requirements.
Matrix of Registration Mode Support
The following table is based on a code audit of RDMA providers in the 3.13 kernel and one (the Soft RoCE driver, rxe) that is only in OFED.
memreg mode/provider | BOUNCEBUFFERS | REGISTER (reg_phys_mr verb present) | MEMWINDOWS (device mem_window flag set) | MEMWINDOWS_ASYNC (device mem_window flag set) | MTHCAFMR (alloc_fmr verb present) | FRMR (device mem_mgt_ext and loc_dma_key flags set) | ALLPHYSICAL |
---|---|---|---|---|---|---|---|
amso1100 | yes | yes | yes | yes | no | no | yes |
cxgb3 | yes | yes | yes | yes | no | yes | yes |
cxgb4 | yes | yes | yes | yes | no | yes | yes |
ehca | yes | yes | no | no | yes | no | yes |
ipath | yes | yes | no | no | yes | no | yes |
mlx4 | yes | no | yes | yes | yes | yes | yes |
mlx5 | yes | no | no | no | no | yes | yes |
mthca | yes | yes | no | no | yes | no | yes |
nes | yes | yes | yes | yes | no | yes | yes |
ocrdma | yes | yes | no | no | no | yes | yes |
qib | yes | yes | no | no | yes | no | yes |
rxe | yes | yes | no | no | yes | no | yes |
Summary of memory registration modes
From Tom Talpey.
Registration mode | Pros | Cons |
---|---|---|
RPCRDMA_BOUNCEBUFFERS |
|
|
RPCRDMA_REGISTER |
|
|
RPCRDMA_MEMWINDOWS |
|
|
RPCRDMA_MEMWINDOWS_ASYNC |
|
|
RPCRDMA_MTHCAFMR |
|
Strongly suggest not relying on mthca except for purpose of supporting antique mellanox adapters |
RPCRDMA_FRMR |
|
|
RPCRDMA_ALLPHYSICAL |
|
|
Long-term Plans
After some discussion on linux-rdma and linux-nfs, we might be able to justify removing unsafe and deprecated memory registration modes.
Mode | Safe | Fast | Long-term | Comments |
---|---|---|---|---|
RPCRDMA_BOUNCEBUFFERS | yes | no | Remove | Never intended for production |
RPCRDMA_REGISTER | yes | no | Remove | Safe, but not fast |
RPCRDMA_MEMWINDOWS | yes | no | Remove | No value to keeping this |
RPCRDMA_MEMWINDOWS_ASYNC | no | yes | Remove | Generally unsafe |
RPCRDMA_MTHCAFMR | no | yes | Keep | Supported in Xen guests; RDMA protection on page boundaries only |
RPCRDMA_FRMR | yes | yes | Keep | Generally fast and safe, should remain our default |
RPCRDMA_ALLPHYSICAL | no | yes | Keep | Can be unsafe, but is broadly compatible |