ucx
ucx copied to clipboard
openucx
OpenMPI 4.1.3 + UCX 1.12.1 - Failed to receive UCX worker address
Describe the bug
We are experiencing issues difficult to identify at "high" number of ranks (8k, when running with dc), while everything works perfectly at lower rank number (4k and bellow, running with rc)
The run would fail with a message typically of:
pml_ucx.c:178 Error: Failed to receive UCX worker address: Not found (-13)
pml_ucx.c:472 Error: Failed to resolve UCX endpoint for rank 25
The following backtrace is then available, which shows an error in the MPI_Testsome() function.
==== backtrace (tid: 17620) ====
25957 0 0x000000000002577f uct_ib_mlx5_completion_with_err() /dev/shm/build/UCX/1.12.1/GCCcore-10.3.0/ucx-1.12.1/src/uct/ib/mlx5/ib_mlx5_log.c:162
25958 1 0x0000000000051c37 uct_dc_mlx5_iface_set_ep_failed() /dev/shm/build/UCX/1.12.1/GCCcore-10.3.0/ucx-1.12.1/src/uct/ib/dc/dc_mlx5.c:1605
25959 2 0x000000000004b8c8 uct_dc_mlx5_ep_handle_failure() /dev/shm/build/UCX/1.12.1/GCCcore-10.3.0/ucx-1.12.1/src/uct/ib/dc/dc_mlx5_ep.c:1537
25960 3 0x000000000004d822 uct_ib_mlx5_poll_cq() /dev/shm/build/UCX/1.12.1/GCCcore-10.3.0/ucx-1.12.1/src/uct/ib/mlx5/ib_mlx5.inl:91
25961 4 0x000000000004d822 uct_dc_mlx5_iface_progress() /dev/shm/build/UCX/1.12.1/GCCcore-10.3.0/ucx-1.12.1/src/uct/ib/dc/dc_mlx5.c:275
25962 5 0x000000000004d822 uct_dc_mlx5_iface_progress_ll() /dev/shm/build/UCX/1.12.1/GCCcore-10.3.0/ucx-1.12.1/src/uct/ib/dc/dc_mlx5.c:285
25963 6 0x000000000003875a ucs_callbackq_dispatch() /dev/shm/build/UCX/1.12.1/GCCcore-10.3.0/ucx-1.12.1/src/ucs/datastruct/callbackq.h:211
25964 7 0x000000000003875a uct_worker_progress() /dev/shm/build/UCX/1.12.1/GCCcore-10.3.0/ucx-1.12.1/src/uct/api/uct.h:2589
25965 8 0x000000000003875a ucp_worker_progress() /dev/shm/build/UCX/1.12.1/GCCcore-10.3.0/ucx-1.12.1/src/ucp/core/ucp_worker.c:2636
25966 9 0x00000000000317dc opal_progress() /dev/shm/build/OpenMPI/4.1.3/GCC-10.3.0/openmpi-4.1.3/opal/runtime/opal_progress.c:231
25967 10 0x0000000000057523 ompi_request_default_test_some() /dev/shm/build/OpenMPI/4.1.3/GCC-10.3.0/openmpi-4.1.3/ompi/request/req_test.c:321
25968 11 0x000000000007f1ed PMPI_Testsome() /dev/shm/build/OpenMPI/4.1.3/GCC-10.3.0/openmpi-4.1.3/ompi/mpi/c/profile/ptestsome.c:85
25969 12 0x00000000004308b2 SendRecv() /project/scratch/p200053/weak_scaling_ompi_4.1.3_2022-06-16-0025-109d23f4/flups/src/SwitchTopoX_nb.cpp:280
25970 13 0x000000000042ecb1 SwitchTopoX_nb::execute() /project/scratch/p200053/weak_scaling_ompi_4.1.3_2022-06-16-0025-109d23f4/flups/src/SwitchTopoX_nb.cpp:164
25971 14 0x00000000004250a5 Solver::cmptGreenFunction_() /project/scratch/p200053/weak_scaling_ompi_4.1.3_2022-06-16-0025-109d23f4/flups/src/Solver.cpp:1060
25972 15 0x000000000042185b Solver::setup() /project/scratch/p200053/weak_scaling_ompi_4.1.3_2022-06-16-0025-109d23f4/flups/src/Solver.cpp:429
25973 16 0x0000000000413cea flups_setup() /project/scratch/p200053/weak_scaling_ompi_4.1.3_2022-06-16-0025-109d23f4/flups/src/flups.cpp:209
25974 17 0x00000000004071a2 validation_3d() /project/scratch/p200053/weak_scaling_ompi_4.1.3_2022-06-16-0025-109d23f4/flups/samples/validation/src/validation_3d.cpp:107
25975 18 0x000000000040afa3 main() /project/scratch/p200053/weak_scaling_ompi_4.1.3_2022-06-16-0025-109d23f4/flups/samples/validation/src/main.cpp:149
25976 19 0x0000000000023493 __libc_start_main() ???:0
25977 20 0x0000000000405cfe _start() ???:0
25978 =================================
[mel0289:17620] *** Process received signal ***
[mel0289:17620] Signal: Aborted (6)
[mel0289:17620] Signal code: (-6)
[mel0289:17620] [ 0] /lib64/libpthread.so.0(+0x12c20)[0x14cb45761c20]
[mel0289:17620] [ 1] /lib64/libc.so.6(gsignal+0x10f)[0x14cb453c137f]
[mel0289:17620] [ 2] /lib64/libc.so.6(abort+0x127)[0x14cb453abdb5]
[mel0289:17620] [ 3] /apps/USE/easybuild/staging/2021.5/software/UCX/1.12.1-GCCcore-10.3.0/lib/libucs.so.0(+0x2646d)[0x14cb42b6246d]
[mel0289:17620] [ 4] /apps/USE/easybuild/staging/2021.5/software/UCX/1.12.1-GCCcore-10.3.0/lib/libucs.so.0(ucs_log_default_handler+0x7b9)[0x14cb42b664d9]
[mel0289:17620] [ 5] /apps/USE/easybuild/staging/2021.5/software/UCX/1.12.1-GCCcore-10.3.0/lib/libucs.so.0(ucs_log_dispatch+0xd1)[0x14cb42b66651]
[mel0289:17620] [ 6] /apps/USE/easybuild/staging/2021.5/software/UCX/1.12.1-GCCcore-10.3.0/lib/ucx/libuct_ib.so.0(uct_ib_mlx5_completion_with_err+0x4ff)[0x14cb4291577f]
[mel0289:17620] [ 7] /apps/USE/easybuild/staging/2021.5/software/UCX/1.12.1-GCCcore-10.3.0/lib/ucx/libuct_ib.so.0(uct_dc_mlx5_iface_set_ep_failed+0x67)[0x14cb42941c37]
[mel0289:17620] [ 8] /apps/USE/easybuild/staging/2021.5/software/UCX/1.12.1-GCCcore-10.3.0/lib/ucx/libuct_ib.so.0(uct_dc_mlx5_ep_handle_failure+0x198)[0x14cb4293b8c8]
[mel0289:17620] [ 9] /apps/USE/easybuild/staging/2021.5/software/UCX/1.12.1-GCCcore-10.3.0/lib/ucx/libuct_ib.so.0(+0x4d822)[0x14cb4293d822]
[mel0289:17620] [10] /apps/USE/easybuild/staging/2021.5/software/UCX/1.12.1-GCCcore-10.3.0/lib/libucp.so.0(ucp_worker_progress+0x5a)[0x14cb42c0675a]
[mel0289:17620] [11] /apps/USE/easybuild/staging/2021.5/software/OpenMPI/4.1.3-GCC-10.3.0/lib/libopen-pal.so.40(opal_progress+0x2c)[0x14cb450467dc]
[mel0289:17620] [12] /apps/USE/easybuild/staging/2021.5/software/OpenMPI/4.1.3-GCC-10.3.0/lib/libmpi.so.40(ompi_request_default_test_some+0x1c3)[0x14cb466f7523]
[mel0289:17620] [13] /apps/USE/easybuild/staging/2021.5/software/OpenMPI/4.1.3-GCC-10.3.0/lib/libmpi.so.40(MPI_Testsome+0x1d)[0x14cb4671f1ed]
[mel0289:17620] [14] ./flups_validation_nb[0x4308b2]
[mel0289:17620] [15] ./flups_validation_nb[0x42ecb1]
[mel0289:17620] [16] ./flups_validation_nb[0x4250a5]
[mel0289:17620] [17] ./flups_validation_nb[0x42185b]
[mel0289:17620] [18] ./flups_validation_nb[0x413cea]
[mel0289:17620] [19] ./flups_validation_nb[0x4071a2]
[mel0289:17620] [20] ./flups_validation_nb[0x40afa3]
[mel0289:17620] [21] /lib64/libc.so.6(__libc_start_main+0xf3)[0x14cb453ad493]
[mel0289:17620] [22] ./flups_validation_nb[0x405cfe]
[mel0289:17620] *** End of error message ***
It's not clear to us whether this issues comes from MPI, UCX, or the code itself.
Several valgrind traces point towards an issue in the UCX, but it's all mixed up and difficult to isolate.
Along the way I have also tried with OpenMPI 4.1.4 and UCX 1.13.0rc1 + valgrind which would give us the following trace:
[1,4000]<stderr>:[mel0537:6788 :0:6788] ib_mlx5_log.c:171 Remote access on mlx5_0:1/IB (synd 0x13 vend 0x88 hw_synd 0/0)
[1,4000]<stderr>:[mel0537:6788 :0:6788] ib_mlx5_log.c:171 DCI QP 0x97e5 wqe[35]: RDMA_READ s-- [rqpn 0x13a98 rlid 463] [rva 0x1791370 rkey 0x1b0682] [va 0x2b01280 len 3968 lkey 0x241e07]
[1,4000]<stderr>:
[1,4000]<stderr>:/project/scratch/p200053/tmp_dbs-2022-06-15-1659-c3f2871c/ucx-1.13.0/src/uct/ib/mlx5/ib_mlx5_log.c: [ uct_ib_mlx5_completion_with_err() ]
[1,4000]<stderr>: ...
[1,4000]<stderr>: 168 uct_ib_mlx5_cqe_err_opcode(ecqe));
[1,4000]<stderr>: 169 }
[1,4000]<stderr>: 170
[1,4000]<stderr>:==> 171 ucs_log(log_level,
[1,4000]<stderr>: 172 "%s on " UCT_IB_IFACE_FMT " (synd 0x%x vend 0x%x hw_synd %d/%d)\n"
[1,4000]<stderr>: 173 "%s QP 0x%x wqe[%d]: %s %s",
[1,4000]<stderr>: 174 err_info, UCT_IB_IFACE_ARG(iface), ecqe->syndrome,
[1,4000]<stderr>:
[1,4000]<stderr>:==== backtrace (tid: 6788) ====
[1,4000]<stderr>: 0 0x00000000000266f9 uct_ib_mlx5_completion_with_err() /project/scratch/p200053/tmp_dbs-2022-06-15-1659-c3f2871c/ucx-1.13.0/src/uct/ib/mlx5/ib_mlx5_log.c:171
[1,4000]<stderr>: 1 0x000000000004e3c7 uct_dc_mlx5_ep_handle_failure() /project/scratch/p200053/tmp_dbs-2022-06-15-1659-c3f2871c/ucx-1.13.0/src/uct/ib/dc/dc_mlx5_ep.c:1606
[1,4000]<stderr>: 2 0x00000000000275b5 uct_ib_mlx5_check_completion() /project/scratch/p200053/tmp_dbs-2022-06-15-1659-c3f2871c/ucx-1.13.0/src/uct/ib/mlx5/ib_mlx5.c:492
[1,4000]<stderr>: 3 0x0000000000050332 uct_ib_mlx5_poll_cq() /project/scratch/p200053/tmp_dbs-2022-06-15-1659-c3f2871c/ucx-1.13.0/src/uct/ib/mlx5/ib_mlx5.inl:115
[1,4000]<stderr>: 4 0x0000000000050332 uct_dc_mlx5_poll_tx() /project/scratch/p200053/tmp_dbs-2022-06-15-1659-c3f2871c/ucx-1.13.0/src/uct/ib/dc/dc_mlx5.c:236
[1,4000]<stderr>: 5 0x0000000000050332 uct_dc_mlx5_iface_progress() /project/scratch/p200053/tmp_dbs-2022-06-15-1659-c3f2871c/ucx-1.13.0/src/uct/ib/dc/dc_mlx5.c:281
[1,4000]<stderr>: 6 0x0000000000050332 uct_dc_mlx5_iface_progress_ll() /project/scratch/p200053/tmp_dbs-2022-06-15-1659-c3f2871c/ucx-1.13.0/src/uct/ib/dc/dc_mlx5.c:291
[1,4000]<stderr>: 7 0x00000000000428ba ucs_callbackq_dispatch() /project/scratch/p200053/tmp_dbs-2022-06-15-1659-c3f2871c/ucx-1.13.0/src/ucs/datastruct/callbackq.h:211
[1,4000]<stderr>: 8 0x00000000000428ba uct_worker_progress() /project/scratch/p200053/tmp_dbs-2022-06-15-1659-c3f2871c/ucx-1.13.0/src/uct/api/uct.h:2638
[1,4000]<stderr>: 9 0x00000000000428ba ucp_worker_progress() /project/scratch/p200053/tmp_dbs-2022-06-15-1659-c3f2871c/ucx-1.13.0/src/ucp/core/ucp_worker.c:2782
[1,4000]<stderr>:10 0x000000000003166c opal_progress() ???:0
[1,4000]<stderr>:11 0x0000000000057193 ompi_request_default_test_some() ???:0
[1,4000]<stderr>:12 0x0000000000093bcb MPI_Testsome() ???:0
[1,4000]<stderr>:13 0x00000000004397a9 SendRecv() /project/scratch/p200053/weak_scaling_ompi_4.1.3_2022-06-15-1810-a5561eb8/flups/src/SwitchTopoX_nb.cpp:280
[1,4000]<stderr>:14 0x000000000043abe6 SwitchTopoX_nb::execute() /project/scratch/p200053/weak_scaling_ompi_4.1.3_2022-06-15-1810-a5561eb8/flups/src/SwitchTopoX_nb.cpp:164
[1,4000]<stderr>:15 0x000000000042275c Solver::cmptGreenFunction_() /project/scratch/p200053/weak_scaling_ompi_4.1.3_2022-06-15-1810-a5561eb8/flups/src/Solver.cpp:1060
[1,4000]<stderr>:16 0x0000000000423978 Solver::setup() /project/scratch/p200053/weak_scaling_ompi_4.1.3_2022-06-15-1810-a5561eb8/flups/src/Solver.cpp:429
[1,4000]<stderr>:17 0x0000000000409038 validation_3d() /project/scratch/p200053/weak_scaling_ompi_4.1.3_2022-06-15-1810-a5561eb8/flups/samples/validation/src/validation_3d.cpp:107
[1,4000]<stderr>:18 0x0000000000406de0 main() /project/scratch/p200053/weak_scaling_ompi_4.1.3_2022-06-15-1810-a5561eb8/flups/samples/validation/src/main.cpp:149
[1,4000]<stderr>:19 0x0000000000023493 __libc_start_main() ???:0
[1,4000]<stderr>:20 0x000000000040734e _start() ???:0
Could you help us to track down the issue and identify a fix? Thank you very much for your help :-)
Setup and versions
rpm -q rdma-core
rdma-core-54mlnx1-1.54310.x86_64
rpm -q libibverbs
libibverbs-54mlnx1-1.54310.x86_64
ofed_info -s
OFED-internal-5.4-3.1.0:
-
ompi 4.1.3 -
ucx-1.12.1 -
ucx_info:
# Memory domain: posix
# Component: posix
# allocate: <= 448491980K
# remote key: 24 bytes
# rkey_ptr is supported
#
# Transport: posix
# Device: memory
# Type: intra-node
# System device: <unknown>
#
# capabilities:
# bandwidth: 0.00/ppn + 12179.00 MB/sec
# latency: 80 nsec
# overhead: 10 nsec
# put_short: <= 4294967295
# put_bcopy: unlimited
# get_bcopy: unlimited
# am_short: <= 100
# am_bcopy: <= 8256
# domain: cpu
# atomic_add: 32, 64 bit
# atomic_and: 32, 64 bit
# atomic_or: 32, 64 bit
# atomic_xor: 32, 64 bit
# atomic_fadd: 32, 64 bit
# atomic_fand: 32, 64 bit
# atomic_for: 32, 64 bit
# atomic_fxor: 32, 64 bit
# atomic_swap: 32, 64 bit
# atomic_cswap: 32, 64 bit
# connection: to iface
# device priority: 0
# device num paths: 1
# max eps: inf
# device address: 8 bytes
# iface address: 8 bytes
# error handling: ep_check
#
#
# Memory domain: sysv
# Component: sysv
# allocate: unlimited
# remote key: 12 bytes
# rkey_ptr is supported
#
# Transport: sysv
# Device: memory
# Type: intra-node
# System device: <unknown>
#
# capabilities:
# bandwidth: 0.00/ppn + 12179.00 MB/sec
# latency: 80 nsec
# overhead: 10 nsec
# put_short: <= 4294967295
# put_bcopy: unlimited
# get_bcopy: unlimited
# am_short: <= 100
# am_bcopy: <= 8256
# domain: cpu
# atomic_add: 32, 64 bit
# atomic_and: 32, 64 bit
# atomic_or: 32, 64 bit
# atomic_xor: 32, 64 bit
# atomic_fadd: 32, 64 bit
# atomic_fand: 32, 64 bit
# atomic_for: 32, 64 bit
# atomic_fxor: 32, 64 bit
# atomic_swap: 32, 64 bit
# atomic_cswap: 32, 64 bit
# connection: to iface
# device priority: 0
# device num paths: 1
# max eps: inf
# device address: 8 bytes
# iface address: 8 bytes
# error handling: ep_check
#
#
# Memory domain: self
# Component: self
# register: unlimited, cost: 0 nsec
# remote key: 0 bytes
#
# Transport: self
# Device: memory0
# Type: loopback
# System device: <unknown>
#
# capabilities:
# bandwidth: 0.00/ppn + 6911.00 MB/sec
# latency: 0 nsec
# overhead: 10 nsec
# put_short: <= 4294967295
# put_bcopy: unlimited
# get_bcopy: unlimited
# am_short: <= 8K
# am_bcopy: <= 8K
# domain: cpu
# atomic_add: 32, 64 bit
# atomic_and: 32, 64 bit
# atomic_or: 32, 64 bit
# atomic_xor: 32, 64 bit
# atomic_fadd: 32, 64 bit
# atomic_fand: 32, 64 bit
# atomic_for: 32, 64 bit
# atomic_fxor: 32, 64 bit
# atomic_swap: 32, 64 bit
# atomic_cswap: 32, 64 bit
# connection: to iface
# device priority: 0
# device num paths: 1
# max eps: inf
# device address: 0 bytes
# iface address: 8 bytes
# error handling: ep_check
#
#
# Memory domain: tcp
# Component: tcp
# register: unlimited, cost: 0 nsec
# remote key: 0 bytes
#
# Transport: tcp
# Device: enp225s0f0
# Type: network
# System device: <unknown>
#
# capabilities:
# bandwidth: 113.16/ppn + 0.00 MB/sec
# latency: 5776 nsec
# overhead: 50000 nsec
# put_zcopy: <= 18446744073709551590, up to 6 iov
# put_opt_zcopy_align: <= 1
# put_align_mtu: <= 0
# am_short: <= 8K
# am_bcopy: <= 8K
# am_zcopy: <= 64K, up to 6 iov
# am_opt_zcopy_align: <= 1
# am_align_mtu: <= 0
# am header: <= 8037
# connection: to ep, to iface
# device priority: 1
# device num paths: 1
# max eps: 256
# device address: 6 bytes
# iface address: 2 bytes
# ep address: 10 bytes
# error handling: peer failure, ep_check, keepalive
#
# Transport: tcp
# Device: lo
# Type: network
# System device: <unknown>
#
# capabilities:
# bandwidth: 11.91/ppn + 0.00 MB/sec
# latency: 10960 nsec
# overhead: 50000 nsec
# put_zcopy: <= 18446744073709551590, up to 6 iov
# put_opt_zcopy_align: <= 1
# put_align_mtu: <= 0
# am_short: <= 8K
# am_bcopy: <= 8K
# am_zcopy: <= 64K, up to 6 iov
# am_opt_zcopy_align: <= 1
# am_align_mtu: <= 0
# am header: <= 8037
# connection: to ep, to iface
# device priority: 1
# device num paths: 1
# max eps: 256
# device address: 18 bytes
# iface address: 2 bytes
# ep address: 10 bytes
# error handling: peer failure, ep_check, keepalive
#
# Transport: tcp
# Device: ib0
# Type: network
# System device: <unknown>
#
# capabilities:
# bandwidth: 22285.02/ppn + 0.00 MB/sec
# latency: 5203 nsec
# overhead: 50000 nsec
# put_zcopy: <= 18446744073709551590, up to 6 iov
# put_opt_zcopy_align: <= 1
# put_align_mtu: <= 0
# am_short: <= 8K
# am_bcopy: <= 8K
# am_zcopy: <= 64K, up to 6 iov
# am_opt_zcopy_align: <= 1
# am_align_mtu: <= 0
# am header: <= 8037
# connection: to ep, to iface
# device priority: 0
# device num paths: 1
# max eps: 256
# device address: 6 bytes
# iface address: 2 bytes
# ep address: 10 bytes
# error handling: peer failure, ep_check, keepalive
#
#
# Connection manager: tcp
# max_conn_priv: 2064 bytes
#
# Memory domain: mlx5_0
# Component: ib
# register: unlimited, cost: 180 nsec
# remote key: 8 bytes
# local memory handle is required for zcopy
#
# Transport: rc_verbs
# Device: mlx5_0:1
# Type: network
# System device: mlx5_0 (0)
#
# capabilities:
# bandwidth: 23588.47/ppn + 0.00 MB/sec
# latency: 600 + 1.000 * N nsec
# overhead: 75 nsec
# put_short: <= 124
# put_bcopy: <= 8256
# put_zcopy: <= 1G, up to 5 iov
# put_opt_zcopy_align: <= 512
# put_align_mtu: <= 4K
# get_bcopy: <= 8256
# get_zcopy: 65..1G, up to 5 iov
# get_opt_zcopy_align: <= 512
# get_align_mtu: <= 4K
# am_short: <= 123
# am_bcopy: <= 8255
# am_zcopy: <= 8255, up to 4 iov
# am_opt_zcopy_align: <= 512
# am_align_mtu: <= 4K
# am header: <= 127
# domain: device
# atomic_add: 64 bit
# atomic_fadd: 64 bit
# atomic_cswap: 64 bit
# connection: to ep
# device priority: 50
# device num paths: 1
# max eps: 256
# device address: 3 bytes
# ep address: 5 bytes
# error handling: peer failure, ep_check
#
#
# Transport: rc_mlx5
# Device: mlx5_0:1
# Type: network
# System device: mlx5_0 (0)
#
# capabilities:
# bandwidth: 23588.47/ppn + 0.00 MB/sec
# latency: 600 + 1.000 * N nsec
# overhead: 40 nsec
# put_short: <= 2K
# put_bcopy: <= 8256
# put_zcopy: <= 1G, up to 14 iov
# put_opt_zcopy_align: <= 512
# put_align_mtu: <= 4K
# get_bcopy: <= 8256
# get_zcopy: 65..1G, up to 14 iov
# get_opt_zcopy_align: <= 512
# get_align_mtu: <= 4K
# am_short: <= 2046
# am_bcopy: <= 8254
# am_zcopy: <= 8254, up to 3 iov
# am_opt_zcopy_align: <= 512
# am_align_mtu: <= 4K
# am header: <= 186
# domain: device
# atomic_add: 32, 64 bit
# atomic_and: 32, 64 bit
# atomic_or: 32, 64 bit
# atomic_xor: 32, 64 bit
# atomic_fadd: 32, 64 bit
# atomic_fand: 32, 64 bit
# atomic_for: 32, 64 bit
# atomic_fxor: 32, 64 bit
# atomic_swap: 32, 64 bit
# atomic_cswap: 32, 64 bit
# connection: to ep
# device priority: 50
# device num paths: 1
# max eps: 256
# device address: 3 bytes
# ep address: 7 bytes
# error handling: buffer (zcopy), remote access, peer failure, ep_check
#
#
# Transport: dc_mlx5
# Device: mlx5_0:1
# Type: network
# System device: mlx5_0 (0)
#
# capabilities:
# bandwidth: 23588.47/ppn + 0.00 MB/sec
# latency: 660 nsec
# overhead: 40 nsec
# put_short: <= 2K
# put_bcopy: <= 8256
# put_zcopy: <= 1G, up to 11 iov
# put_opt_zcopy_align: <= 512
# put_align_mtu: <= 4K
# get_bcopy: <= 8256
# get_zcopy: 65..1G, up to 11 iov
# get_opt_zcopy_align: <= 512
# get_align_mtu: <= 4K
# am_short: <= 2046
# am_bcopy: <= 8254
# am_zcopy: <= 8254, up to 3 iov
# am_opt_zcopy_align: <= 512
# am_align_mtu: <= 4K
# am header: <= 138
# domain: device
# atomic_add: 32, 64 bit
# atomic_and: 32, 64 bit
# atomic_or: 32, 64 bit
# atomic_xor: 32, 64 bit
# atomic_fadd: 32, 64 bit
# atomic_fand: 32, 64 bit
# atomic_for: 32, 64 bit
# atomic_fxor: 32, 64 bit
# atomic_swap: 32, 64 bit
# atomic_cswap: 32, 64 bit
# connection: to iface
# device priority: 50
# device num paths: 1
# max eps: inf
# device address: 3 bytes
# iface address: 5 bytes
# error handling: buffer (zcopy), remote access, peer failure, ep_check
#
#
# Transport: ud_verbs
# Device: mlx5_0:1
# Type: network
# System device: mlx5_0 (0)
#
# capabilities:
# bandwidth: 23588.47/ppn + 0.00 MB/sec
# latency: 630 nsec
# overhead: 105 nsec
# am_short: <= 116
# am_bcopy: <= 4088
# am_zcopy: <= 4088, up to 5 iov
# am_opt_zcopy_align: <= 512
# am_align_mtu: <= 4K
# am header: <= 3952
# connection: to ep, to iface
# device priority: 50
# device num paths: 1
# max eps: inf
# device address: 3 bytes
# iface address: 3 bytes
# ep address: 6 bytes
# error handling: peer failure, ep_check
#
#
# Transport: ud_mlx5
# Device: mlx5_0:1
# Type: network
# System device: mlx5_0 (0)
#
# capabilities:
# bandwidth: 23588.47/ppn + 0.00 MB/sec
# latency: 630 nsec
# overhead: 80 nsec
# am_short: <= 180
# am_bcopy: <= 4088
# am_zcopy: <= 4088, up to 3 iov
# am_opt_zcopy_align: <= 512
# am_align_mtu: <= 4K
# am header: <= 132
# connection: to ep, to iface
# device priority: 50
# device num paths: 1
# max eps: inf
# device address: 3 bytes
# iface address: 3 bytes
# ep address: 6 bytes
# error handling: peer failure, ep_check
#
#
# Connection manager: rdmacm
# max_conn_priv: 54 bytes
#
# Memory domain: cma
# Component: cma
# register: unlimited, cost: 9 nsec
#
# Transport: cma
# Device: memory
# Type: intra-node
# System device: <unknown>
#
# capabilities:
# bandwidth: 0.00/ppn + 11145.00 MB/sec
# latency: 80 nsec
# overhead: 2000 nsec
# put_zcopy: unlimited, up to 16 iov
# put_opt_zcopy_align: <= 1
# put_align_mtu: <= 1
# get_zcopy: unlimited, up to 16 iov
# get_opt_zcopy_align: <= 1
# get_align_mtu: <= 1
# connection: to iface
# device priority: 0
# device num paths: 1
# max eps: inf
# device address: 8 bytes
# iface address: 4 bytes
# error handling: peer failure, ep_check
#
