How do I use OpenMpi in AWS EFA

[chenshixinnb]

I use a normal node and it works fine, but I use a node that supports EFA network and report the following error:

[chenshixinnb]

[1641461840.238519] [c-96-4-worker0001:3595 :0] rc_iface.c:492 UCX ERROR ibv_create_srq() failed: Operation not supported
[1641461840.238244] [c-96-4-worker0001:3598 :0] rc_iface.c:492 UCX ERROR ibv_create_srq() failed: Operation not supported
[c-96-4-worker0001:03640] pml_ucx.c:291 Error: Failed to create UCP worker
[c-96-4-worker0001:03646] pml_ucx.c:291 Error: Failed to create UCP worker
[c-96-4-worker0001:03671] pml_ucx.c:291 Error: Failed to create UCP worker
[c-96-4-worker0001:03603] pml_ucx.c:291 Error: Failed to create UCP worker
[c-96-4-worker0001:03604] pml_ucx.c:291 Error: Failed to create UCP worker
[c-96-4-worker0001:03610] pml_ucx.c:291 Error: Failed to create UCP worker
[c-96-4-worker0001:03615] pml_ucx.c:291 Error: Failed to create UCP worker
[c-96-4-worker0001:03616] pml_ucx.c:291 Error: Failed to create UCP worker
[1641461840.237779] [c-96-4-worker0001:3605 :0] rc_iface.c:492 UCX ERROR ibv_create_srq() failed: Operation not supported
[1641461840.239128] [c-96-4-worker0001:3606 :0] rc_iface.c:492 UCX ERROR ibv_create_srq() failed: Operation not supported
[1641461840.239325] [c-96-4-worker0001:3610 :0] rc_iface.c:492 UCX ERROR ibv_create_srq() failed: Operation not supported
[1641461840.237871] [c-96-4-worker0001:3619 :0] rc_iface.c:492 UCX ERROR ibv_create_srq() failed: Operation not supported
[1641461840.238154] [c-96-4-worker0001:3616 :0] rc_iface.c:492 UCX ERROR ibv_create_srq() failed: Operation not supported
[1641461840.237974] [c-96-4-worker0001:3637 :0] rc_iface.c:492 UCX ERROR ibv_create_srq() failed: Operation not supported
[1641461840.237984] [c-96-4-worker0001:3639 :0] rc_iface.c:492 UCX ERROR ibv_create_srq() failed: Operation not supported
[1641461840.239203] [c-96-4-worker0001:3647 :0] rc_iface.c:492 UCX ERROR ibv_create_srq() failed: Operation not supported
[1641461840.238474] [c-96-4-worker0001:3664 :0] rc_iface.c:492 UCX ERROR ibv_create_srq() failed: Operation not supported
[1641461840.238478] [c-96-4-worker0001:3660 :0] rc_iface.c:492 UCX ERROR ibv_create_srq() failed: Operation not supported
[1641461840.238485] [c-96-4-worker0001:3661 :0] rc_iface.c:492 UCX ERROR ibv_create_srq() failed: Operation not supported
[1641461840.240738] [c-96-4-worker0001:3676 :0] rc_iface.c:492 UCX ERROR ibv_create_srq() failed: Operation not supported
[1641461840.242309] [c-96-4-worker0001:3680 :0] rc_iface.c:492 UCX ERROR ibv_create_srq() failed: Operation not supported
[1641461840.238529] [c-96-4-worker0001:3674 :0] rc_iface.c:492 UCX ERROR ibv_create_srq() failed: Operation not supported
[1641461840.241533] [c-96-4-worker0001:3678 :0] rc_iface.c:492 UCX ERROR ibv_create_srq() failed: Operation not supported
[1641461840.241750] [c-96-4-worker0001:3704 :0] rc_iface.c:492 UCX ERROR ibv_create_srq() failed: Operation not supported
[c-96-4-worker0001:03593] pml_ucx.c:291 Error: Failed to create UCP worker
[c-96-4-worker0001:03595] pml_ucx.c:291 Error: Failed to create UCP worker
[c-96-4-worker0001:03597] pml_ucx.c:291 Error: Failed to create UCP worker
[c-96-4-worker0001:03600] pml_ucx.c:291 Error: Failed to create UCP worker
[c-96-4-worker0001:03612] pml_ucx.c:291 Error: Failed to create UCP worker
[c-96-4-worker0001:03613] pml_ucx.c:291 Error: Failed to create UCP worker
[c-96-4-worker0001:03619] pml_ucx.c:291 Error: Failed to create UCP worker
[c-96-4-worker0001][[52336,1],41][btl_tcp_endpoint.c:625:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[52336,1],29]
[c-96-4-worker0001][[52336,1],31][btl_tcp_endpoint.c:625:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[52336,1],28]
[c-96-4-worker0001][[52336,1],32][btl_tcp_endpoint.c:625:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[52336,1],0]

[brminich]

Can you please post the commands you used to run your application?

[chenshixinnb]

I added the following parameters via this link(#6795 ):
--mca pml cm --mca mtl ofi --mca pml_base_verbose 10 --mca mtl_base_verbose 10,
The following error has occurred：
[c-96-4-worker0002:04711] mca: base: components_register: component cm register function successful
[c-96-4-worker0002:04711] mca: base: components_open: opening pml components
[c-96-4-worker0002:04711] mca: base: components_open: found loaded component cm
[c-96-4-worker0002:04710] mca: base: components_register: registering framework pml components
[c-96-4-worker0002:04710] mca: base: components_register: found loaded component cm
[c-96-4-worker0002:04710] mca: base: components_register: component cm register function successful
[c-96-4-worker0002:04710] mca: base: components_open: opening pml components
[c-96-4-worker0002:04710] mca: base: components_open: found loaded component cm
[c-96-4-worker0001:04704] mca: base: components_register: registering framework pml components
[c-96-4-worker0001:04704] mca: base: components_register: found loaded component cm
[c-96-4-worker0001:04704] mca: base: components_register: component cm register function successful
[c-96-4-worker0001:04704] mca: base: components_open: opening pml components
[c-96-4-worker0001:04704] mca: base: components_open: found loaded component cm
[c-96-4-worker0002:04708] mca: base: components_register: registering framework pml components
[c-96-4-worker0002:04716] mtl_ofi_component.c:315: mtl:ofi:provider_include = "(null)"
[c-96-4-worker0002:04716] mtl_ofi_component.c:318: mtl:ofi:provider_exclude = "shm,sockets,tcp,udp,rstream"
[c-96-4-worker0002:04716] mtl_ofi_component.c:336: mtl:ofi: "tcp;ofi_rxm" in exclude list
[c-96-4-worker0002:04716] mtl_ofi_component.c:336: mtl:ofi: "tcp;ofi_rxm" in exclude list
[c-96-4-worker0002:04716] mtl_ofi_component.c:336: mtl:ofi: "tcp;ofi_rxm" in exclude list
[c-96-4-worker0002:04716] mtl_ofi_component.c:336: mtl:ofi: "tcp;ofi_rxm" in exclude list
[c-96-4-worker0002:04716] mtl_ofi_component.c:336: mtl:ofi: "tcp;ofi_rxm" in exclude list
[c-96-4-worker0002:04716] mtl_ofi_component.c:336: mtl:ofi: "tcp;ofi_rxm" in exclude list
[c-96-4-worker0002:04716] mtl_ofi_component.c:336: mtl:ofi: "tcp;ofi_rxm" in exclude list
[c-96-4-worker0002:04716] mtl_ofi_component.c:336: mtl:ofi: "tcp;ofi_rxm" in exclude list
[c-96-4-worker0002:04716] mtl_ofi_component.c:336: mtl:ofi: "tcp;ofi_rxm" in exclude list
[c-96-4-worker0002:04716] mtl_ofi_component.c:336: mtl:ofi: "tcp;ofi_rxm" in exclude list
[c-96-4-worker0002:04716] mtl_ofi_component.c:336: mtl:ofi: "tcp;ofi_rxm" in exclude list
[c-96-4-worker0002:04716] mtl_ofi_component.c:336: mtl:ofi: "tcp;ofi_rxm" in exclude list

[chenshixinnb]

Can you please post the commands you used to run your application?

#!/bin/bash
module add GROMACS/2021-foss-2020b
mpirun -v gmx_mpi mdrun -v -cpi nvt_gpu -deffnm nvt_gpu

[chenshixinnb]

I added the following parameters via this link(#6795 ): --mca pml cm --mca mtl ofi --mca pml_base_verbose 10 --mca mtl_base_verbose 10, The following error has occurred： [c-96-4-worker0002:04711] mca: base: components_register: component cm register function successful [c-96-4-worker0002:04711] mca: base: components_open: opening pml components [c-96-4-worker0002:04711] mca: base: components_open: found loaded component cm [c-96-4-worker0002:04710] mca: base: components_register: registering framework pml components [c-96-4-worker0002:04710] mca: base: components_register: found loaded component cm [c-96-4-worker0002:04710] mca: base: components_register: component cm register function successful [c-96-4-worker0002:04710] mca: base: components_open: opening pml components [c-96-4-worker0002:04710] mca: base: components_open: found loaded component cm [c-96-4-worker0001:04704] mca: base: components_register: registering framework pml components [c-96-4-worker0001:04704] mca: base: components_register: found loaded component cm [c-96-4-worker0001:04704] mca: base: components_register: component cm register function successful [c-96-4-worker0001:04704] mca: base: components_open: opening pml components [c-96-4-worker0001:04704] mca: base: components_open: found loaded component cm [c-96-4-worker0002:04708] mca: base: components_register: registering framework pml components [c-96-4-worker0002:04716] mtl_ofi_component.c:315: mtl:ofi:provider_include = "(null)" [c-96-4-worker0002:04716] mtl_ofi_component.c:318: mtl:ofi:provider_exclude = "shm,sockets,tcp,udp,rstream" [c-96-4-worker0002:04716] mtl_ofi_component.c:336: mtl:ofi: "tcp;ofi_rxm" in exclude list [c-96-4-worker0002:04716] mtl_ofi_component.c:336: mtl:ofi: "tcp;ofi_rxm" in exclude list [c-96-4-worker0002:04716] mtl_ofi_component.c:336: mtl:ofi: "tcp;ofi_rxm" in exclude list [c-96-4-worker0002:04716] mtl_ofi_component.c:336: mtl:ofi: "tcp;ofi_rxm" in exclude list [c-96-4-worker0002:04716] mtl_ofi_component.c:336: mtl:ofi: "tcp;ofi_rxm" in exclude list [c-96-4-worker0002:04716] mtl_ofi_component.c:336: mtl:ofi: "tcp;ofi_rxm" in exclude list [c-96-4-worker0002:04716] mtl_ofi_component.c:336: mtl:ofi: "tcp;ofi_rxm" in exclude list [c-96-4-worker0002:04716] mtl_ofi_component.c:336: mtl:ofi: "tcp;ofi_rxm" in exclude list [c-96-4-worker0002:04716] mtl_ofi_component.c:336: mtl:ofi: "tcp;ofi_rxm" in exclude list [c-96-4-worker0002:04716] mtl_ofi_component.c:336: mtl:ofi: "tcp;ofi_rxm" in exclude list [c-96-4-worker0002:04716] mtl_ofi_component.c:336: mtl:ofi: "tcp;ofi_rxm" in exclude list [c-96-4-worker0002:04716] mtl_ofi_component.c:336: mtl:ofi: "tcp;ofi_rxm" in exclude list

#!/bin/bash
module add GROMACS/2021-foss-2020b
mpirun -v gmx_mpi mdrun -v -cpi nvt_gpu -deffnm nvt_gpu --mca pml cm --mca mtl ofi --mca pml_base_verbose 10

[brminich]

what is the command which produces the original errors from pml_ucx? Can you please also post the whole log?
Also why do you specify psm2 provider for running with efa?

[wzamazon]

Hi @chenshixinnb,

I have a few questions:

1. What version of open mpi are you using?
2. What version of libfabric are you using (you can get the info by running fi_info)?
3. Are you trying to use GPU version of GROMACS?

[chenshixinnb]

Hi @chenshixinnb,

I have a few questions:

1. What version of open mpi are you using?
2. What version of libfabric are you using (you can get the info by running fi_info)?
3. Are you trying to use GPU version of GROMACS?

thanks，
1.openmpi4.0.5
2.[cloudam@c-96-4-worker0001 ~]$ fi_info -p efa
provider: efa
fabric: EFA-fe80::4fc:a6ff:fe55:bb98
domain: rdmap0s6-rdm
version: 111.20
type: FI_EP_RDM
protocol: FI_PROTO_EFA
provider: efa
fabric: EFA-fe80::4fc:a6ff:fe55:bb98
domain: rdmap0s6-dgrm
version: 111.20
type: FI_EP_DGRAM
protocol: FI_PROTO_EFA
3.no,I use MPI parallelism, -cpi nvt_gpu -deffnm nvt_gpu,"nvt_gpu" is input file name

[chenshixinnb]

what is the command which produces the original errors from pml_ucx? Can you please also post the whole log? Also why do you specify psm2 provider for running with efa?

what is the command which produces the original errors from pml_ucx? Can you please also post the whole log? Also why do you specify psm2 provider for running with efa?

sorry,I pasted it wrong，
--mca pml cm --mca mtl ofi --mca pml_base_verbose 10

[wckzhang]

Those errors look like ucx errors, which wouldn't appear if the ofi mtl was properly selected. EFA is only supported with the ofi mtl. IIRC, specifying --mca pml cm doesn't fail if the cm pml cannot find a proper mtl. Try excluding the ucx pml --mca pml ^ucx. My suspicion is that the ofi mtl is not properly selecting the EFA provider and is causing a fallback to ucx.

[wckzhang]

EDIT: My recollection was wrong, specifying --mca pml cm should force the cm pml to be selected or an error will be thrown if it cannot select an MTL. I misread and realized you added --mca pml cm later.

[wzamazon]

Can you add --mca mtl_base_verbose 100 to your mpirun command line and share the output?

[chenshixinnb]

slurm2-out.txt

#!/bin/bash
module add GROMACS/2021-foss-2020b
mpirun -v gmx_mpi mdrun -v -cpi nvt_gpu -deffnm nvt_gpu --mca pml cm --mca mtl ofi --mca pml_base_verbose 10 --mca mtl_base_verbose 100

Can you add --mca mtl_base_verbose 100 to your mpirun command line and share the output?