MPI library used is the one compiled with the application and not the one used for mpirun.

[wckzhang]

Moved ticket from - openpmix/openpmix#2898

[ec2-user@ip-10-0-0-28 ~]$ ~/ompi5install/bin/mpirun -np 1 ldd ~/osu4/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_latency
<snip>
	libmpi.so.40 => /opt/amazon/openmpi/lib64/libmpi.so.40 (0x00007f6e0e757000)
<snip>
	libopen-pal.so.40 => /opt/amazon/openmpi/lib64/libopen-pal.so.40 (0x00007f6e0d358000)
<snip>
[ec2-user@ip-10-0-0-28 ~]$ ~/ompi5install/bin/mpirun -np 1 ldd ~/osu5/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_latency
<snip>
	libmpi.so.80 => /home/ec2-user/ompi5install/lib/libmpi.so.80 (0x00007f325d42a000)
<snip>
	libopen-pal.so.80 => /home/ec2-user/ompi5install/lib/libopen-pal.so.80 (0x00007f325c08d000)
<snip>

Summary:
I compiled OMB against both Open MPI 5.0.x and 4.1.4. Since no ABI break should have occurred, I expected to be able to run the 5.0.x mpirun to launch the 4.1.4 compiled OMB. However, this did not work. The root cause of the issue is that the MPI library used will be the one the application compiles against. I'm not convinced this is the behavior that we want to have.

[rhc54]

I think the problem is that prterun picks up the various "prefix" settings you push into the environment in your mpirun wrapper and is passing them down to the app. Since both of us use PMIx, that's where the intersection generates a problem.

The fix likely is in the PRRTE project - basically, omitting the "prefix" settings from those passed down to the application. Unfortunately, you might have a case where that needs to be done, so perhaps the correct answer is to define a new "application" prefix that is passed specifically down to the app and ignored by PRRTE? Or maybe the other way around if that helps with backward compatibility.

[ggouaillardet]

you must also make sure Open MPI was built with -enable-new-dtags otherwise the rpath used at build time takes precedence over $LD_LIBRARY_PATH

[jsquyres]

This docs section may be helpful: https://docs.open-mpi.org/en/v5.0.x/installing-open-mpi/configure-cli-options/rpath-and-runpath.html

[bwbarrett]

William should go verify that we did default to using runpath, but I'm pretty sure we did.

The larger issue is that when we rewrote the mpirun executable to deal with prrte, we didn't appear to add any of the code to have PRRTE set the LD_LIBRARY_PATH for applications when prefix is set (either explicitly or, in this case, because full path to mpirun was given), which means that that the linker is falling back to the runpath.

[wckzhang]

Looks like it's defaulted to rpath:

[ec2-user@ip-10-0-0-28 lib]$ readelf -d ./libmpi.so | egrep -i 'rpath|runpath'
 0x000000000000000f (RPATH)              Library rpath: [/home/ec2-user/ompi5install/lib]
[ec2-user@ip-10-0-0-28 lib]$ readelf -d /opt/amazon/openmpi/lib64/libmpi.so | egrep -i 'rpath|runpath'
 0x000000000000000f (RPATH)              Library rpath: [/opt/amazon/openmpi/lib64]

[bwbarrett]

That's... not awesome. Can you include your config.log?

[wckzhang]

From the compile command in config.log

  $ ./configure --prefix=/home/ec2-user/ompi5install --with-libfabric=/opt/amazon/efa/

Do you need the entire log?

[bwbarrett]

Yes, or at least the part when it is figuring out runpath vs. rpath. And probably throw in mpicc -showme as well.

[wckzhang]

[ec2-user@ip-10-0-0-28 ~]$ ./ompi5install/bin/mpicc -showme
gcc -I/home/ec2-user/ompi5install/include -pthread -L/home/ec2-user/ompi5install/lib -Wl,-rpath -Wl,/home/ec2-user/ompi5install/lib -Wl,--enable-new-dtags -lmpi

config.log

[wckzhang]

@rhc54 Hi Ralph, reading our mpirun shim code, it looks like we set the local libdir as an env used by prrte:

    /*
     * set environment variable for our install location
     * used within the OMPI prrte schizo component
     */

    setenv("OMPI_LIBDIR_LOC", opal_install_dirs.libdir, 1);

I can confirm that this libdir is the libdir of the mpirun, but it looks like LD_LIBRARY_PATH does not match on the compute nodes. Is PRRTE supposed to be using this env to set LD_LIBRARY_PATH or is there another way to propagate this information to PRRTE?

[bwbarrett]

I think that was supposed to be added (by OMPI developers, not Ralph), to the schizo framework. Maybe it never did?

[rhc54]

I see it there:

$ ack OMPI_LIBDIR_LOC ~/pmix/prrte
/Users/rhc/pmix/prrte/src/mca/schizo/ompi/schizo_ompi.c
289:    ompi_install_dirs_libdir = getenv("OMPI_LIBDIR_LOC");
$

and then I see it used to deal with jar files:

251:static char *ompi_install_dirs_libdir = NULL;
263:        asprintf(&str, fmt, app->app.argv[index], ompi_install_dirs_libdir, jarfile);
289:    ompi_install_dirs_libdir = getenv("OMPI_LIBDIR_LOC");
290:    if (NULL == ompi_install_dirs_libdir) {
310:            if (NULL == strstr(app->app.argv[i], ompi_install_dirs_libdir)) {
313:                    asprintf(&value, "-Djava.library.path=%s%s", dptr, ompi_install_dirs_libdir);
315:                    asprintf(&value, "-Djava.library.path=%s:%s", dptr, ompi_install_dirs_libdir);
326:        asprintf(&value, "-Djava.library.path=%s", ompi_install_dirs_libdir);
342:            value = pmix_os_path(false, ompi_install_dirs_libdir, "mpi.jar", NULL);
364:                value = pmix_os_path(false, ompi_install_dirs_libdir, "mpi.jar", NULL);
388:            value = pmix_os_path(false, ompi_install_dirs_libdir, "mpi.jar", NULL);
$

but nothing else.