[mpi4py] Regression with ompi@main

[dalcinl]

https://github.com/mpi4py/mpi4py-testing/actions/runs/7294717237/job/19880980395

...
testConstructor (test_comm_inter.TestIntercomm.testConstructor) ... testConstructor (test_comm_inter.TestIntercomm.testConstructor) ... testConstructor (test_comm_inter.TestIntercomm.testConstructor) ... ok
ok
ok

***
An unrecoverable error occurred in the gds/shmem component.
Resolve this issue by disabling it. Set in your environment the following:
PMIX_MCA_gds=hash
***
: Success
/home/runner/work/_temp/a134b824-0923-4f27-82ae-d86582046489.sh: line 1: 144454 Aborted                 (core dumped) mpiexec -n 3 python test/main.py -v -f
Error: Process completed with exit code 134.

[dalcinl]

@rhc54 I'm also getting this output at the beginning of my runs:

--------------------------------------------------------------------------
    This help section is empty because PRRTE was built without Sphinx.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
We attempted to remove a file, but were unable to do so:

  Path:  /tmp/prte.fv-az1543-731.1001/dvm.143695
  Error: Is a directory
--------------------------------------------------------------------------

[wenduwan]

I also see similar warnings in my local run on main branch

We attempted to remove a file, but were unable to do so:

  Path:  /tmp/prte.fv-az1543-731.1001/dvm.[14](https://github.com/mpi4py/mpi4py-testing/actions/runs/7294717237/job/19880980395#step:20:15)3695
  Error: Is a directory

[dalcinl]

@wenduwan I believe the regression comes from #12179, the last good mpi4py run was from commit e6a294b.

[rhc54]

I see how to disable that warning - Linux does something not POSIX compliant, but I can cover it. I'm afraid there isn't enough info about the primary error being reported here for me to do anything about it.

[rhc54]

Here's the warning fix: openpmix/openpmix#3252

[dalcinl]

Here's the warning fix: openpmix/openpmix#3252

Many thanks, Ralph.

I'm afraid there isn't enough info about the primary error being reported here for me to do anything about it.

Any chance that the error was actually related to the fix you pushed?
Any tips about how to get more verbose output that would help you?
If it is not much work, could you please setup a spare branch with updated submodule pointers in your personal ompi repository such that I can test your PMIx fix?

[rhc54]

Can't say - you haven't yet told me what the error is 😉

[dalcinl]

The error is in the description of this issue. "An unrecoverable error occurred in the gds/shmem component. ..." Looks like this happens while trying to construct an intercommunicator with MPI_Intercomm_create.

This happens when running with oversubscription (GitHub Actions runners have 2 virtual cores, and I'm running with 3 MPI processes). Maybe oversubscription has nothing to do with the error, and the issue triggers just because I'm using a small odd number of processes or some corner case like that. Unfortunately, I'm not able to run this locally right now.

[rhc54]

Okay, I wasn't sure if that was the error you were talking about. Hard to figure what could cause that as the procs would all have been started before the intercomm create. What does your test actually do - have a proc that spawns more procs and then tries to create a communicator between them? Is the error coming from one of the spawned procs - or all of them?

Once I know what your test does and where the error comes from, I can try to replicate it here. I doubt it has anything to do with the oversubscription.

[rhc54]

I see where the error is coming out and why - what I don't understand is what is triggering it. I'm not able to reproduce the problem here when running a simple test that does a spawn and then creates the intercommunicator.

Might help to also know what environment your CI is running.

I could disable the shmem component or revert the commit that added the code behind the error report. However, the person who supports that component may be out this week and I'd prefer to get his input on it. We have a meeting Thurs - I'll raise it with him if he attends.

[dalcinl]

The failing test may actually be one involving MPI_Intercomm_create_from_groups from MPI-4. Sorry I cannot be more assertive about this. As I said before, I'm not able to run things locally right now.

The MPI_Intercomm_create_from_groups API is usually used with sessions. However, this is not what my test does. The test steps are.

1. MPI_Comm_split the WORLD comm in two halves to get intracomms with disjoint groups.
2. Create a new intercomm1 connecting the two disjoint intracomms with MPI_Intercomm_create.
3. Get the the local and remote groups of intercomm1.
4. Create a new intercomm2 from the local and remote groups with MPI_Intercomm_create_from_groups.
5. Check that intercomm1 and intercomm2 are congruent with MPI_Comm_compare.

If you can read some slightly convoluted Python, the test code is split in two hunks here and here.

PS: This particular test has had regressions at least two or three times in the past after PMIx submodule pointer updates. Looks like Open MPI folks should add a test into their suite to prevent this issue to resurfacing over and over.

[dalcinl]

Might help to also know what environment your CI is running.

GitHub Actions with GitHub-hosted runners running their Ubuntu 22.04 image.

... I'd prefer to get his input on it. ...

Yes, better to wait for the expert and get to the bottom of it.

[rhc54]

@hppritcha @samuelkgutierrez This may be hard to reproduce as it sounds like quite a convoluted process. Since it might have something to do with PMIx (it isn't immediately clear to me how PMIx is involved here, nor why gds/shmem would impact it), it would be good if we could distill from the test the specific PMIx calls (and their order) being made so we could create a PMIx-only model of it.