ARMCI-MPI hangs when datatypes are used

[jeffhammond]

See https://travis-ci.org/jeffhammond/armci-mpi/builds/99887902 for full details. ARMCI-MPI passes all the tests with MPICH 3.2. When I use Open-MPI 1.10, tests pass when I disable user-defined datatypes (e.g. MPI_Type_subarray), but otherwise fail.

[ggouaillardet]

@jeffhammond 1.10.2rc3 fixes bugs (missing resize) for subarray and darray.
could you give it a try ?

[jeffhammond]

@ggouaillardet It does not appear to help: https://travis-ci.org/jeffhammond/armci-mpi/jobs/99913181. While it has not failed yet, it's almost certainly hung, since successful builds always finish in under 10 minutes.

ARMCI tests (armci-perf and armci-test) were written by PNNL and take liberties with ISO C and even the ARMCI interface (which is easy to do when there is no proper specification), so it is possible that Open-MPI is not at fault, but MPICH is pretty strict about RMA and it has no problem here. I guess I will start debugging locally if none of the Open-MPI devs knows a reason why it would hang.

I guess the other reason to blame it on an Open-MPI datatypes bug is that without user-defined datatypes, the Open-MPI builds pass.

[ggouaillardet]

I confirm the hang even with master.
The test is made of two tasks, and they are doing at the same time into the other task
MPI_Accumulate(MPI_REPLACE)
MPI_Get
MPI_Win_flush_all

I always get a hang on one node with --mca btl tcp,self
It occurs at various places, but when datatype is a subarray of dimension 5 or more.

@hjelmn can you please confirm this should work, even without a fence between accumulate and get ?
Note if i replace MPI_Accumulate with MPI_Put, i got the same hang.

I initially suspected ordering is not correctly enforced, but i cannot make sense of the wrong results i am seeing... so that could indicate something trickier

[ggouaillardet]

my bad, there is no MPI_Win_flush_all at all

[jeffhammond]

@ggouaillardet Yes, there is. ARMCI_AllFence maps directly to MPI_Win_flush_all.

Can you try running with each of the environment variables:

• ARMCI_NO_FLUSH_LOCAL=1? This turns MPI_Win_flush_local calls inside of all the ARMCI blocking operations into MPI_Win_flush.
• ARMCI_RMA_ATOMICITY=1 This turns MPI_Get into MPI_Get_accumulate(MPI_NO_OP) and MPI_Put into MPI_Accumulate(REPLACE).
  I will also try with these...

I will talk to Pavan and Jim about whether or not our default implementation is fully compliant with the MPI standard or if we are are unintentionally relying upon MPICH's behavior (Ch3 is an ordered channel) for correctness.

[ggouaillardet]

the main loop is the for loop starting at line 387, and unless ARMCI_{PutS,GetS} invoke a fence under the hood, I could not find any fence.
I will try flush_local=1
by default, and iirc, rma_atomicity is 1 when MPI data types are used.
I will also rewrite the test_dim test as a pure MPI program (e.g. no more armci) and compare its behavior openmpi vs mpich

[jeffhammond]

All ARMCI blocking operations - ARMCI_{Put,Get,Acc}{,S,V} and ARMCI_Rmw call MPI_Win_flush_local internally.

You are right. ARMCI_RMA_ATOMICITY defaults to 1 in all cases. In this case, we are using MPI accumulate operations for all communication, and these are specified by MPI-3 to be ordered unless the relevant info key is set, and we do not set any info keys related to ordering in ARMCI-MPI.

[jeffhammond]

@ggouaillardet Travis shows ARMCI_NO_FLUSH_LOCAL=1 fixes this. This seems to suggest that Open-MPI does not order RMA by default.

[ggouaillardet]

@jeffhammond i found some other race conditions that can be evidenced by running with --mca btl tcp,self
and it runs fine even with MPI_Win_flush_local

i rewrote the test as a pure MPI app, it is available at https://gist.github.com/ggouaillardet/ae606f6ff90c501f4123

@hjelmn i made #1278 as a proof of concept.
the issue occurs when accumulating "long" message (e.g. ompi_osc_pt2pt_frag_alloc fails, which occurs at random places)
under the hood, MPI_Accumulate and MPI_Get_accumulate ended up using the same tags for isend/irecv. i fixed that by using two distinct "namespaces" for tags.
it is very likely similar fixes should be made in other places.

[jeffhammond]

@ggouaillardet

Excellent work!

I'd love it if you would submit a PR to add the pure MPI version of that test to the ARMCI-MPI test suite. We have MPI-only tests for MPI-3 RMA feature support already, but they are pretty modest.

[jeffhammond]

I have not observed this issue in many years and OMPI 5.0 appears to have fixed all of the RMA datatype issues.