Attempt to free memory that is still in use by an ongoing MPI communication

[artpol84]

We observed this error only once in our MTT. We are running v2.x with SLURM/pmix there. It is possible that it is somehow related to this configuration, though I doubt that.
Here is the error message:

export OMPI_MCA_btl_openib_if_include=mlx4_0:1
OMPI_MCA_btl_openib_if_include=mlx4_0:1
OMPI_MCA_mpi_add_procs_cutoff=0+ OMPI_MCA_pmix_base_async_modex=1
OMPI_MCA_pmix_base_collect_data=0
/tmp/mtt_116453_slurm/bin/srun -N 8 -n 64 --mpi=pmix_v1 -p pmellanox <mtt-base>/installs/T8JL/tests/mpich_tests/mpich-mellanox.git/test/mpi/coll/allgather2
[boo13:10605] Attempt to free memory that is still in use by an ongoing MPI communication (buffer 0xa89000, size 9302016).  MPI job will now abort.
srun: error: boo13: task 9: Exited with exit code 1
srun: Terminating job step 1592.0

[artpol84]

Update: this has triggered today again.
First time I noticed that was on Thursday, 03/30/2017.

[artpol84]

I always observe this with allgather2 test (but it's not 100% reproducible), so in conjunction with the time it first appeared and the commit history, I can say that this may be related to #3159

[artpol84]

@vspetrov

[hppritcha]

These MTT runs were done with vanilla Open MPI, no UCX, MXM, etc.

[gpaulsen]

From web-ex 04/11 - Not 100% reproducible, so lets keep this issue open a bit longer.

[artpol84]

still reproducible:
recent mtt report

[jsquyres]

Artem -- don't forget that you can click on the "Absolute date range" link on the right in MTT and get a short link: https://mtt.open-mpi.org/index.php?do_redir=2412

[artpol84]

Thanks, I didn't know that: https://mtt.open-mpi.org/index.php?do_redir=2412

[jsquyres]

@artpol84 Where are we on this issue?