One method of debugger attach broken

[jsquyres]

In master and v2.x, debugger attach for TotalView is currently broken. The problem is that when we upgraded to PMIx, we removed the OOB support from apps as it was no longer necessary. However, we currently send a message from mpirun to rank0 indicating that the debugger is ready, and therefore releasing rank0 to complete a barrier.

There are several ways of fixing this; there's ongoing discussion to pick the best one.

Just to summarize:

• stat: works (anything based on LaunchMon works)
• DDT: works
• TotalView: doesn't work

[hppritcha]

@rhc54 have you had a chance to look at this?

[rhc54]

I've looked at it and have been working on a fix, but no ETA for committing it.

[jsquyres]

Discussion from today's call...

There was a webex discussing this issue, and what to do about it. Two main options emerged:

1. Restore OOB functionality (including usock component).
2. Use PMIX notification system.

Restoring the OOB functionality for this one message seems like a lot of work, and it also seems like a step backwards.

The consensus seems to be to move forward and use the PMIx notification system (which means: finish implementing the PMIx notification system). @rhc54 is working on it. This will likely take a little time to finish and test. @gpaulsen thinks that IBM may be able to contribute some resources to help.

Note, too, that the PMIx notification stuff will be part of PMIx v1.2. Meaning: if we want TotalView attach to work in Open MPI v2.0.0 (as of today: we do), we'll need to update the v2.x branch with PMIx v1.

[gpaulsen]

From Telcon Call: https://github.com/open-mpi/ompi/wiki/WeeklyTelcon_20160112
Decided this is NOT a blocker for 2.0.0, but want fixed in next release.

[jsquyres]

When this issue is fixed, please also revert the change (on v2.x) from open-mpi/ompi-release#905.

[jsquyres]

Per discussion on 24 Feb 2016, moving this milestone back to v2.0.1. Rationale: it's a bug fix, and it does not affect our backwards compatibility promises for the 2.x series.