Move --host ordering fix to v3.0.x, v3.1.x, v4.0.x?

[jsquyres]

Per #6298, we had an accidental change in behavior of mpirun --host aaa,bbb between version v2.1.x and v3.0.x. A fix just went in to master in #6493.

Here's what happened:

• v2.0.x: behavior X
• v2.1.x: behavior X
• v3.0.x: switch to behavior Y
• v3.1.x: behavior Y
• v4.0.x: behavior Y
• master (to become v5.0.x): after PR Ensure that nodes are always used in order provided #6493, back to behavior X

The question is: should we put this fix on any of v3.0.x, v3.1.x, and/or v4.0.x?

Summary of behavior change

Behavior X

The ordering of hosts in the --host list matters:

$ mpirun --host aaa,bbb rank_test
aaa: MCW rank 0
bbb: MCW rank 1
$ mpirun --host bbb,aaa rank_test
aaa: MCW rank 1
bbb: MCW rank 0

Behavior Y

The ordering of hosts in the --host list does not matter (note: this behavior was unintentional. It was always intended that we honor the ordering of hosts in the --host list):

$ mpirun --host aaa,bbb rank_test
aaa: MCW rank 0
bbb: MCW rank 1
$ mpirun --host bbb,aaa rank_test
aaa: MCW rank 0
bbb: MCW rank 1

Discussion points

We need to discuss this and decide what to do. Points (in no particular order):

1. This is a fairly minor change in behavior.
2. Apparently no one noticed this change in behavior between v2.1.x and v3.0.x. It was only discovered recently by @bturrubiates, a Cisco employee (while using Open MPI for other / unrelated testing).
3. The fix is probably not worth putting into v3.0.x or v3.1.x.
4. But it might be worthwhile to put in to v4.0.x...?
5. That being said, even putting it in v4.0.x is at least sorta breaking backwards compatibility. You could squint at this and call it a bug and therefore allow it in. Or you could say that it was effectively the behavior of all the v3.x/v4.x releases, and they're backwards compatible with each other, so we should maintain that behavior in v4.0.x.

[jjhursey]

I think we noticed in PR #4327, but never got the cycles to get back to it. I'd like to verify this change also fixes that issue (I think it should)

[jsquyres]

Two good points were made on the webex:

1. We should be consistent across all of our releases. I.e., we should acknowledge that the v3.0.x and v3.1.x and v4.0.0 behavior was a bug, and was not intended (i.e., the v2.1.x behavior is the intended behavior).
   • Meaning: we should port these fixes back to v3.0.x, v3.1.x, and v4.0.x.
   • Meaning: the last releases of all of these series will make Open MPI's --host (and --hostfile) ordering semantics consistent across all of these release series.
2. Users who depend on the default ordering given by v3.0.0, v3.1.0, and v4.0.0 can still get that same ordering after the bug fixes are applied. The ordering they want may not be the default, but the desired order will definitely be obtainable.

With these arguments, it seems to make sense to back-port the commits from #6493 to v3.0.x, v3.1.x, and v4.0.x.

[jsquyres]

I attempted to make PRs for v3.0.x, v3.1.x, and v4.0.x. Unfortunately, master's nidmap.c has diverged quite a bit from each of these 3 branches. @rhc54 graciously said he would work on the ports for these branches.

[jsquyres]

#6508 is the PR for v4.0.x. It's becoming a bit of a bear, both in terms of size and complexity. We're working the issue, with the intent that it'll catch whatever v4.0.x train it can. After that, we can hopefully apply a similar back-port to v3.1.x and v3.0.x, and trigger new releases there, too (i.e., with the goal that we can close out the v3.x.y series with this fix).

[jsquyres]

Removed the "Target v2.x" label -- the fix is not needed for the v2.x series.