Move PREDEFINED_COMMUNICATOR_PAD back to 512

[jsquyres]

We want v4.x to be ABI compatible with v5.x.

Per #11365 (comment), it looks like MPI Sessions (#9097) increased PREDEFINED_COMMUNICATOR_PAD from 512 to 1024, thereby breaking ABI.

Unfortunately, it looks like sizeof a communicator struct is over 512 bytes these days. If we want to preserve 4.x and 5.x ABI, we will need to shrink this down somehow. Brian suggests moving some of the communicator data out into a secondary location that can be accessed through a pointer.

[edgargabriel]

I instrumented ompi and ran a few tests, and I am not convinced that increasing the predefined_communicator_pad from 512 to 1024 bytes was necessary. Here are the results of some tests with 4.1.4 and main:

in 4.1.4:

sizeof ompi_communicator_t :  352 bytes

in main:

sizeof ompi_communicator_t without ft support:  424 bytes
sizeof ompi_communicator_t with ft support: 448 bytes
sizeof ompi_communicator_t with ft + peruse: 456 bytes (estimated, I do not have peruse, but its one additional pointer)

I can run some tests changing the predefined_communicator_pad back to 512 and see whether I run into some problems.
Otherwise, if we would need to save a few bytes, the easiest option might be to convert the c_name from

 char  c_name[MPI_MAX_OBJECT_NAME];

(which is 64 bytes), to

char *c_name

and allocate the element dynamically. Its not something in the critical path.

[edgargabriel]

I ran a bunch of tests setting ompi_predefined_communicator_pad to 512, and haven't observed any issues.

[edgargabriel]

An update, @jsquyres suggested I compile ompi also with --enable-debug, and that did in fact put us over the limit:

sizeof ompi_communicator_t without ft: 512 bytes
sizeof ompi_communicator_t with ft : 536 bytes

Even if we add a pointer for peruse (8 bytes), I think the solution of making c_name a dynamic array should put us under the 512 bytes limit. I will try to implement that in the next 1-2 days

[bosilca]

I reviewed the communicator structure, and cleaned up the many unnecessary things that made their way in. As a result, I was able to bring it down to 520 bytes, in what I think is the worst case (ULFM + debug + peruse). We're still above the 512 limit, and without applying @edgargabriel suggestions I don't see how we can bring this further down.

I will make a PR out of my patch, and together with #11405 this will give us a clean slate to work from.

[edgargabriel]

@bosilca do you want to file your pr maybe against #11405 to simplify review and the cherry-picking for the 5.0 branch?

[bosilca]

@edgargabriel I was able to push a commit in your branch. The result is that sizeof(ompi_communicator_t) is at most 464 bytes in debug mode with PERUSE and ULFM enabled.

More than half of this, precisely 272 bytes, is accounted on two structures, that are mostly unused: the opal_infosubscriber_t with 144 bytes and opal_mutex_t with 128 bytes.