OSHMEM yoda spml failures: need to update to BTL v3.0

[jsquyres]

Cisco just added OSHMEM testing to its MTT 2 weeks ago (at the Dallas engineering meeting).

We're seeing a large failure rate on v2.x with OSHMEM testing using TCP,vader,self. For example: https://mtt.open-mpi.org/index.php?do_redir=2347

This shows 1,624 failures and 6,546 passes. I.e., a nearly 20% failure rate. 😱

Many of the failures show this kind of error message:

[mpi006:31821] Error base/memheap_base_mkey.c:162 - memheap_attach_segment() tr_id: 1 key 54ba0015
attach failed: errno = 12

Does anyone know what this means?

@artpol84 @jladd-mlnx

[jsquyres]

I also see these kinds of errors:

[mpi006:18246] Error spml_yoda.c:822 - mca_spml_yoda_put_internal() src=0xff0000d8 nfrags = 1 frag_size=-1
[mpi006:18246] Error spml_yoda.c:823 - mca_spml_yoda_put_internal() shmem OOM error need 262144 bytes
--------------------------------------------------------------------------
'Put' operation failed. Unable to allocate buffer, need 262144 bytes.
Try increasing 'spml_yoda_bml_alloc_threshold' value or setting it to '0' to
force waiting for all puts completion.

  spml_yoda_bml_alloc_threshold: 3
---------------------------------------------------------------------------

[jsquyres]

Am I running these tests wrong? In most cases, the tests are run with 32 procs across 2 nodes (16 cores each); each node has 128GB RAM.

[jsquyres]

@artpol84 @jladd-mlnx @igor-ivanov Any advice here?

[hppritcha]

I'll do some testing, I'd like to see if yoda is busted over other BTLs - like gni.

[hppritcha]

Since open shmem 1.3 compliance is going to one of the major features of the 2.1 release, we want for oshmem on as many system configs as possible. Marking this as a blocker.

[mike-dubman]

@alex-mikheev - could you please comment.

[jladd-mlnx]

@hppritcha @jsquyres We don't maintain Yoda at all and it's very likely that many of the tests fail because (some of) the BTLs can't make asynchronous progress or do true one-sided RDMA. MXM and UCX have all of these features. I would suggest we replace Yoda SPML entirely and move to UCX. There is support for multiple transports and eliminates the BTL mess altogether. UCX will be in a near GA state come late October. We also proposed rolling UCX into OMPI some time ago, perhaps this provides further motivation to do so.

[jsquyres]

One of the requirements for OSHMEM to come into the Open MPI code base was that it needs to be able to handle all network types. AFAIK, UCX does not handle all network types (e.g., portals, usNIC). As such, Yoda needs to be fixed before v2.1.0 can be released.

[jladd-mlnx]

I don't know if it's possible to fix it for TCP BTL. We have no knowledge here. This is not a regression.

[jladd-mlnx]

How does this look on 1.10.3?

[hppritcha]

I did some spot checking on a Cray XE and get "registration errors" for what looks to be the BSS export portion if I try to use 8 or more PEs:

[nid00060:07656] base/memheap_base_alloc.c:38 - mca_memheap_base_alloc_init() Memheap alloc memory: 270532608 byte(s), 1 segments by method: 2
[nid00060:07656] base/memheap_base_static.c:209 - _load_segments() add: 00601000-00602000 rw-p 00001000 00:0e 25175784                           /var/opt/cray/alps/spool/18132313/barrier_performance.x
[nid00060:07656] base/memheap_base_static.c:75 - mca_memheap_base_static_init() Memheap static memory: 160 byte(s), 2 segments
[nid00060:07656] base/memheap_base_register.c:39 - mca_memheap_base_reg() register seg#00: 0x0xff000000 - 0x0x10f200000 270532608 bytes type=0x2 id=0x39E38006
[nid00060:07656] Error spml_yoda.c:439 - mca_spml_yoda_register() ugni: failed to register source memory: addr: 0xff000000, size: 270532608
[nid00060:07656] Error base/memheap_base_register.c:131 - _reg_segment() Failed to register segment

I don't think this is a BTL specific issue.

EDIT: Added verbatim

[jladd-mlnx]

@hppritcha, can you try on 1.10.3? We don't have access to a Cray.