Problems with integer conversions (again)

[mrzv]

I have a new set of problems with integer conversions. It's not clear to me what's going on (and it requires another library to reproduce), but hopefully it will make sense to you.

You need mpi4py for this example.

Define a function through pybind11:

m.def("print_comm", [](int x) { std::cout << "Got comm: " << x << std::endl; });

Now test it from Python:

from mpi4py import MPI
w = MPI.COMM_WORLD
a = MPI._addressof(w)
print type(a)
# prints: <type 'int'>

print_comm(a)

The last command produces:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-7-16f94faa6020> in <module>()
----> 1 h.print_comm(a)

TypeError: Incompatible function arguments. The following argument types are supported:
    1. (int) -> None

I'm not sure how to debug this. If you have advice or any idea what's going on, I'd appreciate it.

[wjakob]

Hi Dmitry,

can you submit a reproducible example that does not depend on mpi4py? (Does it still happen if you bind a dummy function that just has the same function signature but no external dependencies?)

Also, it would be good to know what Python version this is.

Thanks,
Wenzel

[mrzv]

Hi Wenzel,

Everything I describe was under Python 2.7.11 on OS X (El Capitan). I'll try to create a stand-alone example. I don't quite understand what you mean by "binding a dummy function that just has the same function signature but no external dependencies"?

Dmitriy

[wjakob]

Never mind that last bit, I realize now that it did not make sense.

[mrzv]

Hm, so it seems Python reports the type of a incorrectly. It's actually a void*, which ought to be a long I think. If I change print_comm to accept either void* or long everything works — well, this part of the code anyway.

In short, it's not a pybind11 issue, so this can be closed. Sorry for the hassle.

[wjakob]

no problem.

[mrzv]

Actually, there is something else I don't understand. Maybe you have an idea what's going on.

What _addressof returns is the result of PyLong_FromVoidPtr(ptr) from some void* ptr.

If I change my function to accept void*, it compiles and runs, but the pointer that I get is clearly garbage. If I change it to accept long, and then do a reinterpret_cast<void*>, then I get the right pointer.

Any idea why this would be happening? Is there a conversion missing somewhere?

[wjakob]

Not 100% sure -- I can take a look if you could submit a self-contained example that demonstrates the issue.

[mrzv]

Ok, I'll try to cook one up.

[mrzv]

Actually, I didn't have to dig deep. The following already creates a problem (Python 2.7.11, OS X El Capitan):

    m.def("get_val",     [&m]()        {
                                          void* x = &m;
                                          std::cout << x << std::endl;
                                          py::handle h = PyLong_FromVoidPtr(x);
                                          return h;
                                       });
    m.def("print_val",   [](void* x)  { std::cout << x << std::endl; });
    m.def("print_val2",  [](long x)   { std::cout << reinterpret_cast<void*>(x) << std::endl; });

x = m.get_val()
m.print_val(x)
m.print_val2(x)

I get:

0x7fff52c5ca98
0x7fff52c5d0c0
0x7fff52c5ca98

It's also worth noting that it's impossible to return a void* directly from a function, but that's probably just a matter of adding a conversion.

[wjakob]

Minor thing: better to be explicit about borrow status, or you may get reference count leaks.

py::object h = py::object(PyLong_FromVoidPtr(x), false);
return h;

[mrzv]

Ah, good to know. Thanks.

[wjakob]

I've committed a fix which disallows declarations of the form

m.def("print_val",   [](void* x)  { std::cout << x << std::endl; });

(i.e. void* as a function argument)

Not knowing the type of an argument is a bottomless pit of trouble with regards to casting and reference counting, so the way to go will be to pass longs.

[mrzv]

I obviously don't understand all the nuances, but does it not make sense to add casts for void* to long when returning (using PyLong_FromVoidPtr) and from long to void* when accepting (using PyLong_AsVoidPtr). In some sense, the correct version of print_val2 should be:

    m.def("print_val2",  [](py::object x)   { std::cout << PyLong_AsVoidPtr(x.ptr()) << std::endl; });

Maybe this conversion could be done implicitly if the argument is void*? It feels dirty from the Python point of view, but it's probably the cleanest way to exchange data between C codes exposed to Python in different ways.

[ax3l]

@mrzv do you have a final snippet how you pass an mpi4py communicator (for both MPICH (int) and OpenMPI (void*)) to pybind11? I am a bit puzzled how to implement this gently in the end.

Update: went for openPMD/openPMD-api#454

struct openPMD_PyMPICommObject
{
    PyObject_HEAD
    MPI_Comm ob_mpi;
    unsigned int flags;
};
using openPMD_PyMPIIntracommObject = openPMD_PyMPICommObject;

// in .def(py::init([](py::object &comm){
// several checks on comm...
MPI_Comm* mpiCommPtr = &((openPMD_PyMPIIntracommObject*)(comm.ptr()))->ob_mpi;

(note that MPI_Comm is unfortunately implementation dependent, so compile in one consistent stack)