Automatic custom holder type encapsulation

[RichardPat]

I apologise for the duplication of this question, I had already inquired about this on Gitter but with no success.

I've recently been experimenting with pybind with the intent to use it in wrapping a fairly substantial existing codebase. Certain methods, bound in python, will return objects that can potentially be concurrently destroyed in c++ during script execution. Smart pointers are not used in the c++ codebase, the objects are destroyed using delete on raw pointers. I was hoping to find a mechanic that would allow me to automatically encapsulate these objects, in python, in a container that would be notified upon object destruction and set its internal reference to None. I want to minimise as much as possible the need to modify existing methods to accommodate the binding process. The custom holder type described here appeared to fill this criteria but I can not get this to work as I desire. I think I haven't fully understood their purpose and/or underlying mechanics, so I'd be happy with any help regarding custom holder types or any other feature that might achieve the same result.

Here is my c++ code + binding declaration for testing the idea

template< typename T >
class SmartPointer{
public:
    SmartPointer(T * pointer)
        : m_Pointer(pointer) {
        pybind11::print("c++ - ", (size_t)(void*)this, "SmartPointer::SmartPointer(", (size_t)m_Pointer, ")");
        m_Pointer->DeleteEvent = [&] () {
            pybind11::print("c++ - ", (size_t)(void*)this, "Foo::DeleteEvent");
            m_Pointer = nullptr;
        };
    }

    ~SmartPointer(void) {
        pybind11::print("c++ - ", (size_t)(void*)this, "SmartPointer::~SmartPointer() -", (size_t)m_Pointer);
    }

    T * get(void) const {
        pybind11::print("c++ - ", (size_t)(void*)this, "SmartPointer::get() const ->", (size_t)m_Pointer);
        return m_Pointer;
    }

private:
    T * m_Pointer;
};

PYBIND11_DECLARE_HOLDER_TYPE(T, SmartPointer< T >);

class Foo {
public:

    Foo(const std::string & name)
        : m_Name(name) {
        pybind11::print("c++ - ", (size_t)this, " Foo::Foo(", name, ")");
    }

    ~Foo(void) {
        DeleteEvent();
        pybind11::print("c++ - ", (size_t)this, " Foo::~Foo() - ", m_Name);
    }

    const std::string & GetName(void) const {
        pybind11::print("c++ - ", (size_t)this, " Foo::GetName() - ", m_Name);
        return m_Name;
    }

    void SetName(const std::string & name) {
        m_Name = name;
        pybind11::print("c++ - ", (size_t)this, " Foo::SetName(", name, ")");
    }

    std::function< void (void) > DeleteEvent; // used to notify its custom smart pointer that this Foo instance has been destroyed

private:
    std::string m_Name;
};

void Destroy(Foo * foo) {
    pybind11::print("c++ - Destroy(", (size_t)foo, ")");
    delete foo;
}


PYBIND11_MODULE(test_pybind, module) {
    module.doc() = "pybind11 example plugin";

    pybind11::class_< Foo, SmartPointer< Foo > >(module, "Foo")
        .def(pybind11::init< const std::string & >())
        .def("GetName", &Foo::GetName)
        .def("SetName", &Foo::SetName);

    module.def("Destroy", &Destroy, "Destroys a Foo instance");
}

In the following python test code, I create a Foo instance and then destroy it, but python's reference still points to the instance, access to this reference does not use the get method of the custom holder type "SmartPointer".

from test_pybind import *
f = Foo("moo")
f.GetName()
Destroy(f)
f.GetName() # reference should be none but is dangling instead, get method not used

I have added a test project to illustrate this issue. TestPyBind11.zip

Any help or clarification would be greatly appreciated, many thanks in advance.

[jagerman]

The main issue here is that there's no notification mechanism from the class to the holder. Consider this example (which completely takes pybind out of the picture):

struct A {};

int main() {
    auto *a = new A();
    auto b = std::unique_ptr<A>(a);
    delete a;
    // ...
    // maybe other stuff happens
    // ...
    // Is b still valid?
}

The choices here are basically to use shared ownership (like std::shared_ptr); to let the holder manage the object (i.e. the delete a; above isn't allowed); to just hope that delete a hasn't been called; or to build some notification mechanism into the A class that lets you write some code to answer the // Is b still valid? part.

(This isn't really specific to holders at all--that is, if you had: auto *b = a;, you'd still have the issue that there's no way to tell whether *b is still pointing at a valid instance of A.)

[RichardPat]

@jagerman My understanding of your example is that b is a unique_ptr holder. I wish to define my own custom holder, SmartPointer in my contrived example, and in my example there is a notification mechanism between the class Foo and my SmartPointer holder. The Foo destructor calls its DeleteEvent() lambda function which in turn sets the SmartPointer's internal pointer to nullptr.

Destructor calling lambda notification function

    ~Foo(void) {
        DeleteEvent();
        pybind11::print("c++ - ", (size_t)this, " Foo::~Foo() - ", m_Name);
    }

Lambda function defined during SmartPointer construction

    SmartPointer(T * pointer)
        : m_Pointer(pointer) {
        pybind11::print("c++ - ", (size_t)(void*)this, "SmartPointer::SmartPointer(", (size_t)m_Pointer, ")");
        m_Pointer->DeleteEvent = [&] () {
            pybind11::print("c++ - ", (size_t)(void*)this, "Foo::DeleteEvent");
            m_Pointer = nullptr;
        };
    }

In the actual codebase I plan to bind, there is an event system that I can use to the same effect.

My issue is that I can not find a proper means of getting SmartPointer to mediate access in python. The SmartPointer get() method is never called. I have gone so far as casting the Foo pointer to a PyObject using pybind11::cast(), and then calling the release() method on the PyObject. Even after doing this, the python reference still dangles on the deleted object. My understanding is lacking on the python side of reference management. I have the notification mechanism but I do not have a means of controlling python references via a custom holder type.

[RichardPat]

I'm still searching, with little success, for a means of handling dangling python references after c++ code deletes the underlying object during script execution. In my use case Python can not be allowed to keep the object alive, and the script failing is acceptable so long as it does not pass the dangling pointer back into c++ code potentially causing the core program to crash.

I had been looking for a means to automatically encapsulate a bound class, using an event system to notify the encapsulating class that the internal object no longer exists and to set its value to None, but any approach that would allow me to neutralise these dangling python references would be greatly appreciated.

[ehanoc]

Any updates on this? I have similar requirements

[Jmbryan]

I am interested as well. I would like to use a custom weak-pointer as the holder, but this results in a dangling pointer in Python when the c++ object's strong ref count reaches zero.

[VaradDerivative]

I'm also in a very similar boat in trying to work through this issue of python holding on to a dangling references

In my application, I don't want to give ownership of my c++ class instance to python, I want it to remain with c++. But one thing that happens is, when I delete my c++ class instance, the pybind instance doesn't get notified that this has happened and continues pointing to deallocated memory.

My workflow involves using py::cast() to cast my c++ instance to a python instance, after wrapping it in my custom SmartPointer.

It seems like the smart pointer's .get() gets called only during py::cast(). I hoped that it would get called while my python class methods are actually being called, as at that time my Smart pointer can check if the instance is still alive.