Add upcast_hook for exposing non-primary base relationships

[oremanj]

I think this is the minimal internals change that would allow users to roll their own emulation of some multiple inheritance semantics. Maybe it's still more catering to MI than you're interested in, but I figured I'd give it a shot.

[wjakob]

Hi @oremanj,

While indeed very compact, I am not in looking forward to include a feature to specifically enable multiple inheritance (or approximations thereof).

However, there is another feature that I am looking to add at some point. And it might be possible to implement this feature in such a way that it subsumes both use cases, so I thought it might be good to discuss it here. Specifically, I would like to add a compatibility layer that enables interoperability between pybind11 and nanobind-based bindings. (E.g. a nanobind function accepting an instance of a type that has been bound in pybind11). This, too, requires some kind of hook to check type compatibility and return a pointer, which seems quite similar to what you are doing here.

A first small step towards this is already done: both nanobind and pybind11 (on master) switched to a new and more relaxed ABI tag that is now consistent between those two projects (so binding layers aside, we can check if conversion of an instance from one framework to the other risks a crash).

The next step will be to add a hook to bindings that extracts the C++ pointer associated with a Python object, e.g. a callback with signature

void* (*caster)(PyObject *obj, const char *abi_tag, const std::type_info *target);

Since we can't stash this in the type object (pybind11 doesn't have any scratch space there), this might need to go into a static capsule or similar, e.g. bound_class.__abi_support__ (needing an extra dictionary lookup, which is a bummer but I can't think of a better way). I think that the frame of reference (i.e. class where this would need to be installed) is flipped compared to your approach. Do you have any thoughts/suggestions regarding this?

[oremanj]

Storing the foreign cast hook on the Python instance seems like a good convention if you only need to support from-Python conversions. If you need to support to-Python conversions (or implicit conversions in from-Python), you need a registry of some sort in order to be able to look up the converter by its C++ type. That's of course more complex and heavier-weight in terms of code size, though plausibly faster at runtime.

It looks like pybind did something similar to your proposal pybind/pybind11#5296 but theirs is defined as a Python callable rather than a capsule. The capsule is certainly more "nano" and it would be nice if both projects could agree on a unified approach there.

I have a local implementation of the registry-based solution for nanobind/pybind interop specifically, which I'd be happy to clean up and upload if it's of interest. It involves a structure

struct foreign_caster_hooks {
    // Context pointer to be passed in calls to from_python/to_python;
    // in pybind this is a detail::type_info pointer
    void* context;

    // Pointer to the Python type object that this C++ type maps to.
    // This is used mainly to allow determining the Python type name.
    PyTypeObject* pytype;

    // Extract a C++ object of this type from `pyobj`. Return a
    // pointer to it, or null if not possible. If `convert` is true,
    // be more willing to perform conversions. If `keep_referenced` is
    // not nullptr, then `from_python` may make calls to
    // `keep_referenced` to request that some Python objects remain
    // referenced until the returned C++ object is no longer needed.
    // `keep_referenced` should incref its argument immediately and
    // remember that it should be decref'ed later, for no net change
    // in refcount.
    void* (*from_python)(void* context, PyObject* pyobj, bool convert,
                         void (*keep_referenced)(PyObject*) noexcept) noexcept;

    // Wrap `cppobj` into a Python object using the given return value
    // policy. `parent` is relevant only if rvp == reference_internal.
    // rvp must be one of take_ownership (2), copy (3), move (4), reference (5),
    // reference_internal (6), none (7); automatic and automatic_reference are
    // not allowed. Returns null with the Python error indicator potentially set
    // if the cast is not possible, or a new reference otherwise.
    PyObject* (*to_python)(void* context, void* cppobj, uint8_t rvp,
                           PyObject* parent) noexcept;

    // Request that a PyObject reference be dropped, or that a callback
    // be invoked, when `nurse` is destroyed. If `cb` is nullptr, then
    // `payload` is a PyObject* to decref; otherwise `payload` will
    // be passed as the argument to `cb`. Returns true if successful,
    // false and sets the Python error indicator on error.
    bool (*keep_alive)(PyObject* nurse, void* payload,
                       void (*cb)(void*) noexcept) noexcept;
};

along with "import/export foreign type" functions defined in each binding library that accept or produce copies of this structure. You ask the native library for a certain type to export hooks for it, and then pass them to the import side of the other library/ies you want to use it with. Each library's internals structure has a map from C++ type to foreign_caster_hooks, which it consults when it would otherwise be about to fail a conversion. There are some other details that arise around exception translation and synchronizing the full list of types between the libraries (so you don't have to write a separate import/export statement for each one). We've been using this in production for a good year now and it's allowed us to convert some huge extension modules from pybind to nanobind pretty much one compilation unit at a time, with relatively few surprises. I don't think the capsule/conduit approach could do that, although it seems like it would be far easier to implement and would probably work great for simple cases.

I've been working on refining the registry-based approach some and I think it's plausible to support automatic cross-library (including cross-ABI-version-of-same-library) type discovery and exception translation with reasonable performance and with almost no overhead when foreign types are not used. I'm reluctant to put too much more effort in without knowing what level of solution the relevant library maintainers are interested in though.

[wjakob]

Dear @oremanj,

this sounds really exciting! I think there is a general question of how such a feature could be incorporated. Failing a type cast is a pretty common situation, so it's good if extra logic in there does not unduly slow down nanobind. If the extra code path is costly, then it might be necessary to place the extra code into an optional NANOBIND_ENABLE_INTEROP mode flag or similar.

But of course this is an extra flag which is nice to avoid. Therefore I'm also really curious to hear what you mean by "type discovery and exception translation with reasonable performance and with almost no overhead when foreign types are not used" (i.e. how such a mechanism could be added with almost no overhead).

pybind11 is currently facing significant ABI incompatibility issues (as in ABI incompatibility with itself, there were 2 ABI bumps and another one is coming up). The "conduit" feature was intended to smooth over such transitions but was found to be insufficient. The main problem is that it only handles the from_python direction. I am pretty sure that better interop solution would be very well received, especially if it also addresses interop in a more general sense.

[wjakob]

Regarding the registry approach: this does make sense to me and seems necessary if to_python is to be supported. I have a question regarding this:

Each library's internals structure has a map from C++ type to foreign_caster_hooks, which it consults when it would otherwise be about to fail a conversion.

What is the advantage compared to a Python dictionary with std::type_info::name() as keys? Faster lookups? How is the library internals data structure updated when a new type is bound by another binding library?

[oremanj]

I'm glad to hear this would be of interest!

Therefore I'm also really curious to hear what you mean by "type discovery and exception translation with reasonable performance and with almost no overhead when foreign types are not used" (i.e. how such a mechanism could be added with almost no overhead).

The basic idea is that if you only have one binding library in the address space, or if you haven't told it to go look for other libraries' types, then the extra logic when you're about to fail a cast is just testing a flag ("any foreign types?"). If you have enabled the interop at runtime then it will check for new types it doesn't yet know about (one word read) and, assuming there are none, do a map lookup along the lines of type_c2p_fast/type_c2p_slow. The map lookup is not costless but it's low-cost and only kicks in if there are actually other binding libraries to interoperate with.

What is the advantage compared to a Python dictionary with std::type_info::name() as keys? Faster lookups?

You could use such a dictionary if you want. I figure each binding library already has its own preferred way of looking up information from a C++ type (such as nanobind's combination of type_c2p_fast and type_c2p_slow) which might be faster than a Python dictionary in expectation.

How is the library internals data structure updated when a new type is bound by another binding library?

This is the part still under development -- in the fully-implemented version we just call an update_cross_casts() function which inspects both binding libraries' internals structures for new types since the last call and tells them about each other, but that wouldn't scale well to >2 binding libraries. The idea I'm working on is to have a separate "interop internals" structure (of a format commonly understood by all participating binding libraries, and unlikely to need to ever change) which holds a linked list of framework records (one structure per binding library ABI version in the address space) with each framework holding a linked list of type records. The framework/type records are linked intrusively so it's up to the individual framework whether to store them inline (e.g. in nb::detail::type_data) or in separate storage. When binding a new type, you append it to your own framework's list of type records and then set a flag in the structure for every other framework saying that you added a new type. Each framework knows the last type it's seen from each other framework, so can quickly process just the new portion of the linked list. There's some logic for avoiding dangling if a type is destroyed.

I'll let you know when I get to the point of uploading something usable!

[wjakob]

I'll let you know when I get to the point of uploading something usable!

This sounds great! If there is such a registry that lives outside of nanobind/pybind11, then it seems to me that this would either require:

• A specification
• An implementation, whose implementation is the spec (preferable if the spec is too complex for prose). Ideally this would be C (not C++) to make it more broadly accessible.

Thinking a bit more about this..

I figure each binding library already has its own preferred way of looking up information from a C++ type (such as nanobind's combination of type_c2p_fast and type_c2p_slow) which might be faster than a Python dictionary in expectation.

This is an excellent point. I think we basically don't want the registry to implement any fancy data structures. It should just organize what's available and notify binding libraries if something changed. So this can be just a simple list together with some callback mechanism. Here is an idea on what such an API could look like (cheesily namedafter the Rosetta stone)

#if defined(__cplusplus)
extern "C" {
#endif

struct RosettaBinding;
struct RosettaCallback;

/*
 * The registry holds bindings and callbacks to notify other frameworks about
 * potential changes. It is protected by a mutex in free-threaded builds, and by
 * the GIL in regular builds.
 *
 * The pointer to the registry is stored in some place where extension
 * frameworks can access it while being hidden from Python "userspace"
 * (e.g. PyInterpreterState_GetDict). .. and the link to the `RosettaRegistry`
 * can be stored under some name that bakes in the new
 * inter-pybind11-nanobind relaxed ABI name
 * (`NB_PLATFORM_ABI_TAG` in `src/nb_abi.h`). 
 */
struct RosettaRegistry {
#if defined(Py_GIL_DISABLED)
    PyMutex mutex;
#endif
    RosettaBinding *bindings;
    RosettaCallback *callbacks;
};

/*
 * List of type bindings and supported functionality. This is a C doubly linked
 * list to provide a lowest-common-denominator ABI and permit cheap addition
 * and removal. Each framework *owns* its associated records and is, e.g.,
 * responsible for eventually deallocating 'RosettaBinding::name'. After
 * unregistering a type, it should call the associated removal callabck.
 */
struct RosettaBinding {
    const char *name;

    /* std::type_info pointer and mangled name */
    const void *type;
    const char *type_name_mangled;

    /* copied from your code, I changed 'bool' -> uint8_t */
    void* (*from_python)(void* context, PyObject* pyobj, uint8_t convert,
                         void (*keep_referenced)(PyObject*));

    PyObject* (*to_python)(void* context, void* cppobj, uint8_t rvp,
                           PyObject* parent);

    /* potentially more stuff here */

    /* doubly linked list */
    struct RosettaBinding *prev, *next;
};

/* Conventions: callbacks provide the framework with the opportunity to
 * update internal data structures based on newly added or removed types.
 *
 * They should only be called while RosettaRegistry::mutex or the GIL is held.
 *
 * The callee should not perform Python C API calls, and it should not
 * modify the registry while executing the callback. */
struct RosettaCallback {
    void (*notify_added)(RosettaBinding *binding);
    void (*notify_removed)(RosettaBinding *binding);
    RosettaCallback *prev, *next;
};


#if defined(__cplusplus)
};
#endif

[oremanj]

An implementation, whose implementation is the spec (preferable if the spec is too complex for prose).

Yep, this is my plan.

Ideally this would be C (not C++) to make it more broadly accessible.

I considered this initially, but ran into the fact that a C library would have difficulty specifying the RosettaBinding::type in a way that would be recognizable to C++ libraries that want to look up a std::type_info.

Still, we might be able to figure out a C structure that's sufficiently ABI-compatible with type_info for our purposes. We only need the name/pointer, it doesn't have to support dynamic_cast or anything. This is fairly easy on the Itanium ABI:

struct type_info_vtable {
    unsigned long offset_to_top;
    struct type_info *type;
    void (*destroy)(struct type_info*);
    void (*destroy_and_delete)(struct type_info*);
};

struct type_info {
    void *vptr;
    const char *name;
};

static struct type_info_vtable ti_vtable;
static struct type_info ti_ti = {&ti_vtable.destroy, "11c_type_info"};
/* expect no virtual dtor calls */
static struct type_info_vtable ti_vtable = {0, &ti_ti, NULL, NULL};

I don't know enough about the MSVC ABI to make it portable to that. It looks somewhat more involved but still possible.

The name "Rosetta" is appealing but it collides with https://www.pyrosetta.org/ and with Apple's dynamic binary translation tool. I've been using "pymetabind" so far which is less fun but probably won't collide with anything...

[wjakob]

pymetabind sounds good too! Producing a std::type_info record might be difficult for a C-only extension (though in principle it is definitely possible using approaches like the one you demonstrated above). What a C interface would already enable is for a C binding library to reuse bindings made by another library (i.e., without registering new ones).

In general, I do think that we want the std::type_info * pointer to be available since it can accelerate lookups (type_c2p_fast) even though it is a C++-specific data structure. Maybe we could add a language/ABI specifier, e.g.

#define PYMETABIND_ABI_C 0
#define PYMETABIND_ABI_CPP 1
#define PYMETABIND_ABI_RUST 2
#define PYMETABIND_ABI_GO 3

struct pymetabind_binding {
    int abi_lang; /* Language ABI: one of PYMETABIND_ABI_C, PYMETABIND_ABI_CPP, etc. */
    const char *abi_extra; /* Additional language-specific ABI tag, e.g. NB_PLATFORM_ABI_TAG for C++ bindings with pybind11/nanobind */
    char *name;
    char *mangled_name;
    void *payload;
    PyTypeObject *py_type;

   // ...
};

And the payload field is language-defined: unused/NULL for C and std::type_info * for C++. In this case, the registry is shared by all languages/frameworks, and each entry is individually tagged with the language/platform ABI (e.g. compiler/standard library used to compile an extension). A new framework can then (at initialization time) walk through the list and register compatible entries, and then install callbacks to be notified of any further changes.

[oremanj]

Leaving some notes-to-self about MSVC typeinfo ABI for further investigation:
https://github.com/ojdkbuild/tools_toolchain_vs2017bt_1416/blob/master/VC/Tools/MSVC/14.16.27023/include/vcruntime_typeinfo.h
https://github.com/ojdkbuild/tools_toolchain_vs2017bt_1416/blob/master/VC/Tools/MSVC/14.16.27023/crt/src/vcruntime/std_type_info.cpp
https://github.com/ojdkbuild/tools_toolchain_vs2017bt_1416/blob/d3cdb9a6cb3d92f6081290849f4386a9ff2ce30a/VC/Tools/MSVC/14.16.27023/crt/src/vcruntime/undname.cxx#L715
https://en.wikiversity.org/wiki/Visual_C%2B%2B_name_mangling

struct This::Is::A::Test becomes a typeinfo name of .?AUTest@A@Is@This@@
struct Test becomes a typeinfo name of .?AUTest@@
The leading dot is not part of the mangling scheme (it's skipped before passing the name to the mangler).

What a C interface would already enable is for a C binding library to reuse bindings made by another library (i.e., without registering new ones).

Mm, I see how providing both the typeinfo ptr and the name would help with that. Even without typeinfo, a C library could look up by the name of what it's trying to extract.