[API Proposal]: APIs for source generating interop stubs for unmanaged virtual function tables

[jkoritzinsky]

Background and motivation

As part of our COM interop source generator work, we decided to start with implementing support first for the general concept of virtual function tables. Many native APIs, including JNI, MSQuic, and COM are either implemented or presented to users using tables of function pointers, either explicitly like MSQuic or implicitly through abstract member functions like JNI or COM. Providing this source generator would enable developers to use native APIs like these ones with user-friendly types using the new source-generated marshalling model instead of being forced to use the built-in marshalling model with Marshal.GetDelegateForFunctionPointer to manually marshal back a virtual method table or dropping down to manual marshalling with function pointers to get decent performance.

This source generator would enable generating code to call an unmanaged API projected to a managed interface and call a managed interface projected to an unmanaged table of function pointers.

We plan on providing guidance directing users to use these APIs to override behavior from the COM source generator (they will integrate cleanly)

API Proposal

These APIs fall into two categories:

1. APIs that allow developers to define managed APIs that represent unmanaged virtual method tables (such as COM, JNI, MsQuic).
2. APIs that represent concepts that the COM source generator could use as building blocks that are also designed to be seamlessly integrated with the APIs in the first category.

// Types that are only needed for the VTable source generator or to provide abstract concepts that the COM generator would use under the hood.
// These are types that we can exclude from the API proposals and either inline into the generated code, provide as file-scoped types, or not provide publicly (indicated by comments on each type).

using System.Numerics;

namespace System.Runtime.InteropServices.Marshalling;

/// <summary>
/// A factory to create an unmanaged "this pointer" from a managed object and to get a managed object from an unmanaged "this pointer".
/// </summary>
/// <remarks>
/// This interface would be used by the VTable source generator to enable users to indicate how to get the managed object from the "this pointer".
/// We can hard-code the ComWrappers logic here if we don't want to ship this interface.
/// </remarks>
public unsafe interface IUnmanagedObjectUnwrapper
{
    /// <summary>
    /// Get the object wrapped by <paramref name="ptr"/>.
    /// </summary>
    /// <param name="ptr">A an unmanaged "this pointer".</param>
    /// <returns>The object wrapped by <paramref name="ptr"/>.</returns>
    public static abstract object GetObjectForUnmanagedWrapper(void* ptr);
}

// This attribute provides the mechanism for the VTable source generator to know which type to use to get the managed object
// from the unmanaged "this" pointer. If we decide to not expose VirtualMethodIndexAttribute, we don't need to expose this.
[AttributeUsage(AttributeTargets.Interface)]
public class UnmanagedObjectUnwrapperAttribute<TMapper> : Attribute
    where TMapper : IUnmanagedObjectUnwrapper
{
}

// This type implements the logic to get the managed object from the unmanaged "this" pointer.
// If we decide to not expose the VTable source generator, we don't need to expose this and we can just inline the logic
// into the generated code in the source generator.
public sealed unsafe class ComWrappersUnwrapper : IUnmanagedObjectUnwrapper
{
    public static object GetObjectForUnmanagedWrapper(void* ptr)
    {
        return ComWrappers.ComInterfaceDispatch.GetInstance<object>((ComWrappers.ComInterfaceDispatch*)ptr);
    }
}

/// <summary>
/// Marshals an exception object to the value of its <see cref="Exception.HResult"/> converted to <typeparamref name="T"/>.
/// </summary>
/// <typeparam name="T">The unmanaged type to convert the HResult to.</typeparam>
/// <remarks>
/// This type is used by the COM source generator to enable marshalling exceptions to the HResult of the exception.
/// We can skip the exposing the exception marshallers if we decide to not expose the VTable source generator.
/// In that case, we'd hard-code the implementations of these marshallers into the COM source generator.
/// </remarks>
[CustomMarshaller(typeof(Exception), MarshalMode.UnmanagedToManagedOut, typeof(ExceptionHResultMarshaller<>))]
public static class ExceptionHResultMarshaller<T>
    where T : unmanaged, INumber<T>
{
    /// <summary>
    /// Marshals an exception object to the value of its <see cref="Exception.HResult"/> converted to <typeparamref name="T"/>.
    /// </summary>
    /// <param name="e">The exception.</param>
    /// <returns>The HResult of the exception, converted to <typeparamref name="T"/>.</returns>
    public static T ConvertToUnmanaged(Exception e);
}

[CustomMarshaller(typeof(Exception), MarshalMode.UnmanagedToManagedOut, typeof(ExceptionNaNMarshaller<>))]
public static class ExceptionNaNMarshaller<T>
    where T : unmanaged, IFloatingPointIeee754<T>
{
    /// <summary>
    /// Marshals an exception object to <see cref="T.NaN"/>.
    /// </summary>
    /// <param name="e">The exception.</param>
    /// <returns><typeparamref name="T.NaN"/>.</returns>
    public static T ConvertToUnmanaged(Exception e);
}

[CustomMarshaller(typeof(Exception), MarshalMode.UnmanagedToManagedOut, typeof(ExceptionDefaultMarshaller<>))]
public static class ExceptionDefaultMarshaller<T>
    where T : unmanaged
{
    /// <summary>
    /// Marshals an exception object to the default value of <typeparamref name="T"/>.
    /// </summary>
    /// <param name="e">The exception.</param>
    /// <returns>The default value of <typeparamref name="T"/>.</returns>
    public static T ConvertToUnmanaged(Exception e);
}

[CustomMarshaller(typeof(Exception), MarshalMode.UnmanagedToManagedOut, typeof(SwallowExceptionMarshaller))]
public static class SwallowExceptionMarshaller
{
    /// <summary>
    /// Swallows the exception.
    /// </summary>
    /// <param name="e">The exception.</param>
    public static void ConvertToUnmanaged(Exception e);
}

public enum ExceptionMarshalling
{
    Custom = 0,
    Com = 1
}

    public enum MarshalDirection
    {
        ManagedToUnmanaged = 0,
        UnmanagedToManaged = 1,
        Bidirectional = 2
    }

// This is the trigger attribute for the VTable source generator.
// If we decide we want to only expose the COM source generator, then we would keep this attribute internal.
// The current plan is to use this attribute to provide the "don't use the defaults, use this custom logic" options
// for the COM source generator, so if we decide to not expose this, we should provide a different mechanism.
[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class VirtualMethodIndexAttribute : Attribute
{
    public VirtualMethodIndexAttribute(int index);

    public int Index { get; }

    public bool ImplicitThisParameter { get; set; } = true;

    /// <summary>
    /// Gets or sets how to marshal string arguments to the method.
    /// </summary>
    /// <remarks>
    /// If this field is set to a value other than <see cref="StringMarshalling.Custom" />,
    /// <see cref="StringMarshallingCustomType" /> must not be specified.
    /// </remarks>
    public StringMarshalling StringMarshalling { get; set; }

    /// <summary>
    /// Gets or sets the <see cref="Type"/> used to control how string arguments to the method are marshalled.
    /// </summary>
    /// <remarks>
    /// If this field is specified, <see cref="StringMarshalling" /> must not be specified
    /// or must be set to <see cref="StringMarshalling.Custom" />.
    /// </remarks>
    public Type? StringMarshallingCustomType { get; set; }

    /// <summary>
    /// Gets or sets whether the callee sets an error (SetLastError on Windows or errno
    /// on other platforms) before returning from the attributed method.
    /// </summary>
    public bool SetLastError { get; set; }
    
    public MarshalDirection Direction { get; set; } = MarshalDirection.Bidirectional;

    public ExceptionMarshalling ExceptionMarshalling { get; set; }

    /// <summary>
    /// Gets or sets the <see cref="Type"/> used to control how an exception is marshalled to the return value.
    /// </summary>
    /// <remarks>
    /// If this field is specified, <see cref="ExceptionMarshalling" /> must not be specified
    /// or must be set to <see cref="ExceptionMarshalling.Custom" />.
    /// </remarks>
    public Type? ExceptionMarshallingType { get; set; }
}

API Usage

[UnmanagedObjectUnwrapper<MyObjectUnwrapper>]
partial interface INativeAPI : IUnmanagedInterfaceType
{
       [VirtualMethodIndex(0)]
        void Method(string param);
}

[UnmanagedObjectUnwrapper<MyObjectUnwrapper>]
unsafe partial interface INativeAPI2 : IUnmanagedInterfaceType
{
		private static void* _table;
		public static void* IUnmanagedInterface.VirtualMethodTableManagedImplementation => _table;

		static INativeAPI2()
       {
              _table = RuntimeHelpers.AllocateTypeAssociatedMemory(typeof(INativeAPI2), sizeof(void*));
			 Native.FillManagedVirtualMethodTableImplementation(_table);
       }

       [VirtualMethodIndex(0)]
        void Method(string param);
}

class NativeAPI : IUnmanagedVirtualMethodTableProvider, INativeAPI.Native
{
		private void* _this;

        public NativeAPI(void* thisPtr) { _this = thisPtr; }

        VirtualMethodTableInfo IUnmanagedVirtualMethodTableProvider.GetVirtualMethodTableInfoForKey(Type type)
        {
				Debug.Assert(type == typeof(INativeAPI));
                return new(_this, *(void***)_this);
        }
}

sealed unsafe class MyObjectUnwrapper : IUnmanagedObjectUnwrapper
{
    public static object GetObjectForUnmanagedWrapper(void* ptr)
    {
        throw new NotImplementedException();
    }
}

Generated code shape:

partial interface INativeAPI
{
		public static void* IUnmanagedInterface.VirtualMethodTableManagedImplementation => /* implementation */ throw null;

        internal partial interface Native : INativeAPI
        {
               // DIM implementations for every method in INativeAPI with a [VirtualMethodIndexAttribute] attribute.
        }
}

partial interface INativeAPI2
{
        internal partial interface Native : INativeAPI2
        {
               // DIM implementations for every method in INativeAPI with a [VirtualMethodIndexAttribute] attribute.

               // Provided only when the user provides their own implementation of IUnmanagedInterface.VirtualMethodTableManagedImplementation.
               // It's extremely annoying to have to know the unmanaged function pointer signatures to take the address of the
              // implemented stubs, so provide this method to fill the table so users don't need to.
               internal static void FillManagedVirtualMethodTableImplementation(void* vtable) {} 
        }
}

Original API Proposal (kept for history to match to comments)

namespace System.Runtime.InteropServices.Marshalling;

/// <summary>
/// Information about a virtual method table and the unmanaged instance pointer.
/// </summary>
public readonly ref struct VirtualMethodTableInfo
{
    /// <summary>
    /// Construct a <see cref="VirtualMethodTableInfo"/> from a given instance pointer and table memory.
    /// </summary>
    /// <param name="thisPointer">The pointer to the instance.</param>
    /// <param name="virtualMethodTable">The block of memory that represents the virtual method table.</param>
    public VirtualMethodTableInfo(IntPtr thisPointer, ReadOnlySpan<IntPtr> virtualMethodTable)
    {
        ThisPointer = thisPointer;
        VirtualMethodTable = virtualMethodTable;
    }

    /// <summary>
    /// The unmanaged instance pointer
    /// </summary>
    public IntPtr ThisPointer { get; }

    /// <summary>
    /// The virtual method table.
    /// </summary>
    public ReadOnlySpan<IntPtr> VirtualMethodTable { get; }

    /// <summary>
    /// Deconstruct this structure into its two fields.
    /// </summary>
    /// <param name="thisPointer">The <see cref="ThisPointer"/> result</param>
    /// <param name="virtualMethodTable">The <see cref="VirtualMethodTable"/> result</param>
    public void Deconstruct(out IntPtr thisPointer, out ReadOnlySpan<IntPtr> virtualMethodTable)
    {
        thisPointer = ThisPointer;
        virtualMethodTable = VirtualMethodTable;
    }
}

/// <summary>
/// This interface allows an object to provide information about a virtual method table for a managed interface that implements <see cref="IUnmanagedInterfaceType{TInterface}"/> to enable invoking methods in the virtual method table.
/// </summary>
/// <typeparam name="T">The type to use to represent the the identity of the unmanaged type.</typeparam>
public unsafe interface IUnmanagedVirtualMethodTableProvider
{
    /// <summary>
    /// Get the information about the virtual method table for a given unmanaged interface type represented by <paramref name="type"/>.
    /// </summary>
    /// <param name="type">The managed type for the unmanaged interface.</param>
    /// <returns>The virtual method table information for the unmanaged interface.</returns>
    protected VirtualMethodTableInfo GetVirtualMethodTableInfoForKey(Type type);

    /// <summary>
    /// Get the information about the virtual method table for the given unmanaged interface type.
    /// </summary>
    /// <typeparam name="TUnmanagedInterfaceType">The managed interface type that represents the unmanaged interface.</typeparam>
    /// <returns>The virtual method table information for the unmanaged interface.</returns>
    public sealed VirtualMethodTableInfo GetVirtualMethodTableInfoForKey<TUnmanagedInterfaceType>()
        where TUnmanagedInterfaceType : IUnmanagedInterfaceType<TUnmanagedInterfaceType>
    {
        return GetVirtualMethodTableInfoForKey(typeof(TUnmanagedInterfaceType));
    }

    /// <summary>
    /// Get the length of the virtual method table for the given unmanaged interface type.
    /// </summary>
    /// <typeparam name="TUnmanagedInterfaceType">The managed interface type that represents the unmanaged interface.</typeparam>
    /// <returns>The length of the virtual method table for the unmanaged interface.</returns>
    public static int GetVirtualMethodTableLength<TUnmanagedInterfaceType>()
        where TUnmanagedInterfaceType : IUnmanagedInterfaceType<TUnmanagedInterfaceType>
    {
        return TUnmanagedInterfaceType.VirtualMethodTableLength;
    }

    /// <summary>
    /// Get a pointer to the virtual method table  of managed implementations of the unmanaged interface type.
    /// </summary>
    /// <typeparam name="TUnmanagedInterfaceType">The managed interface type that represents the unmanaged interface.</typeparam>
    /// <returns>A pointer to the virtual method table  of managed implementations of the unmanaged interface type</returns>
    public static void* GetVirtualMethodTableManagedImplementation<TUnmanagedInterfaceType>()
        where TUnmanagedInterfaceType : IUnmanagedInterfaceType<TUnmanagedInterfaceType>
    {
        return TUnmanagedInterfaceType.VirtualMethodTableManagedImplementation;
    }

    /// <summary>
    /// Get a pointer that wraps a managed implementation of an unmanaged interface that can be passed to unmanaged code.
    /// </summary>
    /// <typeparam name="TUnmanagedInterfaceType">The managed type that represents the unmanaged interface.</typeparam>
    /// <param name="obj">The managed object that implements the unmanaged interface.</param>
    /// <returns>A pointer-sized value that can be passed to unmanaged code that represents <paramref name="obj"/></returns>
    public static void* GetUnmanagedWrapperForObject<TUnmanagedInterfaceType>(TUnmanagedInterfaceType obj)
        where TUnmanagedInterfaceType : IUnmanagedInterfaceType<TUnmanagedInterfaceType>
    {
        return TUnmanagedInterfaceType.GetUnmanagedWrapperForObject(obj);
    }

    /// <summary>
    /// Get the object wrapped by <paramref name="ptr"/>.
    /// </summary>
    /// <typeparam name="TUnmanagedInterfaceType">The managed type that represents the unmanaged interface.</typeparam>
    /// <param name="ptr">A pointer-sized value returned by <see cref="GetUnmanagedWrapperForObject{TUnmanagedInterfaceType}(TUnmanagedInterfaceType)"/> or <see cref="IUnmanagedInterfaceType{TInterface, TKey}.GetUnmanagedWrapperForObject(TInterface)"/>.</param>
    /// <returns>The object wrapped by <paramref name="ptr"/>.</returns>
    public static TUnmanagedInterfaceType GetObjectForUnmanagedWrapper<TUnmanagedInterfaceType>(void* ptr)
        where TUnmanagedInterfaceType : IUnmanagedInterfaceType<TUnmanagedInterfaceType>
    {
        return TUnmanagedInterfaceType.GetObjectForUnmanagedWrapper(ptr);
    }
}

/// <summary>
/// This interface allows another interface to define that it represents a managed projection of an unmanaged interface from some unmanaged type system.
/// </summary>
/// <typeparam name="TInterface">The managed interface.</typeparam>
/// <typeparam name="TKey">The type of a value that can represent types from the corresponding unmanaged type system.</typeparam>
public unsafe interface IUnmanagedInterfaceType<TInterface>
    where TInterface : IUnmanagedInterfaceType<TInterface>
{
    /// <summary>
    /// Get the length of the virtual method table for the given unmanaged interface type.
    /// </summary>
    /// <returns>The length of the virtual method table for the unmanaged interface.</returns>
    public static abstract int VirtualMethodTableLength { get; }

    /// <summary>
    /// Get a pointer to the virtual method table  of managed implementations of the unmanaged interface type.
    /// </summary>
    /// <returns>A pointer to the virtual method table  of managed implementations of the unmanaged interface type</returns>
    public static abstract void* VirtualMethodTableManagedImplementation { get; }

    /// <summary>
    /// Get a pointer that wraps a managed implementation of an unmanaged interface that can be passed to unmanaged code.
    /// </summary>
    /// <param name="obj">The managed object that implements the unmanaged interface.</param>
    /// <returns>A pointer-sized value that can be passed to unmanaged code that represents <paramref name="obj"/></returns>
    public static abstract void* GetUnmanagedWrapperForObject(TInterface obj);

    /// <summary>
    /// Get the object wrapped by <paramref name="ptr"/>.
    /// </summary>
    /// <param name="ptr">A pointer-sized value returned by <see cref="IUnmanagedVirtualMethodTableProvider{TKey}.GetUnmanagedWrapperForObject{IUnmanagedInterfaceType{TInterface, TKey}}(IUnmanagedInterfaceType{TInterface, TKey})"/> or <see cref="GetUnmanagedWrapperForObject(TInterface)"/>.</param>
    /// <returns>The object wrapped by <paramref name="ptr"/>.</returns>
    public static abstract TInterface GetObjectForUnmanagedWrapper(void* ptr);
}

API Usage

partial interface INativeAPI : IUnmanagedInterfaceType<INativeAPI>
{
       static int IUnmanagedInterfaceType<INativeAPI>.VirtualMethodTableLength => 1;

                private static void** s_vtable = (void**)RuntimeHelpers.AllocateTypeAssociatedMemory(typeof(INativeAPI), sizeof(void*) * IUnmanagedVirtualMethodTableProvider.GetVirtualMethodTableLength<INativeAPI>());
                static void* IUnmanagedInterfaceType<INativeAPI>.VirtualMethodTableManagedImplementation
                {
                    get
                    {
                        if (s_vtable[0] == null)
                        {
                            Native.PopulateUnmanagedVirtualMethodTable(new Span<IntPtr>(s_vtable, IUnmanagedVirtualMethodTableProvider.GetVirtualMethodTableLength<INativeAPI>()));
                        }
                        return s_vtable;
                    }
                }

       static void* IUnmanagedInterfaceType<INativeAPI>.GetUnmanagedWrapperForObject(INativeAPI api) => throw new NotImplementedException();

        static INativeAPI IUnmanagedInterfaceType<INativeAPI>.GetObjectForUnmanagedWrapper(void* ptr) => throw new NotImplementedException();

       static INativeAPI()
       {
			
       }

       [VirtualMethodIndex(0)]
        void Method(string param);
}

class NativeAPI : IUnmanagedVirtualMethodTableProvider, INativeAPI.Native
{
		private void* _this;

        public NativeAPI(void* thisPtr) { _this = thisPtr; }

        VirtualMethodTableInfo IUnmanagedVirtualMethodTableProvider.GetVirtualMethodTableInfoForKey(Type type)
        {
				Debug.Assert(type == typeof(INativeAPI));
                return new((IntPtr)_this, new ReadOnlySpan<IntPtr>(*(void**)_this,  IUnmanagedVirtualMethodTableProvider.GetVirtualMethodTableLength<INativeAPI>()));
        }
}

Generated code shape:

partial interface INativeAPI
{
        internal partial interface Native : INativeAPI
        {
               // DIM implementations for every method in INativeAPI with a [VirtualMethodIndexAttribute] attribute.
               internal static PopulateUnmanagedVirtualMethodTable(Span<nint> vtable) { /* fill the vtable with function pointers for the unmanaged->managed stubs */
        }
}

Alternative Designs

Risks

The DIM-implemented methods aren't visible when using the implementing types directly; they're only visible through the interface. As a result, the user experience is a little weird for the cases where only one interface is implemented, as the interface methods can only be called on an object wrapping the native API through the interface, not through the wrapping class.

Tagging subscribers to this area: @dotnet/interop-contrib
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and motivation

As part of our COM interop source generator work, we decided to start with implementing support first for the general concept of virtual function tables. Many native APIs, including JNI, MSQuic, and COM are either implemented or presented to users using tables of function pointers, either explicitly like MSQuic or implicitly through abstract member functions like JNI or COM. Providing this source generator would enable developers to use native APIs like these ones with user-friendly types using the new source-generated marshalling model instead of being forced to use the built-in marshalling model with Marshal.GetDelegateForFunctionPointer to manually marshal back a virtual method table or dropping down to manual marshalling with function pointers to get decent performance.

This source generator would enable generating code to call an unmanaged API projected to a managed interface and call a managed interface projected to an unmanaged table of function pointers.

We plan on providing guidance directing users to use these APIs to override behavior from the COM source generator (they will integrate cleanly)

API Proposal

namespace System.Runtime.InteropServices.Marshalling;

/// <summary>
/// Information about a virtual method table and the unmanaged instance pointer.
/// </summary>
public readonly ref struct VirtualMethodTableInfo
{
    /// <summary>
    /// Construct a <see cref="VirtualMethodTableInfo"/> from a given instance pointer and table memory.
    /// </summary>
    /// <param name="thisPointer">The pointer to the instance.</param>
    /// <param name="virtualMethodTable">The block of memory that represents the virtual method table.</param>
    public VirtualMethodTableInfo(IntPtr thisPointer, ReadOnlySpan<IntPtr> virtualMethodTable)
    {
        ThisPointer = thisPointer;
        VirtualMethodTable = virtualMethodTable;
    }

    /// <summary>
    /// The unmanaged instance pointer
    /// </summary>
    public IntPtr ThisPointer { get; }

    /// <summary>
    /// The virtual method table.
    /// </summary>
    public ReadOnlySpan<IntPtr> VirtualMethodTable { get; }

    /// <summary>
    /// Deconstruct this structure into its two fields.
    /// </summary>
    /// <param name="thisPointer">The <see cref="ThisPointer"/> result</param>
    /// <param name="virtualMethodTable">The <see cref="VirtualMethodTable"/> result</param>
    public void Deconstruct(out IntPtr thisPointer, out ReadOnlySpan<IntPtr> virtualMethodTable)
    {
        thisPointer = ThisPointer;
        virtualMethodTable = VirtualMethodTable;
    }
}

/// <summary>
/// This interface allows an object to provide information about a virtual method table for a managed interface that implements <see cref="IUnmanagedInterfaceType{TInterface}"/> to enable invoking methods in the virtual method table.
/// </summary>
/// <typeparam name="T">The type to use to represent the the identity of the unmanaged type.</typeparam>
public unsafe interface IUnmanagedVirtualMethodTableProvider
{
    /// <summary>
    /// Get the information about the virtual method table for a given unmanaged interface type represented by <paramref name="type"/>.
    /// </summary>
    /// <param name="type">The managed type for the unmanaged interface.</param>
    /// <returns>The virtual method table information for the unmanaged interface.</returns>
    protected VirtualMethodTableInfo GetVirtualMethodTableInfoForKey(Type type);

    /// <summary>
    /// Get the information about the virtual method table for the given unmanaged interface type.
    /// </summary>
    /// <typeparam name="TUnmanagedInterfaceType">The managed interface type that represents the unmanaged interface.</typeparam>
    /// <returns>The virtual method table information for the unmanaged interface.</returns>
    public sealed VirtualMethodTableInfo GetVirtualMethodTableInfoForKey<TUnmanagedInterfaceType>()
        where TUnmanagedInterfaceType : IUnmanagedInterfaceType<TUnmanagedInterfaceType>
    {
        return GetVirtualMethodTableInfoForKey(typeof(TUnmanagedInterfaceType));
    }

    /// <summary>
    /// Get the length of the virtual method table for the given unmanaged interface type.
    /// </summary>
    /// <typeparam name="TUnmanagedInterfaceType">The managed interface type that represents the unmanaged interface.</typeparam>
    /// <returns>The length of the virtual method table for the unmanaged interface.</returns>
    public static int GetVirtualMethodTableLength<TUnmanagedInterfaceType>()
        where TUnmanagedInterfaceType : IUnmanagedInterfaceType<TUnmanagedInterfaceType>
    {
        return TUnmanagedInterfaceType.VirtualMethodTableLength;
    }

    /// <summary>
    /// Get a pointer to the virtual method table  of managed implementations of the unmanaged interface type.
    /// </summary>
    /// <typeparam name="TUnmanagedInterfaceType">The managed interface type that represents the unmanaged interface.</typeparam>
    /// <returns>A pointer to the virtual method table  of managed implementations of the unmanaged interface type</returns>
    public static void* GetVirtualMethodTableManagedImplementation<TUnmanagedInterfaceType>()
        where TUnmanagedInterfaceType : IUnmanagedInterfaceType<TUnmanagedInterfaceType>
    {
        return TUnmanagedInterfaceType.VirtualMethodTableManagedImplementation;
    }

    /// <summary>
    /// Get a pointer that wraps a managed implementation of an unmanaged interface that can be passed to unmanaged code.
    /// </summary>
    /// <typeparam name="TUnmanagedInterfaceType">The managed type that represents the unmanaged interface.</typeparam>
    /// <param name="obj">The managed object that implements the unmanaged interface.</param>
    /// <returns>A pointer-sized value that can be passed to unmanaged code that represents <paramref name="obj"/></returns>
    public static void* GetUnmanagedWrapperForObject<TUnmanagedInterfaceType>(TUnmanagedInterfaceType obj)
        where TUnmanagedInterfaceType : IUnmanagedInterfaceType<TUnmanagedInterfaceType>
    {
        return TUnmanagedInterfaceType.GetUnmanagedWrapperForObject(obj);
    }

    /// <summary>
    /// Get the object wrapped by <paramref name="ptr"/>.
    /// </summary>
    /// <typeparam name="TUnmanagedInterfaceType">The managed type that represents the unmanaged interface.</typeparam>
    /// <param name="ptr">A pointer-sized value returned by <see cref="GetUnmanagedWrapperForObject{TUnmanagedInterfaceType}(TUnmanagedInterfaceType)"/> or <see cref="IUnmanagedInterfaceType{TInterface, TKey}.GetUnmanagedWrapperForObject(TInterface)"/>.</param>
    /// <returns>The object wrapped by <paramref name="ptr"/>.</returns>
    public static TUnmanagedInterfaceType GetObjectForUnmanagedWrapper<TUnmanagedInterfaceType>(void* ptr)
        where TUnmanagedInterfaceType : IUnmanagedInterfaceType<TUnmanagedInterfaceType>
    {
        return TUnmanagedInterfaceType.GetObjectForUnmanagedWrapper(ptr);
    }
}

/// <summary>
/// This interface allows another interface to define that it represents a managed projection of an unmanaged interface from some unmanaged type system.
/// </summary>
/// <typeparam name="TInterface">The managed interface.</typeparam>
/// <typeparam name="TKey">The type of a value that can represent types from the corresponding unmanaged type system.</typeparam>
public unsafe interface IUnmanagedInterfaceType<TInterface>
    where TInterface : IUnmanagedInterfaceType<TInterface>
{
    /// <summary>
    /// Get the length of the virtual method table for the given unmanaged interface type.
    /// </summary>
    /// <returns>The length of the virtual method table for the unmanaged interface.</returns>
    public static abstract int VirtualMethodTableLength { get; }

    /// <summary>
    /// Get a pointer to the virtual method table  of managed implementations of the unmanaged interface type.
    /// </summary>
    /// <returns>A pointer to the virtual method table  of managed implementations of the unmanaged interface type</returns>
    public static abstract void* VirtualMethodTableManagedImplementation { get; }

    /// <summary>
    /// Get a pointer that wraps a managed implementation of an unmanaged interface that can be passed to unmanaged code.
    /// </summary>
    /// <param name="obj">The managed object that implements the unmanaged interface.</param>
    /// <returns>A pointer-sized value that can be passed to unmanaged code that represents <paramref name="obj"/></returns>
    public static abstract void* GetUnmanagedWrapperForObject(TInterface obj);

    /// <summary>
    /// Get the object wrapped by <paramref name="ptr"/>.
    /// </summary>
    /// <param name="ptr">A pointer-sized value returned by <see cref="IUnmanagedVirtualMethodTableProvider{TKey}.GetUnmanagedWrapperForObject{IUnmanagedInterfaceType{TInterface, TKey}}(IUnmanagedInterfaceType{TInterface, TKey})"/> or <see cref="GetUnmanagedWrapperForObject(TInterface)"/>.</param>
    /// <returns>The object wrapped by <paramref name="ptr"/>.</returns>
    public static abstract TInterface GetObjectForUnmanagedWrapper(void* ptr);
}

API Usage

partial interface INativeAPI : IUnmanagedInterfaceType<INativeAPI>
{
       static int IUnmanagedInterfaceType<INativeAPI>.VirtualMethodTableLength => 1;

                private static void** s_vtable = (void**)RuntimeHelpers.AllocateTypeAssociatedMemory(typeof(INativeAPI), sizeof(void*) * IUnmanagedVirtualMethodTableProvider.GetVirtualMethodTableLength<INativeAPI>());
                static void* IUnmanagedInterfaceType<INativeAPI>.VirtualMethodTableManagedImplementation
                {
                    get
                    {
                        if (s_vtable[0] == null)
                        {
                            Native.PopulateUnmanagedVirtualMethodTable(new Span<IntPtr>(s_vtable, IUnmanagedVirtualMethodTableProvider.GetVirtualMethodTableLength<INativeAPI>()));
                        }
                        return s_vtable;
                    }
                }

       static void* IUnmanagedInterfaceType<INativeAPI>.GetUnmanagedWrapperForObject(INativeAPI api) => throw new NotImplementedException();

        static INativeAPI IUnmanagedInterfaceType<INativeAPI>.GetObjectForUnmanagedWrapper(void* ptr) => throw new NotImplementedException();

       static INativeAPI()
       {
			
       }

       [VirtualMethodIndex(0)]
        void Method(string param);
}

class NativeAPI : IUnmanagedVirtualMethodTableProvider, INativeAPI.Native
{
		private void* _this;

        public NativeAPI(void* thisPtr) { _this = thisPtr; }

        VirtualMethodTableInfo IUnmanagedVirtualMethodTableProvider.GetVirtualMethodTableInfoForKey(Type type)
        {
				Debug.Assert(type == typeof(INativeAPI));
                return new((IntPtr)_this, new ReadOnlySpan<IntPtr>(*(void**)_this,  IUnmanagedVirtualMethodTableProvider.GetVirtualMethodTableLength<INativeAPI>()));
        }
}

Generated code shape:

partial interface INativeAPI
{
        internal partial interface Native : INativeAPI
        {
               // DIM implementations for every method in INativeAPI with a [VirtualMethodIndexAttribute] attribute.
               internal static PopulateUnmanagedVirtualMethodTable(Span<nint> vtable) { /* fill the vtable with function pointers for the unmanaged->managed stubs */
        }
}

Alternative Designs

We could keep these APIs for testing and only expose the COM source generator APIs for the opinionated COM source generator and provide more options on those APIs (still to be proposed) instead of providing a lower-level tool like these APIs.

Risks

The DIM-implemented methods aren't visible when using the implementing types directly; they're only visible through the interface. As a result, the user experience is a little weird for the cases where only one interface is implemented, as the interface methods can only be called on an object wrapping the native API through the interface, not through the wrapping class.

Author:	jkoritzinsky
Assignees:	-
Labels:	api-suggestion, area-System.Runtime.InteropServices, source-generator
Milestone:	8.0.0

[AaronRobinsonMSFT]

    /// <summary>
    /// The unmanaged instance pointer
    /// </summary>
    public IntPtr ThisPointer { get; }

We should just use void* in these kinds of places. In VS these will all revert to nint which isn't all that helpful as it really is a void*.

[jkoritzinsky]

We can't update the ReadOnlySpan<IntPtr> because you can't put pointers in generics, but we can update the ThisPointer property and the like.

[AaronRobinsonMSFT]

We can't update the ReadOnlySpan because you can't put pointers in generics,

Unsure if using ReadOnlySpan<IntPtr> is all that helpful though. The first thing people will do is check the length and then ignore the niceties the span provides. Instead, we could have a property for the length and return void*. I assume the use case here is going to be all generated so I don't see the span providing a lot of value other than using a type to capture two values - pointer and length. Something to consider as an alternative.

[AaronRobinsonMSFT]

public unsafe interface IUnmanagedInterfaceType

I think this interface needs to be broken apart. The mapping logic is likely to be identical to the vast majority of types for a generator. This pattern seems like it would create a lot of unnecessary metadata that couldn't be removed.

[jkoritzinsky]

I'm fine splitting it, but I'm at a loss on what to name the one that has the unmanaged->managed logic.

[Sergio0694]

Sharing some thoughts here after discussing this with @jkoritzinsky as well. I think this might overlap at least in part with what @AaronRobinsonMSFT said as well. My concern is that the current proposal results in a UX that's rather cumbersome and error prone for developers, specifically around the setup code to initialize vtables. Jeremy explained that the point of initializing a vtable explicitly and not in a static field initializer is that this allows the linker to potentially trim everything if the type is not used. This is fine, but I feel like we could have something that's at least a little bit more ergonomic to use. Consider this:

partial interface INativeAPI : IUnmanagedInterfaceType<INativeAPI>
{
    static int IUnmanagedInterfaceType<INativeAPI>.VirtualMethodTableLength => 1;

    private static void** s_vtable = (void**)RuntimeHelpers.AllocateTypeAssociatedMemory(typeof(INativeAPI), sizeof(void*) * IUnmanagedVirtualMethodTableProvider.GetVirtualMethodTableLength<INativeAPI>());
    static void* IUnmanagedInterfaceType<INativeAPI>.VirtualMethodTableManagedImplementation
    {
        get
        {
            if (s_vtable[0] == null)
            {
                Native.PopulateUnmanagedVirtualMethodTable(new Span<IntPtr>(s_vtable, IUnmanagedVirtualMethodTableProvider.GetVirtualMethodTableLength<INativeAPI>()));
            }
            return s_vtable;
        }
    }

    static void* IUnmanagedInterfaceType<INativeAPI>.GetUnmanagedWrapperForObject(INativeAPI api) => throw new NotImplementedException();

    static INativeAPI IUnmanagedInterfaceType<INativeAPI>.GetObjectForUnmanagedWrapper(void* ptr) => throw new NotImplementedException();

    static INativeAPI()
    {
        
    }

    [VirtualMethodIndex(0)]
    void Method(string param);
}

That's a lot of relatively non obvious code, and writing that check manually every time seems a bit awkward, not to mention it's one more thing developers could get wrong. One possible solution could be to consider the way the vtable is allocated to just be the "de-fact right way" to do it, and as such move that initialization using RuntimeHelpers.AllocateTypeAssociatedMemory in some shared stub (eg. in a DIM), as well as the check, and then only ask developers implementing the interface to write the code to fill in the pre-allocated vtable, and nothing more. Something like:

partial interface INativeAPI : IUnmanagedInterfaceType<INativeAPI>
{
    static int IUnmanagedInterfaceType<INativeAPI>.VirtualMethodTableLength => 1;

    static void IUnmanagedInterfaceType<INativeAPI>.InitializeVirtualMethodTableManagedImplementation(void** vtable)
    {
        // Fill vtable here
    }
}

The allocation + initialization check could all be moved to a static DIM in the base interface. Authors would just indicate the length of the vtable, and provide the logic to fill in an allocated vtable, and nothing else 🙂

[jkoritzinsky]

Here's a revised API proposal. The "create a wrapper" and "retrieve an object from within a wrapper" APIs have been extracted into another interface and the methods on IUnmanagedInterfaceType<TInterfaceType, TUnmanagedObjectWrapperFactory> that deal with wrappers just forward to the wrapper factory implmentation and add nice casts based on @AaronRobinsonMSFT's feedback and splits the interface into two copies (one for consumption and one for consumption + production). It also encapsulates the allocation/initialization pattern for the managed vtable as per @Sergio0694's feedback.

Let me know what you think!

namespace System.Runtime.InteropServices.Marshalling;

/// <summary>
/// Information about a virtual method table and the unmanaged instance pointer.
/// </summary>
public readonly unsafe struct VirtualMethodTableInfo
{
    /// <summary>
    /// Construct a <see cref="VirtualMethodTableInfo"/> from a given instance pointer and table memory.
    /// </summary>
    /// <param name="thisPointer">The pointer to the instance.</param>
    /// <param name="virtualMethodTable">The block of memory that represents the virtual method table.</param>
    public VirtualMethodTableInfo(void* thisPointer, void** virtualMethodTable)
    {
        ThisPointer = thisPointer;
        VirtualMethodTable = virtualMethodTable;
    }

    /// <summary>
    /// The unmanaged instance pointer
    /// </summary>
    public void* ThisPointer { get; }

    /// <summary>
    /// The virtual method table.
    /// </summary>
    public void** VirtualMethodTable { get; }

    /// <summary>
    /// Deconstruct this structure into its two fields.
    /// </summary>
    /// <param name="thisPointer">The <see cref="ThisPointer"/> result</param>
    /// <param name="virtualMethodTable">The <see cref="VirtualMethodTable"/> result</param>
    public void Deconstruct(out void* thisPointer, out void** virtualMethodTable)
    {
        thisPointer = ThisPointer;
        virtualMethodTable = VirtualMethodTable;
    }
}

/// <summary>
/// This interface allows an object to provide information about a virtual method table for a managed interface that implements <see cref="IUnmanagedInterfaceType{TInterface}"/> to enable invoking methods in the virtual method table.
/// </summary>
/// <typeparam name="T">The type to use to represent the the identity of the unmanaged type.</typeparam>
public unsafe interface IUnmanagedVirtualMethodTableProvider
{
    /// <summary>
    /// Get the information about the virtual method table for a given unmanaged interface type represented by <paramref name="type"/>.
    /// </summary>
    /// <param name="type">The managed type for the unmanaged interface.</param>
    /// <returns>The virtual method table information for the unmanaged interface.</returns>
    protected VirtualMethodTableInfo GetVirtualMethodTableInfoForKey(Type type);

    /// <summary>
    /// Get the information about the virtual method table for the given unmanaged interface type.
    /// </summary>
    /// <typeparam name="TUnmanagedInterfaceType">The managed interface type that represents the unmanaged interface.</typeparam>
    /// <returns>The virtual method table information for the unmanaged interface.</returns>
    public sealed VirtualMethodTableInfo GetVirtualMethodTableInfoForKey<TUnmanagedInterfaceType>()
        where TUnmanagedInterfaceType : IUnmanagedInterfaceType
    {
        return GetVirtualMethodTableInfoForKey(typeof(TUnmanagedInterfaceType));
    }
}

/// <summary>
/// This interface allows another interface to define that it represents a managed projection of an unmanaged interface from some unmanaged type system.
/// </summary>
public unsafe interface IUnmanagedInterfaceType
{
    /// <summary>
    /// Get the length of the virtual method table for the given unmanaged interface type.
    /// </summary>
    /// <returns>The length of the virtual method table for the unmanaged interface.</returns>
    public static abstract int VirtualMethodTableLength { get; }
}

/// <summary>
/// A factory to create an unmanaged "this pointer" from a managed object and to get a managed object from an unmanaged "this pointer".
/// </summary>
public unsafe interface IUnmanagedObjectWrapperFactory
{
    /// <summary>
    /// Get a pointer that wraps a managed implementation of an unmanaged interface that can be passed to unmanaged code.
    /// </summary>
    /// <param name="obj">The managed object that implements the unmanaged interface.</param>
    /// <returns>A unmanaged "this pointer" that can be passed to unmanaged code that represents <paramref name="obj"/></returns>
    public static abstract void* GetUnmanagedWrapperForObject(object obj);

    /// <summary>
    /// Get the object wrapped by <paramref name="ptr"/>.
    /// </summary>
    /// <param name="ptr">A an unmanaged "this pointer".</param>
    /// <returns>The object wrapped by <paramref name="ptr"/>.</returns>
    public static abstract object GetObjectForUnmanagedWrapper(void* ptr);
}

/// <summary>
/// This interface allows another interface to define that it represents a managed projection of an unmanaged interface from some unmanaged type system and supports passing managed implementations of unmanaged interfaces to unmanaged code.
/// </summary>
/// <typeparam name="TInterface">The managed interface.</typeparam>
/// <typeparam name="TUnmanagedObjectWrapperFactory">The factory to create an unmanaged "this pointer" from a managed object and to get a managed object from an unmanaged "this pointer".</typeparam>
public unsafe interface IUnmanagedInterfaceType<TInterface, TUnmanagedObjectWrapperFactory> : IUnmanagedInterfaceType
    where TInterface : IUnmanagedInterfaceType<TInterface, TUnmanagedObjectWrapperFactory>
    where TUnmanagedObjectWrapperFactory: IUnmanagedObjectWrapperFactory, new()
{
    /// <summary>
    /// Get a pointer to the virtual method table of managed implementations of the unmanaged interface type.
    /// </summary>
    /// <returns>A pointer to the virtual method table of managed implementations of the unmanaged interface type</returns>
    public static void* VirtualMethodTableManagedImplementation { get; }
    
    /// <summary>
    /// Fill the entries in the virtual method table of managed implementations of the unmanaged interface type
    /// </summary>
    protected static virtual void FillVirtualMethodTableMangedImplementation(void** vtable);

    /// <summary>
    /// Get a pointer that wraps a managed implementation of an unmanaged interface that can be passed to unmanaged code.
    /// </summary>
    /// <param name="obj">The managed object that implements the unmanaged interface.</param>
    /// <returns>A unmanaged "this pointer" that can be passed to unmanaged code that represents <paramref name="obj"/></returns>
    public static void* GetUnmanagedWrapperForObject(TInterface obj);

    /// <summary>
    /// Get the object wrapped by <paramref name="ptr"/>.
    /// </summary>
    /// <param name="ptr">A an unmanaged "this pointer".</param>
    /// <returns>The object wrapped by <paramref name="ptr"/>.</returns>
    public static TInterface GetObjectForUnmanagedWrapper(void* ptr);
}

[Sergio0694]

Looks much better! I love how the vtable setup is way easier for authors now 🙂

A few random notes:

public sealed VirtualMethodTableInfo GetVirtualMethodTableInfoForKey<TUnmanagedInterfaceType>()
    where TUnmanagedInterfaceType : IUnmanagedInterfaceType
{
    return GetVirtualMethodTableInfoForKey(typeof(TUnmanagedInterfaceType));
}

I understand this is meant to be a convenience method, but it being a DIM feels like it might make codegen worse. Couldn't this be an extension method instead targeting IUnmanagedVirtualMethodTableProvider, so the interface would just have the one method that developers are meant to implement?

2. It's not entirely clear to me why the convenience helper is public, but the non-generic one is protected 🤔
3. The name "VirtualMethodTableManagedImplementation" seems a bit confusing to me. I get that it's pointing to managed method implementations, but the function pointers in that vtable are unmanaged (I mean, they're using an unmanaged call conv)... I don't really have a better name suggestion, just figured I'd throw this note out too while at it 😄

[AaronRobinsonMSFT]

We could keep these APIs for testing and only expose the COM source generator APIs for the opinionated COM source generator and provide more options on those APIs (still to be proposed) instead of providing a lower-level tool like these APIs.

I'm inclined to go with this for now. I would prefer to keep these as implementation/testing details until we have a full end to end for our P1 COM scenario. I am thinking of the late changes that occurred around the marshaller design and the annoyance it was to update them all. This design also permits a level of extensibility that isn't obviously beneficial beyond the current goal of the COM source generator. Once we have the COM source generator working for our stakeholders, let's revisit this API proposal and see where we've landed.