Using [LibraryImport] with a generic delegate parameter (e.g. Func<int, int>) causes an exception

[ascpixi]

Description

When using a generic delegate, like Func<int, int, int> as a parameter to a function that is marked with LibraryImportAttribute, an ArgumentException: The specified Type must not be a generic type is thrown when the source-generated code attempts to invoke GetFunctionPointerForDelegate.

Reproduction Steps

Declare a function like this:

    [LibraryImport("user32.dll")]
    [return: MarshalAs(UnmanagedType.Bool)]
    public static partial bool EnumWindows(
        Func<nint, nint, bool> lpEnumFunc,
        nint lParam
    );

...then invoke it:

        User32.EnumWindows((hwnd, _) => {
            Console.WriteLine(hwnd);
            return true;
        }, default);

The invocation will cause a System.ArgumentException to be thrown, specifically from the last line of this source-generated snippet:

public static partial bool EnumWindows(global::System.Func<nint, nint, bool> lpEnumFunc, nint lParam)
{
    nint __lpEnumFunc_native;
    bool __retVal;
    int __retVal_native;
    // Marshal - Convert managed data to native data.
    __lpEnumFunc_native = lpEnumFunc != null ? global::System.Runtime.InteropServices.Marshal.GetFunctionPointerForDelegate(lpEnumFunc) : default; // <-- HERE

Expected behavior

The source generator should either recognize generic delegates like Func<...> or Action<...>, and include a special case when generating code to marshal parameters with these types, or it should throw a compile-time error to notify the developer that this is not supported.

Actual behavior

A runtime exception is thrown, at the time of the invocation of the imported method.

Regression?

No response

Known Workarounds

Declare a delegate type, like so:

public delegate bool EnumWindowsProc(IntPtr hWnd, IntPtr lParam);

...then use that non-generic delegate type instead of the generic delegate type. This isn't that much of an issue, but it does slightly impact code aesthetics. Declaring the function parameters inside the method declaration takes up less lines and displays the signature of the method when inspecting the [LibraryImport]-ed declaration.

Configuration

.NET 9.0.100, running under Windows x64.

Other information

Might be related to #32963 if we were to change the functionality of Marshal.GetFunctionPointerForDelegate(). I think an easier approach would be to change the source generator itself.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/interop-contrib
See info in area-owners.md if you want to be subscribed.

[huoyaoyuan]

Note: the optimal approach for passing a callback is using function pointer: delegate* unmanaged[Stdcall]<IntPtr, IntPtr, void>.

[ascpixi]

I don't think that an unmanaged function pointer would work with lambda captures, though - or lambda expressions in general, unless there's syntax for this that I'm not aware of. This also requires the explicit use of unsafe code by the callers.

[huoyaoyuan]

The source generator is behaving expected. The built-in marshalling with DllImport doesn't support generic delegate either.

I don't think that an unmanaged function pointer would work with lambda captures, though - or lambda expressions in general

No, but note that capturing with unmanaged callback is explicitly suboptimal. A new thunk of code must be allocated for each instance delegate.

This also requires the explicit use of unsafe code by the callers.

The "unsafe" syntax here is safer than the entire Marshal class.

[Sergio0694]

"with lambda captures"

You can pass arguments via the lParam argument. For unmanaged values, either pass them directly if they fit, or put them in some struct on the stack (eg. a value tuple), and then simply pass its address as parameter. If it's a managed object, you can use a GCHandle to point to it, and pass the handle as parameter, and then just get the object back inside the callback.

[EgorBo]

This also requires the explicit use of unsafe code by the callers.

We're considering marking all pinvokes/libraryimport as "requires unsafe context" regardless of their signatures

[jkotas]

I don't think that an unmanaged function pointer would work with lambda capture.

You can keep the unmanaged function pointer as an internal implementation detail that takes advantage of the lParam argument to pass the context around. The entrypoint used by the rest of the code can be a regular safe delegate that works with lambda captures. Something like (we use this pattern in number of places in this repo):
 

User32.EnumWindows((hwnd) => {
    Console.WriteLine(hwnd);
    return true;
});

unsafe partial class User32
{
#pragma warning disable CS8500
    public static bool EnumWindows(Func<nint, bool> enumFunc)
        => EnumWindows(&EnumWindowsNativeCallback, (nint)(void*)&enumFunc);

    [LibraryImport("user32.dll")]
    [return: MarshalAs(UnmanagedType.Bool)]
    private static partial bool EnumWindows(delegate* unmanaged[Stdcall]<nint, nint, int> lpEnumFunc, nint lParam);

    [UnmanagedCallersOnly(CallConvs = [typeof(CallConvStdcall)])]
    private static int EnumWindowsNativeCallback(nint hWnd, nint lParam)
        => (*(Func<nint, bool>*)lParam)(hWnd) ? 1 : 0;
#pragma warning restore CS8500
}

Our goal with interop evolution is to enable efficient implementation of different interop scenarios. We are not trying to minimize number of lines required to do so at all costs. The interop source generators help to reduce number of lines for common cases, but not for less common cases like this one.

If it's a managed object, you can use a GCHandle to point to it, and pass the handle as parameter, and then just get the object back inside the callback.

FWIW, you do not even need GCHandle in this case as my example shows.

[Sergio0694]

Out of curiosity, is this only safe because the assumption is EnumWindows will only ever invoke the callback on the current thread? Like, this is not a generally safe pattern for callbacks that could run on other threads, due to lack of memory barriers, right? 🤔

[ascpixi]

The source generator is behaving expected. The built-in marshalling with DllImport doesn't support generic delegate either.

I understand. If generic delegates like Func are not planned to be supported, I suggest that the source generator reports a compile-time error when such a parameter is present in the signature of a LibraryImport'ed method.

No, but note that capturing with unmanaged callback is explicitly suboptimal. A new thunk of code must be allocated for each instance delegate.

Wouldn't the performance hit be equal to any other lambda capture - that is, a delegate allocation...? Interop/marshalling across managed/unmanaged boundaries isn't really my domain, so I can't say for certain 😅

You can keep the unmanaged function pointer as an internal implementation detail that takes advantage of the lParam argument to pass the context around.

In this context, that's true - but if we were to have a function that doesn't take something like lParam, things would get more complicated, and a capture would make our lives a whole lot easier - if the (small-ish?) performance hit is tolerable.

Of course, we can still pass capturing delegates to methods marked with LibraryImport - we just have to explicitly declare a delegate type. I feel the best resolution of this issue is to report an error when a generic delegate type is passed as a parameter (since we can't quite marshal those until #32963 is resolved).

[sylveon]

Out of curiosity, is this only safe because the assumption is EnumWindows will only ever invoke the callback on the current thread?

EnumWindows only calls the callback from within itself, it is not stored anywhere and discarded upon returning to the caller.

[jkotas]

Wouldn't the performance hit be equal to any other lambda capture - that is, a delegate allocation...? Interop/marshalling across managed/unmanaged boundaries isn't really my domain, so I can't say for certain

Marshalling a delegate to a function pointer is a lot more expensive operation than a lambda capture. Marshalling a delegate to a function pointer needs to find or create the marshalling thunk for the given signature, allocate a piece of executable memory that will be used as a function pointer entrypoint, allocate a finalizable object to free this piece of executable memory once the delegate is garbage collected, and registers the result in global tables for bookkeeping.

if we were to have a function that doesn't take something like lParam, things would get more complicated, and a capture would make our lives a whole lot easier - if the (small-ish?) performance hit is tolerable.

It is the standard design pattern for unmanaged APIs that take callbacks to take a context parameter like lParam as well. It is necessary to make the APIs usable in unmanaged code. C/C++ does not allow you to convert a lambda capture to a function pointer like .NET delegate marshalling.

[jkotas]

the best resolution of this issue is to report an error when a generic delegate type is passed as a parameter

+1