[QUESTION]: How can I define kernel which depends generic specialized method?

[aka-nse]

Question

Hello.

I want to define a kernel which uses generic specialized method, but it throws InternalCompilerException:
(Please note that GenericSin is external API so that it cannot be migrate closed generic kernel pattern e.g. GenericKernel sample)

using ILGPU;
using ILGPU.Runtime;

using var context = Context.CreateDefault();
using var accelerator = context.GetPreferredDevice(false).CreateAccelerator(context);

// OK: non generic 
var kernel1 = accelerator.LoadAutoGroupedStreamKernel<Index1D, ArrayView<float>, ArrayView<float>>(Kernel1);

// NG: reference to System.Type is denied
var kernel2 = accelerator.LoadAutoGroupedStreamKernel<Index1D, ArrayView<float>, ArrayView<float>>(Kernel2<float>);

static void Kernel1(Index1D index, ArrayView<float> x, ArrayView<float> ans)
{
    ans[index] = MathF.Sin(x[index]);
}


static void Kernel2<T>(Index1D index, ArrayView<T> x, ArrayView<T> ans)
    where T : unmanaged
{
    ans[index] = GenericSin(x[index]);
}

static T GenericSin<T>(T x)
    where T : unmanaged
{
    if (typeof(T) == typeof(float))
        return Unsafe.BitCast<float, T>(MathF.Sin(Unsafe.BitCast<T, float>(x)));
    else if (typeof(T) == typeof(double))
        return Unsafe.BitCast<double, T>(Math.Cos(Unsafe.BitCast<T, double>(x)));
    else
        throw new NotSupportedException();
}

I want to implement any of following technique:

• defines method call interception from GenericSin<float> to MyGenericSin_float
• rewrites IL of GenericSin<float> using System.Reflection.Emit APIs and load kernel with its IL stream directly

Sorry because this is likely an edge case, but I would be grateful if you could provide guidance if possible.

Environment

• ILGPU version: [1.5.3]
• .NET version: [.NET 9]
• Operating system: [Windows 11]
• Hardware (if GPU-related): [NVIDIA GeForce GTX 4070 Ti]

Additional context

No response

[MoFtZ]

hi @aka-nse, as discussed on Discord, the reason why your generic specialization is not working is because you are attempting to perform runtime type detection, to decide which function to call.

While this may be acceptable in standard .NET code, it is not currently possible for ILGPU to translate this to Cuda. This is also not possible in the native C++ Cuda kernel, as RTTI is not supported.

However, you should be able to use ILGPU to create a generic kernel that handles this. Instead of checking the data type, you should be providing a struct that implements your desired function call. And your own .NET code can perform the type detection. You should be creating two kernel instances, that can share the same base code.
for example:

if dataType == typeof(float)
    accelerator.LoadXxxKernel(MyKernel<float, MyFloatFunc>)
else
    accelerator.LoadXxxKernel(MyKernel<double, MyDoubleFunc>)

where, MyFloatFunc and MyDoubleFunc are structures that contain a method implementation.

[MoFtZ]

@aka-nse here is a more concrete example code:

interface IKernelFunction<T>
    where T : struct
{
    T ComputeValue(T value);
}

readonly struct CustomSinFloat : IKernelFunction<float>
{
    public float ComputeValue(float value) =>
        MathF.Sin(value);
}

readonly struct CustomSinDouble : IKernelFunction<double>
{
    public double ComputeValue(double value) =>
        Math.Sin(value);
}


class Program
{
    static void Kernel<T, TKernelFunction>(
        Index1D index,
        ArrayView1D<T, Stride1D.Dense> inputData,
        ArrayView1D<T, Stride1D.Dense> outputData)
        where TKernelFunction : struct, IKernelFunction<T>
        where T : unmanaged
    {
        var function = default(TKernelFunction);
        outputData[index] = function.ComputeValue(inputData[index]);
    }

    static void RunKernel<T, TKernelFunction>(
        Accelerator accelerator,
        ArrayView1D<T, Stride1D.Dense> inputView,
        ArrayView1D<T, Stride1D.Dense> outputView)
        where TKernelFunction : struct, IKernelFunction<T>
        where T : unmanaged
    {
        var kernel = accelerator.LoadAutoGroupedStreamKernel<Index1D, ArrayView1D<T, Stride1D.Dense>, ArrayView1D<T, Stride1D.Dense>>(Kernel<T, TKernelFunction>);
        kernel((int)inputView.Length, inputView, outputView);
    }


    static void Main()
    {
        var rnd = new Random(12345);
        using var context = Context.CreateDefault();

        foreach (var device in context)
        {
            using var accelerator = device.CreateAccelerator(context);
            Console.WriteLine($"Performing operations on {accelerator}");


            const int DataSize = 1024;
            var inputFloatValues = Enumerable.Range(0, DataSize).Select(i => (float)rnd.NextDouble()).ToArray();
            var inputDoubleValues = Enumerable.Range(0, DataSize).Select(i => rnd.NextDouble()).ToArray();

            using var inputFloatBuffer = accelerator.Allocate1D<float>(inputFloatValues);
            using var outputFloatBuffer = accelerator.Allocate1D<float>(DataSize);


            RunKernel<float, CustomSinFloat>(accelerator, inputFloatBuffer.View, outputFloatBuffer.View);

            var floatData = outputFloatBuffer.GetAsArray1D();

            using var inputDoubleBuffer = accelerator.Allocate1D<double>(inputDoubleValues);
            using var outputDoubleBuffer = accelerator.Allocate1D<double>(DataSize);
            RunKernel<double, CustomSinDouble>(accelerator, inputDoubleBuffer.View, outputDoubleBuffer.View);

            var doubleData = outputDoubleBuffer.GetAsArray1D();
        }
    }
}