I have a similar project (in some ways) to this: https://github.com/damageboy/daemaged.managedhell
One of the reasons I had to abandon the use of this (namely allocating .NET objects in unmanaged memory) was that as the use of the managed references to unmanaged memory lead to all kinds of unpredictable GC crashes... Have you ever seen anything like this in with your library?