It's inherently racy to allow this in the first place, since you're racing against the thread being destroyed no matter how you slice it, but perhaps adding some sort of ref count to the structure would help.
Or, perhaps adding some sort of optional handle mechanism to go through which would start the path towards having a proper user space abstraction.