Remove unsafe code

[Insomniak47]

Given the advent of Span/Memory most of the unsafe code that exists in this repo can probably just be turfed without impacting performance. I can take a pass at it if that's of interest