fix: process issues from external security review

[aapoalas]

Two findings:

1. Stack overflow is very easy to trigger. There's quite little I can do about this per se, as Rust does not give the means to avoid it directly. An experimental limit to the JS call stack depth did, however, fix the crashing.
2. Allocating very large objects or arrays could panic due to Nova not using try_reserve for elements and keys storage allocations. I changed the code to always use try_reserve and bubble up any errors.