Replace some uses of `Vec` with `SmallVec`

[LukasKalbertodt]

Currently we use Vec for everything, including use cases in which we expect only a few items. This is pretty inefficient as Vec always heap allocates.

At some point we should fix this...

[jonas-schievink]

Also consider using Vec::with_capacity and replacing unnecessary Vecs with iterators.