Smooth zooming does not work properly

Given enabled smooth scrolling using a viewport controller, eg.

```
...
  controller: {
    scrollZoom: {
        smooth: true,
    },
  },
```

Zooming using the scrollwheel sometimes noticeably jumps/glitches in 3D and 2D.