set_jumps should be a function, and the default implementation should only return true when the mode is normal mode.
In operator mode the cursor will not move, so there is no need to set it to jump list.
And in visual mode, if we do something like 'y' ,'d', or 'c', the cursor will restore, so there is not need to set the jump list, and making this option a function can let the user to decide when to set the jump easily.