This is due to selection remembering in the inspector: if we select a variable in the inspector, then change context in the stack and then go back to the first context where we selected that variable, the debugger remembers the selection and reactivates it again.

Disabling this mechanism makes the stack navigation much much faster and agreeable for users.

All the time is spent in the miller list presenter, where doing selectIndex: to select the remembered selection goes into a whole announcement and widget updating procedure.

More analysis to come.
I suspect that part of the slowness of the debugger comes from that (in general, whenever there is a miller list update).