I can invoke extension buttons using vimium-everywhere to open their corresponding popup windows, but try invoke vimium-everywhere again would make the Firefox window lose focus and close the windows.
A possible x11 solution is to create the overlay window with override-redirect flag, so that window managers won't manage the window, thus there is no focus changes and the popup stays open. However, I don't really know how to do that using the ahk scripting language.
https://tronche.com/gui/x/xlib/window/attributes/override-redirect.html