I'm guessing this happens as the object the VFX is on is destroyed and for some reason it stick around on clients that enter until they reenter the area, but it's super misleading.
Maybe change to 5 min duration when slotting gems and readd if missing when you interact with pedastals