Upscaling reduces resemblance. Is there a way around?

[thinkingkoala]

I've found a perfect method to create an AI clone of myself and the initial images look amazing
But when I upscale them they seem to lose the resemblance and vary quite a bit from the original images.

Is there a way to preserve the same facial features of my character and only increase the resolution?
I already use the same seed, the same prompt, and enabled "Mixing Image Prompt and Vary/Upscale"
Am I missing something?

Here are the initial and the upscaled version. As you can see some features were changed during the upscale process

[ILikeToasters]

What are your settings for faceswap when upscaling? The fast upscale will not really change the image but also will not really add detail. The other two upscales do change the image but usually faceswap will work well with that in my experience. I use a weight of at least .9 on the faceswap.

[thinkingkoala]

It's not a face swap but LoRa and embedding

[ILikeToasters]

Okay. I saw "Mixing Image Prompt and Vary/Upscale" and figured that was what you were talking about.

[bsdesigner]

The image will not change when enlarged if you specify the same seed as in the original image