Torch 2.0.X breaks image generation on GTX1650

[CrisR82]

I've been trying to wrap my head around a problem for about 3 months now and I decided to throw the towel and actually ask for help - a while back after updating to Torch 2.0.0, image generation basically broke on my GTX1650 - it would generate non-hires images SOMETIMES while using hires fix would result in a failure every time with the following error error message at the end of the post.

My setup (desktop PC) is as follows:
CPU: i7-3770
RAM: 16GB DDR3
GPU: GPU: ASUS Phoenix GTX1650 4GB GDDR5 (nVidia driver version: 531.79)
OS: Windows 10 Home 22H2

The working WebUI setup (this works every time):
WebUI: v1.4.1
Python: 3.10.6
Torch: 1.12+cu113
xFormers: N/A
Gradio 3.32.0
Arguments used: --medvram

Results:
512x512: 3.75s/it
1024x1024 (Hires - Latent): 29.56s/it

After running "pip install -r requirements_versions.txt" and adding "--reinstall-torch"

The non-working WebUI setup (fails at the same spot every time):
WebUI: v1.4.1
Python: 3.10.6
Torch: 2.0.1+cu118
xFormers: N/A
Gradio 3.32.0
Arguments used: --medvram

512x512: 5.04s/it
1024x1024 (Hires - Latent): N/A (errored out)

In particular, as soon as it is done with the 512x512 image, it would remain like it's trying to work out something before generating anything from the 1024x1024 one (at the 50% mark in the UI, preview doesn't change) it errors out.
This happened back in Torch 2.0.0 as well.

Attempting it with the argument combinations listed below did not work either (errors out in all of them):
--medvram --no-half (fails to even generate 512x512, different error than just --medvram)
--medvram --disable-nan-check (512x512 works, hires is being generated as a black image)
--medvram --disable-nan-check --precision full --no-half (fails to even generate 512x512, different error than just --medvram)
--medvram --disable-nan-check --precision full (fails to even generate 512x512, different error than just --medvram)
--medvram --disable-nan-check --no-half-vae (512x512 works, hires is being generated as a black image)
(note: even in the few situations where 512x512 works, it's always 30-60% slower than the working setup)

Changing the Upcast cross attention layer to float32 option did not work either.

*** Error completing request
*** Arguments: ('task(a8hvj13tkgqx0f0)', 'Sarah, orange hair, long hair, lips, serious look, solo, SarAtt, hoop earrings, midriff, pink shirt, (denim jeans), shoes, outside, cartoon, night, ', '(easynegative), (badhandv4), greyscale, (muscular:1.5), (abs:1.5),', [], 40, 16, False, False, 1, 1, 7, 1235765650.0, -1.0, 0, 0, 0, False, 512, 512, True, 0.6, 2, 'Latent', 0, 0, 0, 0, '', '', [], 0, True, False, 'LoRA', 'Sarah-ED-10(1a2f2532f9e9)', 0.75, 0.75, 'LoRA', 'None', 0, 0, 'LoRA', 'None', 0, 0, 'LoRA', 'None', 0, 0, 'LoRA', 'None', 0, 0, None, 'Refresh models', False, False, 'positive', 'comma', 0, False, False, '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0) {}
Traceback (most recent call last):
File "D:\Applications\stable-diffusion-webui\modules\call_queue.py", line 55, in f
res = list(func(*args, **kwargs))
File "D:\Applications\stable-diffusion-webui\modules\call_queue.py", line 35, in f
res = func(*args, **kwargs)
File "D:\Applications\stable-diffusion-webui\modules\txt2img.py", line 57, in txt2img
processed = processing.process_images(p)
File "D:\Applications\stable-diffusion-webui\modules\processing.py", line 620, in process_images
res = process_images_inner(p)
File "D:\Applications\stable-diffusion-webui\modules\processing.py", line 739, in process_images_inner
samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
File "D:\Applications\stable-diffusion-webui\modules\processing.py", line 1077, in sample
samples = self.sampler.sample_img2img(self, samples, noise, self.hr_c, self.hr_uc, steps=self.hr_second_pass_steps or self.steps, image_conditioning=image_conditioning)
File "D:\Applications\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 409, in sample_img2img
samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
File "D:\Applications\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 278, in launch_sampling
return func()
File "D:\Applications\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 409, in
samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
File "D:\Applications\stable-diffusion-webui\venv\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "D:\Applications\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 594, in sample_dpmpp_2m
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "D:\Applications\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "D:\Applications\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 190, in forward
devices.test_for_nans(x_out, "unet")
File "D:\Applications\stable-diffusion-webui\modules\devices.py", line 158, in test_for_nans
raise NansException(message)
modules.devices.NansException: A tensor with all NaNs was produced in Unet. This could be either because there's not enough precision to represent the picture, or because your video card does not support half type. Try setting the "Upcast cross attention layer to float32" option in Settings > Stable Diffusion or using the --no-half commandline argument to fix this. Use --disable-nan-check commandline argument to disable this check.

The solution I have to get it back running flawlessly is to follow the steps I found here - https://www.reddit.com/r/StableDiffusion/comments/zsn83k/stable_diffusion_auto111_torchpytorch_install/ (except the xformers part), but I would really love to know if this is something known/expected, if there is an actual way to fix it now, or if this is an issue that someone has to (hopefully) fix at some point in the future.

[dhwz]

Why are you not using xformers? And I think you should use --lowvram not --medvram with 4GB VRAM
Don't use --no-half or --no-half-vae or --precision full or upcast that will increase VRAM usage even more.

[CrisR82]

--medvram has worked for me without any trouble for almost a year up until the change to Torch 2.0.X, putting --lowvram would just unnecessarily slow down the generation speed (as far as I know, if --medvram works, there is no benefit for using --lowvram).

As for xformers, while I haven't tested it myself, from what I heard it had (while extremely minor) impact on the image quality but (more importantly for me) made generating the same image again on a different machine hard/not possible.

[ClashSAN]

you should use --upcast-sampling and --xformers

As for xformers, while I haven't tested it myself, from what I heard it had (while extremely minor) impact on the image quality but (more importantly for me) made generating the same image again on a different machine hard/not possible.

Using the newest xformers, you will create 100% reproducible images for your gpu.

[CrisR82]

That's kind'a the thing - I will soon (well, in less than a year) replace the whole PC so this GPU won't be in use for that much longer. Anyway, is the suggestion just a general one, or a solution to the error message I'm getting in Torch 2.0.X?

[SE4Ldra]

Hi, i had the same issue , The only thing will work is to uninstall torch 2.0 and install Torch 1.13 (Rolling back to old), You cannot use xformers because xformers only compatible with torch 2.0, if u insist installing it , it will replace torch 1.13 to latest torch, you can use arg like --medvram --opt-split-attention
GTX 1650 and Super, had trouble with torch 2.0

[GawainProdigy]

I found that extensions and their requirements and/or dependencies sometimes conflict after you install a new one, leading to performance problems.

A starting point to solve your problem is 'the conflict checker to check for incompatibilities: pip check

1. Navigate to the directory containing your virtual environment:
   cd C:\insert your directory for stable diffusion\stable-diffusion-webui\venv

2. Activate the virtual environment:
   .\Scripts\activate

3. Check for compatibility issues
   pip check

I ran 'pip check' and identified what was conflicting and what versions would be required. It will show some things that need upgrading and some downgrading.

Be prepared, after your first fix, a new conflict might appear, and you deal with them one by one to find the lowest version of each dependency that would be compatible with your A1111 setup and your extensions.

Don't forget to activate an environment in your venv.

Please verify this information before doing this, I am by no means an expert, but I figured it out. So, don't hold me accountable if something goes wrong.

If you need help with how to perform any of those, check with ChatGPT for tips; but, verify everything via google or on here.
Don't believe everything that ChatGPT tells you. The free chatgpt doesn't know anything about A1111, it can only read the code answer some things well, and usually make good guesses because it understands Python code. But sometimes it might tell you the wrong thing to do.

Good luck.

[CrisR82]

Kind'a late of a response from my side, holiday season 'n all that. While your suggestion here did not resolve the issue by itself, it was a bit part in actually reaching a solution - I was able to not only get it to work, but also use a SDXL model on the GTX1650.

As a summary, what I did was:

• update WebUI (I was doing this in general)
• updated PIP (2.23.2)
• reinstalled Torch
• installed pyre-extensions 0.0.29
• enabled "Upcast cross attention layer to float32" in Settings->Stable Diffusion
• added launch parameters "--medvram --no-half --xformers"

NOTE: currently on the nVidia studio driver v546.33, applying the SD-specific fix
NOTE 2: while I have a bunch of extensions installed, none of them are enabled except "tag-autocomplete" (they were updated but I highly doubt that did anything fix-wise)

Once this was completed, I was left with the following version into in WebUI:
version: v1.7.0
python: 3.10.6
torch: 2.0.1+cu118
xformers: 0.0.20
gradio: 3.41.2

With this, I was able to run a full text2image generation at 512x512 and then a hires to 1024x1024, saving both images successfully multiple times.
512x512 was running at 17-19s/it
1024x1012 was running at 103-105s/it

What ended up happening was all 4GB VRAM of my GPU were in-use along with 12-12.5GB of the regular RAM at all times during the process and while saving the images, there was a period of about a minute where the whole PC would become unresponsive/extra laggy while saving it (no crashing or errors). I did have 40+ Chrome tabs open and all of them were pushed out of RAM during this process, I didn't manually close anything (though I DID have hardware acceleration disabled for the browser).

IMPORTANT NOTE: the SDXL model I was interested in using in particular specifically said the "--xformers" or "--no-half" are required so that might have influenced the results, I haven't had the chance to experiment with non-XL (or properly made XL) models yet so there IS a chance one of those parameters or the float32 setting might not be needed and result in a slight speed boost. I'll update this post if I get around to figuring that out.

TL;DR - it CAN work though it's really slow...which is better than nothing so there's that! :)

UPDATE: managed to do some extra tests, the "--no-half" is not needed, removing it gave a decent speed-boost (too bad the model I use specifically requires it...).