Very slow training w/ standardized trainer - epochs take over a minute

[TobykoMusic]

Thanks for taking the time to write a bug report! Use the following prompts to help you describe what's going on. The more info you provide, the easier it'll be to understand how to fix it for you without having to come back and ask you questions.

Describe the bug
When running NAM on my Mac (M1 Pro MacBook Pro), i get the error

NAM encountered a bug in PyTorch's MPS backend and will switch to a fallback.
Your version of PyTorch is 2.5.1.
Please report this in an Issue at:
https://github.com/sdatkinson/neural-amp-modeler/issues/new/choose
so that NAM's dependencies can avoid buggy versions of PyTorch and the associated performance hit.
MRSTFT failed on device; falling back to CPU

I read somewhere that downgrading PyTorch to <2.0 can help but it seemed to do nothing and is still running with pytorch 2.5.1.
Im sure its an easy fix but im technologically illiterate when it comes to any of this stuff.

Cheers!

Desktop (please complete the following information):

• OS macOS 14.0
• Local?
• Version 0.11.0

[sdatkinson]

Hi!

Did it crash, or did it just continue to run? The progress bar in your snapshot suggests that it's doing fine.

This may just be an old error message I need to clean up

[sdatkinson]

neural-amp-modeler/nam/models/base.py

Lines 212 to 222 in b5de1f4

	print(
	"===WARNING===\n"
	"NAM encountered a bug in PyTorch's MPS backend and will "
	"switch to a fallback.\n"
	f"Your version of PyTorch is {torch.__version__}.\n"
	"Please report this in an Issue at:\n"
	"https://github.com/sdatkinson/neural-amp-modeler/issues/new/choose"
	"\n"
	"so that NAM's dependencies can avoid buggy versions of "
	"PyTorch and the associated performance hit."
	)

[TobykoMusic]

Kia ora. It did continue to run fine but incredibly slowly (like 1.5-2 minutes per epoch). That may just be because of my computer but I remember seeing warnings when troubleshooting before about an "MPS backend" thingy being faster than the fallback (cpu). That's what I assumed the report was about. I am not that bright when it comes to programming or anything like this so it could just be me misunderstanding things. Cheers, T

…

On Fri, 13 Dec 2024 at 5:13 AM, Steven Atkinson ***@***.***> wrote: Hi! Did it crash, or did it just continue to run? The progress bar in your snapshot suggests that it's doing fine. This may just be an old error message I need to clean up — Reply to this email directly, view it on GitHub <#517 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BNUWTR6SBPG67CHJQ237AFL2FGY4NAVCNFSM6AAAAABTPLO3LOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMZZGQYDAMJSGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[binky100]

I have had the same issue on my 2023 macbook pro running osx sonoma with an M3 Pro chip . The local trainer doesnt do anything after plotting the latency graph. The local trainer wont start until I save or close the latency graph and then the terminal gives me the same error as the person who started this thread where my mac says its falling back to the CPU.
Screenshot shows the terminal output

[sdatkinson]

Regarding the original issue here: what we can do is modify the warning message.

It would be interesting to warn people that the version of PyTorch that they're using has this MPS conv false positive, and what versions have this Issue.

As I said in #507 (comment), I don't want to constrain this package not to use those versions because overly-constrained packages aren't fun to work with, but I'm happy to keep a list of versions that folks could use as a stronger requirement to have better functionality.

I'm going to make a new Issue to improve the warning message: #520.

Still, over a minute per epoch is slow, even for a Mac. Let me think about that for a bit.

[binky100]

Is there anything I can do on mac so that the trainer doesnt keep falling back to the CPU?

[sdatkinson]

@binky100 Please make a separate issue with your problem including the details that the template asks for.

[sdatkinson]

Renamed this Issue to focus on OP's ">1 minute epoch" behavior.