MPS implementation command buffer errors

[SoylentGraham]

Often -but not always- when getting NaNs when training on a mac, (m1 Max 2020 mac studio) I'm getting this output in xcode (I think only if Metal api validation is on)
These are also coinciding with bad resulting data (presumably because that execution didn't finish and big blobs are left in the data)

Training continues, and sometimes get more of these.
I also think it coincides with a sustained period of 100% gpu usage, so maybe the OS is overheating a bit, but I'm hoping this is an error that can be caught and undo/not commit any changes, re-run a step etc

Execution of the command buffer was aborted due to an error during execution. Internal Error (0000000e:Internal Error)
Error: command buffer exited with error status.
	The Metal Performance Shaders operations encoded on it may not have completed.
	Error: 
	(null)
	Internal Error (0000000e:Internal Error)
	<AGXG13XFamilyCommandBuffer: 0x14c689290>
    label = <none> 
    device = <AGXG13XDevice: 0x13a864400>
        name = Apple M1 Max 
    commandQueue = <AGXG13XFamilyCommandQueue: 0x14c93fa00>
        label = <none> 
        device = <AGXG13XDevice: 0x13a864400>
            name = Apple M1 Max 
    retainedReferences = 1
Step 101: nan (20%)

command buffer exited with error status

Haven't looked into if this message is coming from the OS or from opensplat/libtorch etc, but might be somewhere to start.

[SoylentGraham]

I don't think these are timeouts, but they do often (but again, not always) seem to coincide with runs where my OS ends up with my gpu stuck at 100%* (often dragging down the rest of the os) and often nan loss, I think something is going wrong in a kernel (either in libtorch or in this repos) and maybe getting stuck in something like an infinite loop (not seeing any wild values in dispatch groups)

Typically after ~2000 steps with banana this is happening, but with my own sets (not sure if my input data is very good) it's happening more often

Unfortunetly I cannot use any metal debug tools, as libtorch's internal kernels throws up some api-validation errors. (Will try and fix this once I manage to get libtorch to compile.... no luck so far)

Going to add a step to cut out any invalid gaussians to see if this stops happening.

Additional note; typically stuck on backward(), but sometimes forward

• using https://github.com/exelban/stats to monitor cpu/gpu - gpu just 100% for WindowServer process in activity mon.

[SoylentGraham]

Well, this didn't take too long
519de6652fa9b26795b63bd4b70aa8b267b0d813

Step 840: 0.131933 (42%)
Warning; Found 1 NaNs in restFeatures data
.....
Step 850: 0.0656216 (42%)
Warning; Found 1 NaNs in restFeatures data
Added 20239 gaussians, new count 68073
Culled 5686 gaussians, remaining 62387
Step 900: 0.0740231 (45%)
Step 910: 0.10871 (45%)

Wonder if it got culled...

Also found bad opacities, means, but didn't lock up gpu