Need some suggestions for the debugging

[amos402]

Proton version: Proton Experimental [debug]
A crash caused the hang of the game. Here are some issues I have found while I'm trying to debug the problem.

Regarding the Windows live debugging, the latest SteamOS Devkit Client no longer has Remote Debugger and Wait for attach options on Title Upload tab, as demonstrated https://partner.steamgames.com/doc/steamdeck/debugging. Is it a bug or what? I also found that I can't find the installation address of SteamOS Devkit Service anymore.

Therefore, I have to export STEAM_COMPAT_DATA_PATH by myself, then use python "/home/deck/.local/share/Steam/steamapps/common/Proton - Experimental/proton" runinprefix "Z:/home/deck/.local/share/Steam/steamapps/common/Proton - Experimental/msvsmon2022/x64/msvsmon.exe" /noclrwarn /nowowwarn /nofirewallwarn /anyuser /noauth /nosecuritywarn /silent to launch the remote debugger manually.

But unfortunately, it only shows the last frame of dbghelp.dll. Just to be clear, I added https://proton-archive.steamos.cloud/ to the symbol search paths, as well as the path of my own exe.

I also tried the GDB debugging:

Just a bunch of addresses, nothing helps.
Then I tried to distinguish which modules these frames are inside by matching the info proc mappings, still didn't get any useful information.

#0 /run/host/usr/lib/libc.so.6
#1 /home/deck/.local/share/Steam/steamapps/common/Proton - Experimental/files/lib/wine/x86_64-unix/ntdll.so
#2 ??
#3 /home/deck/.local/share/Steam/steamapps/common/Proton - Experimental/files/lib/wine/x86_64-unix/ntdll.so
#4 ??
#5 /run/host/usr/lib/libc.so.6
#6 ??
#7 ??
#8 /home/deck/.local/share/Steam/steamapps/common/Proton - Experimental/files/lib/wine/x86_64-unix/ntdll.so
#9 ??

So I wonder what I can do next. Any suggestions would be helpful.

[ivyl]

For the debugging options to be available you have to select Runtime: to be Steam Play (Proton).

Looks like it has slightly changed since https://partner.steamgames.com/doc/steamdeck/debugging was written. I've asked if it can be updated.

I'll look into issues with the symbols shortly. Last time I've checked it worked well so this is surprising.

[amos402]

For the debugging options to be available you have to select Runtime: to be Steam Play (Proton).

• We just used the old config for uploading the Title, didn't notice that changed at the begging, thank you for pointing it out.

• Regarding to deploying a Title on Linux PC, I used to use SteamOS Devkit Service to serve as the server, but now, I can't find any place can install it, seems the document no longer mention it. So our new test accounts can't install it directly. Also it can't work properly by default anymore, I have to set it run on Legacy runtime 1.0, otherwise it would complain the system is not Manjaro on it's sandbox. The system is the latest Manjaro 25.0.10
  So is there any new method for it?

• About debugging on Proton, that can't get the Windows callstacks, we encountered similar problem multiple times, yet we don't an effective method to debug it, so is there any technique for dealing with it? I also tried winedbg, it didn't help too, maybe I'm just heading the wrong way.

[ivyl]

Regarding to deploying a Title on Linux PC, I used to use SteamOS Devkit Service to serve as the server, but now, I can't find any place can install it, seems the document no longer mention it. So our new test accounts can't install it directly. Also it can't work properly by default anymore, I have to set it run on Legacy runtime 1.0, otherwise it would complain the system is not Manjaro on it's sandbox. The system is the latest Manjaro 25.0.10
So is there any new method for it?

Looks like that might have been deprecated?

https://gitlab.steamos.cloud/devkit/steamos-devkit/-/blob/main/ChangeLog#L30

I think the alternative is to use something running SteamOS or manually run remote debugging. @TTimo may have some pointers here.

About debugging on Proton, that can't get the Windows callstacks, we encountered similar problem multiple times, yet we don't an effective method to debug it, so is there any technique for dealing with it? I also tried winedbg, it didn't help too, maybe I'm just heading the wrong way.

Running any of the non-debug builds of Proton (e.g Proton Experimental, not set to debug branch, just left on "default") the symbols do exist in https://proton-archive.steamos.cloud/.

I've just tried running a simple program, forced a minidump and:

I have some issues with VS22 currently though. On my setup it doesn't even attempt to query anything from the symbol server - I've set up a test one and there's no incoming HTTP requests at all. It used to work. I'll be looking into that.

The symbols for my app are resolved correctly though:

[amos402]

If I attach VS, make it hit a breakpoint before the crash, the symbols are resolved correctly, and the display of call stacks shows normal.
Generally, our program can generate a dump if it encounters a crash, but not in this case.
It has two cases when I test it if no debugger is attached:

1. The process became zombie, but it won't exit, and the UI just froze.
2. Fullspeed on the main thread, once I attach VS to it, tons of ntdll's frames, the bottom is the dbghelp.dll, which I showed in my first post.

I tried both the non-debug and debug versions of Proton Experimental, same symptoms.
I tried to use procdump, the tool of SysinternalsSuite, to generate a dump, which it's not the zombie process case. It shows only one ntdll frame.

[ivyl]

That's very surprising. Either the stack gets mangled very badly or something crashes on the Linux side in a way that takes the process down. We are resilient against segfault and other typical crashes. It should result in the faux syscall / unix call failing with the appropriate status.

Can you try collect Proton logs with launch command set to PROTON_LOG=1 %command% and see if there's anything suspicious there? Also if there's a core dump generated (coredumpctl). I'm interested in understanding this case and I can help with reading the Proton log or even trying out the build if that's not off limits.

You can find my CodeWeavers email address in git history of this repo if you prefer share things privately.