Godot mono server/headless failed to run dedicated server

[nongvantinh]

Godot version

3.4.4-stable_mono_linux_headless/server
v3.5.stable.mono.official.991bb6ac7 server/headless

System information

ubuntu 20.04

Issue description

Godot mono server/headless can not read exported PCK/ZIP project. Doesn't even run if it is the game project itself.

Read the PCK/ZIP using SERVER version result in ERROR: Cannot instance script because the project assembly is not loaded.

Read the PCK/ZIP using HEADLESS version result in:
ERROR: Failed to open ...
ERROR: The assembly is invalidated and the prebuilt assemblies are missing.
and engine crash

nongvantinh@Monday:/mnt/d/Projects/Godot/Godot_v3.5-stable_mono_linux_server_64$ ./Godot_v3.5-stable_mono_linux_server.64 --main-pack ../can_not_run_pck_zip/builds/server/can_not_read_pck_zip.zip
Godot Engine v3.5.stable.mono.official.991bb6ac7 - https://godotengine.org

Mono: Log file is: '/home/nongvantinh/.local/share/godot/app_userdata/can_not_run_pck_zip/mono/mono_logs/2022-08-19_00.55.41_4034.log'
ERROR: Cannot instance script because the project assembly is not loaded. Script: 'res://TheCSharpScript.cs'.
   at: can_instance (modules/mono/csharp_script.cpp:2889)
^C
nongvantinh@Monday:/mnt/d/Projects/Godot/Godot_v3.5-stable_mono_linux_server_64$

And using headless version, it failed to open some files and crash.

nongvantinh@Monday:/mnt/d/Projects/Godot/Godot_v3.5-stable_mono_linux_headless_64$ ./Godot_v3.5-stable_mono_linux_headless.64 --main-pack ../can_not_run_pck_zip/builds/server/can_not_read_pck_zip.zip
Godot Engine v3.5.stable.mono.official.991bb6ac7 - https://godotengine.org

Mono: Log file is: '/home/nongvantinh/.local/share/godot/mono/mono_logs/2022-08-19_00.55.25_3929.log'
ERROR: Failed to open res://.mono/assemblies/Debug/GodotSharp.xml
   at: copy (core/os/dir_access.cpp:297)
WARNING: Failed to copy 'GodotSharp.xml'.
     at: copy_prebuilt_api_assembly (modules/mono/mono_gd/gd_mono.cpp:676)
ERROR: Failed to open res://.mono/assemblies/Debug/GodotSharp.pdb
   at: copy (core/os/dir_access.cpp:297)
WARNING: Failed to copy 'GodotSharp.pdb'.
     at: copy_prebuilt_api_assembly (modules/mono/mono_gd/gd_mono.cpp:680)
ERROR: Failed to open res://.mono/assemblies/Debug/GodotSharp.dll
   at: copy (core/os/dir_access.cpp:297)
ERROR: Failed to copy 'GodotSharp.dll'.
   at: copy_prebuilt_api_assembly (modules/mono/mono_gd/gd_mono.cpp:684)
ERROR: The assembly is invalidated and the prebuilt assemblies are missing.
   at: _load_api_assemblies (modules/mono/mono_gd/gd_mono.cpp:940)

=================================================================
        Native Crash Reporting
=================================================================
Got a UNKNOWN while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================

=================================================================
        Native stacktrace:
=================================================================
        0xf47e6d - ./Godot_v3.5-stable_mono_linux_headless.64 :
        0xf481f9 - ./Godot_v3.5-stable_mono_linux_headless.64 :
        0xf3a8af - ./Godot_v3.5-stable_mono_linux_headless.64 :
        0xf51e50 - ./Godot_v3.5-stable_mono_linux_headless.64 :
        0x7fe885db23c0 - /lib/x86_64-linux-gnu/libpthread.so.0 :
        0x1398b56 - ./Godot_v3.5-stable_mono_linux_headless.64 : _ZN6GDMono20_load_api_assembliesEv
        0x1398c0d - ./Godot_v3.5-stable_mono_linux_headless.64 : _ZN6GDMono26initialize_load_assembliesEv
        0x16f5857 - ./Godot_v3.5-stable_mono_linux_headless.64 : _ZN14CSharpLanguage4initEv
        0x373fee2 - ./Godot_v3.5-stable_mono_linux_headless.64 : _ZN12ScriptServer14init_languagesEv
        0x11af968 - ./Godot_v3.5-stable_mono_linux_headless.64 : _ZN4Main6setup2Em
        0x11b4751 - ./Godot_v3.5-stable_mono_linux_headless.64 : _ZN4Main5setupEPKciPPcb
        0xe6b63d - ./Godot_v3.5-stable_mono_linux_headless.64 : main
        0x7fe885a720b3 - /lib/x86_64-linux-gnu/libc.so.6 : __libc_start_main
        0xe6ef8e - ./Godot_v3.5-stable_mono_linux_headless.64 :

=================================================================
        Telemetry Dumper:
=================================================================
Pkilling 0x140636585977600x from 0x140636651299840x
Could not exec mono-hang-watchdog, expected on path '/mnt/d/Projects/Godot/Godot_v3.5-stable_mono_linux_headless_64/GodotSharp/Mono/etc/../bin/mono-hang-watchdog' (errno 2)
WARNING: A Thread object has been destroyed without wait_to_finish() having been called on it. Please do so to ensure correct cleanup of the thread.
     at: ~Thread (core/os/thread.cpp:124)
Entering thread summarizer pause from 0x140636651299840x
Finished thread summarizer pause from 0x140636651299840x.
Failed to create breadcrumb file (null)/crash_hash_0x0

Waiting for dumping threads to resume

=================================================================
        External Debugger Dump:
=================================================================
mono_gdb_render_native_backtraces not supported on this platform, unable to find gdb or lldb

=================================================================
        Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x1398b56):0x1398b46  03 bf a9 3a d8 03 e8 2f 8d 29 02 e8 aa b9 26 02  ...:.../.)....&.
0x1398b56  0f 0b 0f 1f 84 00 00 00 00 00 45 31 c9 41 b8 b0  ..........E1.A..
0x1398b66  96 d8 03 b9 1c 5e ce 03 ba c9 03 00 00 e9 11 ff  .....^..........
0x1398b76  ff ff 0f 1f 84 00 00 00 00 00 45 31 c9 41 b8 70  ..........E1.A.p

=================================================================
        Managed Stacktrace:
=================================================================
=================================================================
Aborted
nongvantinh@Monday:/mnt/d/Projects/Godot/Godot_v3.5-stable_mono_linux_headless_64$

Steps to reproduce

Export -> Add Presets -> Linux/X11 -> Export PCK/Zip

Method 1:

1. Zip the folder contain the file in to an archive.
2. Use scp to copy the archive file just created from local machine to the server: scp can_not_read_pck_zip.zip user@host:~/Downloads.
3. ssh to remote server, unzip the file, and download Godot mono Server/Headless.
4. Run the exported PCK/ZIP with command godot3.5.exe --main-pack can_not_read_pck_zip.zip.

Method 2:

1. Setup WSL on Windows, download Godot mono Server/Headless
2. Unzip the Godot mono Server/Headless somewhere near the PCK/ZIP file.
3. Shift -> right click Open Linux shell here.
4. Run the exported PCK/ZIP with command: godot3.5.exe --main-pack ../builds/can_not_read_pck_zip.zip

Minimal reproduction project

[nongvantinh]

@Chaosus Can this problem be priority?

Minimal reproduction project

can_not_run_pck_zip.zip

[Chaosus]

Well, we could add this issue to a PR meeting list, to discuss, there is no other way to speed up the solvation of the problem.

[brahmkshatriya]

serverTest.zip
Here's a minimal reproduction,
All you gotta do is :

• Open the project & export it for Linux as a pck/zip
• Run the exported files with server binary
• results as the above error

weirdly enough, exporting directly as a linux binary(.x86_64 format) & running that export, it doesnt cause this error

[nongvantinh]

First try:
Instead of printing "Hello World" in C#, I changed the script to GDScript.

The mono_linux_server version now can print out the "Hello World" message:

nongvantinh@Monday:/mnt/d/Projects/Godot/Godot_v3.5-stable_mono_linux_server_64$ ./Godot_v3.5-stable_mono_linux_server.64 --main-pack ../3.5_development/builds/server/3.5_development.zip
Godot Engine v3.5.stable.mono.official.991bb6ac7 - https://godotengine.org

Mono: Log file is: '/home/nongvantinh/.local/share/godot/app_userdata/3.5_development/mono/mono_logs/2022-08-19_12.30.18_73.log'
Hello world

But the mono_linux_headless version crashes:

nongvantinh@Monday:/mnt/d/Projects/Godot/Godot_v3.5-stable_mono_linux_headless_64$ ./Godot_v3.5-stable_mono_linux_headless.64 --main-pack ../3.5_development/builds/server/3.5_development.zip
Godot Engine v3.5.stable.mono.official.991bb6ac7 - https://godotengine.org

Mono: Log file is: '/home/nongvantinh/.local/share/godot/mono/mono_logs/2022-08-19_12.30.32_78.log'
ERROR: Could not create directory: res://.mono
   at: make_dir_recursive (core/os/dir_access.cpp:180)
ERROR: Failed to create destination directory for the API assemblies. Error: 2.
   at: copy_prebuilt_api_assembly (modules/mono/mono_gd/gd_mono.cpp:667)
ERROR: The assembly is invalidated and the prebuilt assemblies are missing.
   at: _load_api_assemblies (modules/mono/mono_gd/gd_mono.cpp:940)

=================================================================
        Native Crash Reporting
=================================================================
Got a UNKNOWN while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================

=================================================================
        Native stacktrace:
=================================================================
        0xf47e6d - ./Godot_v3.5-stable_mono_linux_headless.64 :
        0xf481f9 - ./Godot_v3.5-stable_mono_linux_headless.64 :
        0xf3a8af - ./Godot_v3.5-stable_mono_linux_headless.64 :
        0xf51e50 - ./Godot_v3.5-stable_mono_linux_headless.64 :
        0x7fa839b8f3c0 - /lib/x86_64-linux-gnu/libpthread.so.0 :
        0x1398b56 - ./Godot_v3.5-stable_mono_linux_headless.64 : _ZN6GDMono20_load_api_assembliesEv
        0x1398c0d - ./Godot_v3.5-stable_mono_linux_headless.64 : _ZN6GDMono26initialize_load_assembliesEv
        0x16f5857 - ./Godot_v3.5-stable_mono_linux_headless.64 : _ZN14CSharpLanguage4initEv
        0x373fee2 - ./Godot_v3.5-stable_mono_linux_headless.64 : _ZN12ScriptServer14init_languagesEv
        0x11af968 - ./Godot_v3.5-stable_mono_linux_headless.64 : _ZN4Main6setup2Em
        0x11b4751 - ./Godot_v3.5-stable_mono_linux_headless.64 : _ZN4Main5setupEPKciPPcb
        0xe6b63d - ./Godot_v3.5-stable_mono_linux_headless.64 : main
        0x7fa83984e0b3 - /lib/x86_64-linux-gnu/libc.so.6 : __libc_start_main
        0xe6ef8e - ./Godot_v3.5-stable_mono_linux_headless.64 :

=================================================================
        Telemetry Dumper:
=================================================================
Pkilling 0x140360432465664x from 0x140360496092160x
Could not exec mono-hang-watchdog, expected on path '/mnt/d/Projects/Godot/Godot_v3.5-stable_mono_linux_headless_64/GodotSharp/Mono/etc/../bin/mono-hang-watchdog' (errno 2)
Entering thread summarizer pause from 0x140360496092160x
Finished thread summarizer pause from 0x140360496092160x.
Failed to create breadcrumb file (null)/crash_hash_0x0
WARNING: A Thread object has been destroyed without wait_to_finish() having been called on it. Please do so to ensure correct cleanup of the thread.
     at: ~Thread (core/os/thread.cpp:124)

Waiting for dumping threads to resume

=================================================================
        External Debugger Dump:
=================================================================
mono_gdb_render_native_backtraces not supported on this platform, unable to find gdb or lldb

=================================================================
        Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x1398b56):0x1398b46  03 bf a9 3a d8 03 e8 2f 8d 29 02 e8 aa b9 26 02  ...:.../.)....&.
0x1398b56  0f 0b 0f 1f 84 00 00 00 00 00 45 31 c9 41 b8 b0  ..........E1.A..
0x1398b66  96 d8 03 b9 1c 5e ce 03 ba c9 03 00 00 e9 11 ff  .....^..........
0x1398b76  ff ff 0f 1f 84 00 00 00 00 00 45 31 c9 41 b8 70  ..........E1.A.p

=================================================================
        Managed Stacktrace:
=================================================================
=================================================================
Aborted
nongvantinh@Monday:/mnt/d/Projects/Godot/Godot_v3.5-stable_mono_linux_headless_64$

Minimal Reproduction Project

3.5_development - Copy.zip

[stevepryde]

I'm getting the same issue, both when running under WSL2 on Windows 11 and when running under docker (debian bullseye).

Notice the log file: /home/nongvantinh/.local/share/godot/mono/mono_logs/2022-08-19_12.30.32_78.log

I checked this log on my machine and there were errors saying the GodotSharp/Mono/lib/mono/4.5/mscorlib.dll.so file could not be found. Indeed if I view that directory, the files are all .dll. There are no .so files at all.

Is it possible the GodotSharp that gets output is not for the correct OS? I plan to try building this on Linux directly and then copying the GodotSharp directory from that build and seeing if it's different.

[neikeq]

Where is the data_* or GodotSharp directory exported to? Is it next to the executable? Inside or next to the zip?

[nongvantinh]

Where is the data_* or GodotSharp directory exported to?

Beside to the executable file, inside the same folder.

[stevepryde]

I've been running into this randomly. Somehow the issue went away and then came back again.
I discovered that copying the .mono directory from my project dir and pasting it into the zip file (overwriting the .mono directory there) got it working.

[Gleadex]

I'm getting this error too. I've even tried versions as far back as 3.3.4 - error is slightly different on this version:

ERROR: can_instance: Method failed. Returning: __null At: modules/mono/csharp_script.cpp:2907.

At this current time, I've been unable to successfully run a linux server build as far back as version 3.3.4 due to this error.