[Android][CoreCLR] Android.Device_Emulator.JIT.Test crashes with SIGSYS

[simonrozsival]

#115330 showed that Android.Device_Emulator.JIT.Test work item crashes with the following output on several Android API levels:

05-06 11:27:48.155  2909  2925 D DOTNET  : Java_net_dot_MonoRunner_setEnv:
05-06 11:27:48.155  2909  2925 D DOTNET  : Setting env: HOME=/data/user/0/net.dot.Android.Device_Emulator.JIT.Test/files
05-06 11:27:48.155  2909  2925 D DOTNET  : Java_net_dot_MonoRunner_setEnv:
05-06 11:27:48.155  2909  2925 D DOTNET  : Setting env: TMPDIR=/data/user/0/net.dot.Android.Device_Emulator.JIT.Test/cache
05-06 11:27:48.155  2909  2925 D DOTNET  : Java_net_dot_MonoRunner_setEnv:
05-06 11:27:48.155  2909  2925 D DOTNET  : Setting env: TEST_RESULTS_DIR=/storage/emulated/0/Android/data/net.dot.Android.Device_Emulator.JIT.Test/files/Documents
05-06 11:27:48.155  2909  2925 I DOTNET  : MonoRunner initializeRuntime, entryPointLibName=Android.Device_Emulator.JIT.Test.dll
05-06 11:27:48.421  2909  2925 D DOTNET  : Java_net_dot_MonoRunner_initRuntime (CoreCLR):
05-06 11:27:48.421  2909  2925 D DOTNET  : mono_droid_runtime_init (CoreCLR) called with executable: Android.Device_Emulator.JIT.Test.dll
05-06 11:27:48.421  2909  2925 D DOTNET  : Calling coreclr_initialize
--------- beginning of crash
05-06 11:27:48.517  2909  2925 F libc    : Fatal signal 31 (SIGSYS), code 1 in tid 2925 (.dot.MonoRunner)
05-06 11:27:48.712  2931  2931 W crash_dump64: type=1400 audit(0.0:34): avc: denied { search } for name="files" dev="dm-0" ino=98989 scontext=u:r:crash_dump:s0:c512,c768 tcontext=u:object_r:app_data_file:s0:c512,c768 tclass=dir permissive=0
05-06 11:27:48.732  2931  2931 I chatty  : uid=10065(u0_a65) crash_dump64 identical 1 line
05-06 11:27:48.775  2931  2931 I crash_dump64: obtaining output fd from tombstoned
05-06 11:27:48.792  1510  1510 I /system/bin/tombstoned: received crash request for pid 2909
05-06 11:27:48.793  2931  2931 I crash_dump64: performing dump of process 2909 (target tid = 2925)
05-06 11:27:48.793  2931  2931 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
05-06 11:27:48.793  2931  2931 F DEBUG   : Build fingerprint: 'Android/sdk_phone_x86_64/generic_x86_64:8.0.0/OSR1.180418.004/4931640:userdebug/test-keys'
05-06 11:27:48.793  2931  2931 F DEBUG   : Revision: '0'
05-06 11:27:48.793  2931  2931 F DEBUG   : ABI: 'x86_64'
05-06 11:27:48.793  2931  2931 F DEBUG   : pid: 2909, tid: 2925, name: .dot.MonoRunner  >>> net.dot.Android.Device_Emulator.JIT.Test <<<
05-06 11:27:48.793  2931  2931 F DEBUG   : signal 31 (SIGSYS), code 1 (SYS_SECCOMP), fault addr --------
05-06 11:27:48.793  2931  2931 F DEBUG   : Cause: seccomp prevented call to disallowed x86_64 system call 0
05-06 11:27:48.793  2931  2931 F DEBUG   :     rax 0000000000000144  rbx 0000000000000000  rcx ffffffffffffffff  rdx 0000000000000000
05-06 11:27:48.793  2931  2931 F DEBUG   :     rsi 0000000000000000  rdi 0000000000000000
05-06 11:27:48.793  2931  2931 F DEBUG   :     r8  0000000000000000  r9  0000744a0e658860  r10 00000000ffffffff  r11 0000000000000246
05-06 11:27:48.793  2931  2931 F DEBUG   :     r12 0000744a1ab1b580  r13 00000000000001ff  r14 0000744a0e658b00  r15 000000000000001a
05-06 11:27:48.793  2931  2931 F DEBUG   :     cs  0000000000000033  ss  000000000000002b
05-06 11:27:48.793  2931  2931 F DEBUG   :     rip 0000744a1d52a6f8  rbp 0000744a059d2130  rsp 0000744a059d2118  eflags 0000000000000246
05-06 11:27:48.795  2931  2931 F DEBUG   : 
05-06 11:27:48.795  2931  2931 F DEBUG   : backtrace:
05-06 11:27:48.795  2931  2931 F DEBUG   :     #00 pc 00000000000276f8  /system/lib64/libc.so (syscall+24)
05-06 11:27:48.795  2931  2931 F DEBUG   :     #01 pc 00000000005acb5a  /data/app/net.dot.Android.Device_Emulator.JIT.Test-I0FZtG-4sfwHbLVXurirLg==/lib/x86_64/libcoreclr.so
05-06 11:27:48.795  2931  2931 F DEBUG   :     #02 pc 000000000058d9c5  /data/app/net.dot.Android.Device_Emulator.JIT.Test-I0FZtG-4sfwHbLVXurirLg==/lib/x86_64/libcoreclr.so
05-06 11:27:48.795  2931  2931 F DEBUG   :     #03 pc 000000000058dcad  /data/app/net.dot.Android.Device_Emulator.JIT.Test-I0FZtG-4sfwHbLVXurirLg==/lib/x86_64/libcoreclr.so
05-06 11:27:48.795  2931  2931 F DEBUG   :     #04 pc 00000000001c111f  /data/app/net.dot.Android.Device_Emulator.JIT.Test-I0FZtG-4sfwHbLVXurirLg==/lib/x86_64/libcoreclr.so (coreclr_initialize+479)
05-06 11:27:48.795  2931  2931 F DEBUG   :     #05 pc 00000000000017f6  /data/app/net.dot.Android.Device_Emulator.JIT.Test-I0FZtG-4sfwHbLVXurirLg==/lib/x86_64/libmonodroid.so (mono_droid_runtime_init+326)
05-06 11:27:48.795  2931  2931 F DEBUG   :     #06 pc 000000000000153b  /data/app/net.dot.Android.Device_Emulator.JIT.Test-I0FZtG-4sfwHbLVXurirLg==/lib/x86_64/libmonodroid.so (Java_net_dot_MonoRunner_initRuntime+507)
05-06 11:27:48.795  2931  2931 F DEBUG   :     #07 pc 00000000000023ff  /data/app/net.dot.Android.Device_Emulator.JIT.Test-I0FZtG-4sfwHbLVXurirLg==/oat/x86_64/base.odex (offset 0x2000)
05-06 11:27:48.795  2931  2931 F DEBUG   :     #08 pc 000000000000000b  <unknown>
05-06 11:27:49.072  2931  2931 W crash_dump64: type=1400 audit(0.0:36): avc: denied { search } for name="files" dev="dm-0" ino=98989 scontext=u:r:crash_dump:s0:c512,c768 tcontext=u:object_r:app_data_file:s0:c512,c768 tclass=dir permissive=0
05-06 11:27:49.164  1510  1510 E /system/bin/tombstoned: Tombstone written to: /data/tombstones//tombstone_00
05-06 11:27:49.165  1626  1645 I BootReceiver: Copying /data/tombstones/tombstone_00 to DropBox (SYSTEM_TOMBSTONE)
05-06 11:27:49.205  1626  1640 E memtrack: Couldn't load memtrack module
05-06 11:27:49.205  1626  1640 W android.os.Debug: failed to get memory consumption info: -1
05-06 11:27:49.212  1626  1886 I ActivityManager: Process net.dot.Android.Device_Emulator.JIT.Test (pid 2909) has died: fore FGS 
05-06 11:27:49.212  1626  1886 W ActivityManager: Crash of app net.dot.Android.Device_Emulator.JIT.Test running instrumentation ComponentInfo{net.dot.Android.Device_Emulator.JIT.Test/net.dot.MonoRunner}
05-06 11:27:49.213  1626  1886 I ActivityManager: Force stopping net.dot.Android.Device_Emulator.JIT.Test appid=10065 user=0: finished inst
05-06 11:27:49.215  1471  1471 I Zygote  : Process 2909 exited due to signal (31)

Full logs: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1034591&view=ms.vss-test-web.build-test-results-tab

This crash was observed on the following helix queues:

• ubuntu.2204.amd64.android.21.open
• ubuntu.2204.amd64.android.22.open
• ubuntu.2204.amd64.android.26.open
• ubuntu.2204.amd64.android.27.open
• ubuntu.2204.amd64.android.28.open

The test passed on the following helix queues:

• ubuntu.2204.amd64.android.23.open
• ubuntu.2204.amd64.android.24.open
• ubuntu.2204.amd64.android.25.open
• ubuntu.2204.amd64.android.29.open
• ubuntu.2204.amd64.android.30.open
• ubuntu.2204.amd64.android.31.open
• ubuntu.2204.amd64.android.32.open
• ubuntu.2204.amd64.android.33.open
• ubuntu.2204.amd64.android.34.open

It's not apparent why this test passes on some API levels and fails on others. There isn't a clear divide which would show that it only fails before a certain API level, so the failures might be somewhat random.

/cc @kotlarmilos @ivanpovazan @vitek-karas

[ivanpovazan]

I think this might be a variant of: #112633 where x64 emulator is corrupted/offline.

On *.amd64.* queues we supposed to have both x64 and x86 emulators available.
And looking at the log before it tries to run the app it shows the following:

[11:25:54] dbug: Found 2 possible devices
[11:25:54] dbug: Evaluating output line for device serial: emulator-5554          offline transport_id:1
[11:25:54] dbug: Executing command: '/datadisks/disk1/work/B1BA09A3/p/microsoft.dotnet.xharness.cli/10.0.0-prerelease.25225.1/runtimes/any/native/adb/linux/adb -s emulator-5554 shell getprop ro.product.cpu.abilist'
[11:25:54] warn: Device emulator-5554 is offline; retrying up to five minutes
[11:25:54] dbug: Attempt 1 failed, retrying in 10 seconds...
[11:26:04] dbug: Executing command: '/datadisks/disk1/work/B1BA09A3/p/microsoft.dotnet.xharness.cli/10.0.0-prerelease.25225.1/runtimes/any/native/adb/linux/adb -s emulator-5554 shell getprop ro.product.cpu.abilist'
[11:26:04] dbug: Evaluating output line for device serial: emulator-5556          offline transport_id:2
[11:26:04] dbug: Executing command: '/datadisks/disk1/work/B1BA09A3/p/microsoft.dotnet.xharness.cli/10.0.0-prerelease.25225.1/runtimes/any/native/adb/linux/adb -s emulator-5556 shell getprop ro.product.cpu.abilist'
[11:26:04] dbug: Found 1 possible devices. Using 'emulator-5556'
[11:26:04] info: Active Android device set to serial 'emulator-5556'

My assumption is that the emulator that gets resolved -> emulator-5556 is in fact with x86 ABI not x64, so when launched the app crashes (not sure how it gets installed though).

It would be good to connect to these machines and check the state of the emulators (and their ABIs) to verify this. Additionally, the logging from xharness at the point of discovery could be improved so that we include information in the logs of which emulator we are using when running the test.

---

PS in any case, seems like an infra issue.