[mono] android crash : monoeg_g_hash_table_insert_replace [ghashtable.c : 243]

[srxqds]

signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------

    x0  0  x1  53c8  x2  6  x3  6d9f5b8d80

    x4  6064671f6a6d7467  x5  6064671f6a6d7467  x6  6064671f6a6d7467  x7  7f7f7f7f7f7f7f7f

    x8  f0  x9  7166adba98  x10 1  x11 7166b23540

    x12 2025cc74  x13 1af79eb58368  x14 1ea343e7f5dfaa  x15 34155555

    x16 7166b87f38  x17 7166b65370  x18 6d35024000  x19 5209

    x20 53c8  x21 ffffffff  x22 0  x23 b400006ff611d908

    x24 6df3e9e1c8  x25 b400006efaa33578  x26 a000000  x27 116

    x28 b4000070ba053bc3  x29 6d9f5b8e00  lr  7166b14d94  sp  6d9f5b8d60

    pc  7166b14dc0  pst 1000

backtrace:

    #00 pc 000000000005bdc0  /apex/com.android.runtime/lib64/bionic/libc.so (abort+164) (BuildId: d638ba9bdf4cea2cddd9bd06ae04407f)

    #01 pc 0000000000048f1c  /apex/com.android.runtime/lib64/bionic/libc.so (scudo::die()+8) (BuildId: d638ba9bdf4cea2cddd9bd06ae04407f)

    #02 pc 00000000000497d8  /apex/com.android.runtime/lib64/bionic/libc.so (scudo::ScopedErrorReport::~ScopedErrorReport()+32) (BuildId: d638ba9bdf4cea2cddd9bd06ae04407f)

    #03 pc 0000000000049a24  /apex/com.android.runtime/lib64/bionic/libc.so (scudo::reportHeaderCorruption(void*)+96) (BuildId: d638ba9bdf4cea2cddd9bd06ae04407f)

#04 pc 000000000004b6a0  /apex/com.android.runtime/lib64/bionic/libc.so (scudo::Allocator<scudo::AndroidConfig, &(scudo_malloc_postinit)>::deallocate(void*, scudo::Chunk::Origin, unsigned long, unsigned long)+296) (BuildId: d638ba9bdf4cea2cddd9bd06ae04407f)

    #05 pc 00000000001d8298  /data/app/~~UwgwN6v5uxWtMNVFvl8ysA==/com.xxx.yyy-n-bsx4VqiHRzD9C2ehQiSw==/lib/arm64/libmonosgen-2.0.so (monoeg_g_hash_table_insert_replace [src/mono/mono/eglib/ghashtable.c : 243 + 0x0]) (BuildId: 42c8b61931986227e9bad5b46aa5cb7748fd9c9b)

    #06 pc 000000000023f9a4  /data/app/~~UwgwN6v5uxWtMNVFvl8ysA==/com.xxx.yyy-n-bsx4VqiHRzD9C2ehQiSw==/lib/arm64/libmonosgen-2.0.so (mono_get_method_checked [src/mono/mono/metadata/loader.c : 0 + 0x0]) (BuildId: 42c8b61931986227e9bad5b46aa5cb7748fd9c9b)

    #07 pc 00000000000f7594  /data/app/~~UwgwN6v5uxWtMNVFvl8ysA==/com.xxx.yyy-n-bsx4VqiHRzD9C2ehQiSw==/lib/arm64/libmonosgen-2.0.so (mini_get_method [src/mono/mono/mini/method-to-ir.c : 5094 + 0x0]) (BuildId: 42c8b61931986227e9bad5b46aa5cb7748fd9c9b)

    #08 pc 00000000000dd5e0  /data/app/~~UwgwN6v5uxWtMNVFvl8ysA==/com.xxx.yyy-n-bsx4VqiHRzD9C2ehQiSw==/lib/arm64/libmonosgen-2.0.so (mono_method_to_ir [src/mono/mono/mini/method-to-ir.c : 7610 + 0x0]) (BuildId: 42c8b61931986227e9bad5b46aa5cb7748fd9c9b)

#09 pc 00000000000cd1ac  /data/app/~~UwgwN6v5uxWtMNVFvl8ysA==/com.xxx.yyy-n-bsx4VqiHRzD9C2ehQiSw==/lib/arm64/libmonosgen-2.0.so (inline_method [src/mono/mono/mini/method-to-ir.c : 4825 + 0x0]) (BuildId: 42c8b61931986227e9bad5b46aa5cb7748fd9c9b)

    #10 pc 00000000000e6c74  /data/app/~~UwgwN6v5uxWtMNVFvl8ysA==/com.xxx.yyy-n-bsx4VqiHRzD9C2ehQiSw==/lib/arm64/libmonosgen-2.0.so (mono_method_to_ir [src/mono/mono/mini/method-to-ir.c : 7873 + 0x0]) (BuildId: 42c8b61931986227e9bad5b46aa5cb7748fd9c9b)

    #11 pc 00000000000be0a8  /data/app/~~UwgwN6v5uxWtMNVFvl8ysA==/com.xxx.yyy-n-bsx4VqiHRzD9C2ehQiSw==/lib/arm64/libmonosgen-2.0.so (mini_method_compile [src/mono/mono/mini/mini.c : 3498 + 0x0]) (BuildId: 42c8b61931986227e9bad5b46aa5cb7748fd9c9b)

    #12 pc 00000000000c06e4  /data/app/~~UwgwN6v5uxWtMNVFvl8ysA==/com.xxx.yyy-n-bsx4VqiHRzD9C2ehQiSw==/lib/arm64/libmonosgen-2.0.so (mono_jit_compile_method_inner [src/mono/mono/mini/mini.c : 4132 + 0x0]) (BuildId: 42c8b61931986227e9bad5b46aa5cb7748fd9c9b)

    #13 pc 00000000000c53b0  /data/app/~~UwgwN6v5uxWtMNVFvl8ysA==/com.xxx.yyy-n-bsx4VqiHRzD9C2ehQiSw==/lib/arm64/libmonosgen-2.0.so (jit_compile_method_with_opt [src/mono/mono/mini/mini-runtime.c : 2884 + 0x0]) (BuildId: 42c8b61931986227e9bad5b46aa5cb7748fd9c9b)

   #14 pc 00000000000c481c  /data/app/~~UwgwN6v5uxWtMNVFvl8ysA==/com.xxx.yyy-n-bsx4VqiHRzD9C2ehQiSw==/lib/arm64/libmonosgen-2.0.so (mono_jit_compile_method [src/mono/mono/mini/mini-runtime.c : 2903 + 0x0]) (BuildId: 42c8b61931986227e9bad5b46aa5cb7748fd9c9b)

    #15 pc 00000000001559ec  /data/app/~~UwgwN6v5uxWtMNVFvl8ysA==/com.xxx.yyy-n-bsx4VqiHRzD9C2ehQiSw==/lib/arm64/libmonosgen-2.0.so (common_call_trampoline [src/mono/mono/mini/mini-trampolines.c : 628 + 0x0]) (BuildId: 42c8b61931986227e9bad5b46aa5cb7748fd9c9b)

    #16 pc 0000000000155550  /data/app/~~UwgwN6v5uxWtMNVFvl8ysA==/com.xxx.yyy-n-bsx4VqiHRzD9C2ehQiSw==/lib/arm64/libmonosgen-2.0.so (mono_magic_trampoline [src/mono/mono/mini/mini-trampolines.c : 769 + 0x0]) (BuildId: 42c8b61931986227e9bad5b46aa5cb7748fd9c9b)

    #17 pc 0000000000004300  <anonymous:7156a8d000>

[dotnet-policy-service]

Tagging subscribers to 'arch-android': @vitek-karas, @simonrozsival, @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

[steveisok]

/cc @kg @lateralusX

[kg]

This looks like heap corruption

[steveisok]

@srxqds can you provide more info around what your app is doing? Is it possible to provide a small repro?

[kg]

The mono_get_method_checked [src/mono/mono/metadata/loader.c : 0 + 0x0] means we don't have exact location information for the crash, but there are only a few potential call sites for insert_replace in this function.

Depending on whether the token is a method or a methodref, we're either touching the image->method_cache or the image->methodref_cache. This kind of crash inside of a hashtable op - what appears to be a free crashing due to detected heap corruption - could indicate a bunch of things.

• The first thing we do is lock the image. I would expect this to crash if the image pointer is invalid.
• Next we do is look up the method token inside the image's cache, unless the image is dynamic. If we had a corrupt image pointer or a corrupt cache pointer I would expect a crash here. If the image is dynamic the cache isn't used.
• Next, if the lookup didn't find a result, we call mono_get_method_from_token. I would expect this to crash if the image or klass or context are invalid.
• Next, the get_method_from_token must have succeeded, otherwise we'd return. This suggests that we successfully fetched a method, so image, token, and klass must have all been valid.
• Next, we lock the image again.
• Finally, we update the cache with the method we just got. This is where we're crashing. We don't touch the cache if the image is dynamic.
• Finally, we unlock the image.

The most likely explanation at a glance based on this is that either image->xxx_cache is corrupt or that the actual node allocations for the cache got corrupted. The line number information indicates that we're rehashing, since it's line 243:

runtime/src/mono/mono/eglib/ghashtable.c

Line 243 in 0fdf85d

rehash (hash);

For a free to happen during rehash that would suggest the cache being updated already has some items in it and is growing, since by default ghashtables are allocated with space for at least 1 item. it uses g_spaced_primes_closest to pick an actual bucket count, it looks like the smallest one it would choose is 11, which means the caches should start with 11 buckets that can hold ~7 items. So for us to rehash that means we've already put 7 items in the cache and are inserting an 8th one.

I'm not sure why rehash isn't on the stack, or if it was inlined, why we're getting line 243 instead of a line number inside of rehash. Bad debug information maybe. A rehash operation doesn't free anything except the old keys/values arrays (at the end of the rehash operation) which would imply we successfully did a full rehash without crashing and are only falling over at the end, perhaps because of a buffer overrun/underrun on the keys or values arrays.

If we're not rehashing, this insert_replace operation shouldn't ever free nodes, but if there's a key_destroy_func or value_destroy_func it could call those, and those could be free.

[srxqds]

@srxqds can you provide more info around what your app is doing? Is it possible to provide a small repro?

we are embeding mono with UnrealEngine, the project is so big can't shared and reproduced it, but it occured not only one time.

[steveisok]

we are embeding mono with UnrealEngine, the project is so big can't shared and reproduced it, but it occured not only one time.

Since your case is "not typical", any info you can provide will be helpful.