llama model with coreml backend doesn't work on iphone 13 pro

[seayoung1112]

First, the same test worked fine on iphone 15/14, so it's very likely an issue on iphone 13

Test set up:

llama ExecuTorch ios app, built locally
iphone 13 pro, ios version 17.4

llama model was exported using the command:
python -m examples.models.llama2.export_llama -c <model.ckpt> -p <model_params.json> --coreml --use_kv_cache

The app crashed with the following error log:

Compiler failed with XPC_ERROR_CONNECTION_INTERRUPTED
Compiler failed with XPC_ERROR_CONNECTION_INTERRUPTED
Compiler failed with XPC_ERROR_CONNECTION_INTERRUPTED
MTLCompiler: Compilation failed with XPC_ERROR_CONNECTION_INTERRUPTED on 3 try
/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShaders/MPSCore/Utility/MPSLibrary.mm, line 491: error 'MPSLibrary::MPSKey_Create internal error: Unable to get MPS kernel NDArrayVectorMatrixMultiply. Error: Compiler encountered an internal error
'
/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShaders/MPSCore/Utility/MPSLibrary.mm:491: failed assertion `MPSLibrary::MPSKey_Create internal error: Unable to get MPS kernel NDArrayVectorMatrixMultiply. Error: Compiler encountered an internal error

[cccclai]

@cymbalrush @YifanShenSZ any chance you know the reason? Also it will be great to return error instead of fatal exit.

[cymbalrush]

We are investigating the crash, @seayoung1112 could you please share the crash log.

[seayoung1112]

We are investigating the crash, @seayoung1112 could you please share the crash log.

Sorry I didn't describe it clearly, it didn't crash, the app hangs there on the error and I have to close it manually. The only logs are what I added in the report

[cheerss]

I got the error when exporting LLaMA 3 model with an coreml backend.

Enviroment

Mac OS, conda 24.1, python 3.10. I have successfully export LLaMA3 with an xnnpack backend before this.

Command

python -m examples.models.llama2.export_llama --checkpoint $MODEL_PATH/checkpoint.pth -p $MODEL_PATH/params.json  -kv --use_sdpa_with_kv_cache --coreml -qmode 8da4w  --group_size 128 -d fp32 --metadata '{"get_bos_id":128000, "get_eos_id":128001}' --embedding-quantize 4,32 --output_name="llama3_kv_sdpa_coreml_qe_4_32.pte"

Error Log

Running MIL default pipeline: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 78/78 [00:00<00:00, 2445.90 passes/s]
Running MIL backend_mlprogram pipeline: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 4828.90 passes/s]
[INFO 2024-06-12 14:53:33,762 converter.py:515] Converting graph.
Traceback (most recent call last):
  File "/opt/anaconda3/envs/executorch/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/anaconda3/envs/executorch/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/somewhere/executorch/examples/models/llama2/export_llama.py", line 30, in <module>
    main()  # pragma: no cover
  File "/somewhere/executorch/examples/models/llama2/export_llama.py", line 26, in main
    export_llama(modelname, args)
  File "/somewhere/executorch/examples/models/llama2/export_llama_lib.py", line 307, in export_llama
    builder = _export_llama(modelname, args)
  File "/somewhere/executorch/examples/models/llama2/export_llama_lib.py", line 455, in _export_llama
    builder = builder_exported_to_edge.to_backend(partitioners).to_executorch()
  File "/somewhere/executorch/examples/models/llama2/builder.py", line 351, in to_backend
    self.edge_manager = self.edge_manager.to_backend(partitioner)
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/executorch/exir/program/_program.py", line 1166, in to_backend
    new_edge_programs[name] = to_backend(program, partitioner)
  File "/opt/anaconda3/envs/executorch/lib/python3.10/functools.py", line 878, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/executorch/exir/backend/backend_api.py", line 384, in _
    tagged_graph_module = _partition_and_lower(
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/executorch/exir/backend/backend_api.py", line 299, in _partition_and_lower
    partitioned_module = _partition_and_lower_one_graph_module(
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/executorch/exir/backend/backend_api.py", line 230, in _partition_and_lower_one_graph_module
    lowered_submodule = to_backend(
  File "/opt/anaconda3/envs/executorch/lib/python3.10/functools.py", line 878, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/executorch/exir/backend/backend_api.py", line 114, in _
    preprocess_result: PreprocessResult = cls.preprocess(
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/executorch/backends/apple/coreml/compiler/coreml_preprocess.py", line 367, in preprocess
    mlmodel = ct.convert(
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/coremltools/converters/_converters_entry.py", line 581, in convert
    mlmodel = mil_convert(
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/coremltools/converters/mil/converter.py", line 188, in mil_convert
    return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, compute_units, **kwargs)
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/coremltools/converters/mil/converter.py", line 212, in _mil_convert
    proto, mil_program = mil_convert_to_proto(
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/coremltools/converters/mil/converter.py", line 288, in mil_convert_to_proto
    prog = frontend_converter(model, **kwargs)
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/coremltools/converters/mil/converter.py", line 108, in __call__
    return load(*args, **kwargs)
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 82, in load
    return _perform_torch_convert(converter, debug)
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 116, in _perform_torch_convert
    prog = converter.convert()
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/converter.py", line 570, in convert
    raise ValueError(
ValueError: To use fp16 input, please set minimum deployment target to iOS16+

[YifanShenSZ]

Hi @cheerss, this is because fp16 io is used, and Core ML starts to support fp16 io from iOS16.

If it is ok to support only iOS16+ OS for you, then please modify here

     compile_specs = CoreMLBackend.generate_compile_specs(
         compute_precision=ct.precision(ct.precision.FLOAT16.value),
         # using `ComputeUnit.ALL` can increase the model load time, default to `ComputeUnit.CPU_AND_GPU`
         compute_unit=ct.ComputeUnit[ct.ComputeUnit.CPU_AND_GPU.name.upper()],
         model_type=CoreMLBackend.MODEL_TYPE.MODEL,
+        minimum_deployment_target=ct.target.iOS16,
     )

If you need to support older iOS, then please use fp32 io in your model.

[cheerss]

@YifanShenSZ Wow, nice solution and explaination. That's really works. Thanks a lot. While another problem emerges after the backend optimization process T_T:

Running MIL default pipeline: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 78/78 [00:00<00:00, 9429.21 passes/s]
Running MIL backend_mlprogram pipeline: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 22723.09 passes/s]
Traceback (most recent call last):
  File "/opt/anaconda3/envs/executorch/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/anaconda3/envs/executorch/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/somewhere/executorch/examples/models/llama2/export_llama.py", line 30, in <module>
    main()  # pragma: no cover
  File "/somewhere/executorch/examples/models/llama2/export_llama.py", line 26, in main
    export_llama(modelname, args)
  File "/somewhere/executorch/examples/models/llama2/export_llama_lib.py", line 307, in export_llama
    builder = _export_llama(modelname, args)
  File "/somewhere/executorch/examples/models/llama2/export_llama_lib.py", line 455, in _export_llama
    builder = builder_exported_to_edge.to_backend(partitioners).to_executorch()
  File "/somewhere/executorch/examples/models/llama2/builder.py", line 351, in to_backend
    self.edge_manager = self.edge_manager.to_backend(partitioner)
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/executorch/exir/program/_program.py", line 1166, in to_backend
    new_edge_programs[name] = to_backend(program, partitioner)
  File "/opt/anaconda3/envs/executorch/lib/python3.10/functools.py", line 878, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/executorch/exir/backend/backend_api.py", line 384, in _
    tagged_graph_module = _partition_and_lower(
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/executorch/exir/backend/backend_api.py", line 299, in _partition_and_lower
    partitioned_module = _partition_and_lower_one_graph_module(
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/executorch/exir/backend/backend_api.py", line 230, in _partition_and_lower_one_graph_module
    lowered_submodule = to_backend(
  File "/opt/anaconda3/envs/executorch/lib/python3.10/functools.py", line 878, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/executorch/exir/backend/backend_api.py", line 114, in _
    preprocess_result: PreprocessResult = cls.preprocess(
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/executorch/backends/apple/coreml/compiler/coreml_preprocess.py", line 367, in preprocess
    mlmodel = ct.convert(
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/coremltools/converters/_converters_entry.py", line 581, in convert
    mlmodel = mil_convert(
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/coremltools/converters/mil/converter.py", line 188, in mil_convert
    return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, compute_units, **kwargs)
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/coremltools/converters/mil/converter.py", line 212, in _mil_convert
    proto, mil_program = mil_convert_to_proto(
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/coremltools/converters/mil/converter.py", line 288, in mil_convert_to_proto
    prog = frontend_converter(model, **kwargs)
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/coremltools/converters/mil/converter.py", line 108, in __call__
    return load(*args, **kwargs)
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 73, in load
    converter = TorchConverter(
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/converter.py", line 384, in __init__
    self.graph = InternalTorchIRGraph.from_exir(exir=loaded_model)
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/internal_graph.py", line 524, in from_exir
    extract_inputs_from_exir_program(exported_program)
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/exir_utils.py", line 88, in extract_inputs_from_exir_program
    assert isinstance(val, torch.Tensor), "placeholder val must be a tensor or fake tensor"
AssertionError: placeholder val must be a tensor or fake tensor

[YifanShenSZ]

@cheerss There are 2 possible reasons I can guess of:

1. ExecuTorch may have some change since 0.2.0, that the model is partitioned differently thus something non-tensor is delegated to Core ML. Can you retry with ExecuTorch 0.2.0?
2. Your model may be different from the Llama model in ExecuTorch example, and contains some unsupported op in Core ML thus triggered partition. Can you verify by converting directly with coremltools? Here is an example

import torch
import torchvision
import executorch.exir

import coremltools as ct

torch_model = torchvision.models.vit_b_16(weights="IMAGENET1K_V1")

x = torch.rand((1, 3, 224, 224))
example_inputs = (x,)
exported_program_aten = torch.export.export(torch_model, example_inputs)
exported_program_edge = executorch.exir.to_edge(exported_program_aten)

coreml_model = ct.convert(exported_program_edge)

Btw I feel this is relevant to #2769

[jefflewis]

in released versions of my app, I've exported a model with coremltools and see this on some devices some times. I am having a hard time finding what is using NDArrayVectorMatrixMultiply and why it doesn't happen reliably and only on some devices.

Here's a stack trace I have from it (clarification: I'm not using executorch, but am hoping this can help me/others in relating what could be causing the issue:

Thread Stack Trace

Newest

OS Version: iOS 17.6.1 (21G93)
Report Version: 104

Exception Type: EXC_CRASH (SIGABRT)
Crashed Thread: 4

Application Specific Information:
error > MPSLibrary::MPSKey_Create internal error: Unable to get MPS kernel NDArrayVectorMatrixMultiply. Error: Compiler encountered an internal error


Thread 4 Crashed:
0   libsystem_kernel.dylib          0x3b2b042ec         __pthread_kill
1   libsystem_pthread.dylib         0x3da6efc08         pthread_kill
2   libsystem_c.dylib               0x331099b9c         abort
3   libsystem_c.dylib               0x331098ea8         __assert_rtn
4   Metal                           0x320a6aaa0         MTLReportFailure.cold.1
5   Metal                           0x320911d3c         MTLReportFailure
6   MPSCore                         0x404fad824         MPSLibrary::MPSKey_Compile
7   MPSCore                         0x405030d58         MPSLibrary::GetPipelineStateForMPSKey
8   MPSNDArray                      0x44e422978         <redacted>
9   MPSNDArray                      0x44e415e60         <redacted>
10  MPSNDArray                      0x44e415a24         <redacted>
11  MPSNDArray                      0x44e41d430         <redacted>
12  MPSNDArray                      0x44e479154         MPSSetResourcesOnCommandEncoder
13  MPSNDArray                      0x44e47b63c         MPSSetResourcesOnCommandEncoder
14  MetalPerformanceShadersGraph    0x312060bbc         <redacted>
15  MetalPerformanceShadersGraph    0x311e7ff90         <redacted>
16  MetalPerformanceShadersGraph    0x31200d654         <redacted>
17  MetalPerformanceShadersGraph    0x312008668         <redacted>
18  MetalPerformanceShadersGraph    0x312017e28         <redacted>
19  MetalPerformanceShadersGraph    0x312005da8         <redacted>
20  MetalPerformanceShadersGraph    0x3120ee74c         <redacted>
21  MetalPerformanceShadersGraph    0x3120ecc58         <redacted>
22  Espresso                        0x3400e6da8         Espresso::MPSGraphEngine::compiler::__forward_segment
23  Espresso                        0x33fbce0ac         Espresso::net::__forward_network_two_compilers
24  Espresso                        0x33f712e68         EspressoLight::espresso_plan::dispatch_task_on_compute_batch
25  Espresso                        0x33f711408         EspressoLight::espresso_plan::execute_sync
26  Espresso                        0x33f7111e4         espresso_plan_execute_sync
27  Espresso                        0x33f710f34         E5RT::Ops::BaseEirInferenceOperation::Impl::ExecuteSync
28  Espresso                        0x33f710cd0         E5RT::Ops::BaseEirInferenceOperation::ExecuteSync
29  Espresso                        0x33f7108c0         E5RT::ExecutionStreamImpl::ExecuteStreamSyncImpl<T>
30  Espresso                        0x33f7105ec         E5RT::ExecutionStreamImpl::ExecuteStreamSync
31  Espresso                        0x33f73efc0         std::__1::__shared_ptr_pointer<T>::__on_zero_shared_weak
32  Espresso                        0x33f637110         E5RT::ExceptionSafeExecute
33  Espresso                        0x33f73d9c4         e5rt_execution_stream_execute_sync
34  CoreML                          0x3569b4af4         -[MLE5ExecutionStream _executeStream:error:]
35  CoreML                          0x3569b4c9c         -[MLE5ExecutionStream executeWithOptions:error:]
36  CoreML                          0x356982e98         -[MLE5Engine _predictionFromFeatures:options:usingStream:operation:error:]
37  CoreML                          0x356983040         -[MLE5Engine _predictionFromFeatures:options:error:]
38  CoreML                          0x3569833d8         -[MLE5Engine predictionFromFeatures:options:error:]
39  CoreML                          0x35664d8b0         -[MLDelegateModel predictionFromFeatures:options:error:]
40  XOi Vision                      0x202c653a8         <unknown> + 300
41  libswift_Concurrency.dylib      0x33788f760         swift::runJobInEstablishedExecutorContext

Thread 0
0   libobjc.A.dylib                 0x310f7e2a0         getMethodNoSuper_nolock
1   libobjc.A.dylib                 0x310f7de28         [inlined] lookUpImpOrForward
2   libobjc.A.dylib                 0x310f7de28         lookUpImpOrForward
3   libobjc.A.dylib                 0x310f7e9f4         class_respondsToSelector_inst
4   BaseBoard                       0x331254fac         -[BSDescriptionBuilder appendObject:withName:skipIfNil:]
5   UIKitCore                       0x325a3037c         __63-[_UIKeyWindowEvaluator descriptionBuilderWithMultilinePrefix:]_block_invoke_3
6   BaseBoard                       0x331254e34         -[BSDescriptionBuilder appendBodySectionWithName:multilinePrefix:block:]
7   UIKitCore                       0x3259c8014         __63-[_UIKeyWindowEvaluator descriptionBuilderWithMultilinePrefix:]_block_invoke_2
8   BaseBoard                       0x331254e34         -[BSDescriptionBuilder appendBodySectionWithName:multilinePrefix:block:]
9   UIKitCore                       0x325b89590         __63-[_UIKeyWindowEvaluator descriptionBuilderWithMultilinePrefix:]_block_invoke
10  BaseBoard                       0x331269e6c         -[BSDescriptionBuilder modifyBody:]
11  UIKitCore                       0x325a52370         -[_UIKeyWindowEvaluator descriptionBuilderWithMultilinePrefix:]
12  UIKitCore                       0x325a518b4         __30-[_UIKeyWindowEvaluator _init]_block_invoke
13  BaseBoard                       0x33128fa38         ___BSLogAddStateCaptureBlockWithTitle_block_invoke
14  libsystem_trace.dylib           0x3554aaa9c         ___os_state_request_for_self_impl_block_invoke
15  libdispatch.dylib               0x330f99dd0         _dispatch_client_callout
16  libdispatch.dylib               0x330fa9398         _dispatch_async_and_wait_invoke
17  libdispatch.dylib               0x330f99dd0         _dispatch_client_callout
18  libdispatch.dylib               0x330fa85a0         _dispatch_main_queue_drain
19  libdispatch.dylib               0x330fa81b4         _dispatch_main_queue_callback_4CF
20  CoreFoundation                  0x3211a070c         __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__
21  CoreFoundation                  0x32119d910         __CFRunLoopRun
22  CoreFoundation                  0x32119ccd4         CFRunLoopRunSpecific
23  GraphicsServices                0x3aa6891a4         GSEventRunModal
24  UIKitCore                       0x325a58ae4         -[UIApplication _run]
25  UIKitCore                       0x325b0cd94         UIApplicationMain
26  XOi Vision                      0x202c50e30         main
27  <unknown>                       0x1bc7a3154         <redacted>

Thread 1
0   libsystem_pthread.dylib         0x3da6e90c4         start_wqthread

Thread 2
0   Network                         0x321e3c4b0         nw_resolver_deallocate_cache_entry
1   Network                         0x321cfc4e8         nw_protocol_definition_deallocate_cache_entry
2   Network                         0x321cfbd40         __nw_association_is_empty_block_invoke
3   Network                         0x321bf2fec         nw_hash_table_apply
4   Network                         0x321d65b60         nw_association_is_empty
5   Network                         0x321d65760         __nw_endpoint_has_associations_block_invoke
6   Network                         0x321bf2fec         nw_hash_table_apply
7   Network                         0x321d963b8         nw_endpoint_is_active
8   Network                         0x321d95da8         __nw_context_purge_endpoint_hash_table_block_invoke_2
9   Network                         0x321bf2fec         nw_hash_table_apply
10  Network                         0x321d95a54         __nw_context_purge_endpoint_hash_table_block_invoke
11  Network                         0x321bf2fec         nw_hash_table_apply
12  Network                         0x321d970a4         nw_context_purge_endpoints
13  Network                         0x321bd4e58         nw_queue_context_async_if_needed
14  Network                         0x321bd3428         nw_context_purge_all_endpoints
15  libdispatch.dylib               0x330f98138         _dispatch_call_block_and_release
16  libdispatch.dylib               0x330f99dd0         _dispatch_client_callout
17  libdispatch.dylib               0x330fa340c         _dispatch_workloop_invoke
18  libdispatch.dylib               0x330faccb0         _dispatch_root_queue_drain_deferred_wlh
19  libdispatch.dylib               0x330fac524         _dispatch_workloop_worker_thread
20  libsystem_pthread.dylib         0x3da6ec930         _pthread_wqthread

Thread 3 name: com.apple.uikit.eventfetch-thread
0   libsystem_kernel.dylib          0x3b2af96c8         mach_msg2_trap
1   libsystem_kernel.dylib          0x3b2afcec4         mach_msg2_internal
2   libsystem_kernel.dylib          0x3b2afcddc         mach_msg_overwrite
3   libsystem_kernel.dylib          0x3b2afcc1c         mach_msg
4   CoreFoundation                  0x32119df58         __CFRunLoopServiceMachPort
5   CoreFoundation                  0x32119d5fc         __CFRunLoopRun
6   CoreFoundation                  0x32119ccd4         CFRunLoopRunSpecific
7   Foundation                      0x31ef69b58         -[NSRunLoop(NSRunLoop) runMode:beforeDate:]
8   Foundation                      0x31ef699a8         -[NSRunLoop(NSRunLoop) runUntilDate:]
9   UIKitCore                       0x325a6c818         -[UIEventFetcher threadMain]
10  Foundation                      0x31ef80424         __NSThread__start__
11  libsystem_pthread.dylib         0x3da6ee068         _pthread_start

Thread 4 Crashed:
0   libsystem_kernel.dylib          0x3b2b042ec         __pthread_kill
1   libsystem_pthread.dylib         0x3da6efc08         pthread_kill
2   libsystem_c.dylib               0x331099b9c         abort
3   libsystem_c.dylib               0x331098ea8         __assert_rtn
4   Metal                           0x320a6aaa0         MTLReportFailure.cold.1
5   Metal                           0x320911d3c         MTLReportFailure
6   MPSCore                         0x404fad824         MPSLibrary::MPSKey_Compile
7   MPSCore                         0x405030d58         MPSLibrary::GetPipelineStateForMPSKey
8   MPSNDArray                      0x44e422978         <redacted>
9   MPSNDArray                      0x44e415e60         <redacted>
10  MPSNDArray                      0x44e415a24         <redacted>
11  MPSNDArray                      0x44e41d430         <redacted>
12  MPSNDArray                      0x44e479154         MPSSetResourcesOnCommandEncoder
13  MPSNDArray                      0x44e47b63c         MPSSetResourcesOnCommandEncoder
14  MetalPerformanceShadersGraph    0x312060bbc         <redacted>
15  MetalPerformanceShadersGraph    0x311e7ff90         <redacted>
16  MetalPerformanceShadersGraph    0x31200d654         <redacted>
17  MetalPerformanceShadersGraph    0x312008668         <redacted>
18  MetalPerformanceShadersGraph    0x312017e28         <redacted>
19  MetalPerformanceShadersGraph    0x312005da8         <redacted>
20  MetalPerformanceShadersGraph    0x3120ee74c         <redacted>
21  MetalPerformanceShadersGraph    0x3120ecc58         <redacted>
22  Espresso                        0x3400e6da8         Espresso::MPSGraphEngine::compiler::__forward_segment
23  Espresso                        0x33fbce0ac         Espresso::net::__forward_network_two_compilers
24  Espresso                        0x33f712e68         EspressoLight::espresso_plan::dispatch_task_on_compute_batch
25  Espresso                        0x33f711408         EspressoLight::espresso_plan::execute_sync
26  Espresso                        0x33f7111e4         espresso_plan_execute_sync
27  Espresso                        0x33f710f34         E5RT::Ops::BaseEirInferenceOperation::Impl::ExecuteSync
28  Espresso                        0x33f710cd0         E5RT::Ops::BaseEirInferenceOperation::ExecuteSync
29  Espresso                        0x33f7108c0         E5RT::ExecutionStreamImpl::ExecuteStreamSyncImpl<T>
30  Espresso                        0x33f7105ec         E5RT::ExecutionStreamImpl::ExecuteStreamSync
31  Espresso                        0x33f73efc0         std::__1::__shared_ptr_pointer<T>::__on_zero_shared_weak
32  Espresso                        0x33f637110         E5RT::ExceptionSafeExecute
33  Espresso                        0x33f73d9c4         e5rt_execution_stream_execute_sync
34  CoreML                          0x3569b4af4         -[MLE5ExecutionStream _executeStream:error:]
35  CoreML                          0x3569b4c9c         -[MLE5ExecutionStream executeWithOptions:error:]
36  CoreML                          0x356982e98         -[MLE5Engine _predictionFromFeatures:options:usingStream:operation:error:]
37  CoreML                          0x356983040         -[MLE5Engine _predictionFromFeatures:options:error:]
38  CoreML                          0x3569833d8         -[MLE5Engine predictionFromFeatures:options:error:]
39  CoreML                          0x35664d8b0         -[MLDelegateModel predictionFromFeatures:options:error:]
40  XOi Vision                      0x202c653a8         <unknown> + 300
41  libswift_Concurrency.dylib      0x33788f760         swift::runJobInEstablishedExecutorContext

Thread 5
0   libsystem_kernel.dylib          0x3b2aff2ac         __semwait_signal
1   libsystem_c.dylib               0x33103a5ec         nanosleep
2   libsystem_c.dylib               0x331097728         sleep
3   BugfenderSDK                    0x104b2a1f4         monitorCachedData
4   libsystem_pthread.dylib         0x3da6ee068         _pthread_start

Thread 6 name: KSCrash Exception Handler (Secondary)
0   libsystem_kernel.dylib          0x3b2af96c8         mach_msg2_trap
1   libsystem_kernel.dylib          0x3b2afcec4         mach_msg2_internal
2   libsystem_kernel.dylib          0x3b2afac34         thread_suspend
3   BugfenderSDK                    0x104b33b08         handleExceptions
4   libsystem_pthread.dylib         0x3da6ee068         _pthread_start

Thread 7 name: KSCrash Exception Handler (Primary)
0   libsystem_kernel.dylib          0x3b2af96c8         mach_msg2_trap
1   libsystem_kernel.dylib          0x3b2afcec4         mach_msg2_internal
2   libsystem_kernel.dylib          0x3b2afcddc         mach_msg_overwrite
3   libsystem_kernel.dylib          0x3b2afcc1c         mach_msg
4   BugfenderSDK                    0x104b33b34         handleExceptions
5   libsystem_pthread.dylib         0x3da6ee068         _pthread_start

Thread 8 name: com.facebook.react.JavaScript
0   libsystem_kernel.dylib          0x3b2af96c8         mach_msg2_trap
1   libsystem_kernel.dylib          0x3b2afcec4         mach_msg2_internal
2   libsystem_kernel.dylib          0x3b2afcddc         mach_msg_overwrite
3   libsystem_kernel.dylib          0x3b2afcc1c         mach_msg
4   CoreFoundation                  0x32119df58         __CFRunLoopServiceMachPort
5   CoreFoundation                  0x32119d5fc         __CFRunLoopRun
6   CoreFoundation                  0x32119ccd4         CFRunLoopRunSpecific
7   React                           0x10601ecec         <unknown> + 192
8   Foundation                      0x31ef80424         __NSThread__start__
9   libsystem_pthread.dylib         0x3da6ee068         _pthread_start

Thread 9 name: JavaScriptCore libpas scavenger
0   libsystem_kernel.dylib          0x3b2aff08c         __psynch_cvwait
1   libsystem_pthread.dylib         0x3da6eb6e0         _pthread_cond_wait
2   JavaScriptCore                  0x34ea212a0         scavenger_thread_main
3   libsystem_pthread.dylib         0x3da6ee068         _pthread_start

Thread 10 name: H11ANEServicesThread
0   libsystem_kernel.dylib          0x3b2af96c8         mach_msg2_trap
1   libsystem_kernel.dylib          0x3b2afcec4         mach_msg2_internal
2   libsystem_kernel.dylib          0x3b2afcddc         mach_msg_overwrite
3   libsystem_kernel.dylib          0x3b2afcc1c         mach_msg
4   CoreFoundation                  0x32119df58         __CFRunLoopServiceMachPort
5   CoreFoundation                  0x32119d5fc         __CFRunLoopRun
6   CoreFoundation                  0x32119ccd4         CFRunLoopRunSpecific
7   CoreFoundation                  0x32120af00         CFRunLoopRun
8   ANEServices                     0x40666bab4         H11ANE::H11ANEServicesThreadStart
9   libsystem_pthread.dylib         0x3da6ee068         _pthread_start

Thread 11 name: H11ANEServicesThread
0   libsystem_kernel.dylib          0x3b2af96c8         mach_msg2_trap
1   libsystem_kernel.dylib          0x3b2afcec4         mach_msg2_internal
2   libsystem_kernel.dylib          0x3b2afcddc         mach_msg_overwrite
3   libsystem_kernel.dylib          0x3b2afcc1c         mach_msg
4   CoreFoundation                  0x32119df58         __CFRunLoopServiceMachPort
5   CoreFoundation                  0x32119d5fc         __CFRunLoopRun
6   CoreFoundation                  0x32119ccd4         CFRunLoopRunSpecific
7   CoreFoundation                  0x32120af00         CFRunLoopRun
8   ANEServices                     0x40666bab4         H11ANE::H11ANEServicesThreadStart
9   libsystem_pthread.dylib         0x3da6ee068         _pthread_start

Thread 12 name: com.apple.NSURLConnectionLoader
0   libsystem_kernel.dylib          0x3b2af96c8         mach_msg2_trap
1   libsystem_kernel.dylib          0x3b2afcec4         mach_msg2_internal
2   libsystem_kernel.dylib          0x3b2afcddc         mach_msg_overwrite
3   libsystem_kernel.dylib          0x3b2afcc1c         mach_msg
4   CoreFoundation                  0x32119df58         __CFRunLoopServiceMachPort
5   CoreFoundation                  0x32119d5fc         __CFRunLoopRun
6   CoreFoundation                  0x32119ccd4         CFRunLoopRunSpecific
7   CFNetwork                       0x3234b3c78         _CFHostIsDomainTopLevel
8   Foundation                      0x31ef80424         __NSThread__start__
9   libsystem_pthread.dylib         0x3da6ee068         _pthread_start

Thread 13 name: Heap Helper Thread
0   libsystem_kernel.dylib          0x3b2aff08c         __psynch_cvwait
1   libsystem_pthread.dylib         0x3da6eb6e0         _pthread_cond_wait
2   JavaScriptCore                  0x34d53ccc0         WTF::ThreadCondition::timedWait
3   JavaScriptCore                  0x34d50d9a8         WTF::ParkingLot::parkConditionallyImpl
4   JavaScriptCore                  0x34d4c4f7c         WTF::Detail::CallableWrapper<T>::call
5   JavaScriptCore                  0x34d539f0c         WTF::Thread::entryPoint
6   JavaScriptCore                  0x34d53c3ac         WTF::wtfThreadEntryPoint
7   libsystem_pthread.dylib         0x3da6ee068         _pthread_start

Thread 14 name: Heap Helper Thread
0   libsystem_kernel.dylib          0x3b2aff08c         __psynch_cvwait
1   libsystem_pthread.dylib         0x3da6eb6e0         _pthread_cond_wait
2   JavaScriptCore                  0x34d53ccc0         WTF::ThreadCondition::timedWait
3   JavaScriptCore                  0x34d50d9a8         WTF::ParkingLot::parkConditionallyImpl
4   JavaScriptCore                  0x34d4c4f7c         WTF::Detail::CallableWrapper<T>::call
5   JavaScriptCore                  0x34d539f0c         WTF::Thread::entryPoint
6   JavaScriptCore                  0x34d53c3ac         WTF::wtfThreadEntryPoint
7   libsystem_pthread.dylib         0x3da6ee068         _pthread_start

Thread 15 name: Heap Helper Thread
0   libsystem_kernel.dylib          0x3b2aff08c         __psynch_cvwait
1   libsystem_pthread.dylib         0x3da6eb6e0         _pthread_cond_wait
2   JavaScriptCore                  0x34d53ccc0         WTF::ThreadCondition::timedWait
3   JavaScriptCore                  0x34d50d9a8         WTF::ParkingLot::parkConditionallyImpl
4   JavaScriptCore                  0x34d4c4f7c         WTF::Detail::CallableWrapper<T>::call
5   JavaScriptCore                  0x34d539f0c         WTF::Thread::entryPoint
6   JavaScriptCore                  0x34d53c3ac         WTF::wtfThreadEntryPoint
7   libsystem_pthread.dylib         0x3da6ee068         _pthread_start

Thread 16 name: Heap Helper Thread
0   libsystem_kernel.dylib          0x3b2aff08c         __psynch_cvwait
1   libsystem_pthread.dylib         0x3da6eb6e0         _pthread_cond_wait
2   JavaScriptCore                  0x34d53ccc0         WTF::ThreadCondition::timedWait
3   JavaScriptCore                  0x34d50d9a8         WTF::ParkingLot::parkConditionallyImpl
4   JavaScriptCore                  0x34d4c4f7c         WTF::Detail::CallableWrapper<T>::call
5   JavaScriptCore                  0x34d539f0c         WTF::Thread::entryPoint
6   JavaScriptCore                  0x34d53c3ac         WTF::wtfThreadEntryPoint
7   libsystem_pthread.dylib         0x3da6ee068         _pthread_start

Thread 17 name: Heap Helper Thread
0   libsystem_kernel.dylib          0x3b2aff08c         __psynch_cvwait
1   libsystem_pthread.dylib         0x3da6eb6e0         _pthread_cond_wait
2   JavaScriptCore                  0x34d53ccc0         WTF::ThreadCondition::timedWait
3   JavaScriptCore                  0x34d50d9a8         WTF::ParkingLot::parkConditionallyImpl
4   JavaScriptCore                  0x34d4c4f7c         WTF::Detail::CallableWrapper<T>::call
5   JavaScriptCore                  0x34d539f0c         WTF::Thread::entryPoint
6   JavaScriptCore                  0x34d53c3ac         WTF::wtfThreadEntryPoint
7   libsystem_pthread.dylib         0x3da6ee068         _pthread_start

Thread 18 name: H11ANEServicesThread
0   libsystem_kernel.dylib          0x3b2af96c8         mach_msg2_trap
1   libsystem_kernel.dylib          0x3b2afcec4         mach_msg2_internal
2   libsystem_kernel.dylib          0x3b2afcddc         mach_msg_overwrite
3   libsystem_kernel.dylib          0x3b2afcc1c         mach_msg
4   CoreFoundation                  0x32119df58         __CFRunLoopServiceMachPort
5   CoreFoundation                  0x32119d5fc         __CFRunLoopRun
6   CoreFoundation                  0x32119ccd4         CFRunLoopRunSpecific
7   CoreFoundation                  0x32120af00         CFRunLoopRun
8   ANEServices                     0x40666bab4         H11ANE::H11ANEServicesThreadStart
9   libsystem_pthread.dylib         0x3da6ee068         _pthread_start

Thread 19
0   libsystem_kernel.dylib          0x3b2af96c8         mach_msg2_trap
1   libsystem_kernel.dylib          0x3b2afcec4         mach_msg2_internal
2   libsystem_kernel.dylib          0x3b2afcddc         mach_msg_overwrite
3   libsystem_kernel.dylib          0x3b2afcc1c         mach_msg
4   libdispatch.dylib               0x330fb4b14         _dispatch_mach_send_and_wait_for_reply
5   libdispatch.dylib               0x330fb4eb4         dispatch_mach_send_with_result_and_wait_for_reply
6   libxpc.dylib                    0x3da780ac4         xpc_connection_send_message_with_reply_sync
7   Foundation                      0x31ef086a4         __NSXPCCONNECTION_IS_WAITING_FOR_A_SYNCHRONOUS_REPLY__
8   Foundation                      0x31ef04d10         -[NSXPCConnection _sendInvocation:orArguments:count:methodSignature:selector:withProxy:]
9   Foundation                      0x31ef7cf18         -[NSXPCConnection _sendSelector:withProxy:arg1:]
10  Foundation                      0x31ef7cb8c         _NSXPCDistantObjectSimpleMessageSend1
11  OSLog                           0x3c31b3f64         -[OSLogCurrentProcessEnumerator nextObject]
12  BugfenderSDK                    0x104b25c9c         __47-[BFLogStoreLogger bf_processPendingLogEntries]_block_invoke_2
13  libdispatch.dylib               0x330f98138         _dispatch_call_block_and_release
14  libdispatch.dylib               0x330f99dd0         _dispatch_client_callout
15  libdispatch.dylib               0x330fa13fc         _dispatch_lane_serial_drain
16  libdispatch.dylib               0x330fa1f2c         _dispatch_lane_invoke
17  libdispatch.dylib               0x330faccb0         _dispatch_root_queue_drain_deferred_wlh
18  libdispatch.dylib               0x330fac524         _dispatch_workloop_worker_thread
19  libsystem_pthread.dylib         0x3da6ec930         _pthread_wqthread

Thread 20
0   libsystem_pthread.dylib         0x3da6e90c4         start_wqthread

Thread 21
0   libsystem_kernel.dylib          0x3b2aff2ac         __semwait_signal
1   libsystem_c.dylib               0x33103a5ec         nanosleep
2   libsystem_c.dylib               0x331097728         sleep
3   Sentry                          0x1097809f4         <unknown> + 660
4   libsystem_pthread.dylib         0x3da6ee068         _pthread_start

Thread 22 name: SentryCrash Exception Handler (Secondary)
0   libsystem_kernel.dylib          0x3b2af96c8         mach_msg2_trap
1   libsystem_kernel.dylib          0x3b2afcec4         mach_msg2_internal
2   libsystem_kernel.dylib          0x3b2afac34         thread_suspend
3   Sentry                          0x10978ae00         <unknown> + 124
4   libsystem_pthread.dylib         0x3da6ee068         _pthread_start

Thread 23 name: SentryCrash Exception Handler (Primary)
0   libsystem_kernel.dylib          0x3b2af96c8         mach_msg2_trap
1   libsystem_kernel.dylib          0x3b2afcec4         mach_msg2_internal
2   libsystem_kernel.dylib          0x3b2afcddc         mach_msg_overwrite
3   libsystem_kernel.dylib          0x3b2afcc1c         mach_msg
4   Sentry                          0x10978ae2c         <unknown> + 168
5   libsystem_pthread.dylib         0x3da6ee068         _pthread_start

Thread 24 name: io.sentry.app-hang-tracker
0   libsystem_kernel.dylib          0x3b2aff2ac         __semwait_signal
1   libsystem_c.dylib               0x33103a5ec         nanosleep
2   Foundation                      0x31f0bc2e0         +[NSThread sleepForTimeInterval:]
3   Sentry                          0x10976d98c         <unknown> + 492
4   Foundation                      0x31ef80424         __NSThread__start__
5   libsystem_pthread.dylib         0x3da6ee068         _pthread_start

Thread 25
0   libsystem_kernel.dylib          0x3b2af94dc         __ulock_wait2
1   libsystem_platform.dylib        0x3da57f4d4         _os_unfair_lock_lock_slow
2   MPSCore                         0x404fad508         MPSLibrary::MPSKey_Compile
3   libdispatch.dylib               0x330f98138         _dispatch_call_block_and_release
4   libdispatch.dylib               0x330f99dd0         _dispatch_client_callout
5   libdispatch.dylib               0x330fa13fc         _dispatch_lane_serial_drain
6   libdispatch.dylib               0x330fa1f2c         _dispatch_lane_invoke
7   libdispatch.dylib               0x330faccb0         _dispatch_root_queue_drain_deferred_wlh
8   libdispatch.dylib               0x330fac524         _dispatch_workloop_worker_thread
9   libsystem_pthread.dylib         0x3da6ec930         _pthread_wqthread

Thread 26
0   libsystem_pthread.dylib         0x3da6e90c4         start_wqthread

Thread 27 name: JSC Heap Collector Thread
0   libsystem_kernel.dylib          0x3b2aff08c         __psynch_cvwait
1   libsystem_pthread.dylib         0x3da6eb6e0         _pthread_cond_wait
2   JavaScriptCore                  0x34d53ccc0         WTF::ThreadCondition::timedWait
3   JavaScriptCore                  0x34d50d9a8         WTF::ParkingLot::parkConditionallyImpl
4   JavaScriptCore                  0x34d4c4f7c         WTF::Detail::CallableWrapper<T>::call
5   JavaScriptCore                  0x34d539f0c         WTF::Thread::entryPoint
6   JavaScriptCore                  0x34d53c3ac         WTF::wtfThreadEntryPoint
7   libsystem_pthread.dylib         0x3da6ee068         _pthread_start

Thread 28
0   libsystem_kernel.dylib          0x3b2affbcc         __ulock_wait
1   libdispatch.dylib               0x330f9a7c0         _dlock_wait
2   libdispatch.dylib               0x330f9a574         _dispatch_thread_event_wait_slow
3   libdispatch.dylib               0x330fa95ac         __DISPATCH_WAIT_FOR_QUEUE__
4   libdispatch.dylib               0x330fa9158         _dispatch_sync_f_slow
5   libsystem_trace.dylib           0x3554aac88         ___os_state_request_for_self_block_invoke
6   libdispatch.dylib               0x330f98138         _dispatch_call_block_and_release
7   libdispatch.dylib               0x330f99dd0         _dispatch_client_callout
8   libdispatch.dylib               0x330fa13fc         _dispatch_lane_serial_drain
9   libdispatch.dylib               0x330fa1f60         _dispatch_lane_invoke
10  libdispatch.dylib               0x330faccb0         _dispatch_root_queue_drain_deferred_wlh
11  libdispatch.dylib               0x330fac524         _dispatch_workloop_worker_thread
12  libsystem_pthread.dylib         0x3da6ec930         _pthread_wqthread

Thread 29
0   libsystem_pthread.dylib         0x3da6e90c4         start_wqthread

Thread 30
0   libsystem_pthread.dylib         0x3da6e90c4         start_wqthread

Thread 4 crashed with ARM Thread State (64-bit):
    x0: 0x0000000000000000   x1: 0x0000000000000000   x2: 0x0000000000000000   x3: 0x0000000000000000
    x4: 0x0000000000000073   x5: 0x000000000000002e   x6: 0x0000000000000000   x7: 0x0000000000000890
    x8: 0xbbdd41d96f1afa9a   x9: 0xbbdd41d802554a9a  x10: 0x000000000000000a  x11: 0x0000000000000000
   x12: 0x0000000000000034  x13: 0x000000010a99bfb0  x14: 0x01000001f9920e59  x15: 0x00000001f9920e58
   x16: 0x0000000000000148  x17: 0x000000016d4fb000  x18: 0x0000000000000000  x19: 0x0000000000000006
   x20: 0x0000000000005303  x21: 0x000000016d4fb0e0  x22: 0x0000000112004000  x23: 0xffffffffffffffff
   x24: 0x00000001f88d2000  x25: 0x00000002005c1000  x26: 0x0000000198cfb2d5  x27: 0x0000000000000000
   x28: 0x0000000000000000   fp: 0x000000016d4f6f10   lr: 0x00000001f5a4fc0c   sp: 0x000000016d4f6ef0
    pc: 0x00000001e1c5c2ec cpsr: 0x0000000040000000



EOF