The action 'Galaxy Llama3 demo tests' has timed out after 20 minutes.

test_demo_text[wormhole_b0-mesh_device0-device_params0-10-performance-repeat2] AssertionError: More mismatches exist but will not be logged for this comparison (limit reached). assert False

The action 'Galaxy Llama3 8B data-parallel demo tests' has timed out after 20 minutes.

test_demo_text[wormhole_b0-mesh_device0-device_params0-False-performance-ci-b1-DP-4] RuntimeError: TT_THROW @ /project/tt_metal/impl/dispatch/system_memory_manager.cpp:627: tt::exception info: TIMEOUT: device timeout, potential hang detected, the device is unrecoverable backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0xb99868) [0x7f7f3661b868] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x5cf674) [0x7f7f3502e674] --- tt::tt_metal::distributed::FDMeshCommandQueue::read_completion_queue_event(tt::tt_metal::distributed::MeshReadEventDescriptor&) --- tt::tt_metal::distributed::FDMeshCommandQueue::read_completion_queue() --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f7ff68ce253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f7ff99ceac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f7ff9a5fa84]

The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

Process completed with exit code 134.

test_demo_text[wormhole_b0-mesh_device0-device_params0-False-performance-ci-b1-DP-4] RuntimeError: TT_THROW @ /project/tt_metal/impl/dispatch/system_memory_manager.cpp:572: tt::exception info: TIMEOUT: device timeout in fetch queue wait, potential hang detected backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0xb99868) [0x73d17e41b868] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x5cf477) [0x73d17ce2e477] --- tt::tt_metal::SystemMemoryManager::fetch_queue_reserve_back(unsigned char) --- tt::tt_metal::program_dispatch::write_program_command_sequence(tt::tt_metal::ProgramCommandSequence const&, tt::tt_metal::SystemMemoryManager&, unsigned int, tt::CoreType, bool, bool, bool) --- tt::tt_metal::distributed::FDMeshCommandQueue::write_program_cmds_to_subgrid(tt::tt_metal::distributed::MeshCoordinateRange const&, tt::tt_metal::ProgramCommandSequence&, bool, bool, std::unordered_set<unsigned int, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<unsigned int> >&, unsigned int) --- tt::tt_metal::distributed::FDMeshCommandQueue::enqueue_mesh_workload(tt::tt_metal::distributed::MeshWorkload&, bool) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail6launchITkNS0_22DeviceOperationConceptENS_12experimental4prim35RotaryEmbeddingLlamaDeviceOperationEEENT_21tensor_return_value_tERKNS6_22operation_attributes_tERKNS6_13tensor_args_tE+0x365) [0x73d17f1b2555] --- ttnn::prim::rotary_embedding_llama(tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, bool, std::optional<tt::tt_metal::MemoryConfig> const&, std::optional<ttnn::ComputeKernelConfig const> const&) --- ttnn::operations::experimental::transformer::RotaryEmbeddingLlamaOperation::invoke(tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, bool, std::optional<tt::tt_metal::MemoryConfig> const&, std::optional<ttnn::ComputeKernelConfig const>) --- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x4039ce) [0x73d1804d89ce] --- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x2471a5) [0x73d18031c1a5] --- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x63c1cb6ab33d] --- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x63c1cb6c025c] --- /opt/venv/bin/python3(+0x29cd14) [0x63c1cb7c7d14] --- /opt/venv/bin/python3(PyObject_Call+0xbb) [0x63c1cb6c3d3b] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a8f) [0x63c1cb6a27df] --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x63c1cb6b602c] --- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x63c1cb6ab33d] --- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x63c1cb6c025c] --- /opt/venv/bin/python3(+0x29cd14) [0x63c1cb7c7d14] --- /opt/venv/bin/python3(_PyObject_MakeTpCall+0x25b) [0x63c1cb6ac0bb] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x6a59) [0x63c1cb6a67a9] --- /opt/venv/bin/python3(+0x198101) [0x63c1cb6c3101] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x1990) [0x63c1cb6a16e0] --- /opt/venv/bin/python3(+0x198101) [0x63c1cb6c3101] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x1990) [0x63c1cb6a16e0] --- /opt/venv/bin/python3(+0x198101) [0x63c1cb6c3101] --- /opt/venv/bin/python3(PyObject_Call+0x122) [0x63c1cb6c3da2] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a8f) [0x63c1cb6a27df] --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x63c1cb6b602c] --- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x63c1cb6ab33d] --- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x63c1cb6c025c] --- /opt/venv/bin/python3(+0x29cd14) [0x63c1cb7c7d14] --- /opt/venv/bin/python3(_PyObject_MakeTpCall+0x25b) [0x63c1cb6ac0bb] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x6a59) [0x63c1cb6a67a9] --- /opt/venv/bin/python3(+0x198101) [0x63c1cb6c3101] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x1990) [0

Process completed with exit code 134.

test_qwen_demo[wormhole_b0-device_params0-mesh_device0-performance-full] RuntimeError: TT_THROW @ /project/tt_metal/impl/dispatch/system_memory_manager.cpp:572: tt::exception info: TIMEOUT: device timeout in fetch queue wait, potential hang detected backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0xb99868) [0x79330fc1b868] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x5cf477) [0x79330e62e477] --- tt::tt_metal::SystemMemoryManager::fetch_queue_reserve_back(unsigned char) --- tt::tt_metal::program_dispatch::write_program_command_sequence(tt::tt_metal::ProgramCommandSequence const&, tt::tt_metal::SystemMemoryManager&, unsigned int, tt::CoreType, bool, bool, bool) --- tt::tt_metal::distributed::FDMeshCommandQueue::write_program_cmds_to_subgrid(tt::tt_metal::distributed::MeshCoordinateRange const&, tt::tt_metal::ProgramCommandSequence&, bool, bool, std::unordered_set<unsigned int, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<unsigned int> >&, unsigned int) --- tt::tt_metal::distributed::FDMeshCommandQueue::enqueue_mesh_workload(tt::tt_metal::distributed::MeshWorkload&, bool) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29handle_mesh_adapter_cache_hitITkNS0_36DeviceOperationWithMeshDeviceAdapterENS0_26MeshDeviceOperationAdapterINS_4prim22ReshardDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS7_13tensor_args_tERNS7_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceERNSH_13program_cache6detail12ProgramCacheEm+0x154) [0x7933105292c4] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail6launchITkNS0_22DeviceOperationConceptENS_4prim22ReshardDeviceOperationEEENT_21tensor_return_value_tERKNS5_22operation_attributes_tERKNS5_13tensor_args_tE+0x341) [0x79331051bbd1] --- ttnn::prim::reshard(tt::tt_metal::Tensor const&, tt::tt_metal::MemoryConfig const&, std::optional<tt::tt_metal::Tensor> const&) --- ttnn::operations::data_movement::ReshardOperation::invoke(tt::tt_metal::Tensor const&, tt::tt_metal::MemoryConfig const&, std::optional<tt::tt_metal::Tensor> const&) --- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x302f32) [0x793311bd7f32] --- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x2471a5) [0x793311b1c1a5] --- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x58d8fb90c33d] --- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x58d8fb92125c] --- /opt/venv/bin/python3(+0x29cd14) [0x58d8fba28d14] --- /opt/venv/bin/python3(PyObject_Call+0xbb) [0x58d8fb924d3b] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a8f) [0x58d8fb9037df] --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x58d8fb91702c] --- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x58d8fb90c33d] --- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x58d8fb92125c] --- /opt/venv/bin/python3(+0x29cd14) [0x58d8fba28d14] --- /opt/venv/bin/python3(_PyObject_MakeTpCall+0x25b) [0x58d8fb90d0bb] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x6a59) [0x58d8fb9077a9] --- /opt/venv/bin/python3(+0x198101) [0x58d8fb924101] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x1990) [0x58d8fb9026e0] --- /opt/venv/bin/python3(+0x198101) [0x58d8fb924101] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x1990) [0x58d8fb9026e0] --- /opt/venv/bin/python3(+0x198101) [0x58d8fb924101] --- /opt/venv/bin/python3(PyObject_Call+0x122) [0x58d8fb924da2] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a8f) [0x58d8fb9037df] --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x58d8fb91702c] --- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x58d8fb90c33d] --- /opt/venv/bin/python3(_PyObject_Call_Prepend+0xc1) [0x58d8fb9212c1] --- /opt/venv/bin/python3(+0x29cd14) [0x58d8fba28d14] --- /opt/venv/bin/python3(_PyObject_MakeTpCall+0x25b) [0x58d8fb90d0bb] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x6a59) [0x

The action 'Galaxy Whisper demo tests' has timed out after 5 minutes.

test_demo_for_conditional_generation[wormhole_b0-device_params0-None-False-temperatures1-2.4--1.0-0.6-True-transcribe-English-models/demos/audio/whisper/demo/dataset/conditional_generation-32-distil-whisper/distil-large-v3-2-2] RuntimeError: TT_THROW @ /project/tt_metal/impl/dispatch/system_memory_manager.cpp:627: tt::exception info: TIMEOUT: device timeout, potential hang detected, the device is unrecoverable backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0xb99868) [0x73be6e81b868] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x5cf674) [0x73be6d22e674] --- tt::tt_metal::distributed::FDMeshCommandQueue::read_completion_queue_event(tt::tt_metal::distributed::MeshReadEventDescriptor&) --- tt::tt_metal::distributed::FDMeshCommandQueue::read_completion_queue() --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x73bf2ea50253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x73bf31b50ac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x73bf31be1a84]

Process completed with exit code 134.

test_qwen_demo_text[wormhole_b0-mesh_device0-device_params0-10-performance-long-8k-b1] KeyError: (0, 'compile_decode')

The action 'Galaxy Motif-Image-6B-Preview demo tests' has timed out after 15 minutes.

The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

build_verbosity 2 is not supported for build frontend. Ignoring.

Benchmark data directory 'generated/benchmark_data' does not exist

Benchmark data directory 'generated/benchmark_data' does not exist

builtin type SwigPyObject has no __module__ attribute

builtin type SwigPyPacked has no __module__ attribute

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

No files were found with the provided path: /work/generated/test_reports/. No artifacts will be uploaded.

Benchmark data directory 'generated/benchmark_data' does not exist

Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

No files were found with the provided path: /work/generated/test_reports/. No artifacts will be uploaded.

Benchmark data directory 'generated/benchmark_data' does not exist

Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Benchmark data directory 'generated/benchmark_data' does not exist

Use load_torch_state_dict instead

Use load_torch_state_dict instead

builtin type SwigPyObject has no __module__ attribute

builtin type SwigPyPacked has no __module__ attribute

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Benchmark data directory 'generated/benchmark_data' does not exist

builtin type SwigPyObject has no __module__ attribute

builtin type SwigPyPacked has no __module__ attribute

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Test functions should return None, but models/demos/tg/falcon7b/demo_tg.py::test_demo_multichip[wormhole_b0-user_input0-32chipTG-perf_mode_2048_stochastic_verify] returned <class 'tuple'>. Did you mean to use `assert` instead of `return`? See https://docs.pytest.org/en/stable/how-to/assert.html#return-not-none for more information.

Test functions should return None, but models/demos/tg/falcon7b/demo_tg.py::test_demo_multichip[wormhole_b0-user_input0-32chipTG-perf_mode_1024_stochastic_verify] returned <class 'tuple'>. Did you mean to use `assert` instead of `return`? See https://docs.pytest.org/en/stable/how-to/assert.html#return-not-none for more information.

Test functions should return None, but models/demos/tg/falcon7b/demo_tg.py::test_demo_multichip[wormhole_b0-user_input0-32chipTG-perf_mode_128_stochastic_verify] returned <class 'tuple'>. Did you mean to use `assert` instead of `return`? See https://docs.pytest.org/en/stable/how-to/assert.html#return-not-none for more information.

Support for class-based `config` is deprecated, use ConfigDict instead.

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

No files were found with the provided path: /work/generated/test_reports/. No artifacts will be uploaded.

Benchmark data directory 'generated/benchmark_data' does not exist

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

No files were found with the provided path: /work/generated/test_reports/. No artifacts will be uploaded.

Benchmark data directory 'generated/benchmark_data' does not exist

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Benchmark data directory 'generated/benchmark_data' does not exist

invalid value encountered in divide

Use load_torch_state_dict instead

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Benchmark data directory 'generated/benchmark_data' does not exist

builtin type SwigPyObject has no __module__ attribute

builtin type SwigPyPacked has no __module__ attribute

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

No files were found with the provided path: /work/generated/test_reports/. No artifacts will be uploaded.

Benchmark data directory 'generated/benchmark_data' does not exist

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details

Support for class-based `config` is deprecated, use ConfigDict instead.

No files were found with the provided path: /work/generated/test_reports/. No artifacts will be uploaded.

Benchmark data directory 'generated/benchmark_data' does not exist

builtin type SwigPyObject has no __module__ attribute

builtin type SwigPyPacked has no __module__ attribute

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.