Process completed with exit code 1.

test_flux1_pipeline[wormhole_b0-yes_use_cache-not_traced-encoder_device-4x8sp0tp1-dev-1024-1024-28-device_params0-True] OSError: You are trying to access a gated repo. Make sure to have access to it at https://huggingface.co/black-forest-labs/FLUX.1-dev. 401 Client Error. (Request ID: Root=1-698fceb5-67d66f5d16064fad60118129;21f74846-b223-4d22-af43-18e388dd212b) Cannot access gated repo for url https://huggingface.co/black-forest-labs/FLUX.1-dev/resolve/main/tokenizer/tokenizer_config.json. Access to model black-forest-labs/FLUX.1-dev is restricted. You must have access to it and be authenticated to access it. Please log in.

The action 'Galaxy Llama3 8B data-parallel demo tests' has timed out after 20 minutes.

test_demo_text[wormhole_b0-mesh_device0-device_params0-False-performance-ci-b1-DP-4] RuntimeError: TT_THROW @ /project/tt_metal/impl/dispatch/system_memory_manager.cpp:627: tt::exception info: TIMEOUT: device timeout, potential hang detected, the device is unrecoverable backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0xb998c8) [0x73fe05c1a8c8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x5ce0b4) [0x73fe0462f0b4] --- tt::tt_metal::distributed::FDMeshCommandQueue::read_completion_queue_event(tt::tt_metal::distributed::MeshReadEventDescriptor&) --- tt::tt_metal::distributed::FDMeshCommandQueue::read_completion_queue() --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x73fec5eab253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x73fec8fabac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x73fec903ca84]

Process completed with exit code 134.

test_qwen_demo_text[wormhole_b0-mesh_device0-device_params0-10-performance-long-8k-b1] KeyError: (0, 'compile_decode')

Process completed with exit code 134.

test_qwen_demo[wormhole_b0-device_params0-mesh_device0-performance-full] RuntimeError: TT_THROW @ /project/tt_metal/impl/dispatch/system_memory_manager.cpp:572: tt::exception info: TIMEOUT: device timeout in fetch queue wait, potential hang detected backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0xb998c8) [0x793c7b01a8c8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x5cdeb7) [0x793c79a2eeb7] --- tt::tt_metal::SystemMemoryManager::fetch_queue_reserve_back(unsigned char) --- tt::tt_metal::program_dispatch::write_program_command_sequence(tt::tt_metal::ProgramCommandSequence const&, tt::tt_metal::SystemMemoryManager&, unsigned int, tt::CoreType, bool, bool, bool) --- tt::tt_metal::distributed::FDMeshCommandQueue::write_program_cmds_to_subgrid(tt::tt_metal::distributed::MeshCoordinateRange const&, tt::tt_metal::ProgramCommandSequence&, bool, bool, std::unordered_set<unsigned int, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<unsigned int> >&, unsigned int) --- tt::tt_metal::distributed::FDMeshCommandQueue::enqueue_mesh_workload(tt::tt_metal::distributed::MeshWorkload&, bool) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29handle_mesh_adapter_cache_hitITkNS0_36DeviceOperationWithMeshDeviceAdapterENS0_26MeshDeviceOperationAdapterINS_4prim22ReshardDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS7_13tensor_args_tERNS7_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceERNSH_13program_cache6detail12ProgramCacheEm+0x154) [0x793c7b928324] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail6launchITkNS0_22DeviceOperationConceptENS_4prim22ReshardDeviceOperationEEENT_21tensor_return_value_tERKNS5_22operation_attributes_tERKNS5_13tensor_args_tE+0x341) [0x793c7b91ac31] --- ttnn::prim::reshard(tt::tt_metal::Tensor const&, tt::tt_metal::MemoryConfig const&, std::optional<tt::tt_metal::Tensor> const&) --- ttnn::operations::data_movement::ReshardOperation::invoke(tt::tt_metal::Tensor const&, tt::tt_metal::MemoryConfig const&, std::optional<tt::tt_metal::Tensor> const&) --- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x302f62) [0x793c7cfd6f62] --- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x2471d5) [0x793c7cf1b1d5] --- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x584022dbe33d] --- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x584022dd325c] --- /opt/venv/bin/python3(+0x29cd14) [0x584022edad14] --- /opt/venv/bin/python3(PyObject_Call+0xbb) [0x584022dd6d3b] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a8f) [0x584022db57df] --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x584022dc902c] --- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x584022dbe33d] --- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x584022dd325c] --- /opt/venv/bin/python3(+0x29cd14) [0x584022edad14] --- /opt/venv/bin/python3(_PyObject_MakeTpCall+0x25b) [0x584022dbf0bb] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x6a59) [0x584022db97a9] --- /opt/venv/bin/python3(+0x198101) [0x584022dd6101] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x1990) [0x584022db46e0] --- /opt/venv/bin/python3(+0x198101) [0x584022dd6101] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x1990) [0x584022db46e0] --- /opt/venv/bin/python3(+0x198101) [0x584022dd6101] --- /opt/venv/bin/python3(PyObject_Call+0x122) [0x584022dd6da2] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a8f) [0x584022db57df] --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x584022dc902c] --- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x584022dbe33d] --- /opt/venv/bin/python3(_PyObject_Call_Prepend+0xc1) [0x584022dd32c1] --- /opt/venv/bin/python3(+0x29cd14) [0x584022edad14] --- /opt/venv/bin/python3(_PyObject_MakeTpCall+0x25b) [0x584022dbf0bb] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x6a59) [0x

The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

The action 'Galaxy Llama3 70B data-parallel demo tests' has timed out after 20 minutes.

test_demo_text[wormhole_b0-mesh_device0-device_params0-False-performance-ci-b1-DP-4] RuntimeError: TT_THROW @ /project/tt_metal/impl/dispatch/system_memory_manager.cpp:572: tt::exception info: TIMEOUT: device timeout in fetch queue wait, potential hang detected backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0xb998c8) [0x71a5bae1a8c8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x5cdeb7) [0x71a5b982eeb7] --- tt::tt_metal::SystemMemoryManager::fetch_queue_reserve_back(unsigned char) --- tt::tt_metal::program_dispatch::write_program_command_sequence(tt::tt_metal::ProgramCommandSequence const&, tt::tt_metal::SystemMemoryManager&, unsigned int, tt::CoreType, bool, bool, bool) --- tt::tt_metal::distributed::FDMeshCommandQueue::write_program_cmds_to_subgrid(tt::tt_metal::distributed::MeshCoordinateRange const&, tt::tt_metal::ProgramCommandSequence&, bool, bool, std::unordered_set<unsigned int, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<unsigned int> >&, unsigned int) --- tt::tt_metal::distributed::FDMeshCommandQueue::enqueue_mesh_workload(tt::tt_metal::distributed::MeshWorkload&, bool) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail6launchITkNS0_22DeviceOperationConceptENS_12experimental4prim35RotaryEmbeddingLlamaDeviceOperationEEENT_21tensor_return_value_tERKNS6_22operation_attributes_tERKNS6_13tensor_args_tE+0x365) [0x71a5bbbb15c5] --- ttnn::prim::rotary_embedding_llama(tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, bool, std::optional<tt::tt_metal::MemoryConfig> const&, std::optional<ttnn::ComputeKernelConfig const> const&) --- ttnn::operations::experimental::transformer::RotaryEmbeddingLlamaOperation::invoke(tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, bool, std::optional<tt::tt_metal::MemoryConfig> const&, std::optional<ttnn::ComputeKernelConfig const>) --- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x40368e) [0x71a5bced768e] --- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x2471d5) [0x71a5bcd1b1d5] --- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x59f92ce5433d] --- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x59f92ce6925c] --- /opt/venv/bin/python3(+0x29cd14) [0x59f92cf70d14] --- /opt/venv/bin/python3(PyObject_Call+0xbb) [0x59f92ce6cd3b] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a8f) [0x59f92ce4b7df] --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59f92ce5f02c] --- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x59f92ce5433d] --- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x59f92ce6925c] --- /opt/venv/bin/python3(+0x29cd14) [0x59f92cf70d14] --- /opt/venv/bin/python3(_PyObject_MakeTpCall+0x25b) [0x59f92ce550bb] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x6a59) [0x59f92ce4f7a9] --- /opt/venv/bin/python3(+0x198101) [0x59f92ce6c101] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x1990) [0x59f92ce4a6e0] --- /opt/venv/bin/python3(+0x198101) [0x59f92ce6c101] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x1990) [0x59f92ce4a6e0] --- /opt/venv/bin/python3(+0x198101) [0x59f92ce6c101] --- /opt/venv/bin/python3(PyObject_Call+0x122) [0x59f92ce6cda2] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a8f) [0x59f92ce4b7df] --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59f92ce5f02c] --- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x59f92ce5433d] --- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x59f92ce6925c] --- /opt/venv/bin/python3(+0x29cd14) [0x59f92cf70d14] --- /opt/venv/bin/python3(_PyObject_MakeTpCall+0x25b) [0x59f92ce550bb] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x6a59) [0x59f92ce4f7a9] --- /opt/venv/bin/python3(+0x198101) [0x59f92ce6c101] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x1990) [0

Process completed with exit code 139.

test_llama_demo[wormhole_b0-device_params0-mesh_device0-performance-full] RuntimeError: TT_THROW @ /project/tt_metal/impl/dispatch/system_memory_manager.cpp:627: tt::exception info: TIMEOUT: device timeout, potential hang detected, the device is unrecoverable backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0xb998c8) [0x74d554a1a8c8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x5ce0b4) [0x74d55342f0b4] --- tt::tt_metal::distributed::FDMeshCommandQueue::read_completion_queue_event(tt::tt_metal::distributed::MeshReadEventDescriptor&) --- tt::tt_metal::distributed::FDMeshCommandQueue::read_completion_queue() --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x74d614d78253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x74d617e78ac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x74d617f09a84]

Benchmark data directory 'generated/benchmark_data' does not exist

builtin type SwigPyObject has no __module__ attribute

builtin type SwigPyPacked has no __module__ attribute

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Back off 18.344 seconds before retry.

Failed to download action 'https://api.github.com/repos/tenstorrent/tt-metal/tarball/efa39ff222bb1575585140352f202ef4120ca933'. Error: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.

No files were found with the provided path: /work/generated/test_reports/. No artifacts will be uploaded.

Benchmark data directory 'generated/benchmark_data' does not exist

Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

No files were found with the provided path: /work/generated/test_reports/. No artifacts will be uploaded.

Benchmark data directory 'generated/benchmark_data' does not exist

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details

Support for class-based `config` is deprecated, use ConfigDict instead.

No files were found with the provided path: /work/generated/test_reports/. No artifacts will be uploaded.

Benchmark data directory 'generated/benchmark_data' does not exist

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

No files were found with the provided path: /work/generated/test_reports/. No artifacts will be uploaded.

Benchmark data directory 'generated/benchmark_data' does not exist

Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Benchmark data directory 'generated/benchmark_data' does not exist

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.