Process completed with exit code 1.

test_flux1_pipeline[wormhole_b0-yes_use_cache-not_traced-encoder_device-4x8sp0tp1-dev-1024-1024-28-device_params0-True] OSError: You are trying to access a gated repo. Make sure to have access to it at https://huggingface.co/black-forest-labs/FLUX.1-dev. 401 Client Error. (Request ID: Root=1-69900df4-1c8e35d67162103157b9cd15;ab161afe-1305-4a10-a91c-85eeb281dea6) Cannot access gated repo for url https://huggingface.co/black-forest-labs/FLUX.1-dev/resolve/main/tokenizer/tokenizer_config.json. Access to model black-forest-labs/FLUX.1-dev is restricted. You must have access to it and be authenticated to access it. Please log in.

The action 'Galaxy Llama3 8B data-parallel demo tests' has timed out after 20 minutes.

test_demo_text[wormhole_b0-mesh_device0-device_params0-False-performance-ci-b1-DP-4] RuntimeError: TT_THROW @ /project/tt_metal/impl/dispatch/system_memory_manager.cpp:627: tt::exception info: TIMEOUT: device timeout, potential hang detected, the device is unrecoverable backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0xb99868) [0x79e41ea1b868] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x5cf674) [0x79e41d42e674] --- tt::tt_metal::distributed::FDMeshCommandQueue::read_completion_queue_event(tt::tt_metal::distributed::MeshReadEventDescriptor&) --- tt::tt_metal::distributed::FDMeshCommandQueue::read_completion_queue() --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x79e4dec60253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x79e4e1d60ac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x79e4e1df1a84]

Process completed with exit code 139.

Process completed with exit code 134.

test_demo_text[wormhole_b0-mesh_device0-device_params0-False-performance-ci-b1-DP-4] RuntimeError: TT_THROW @ /project/tt_metal/impl/dispatch/system_memory_manager.cpp:572: tt::exception info: TIMEOUT: device timeout in fetch queue wait, potential hang detected backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0xb99868) [0x7738a0e1b868] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x5cf477) [0x77389f82e477] --- tt::tt_metal::SystemMemoryManager::fetch_queue_reserve_back(unsigned char) --- tt::tt_metal::program_dispatch::write_program_command_sequence(tt::tt_metal::ProgramCommandSequence const&, tt::tt_metal::SystemMemoryManager&, unsigned int, tt::CoreType, bool, bool, bool) --- tt::tt_metal::distributed::FDMeshCommandQueue::write_program_cmds_to_subgrid(tt::tt_metal::distributed::MeshCoordinateRange const&, tt::tt_metal::ProgramCommandSequence&, bool, bool, std::unordered_set<unsigned int, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<unsigned int> >&, unsigned int) --- tt::tt_metal::distributed::FDMeshCommandQueue::enqueue_mesh_workload(tt::tt_metal::distributed::MeshWorkload&, bool) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail6launchITkNS0_22DeviceOperationConceptENS_12experimental4prim35RotaryEmbeddingLlamaDeviceOperationEEENT_21tensor_return_value_tERKNS6_22operation_attributes_tERKNS6_13tensor_args_tE+0x365) [0x7738a1bb2555] --- ttnn::prim::rotary_embedding_llama(tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, bool, std::optional<tt::tt_metal::MemoryConfig> const&, std::optional<ttnn::ComputeKernelConfig const> const&) --- ttnn::operations::experimental::transformer::RotaryEmbeddingLlamaOperation::invoke(tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, bool, std::optional<tt::tt_metal::MemoryConfig> const&, std::optional<ttnn::ComputeKernelConfig const>) --- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x4039ce) [0x7738a2ed89ce] --- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x2471a5) [0x7738a2d1c1a5] --- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x62edf65a733d] --- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x62edf65bc25c] --- /opt/venv/bin/python3(+0x29cd14) [0x62edf66c3d14] --- /opt/venv/bin/python3(PyObject_Call+0xbb) [0x62edf65bfd3b] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a8f) [0x62edf659e7df] --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x62edf65b202c] --- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x62edf65a733d] --- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x62edf65bc25c] --- /opt/venv/bin/python3(+0x29cd14) [0x62edf66c3d14] --- /opt/venv/bin/python3(_PyObject_MakeTpCall+0x25b) [0x62edf65a80bb] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x6a59) [0x62edf65a27a9] --- /opt/venv/bin/python3(+0x198101) [0x62edf65bf101] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x1990) [0x62edf659d6e0] --- /opt/venv/bin/python3(+0x198101) [0x62edf65bf101] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x1990) [0x62edf659d6e0] --- /opt/venv/bin/python3(+0x198101) [0x62edf65bf101] --- /opt/venv/bin/python3(PyObject_Call+0x122) [0x62edf65bfda2] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a8f) [0x62edf659e7df] --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x62edf65b202c] --- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x62edf65a733d] --- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x62edf65bc25c] --- /opt/venv/bin/python3(+0x29cd14) [0x62edf66c3d14] --- /opt/venv/bin/python3(_PyObject_MakeTpCall+0x25b) [0x62edf65a80bb] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x6a59) [0x62edf65a27a9] --- /opt/venv/bin/python3(+0x198101) [0x62edf65bf101] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x1990) [0

Process completed with exit code 134.

test_qwen_demo[wormhole_b0-device_params0-mesh_device0-performance-full] RuntimeError: TT_THROW @ /project/tt_metal/impl/dispatch/system_memory_manager.cpp:572: tt::exception info: TIMEOUT: device timeout in fetch queue wait, potential hang detected backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0xb99868) [0x76a3f9a1b868] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x5cf477) [0x76a3f842e477] --- tt::tt_metal::SystemMemoryManager::fetch_queue_reserve_back(unsigned char) --- tt::tt_metal::program_dispatch::write_program_command_sequence(tt::tt_metal::ProgramCommandSequence const&, tt::tt_metal::SystemMemoryManager&, unsigned int, tt::CoreType, bool, bool, bool) --- tt::tt_metal::distributed::FDMeshCommandQueue::write_program_cmds_to_subgrid(tt::tt_metal::distributed::MeshCoordinateRange const&, tt::tt_metal::ProgramCommandSequence&, bool, bool, std::unordered_set<unsigned int, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<unsigned int> >&, unsigned int) --- tt::tt_metal::distributed::FDMeshCommandQueue::enqueue_mesh_workload(tt::tt_metal::distributed::MeshWorkload&, bool) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29handle_mesh_adapter_cache_hitITkNS0_36DeviceOperationWithMeshDeviceAdapterENS0_26MeshDeviceOperationAdapterINS_4prim22ReshardDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS7_13tensor_args_tERNS7_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceERNSH_13program_cache6detail12ProgramCacheEm+0x154) [0x76a3fa3292c4] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail6launchITkNS0_22DeviceOperationConceptENS_4prim22ReshardDeviceOperationEEENT_21tensor_return_value_tERKNS5_22operation_attributes_tERKNS5_13tensor_args_tE+0x341) [0x76a3fa31bbd1] --- ttnn::prim::reshard(tt::tt_metal::Tensor const&, tt::tt_metal::MemoryConfig const&, std::optional<tt::tt_metal::Tensor> const&) --- ttnn::operations::data_movement::ReshardOperation::invoke(tt::tt_metal::Tensor const&, tt::tt_metal::MemoryConfig const&, std::optional<tt::tt_metal::Tensor> const&) --- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x302f32) [0x76a3fb9d7f32] --- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x2471a5) [0x76a3fb91c1a5] --- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x58d8f427d33d] --- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x58d8f429225c] --- /opt/venv/bin/python3(+0x29cd14) [0x58d8f4399d14] --- /opt/venv/bin/python3(PyObject_Call+0xbb) [0x58d8f4295d3b] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a8f) [0x58d8f42747df] --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x58d8f428802c] --- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x58d8f427d33d] --- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x58d8f429225c] --- /opt/venv/bin/python3(+0x29cd14) [0x58d8f4399d14] --- /opt/venv/bin/python3(_PyObject_MakeTpCall+0x25b) [0x58d8f427e0bb] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x6a59) [0x58d8f42787a9] --- /opt/venv/bin/python3(+0x198101) [0x58d8f4295101] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x1990) [0x58d8f42736e0] --- /opt/venv/bin/python3(+0x198101) [0x58d8f4295101] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x1990) [0x58d8f42736e0] --- /opt/venv/bin/python3(+0x198101) [0x58d8f4295101] --- /opt/venv/bin/python3(PyObject_Call+0x122) [0x58d8f4295da2] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a8f) [0x58d8f42747df] --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x58d8f428802c] --- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x58d8f427d33d] --- /opt/venv/bin/python3(_PyObject_Call_Prepend+0xc1) [0x58d8f42922c1] --- /opt/venv/bin/python3(+0x29cd14) [0x58d8f4399d14] --- /opt/venv/bin/python3(_PyObject_MakeTpCall+0x25b) [0x58d8f427e0bb] --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x6a59) [0x

The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

Process completed with exit code 139.

test_llama_demo[wormhole_b0-device_params0-mesh_device0-performance-full] RuntimeError: TT_THROW @ /project/tt_metal/impl/dispatch/system_memory_manager.cpp:627: tt::exception info: TIMEOUT: device timeout, potential hang detected, the device is unrecoverable backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0xb99868) [0x7aa6da81b868] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x5cf674) [0x7aa6d922e674] --- tt::tt_metal::distributed::FDMeshCommandQueue::read_completion_queue_event(tt::tt_metal::distributed::MeshReadEventDescriptor&) --- tt::tt_metal::distributed::FDMeshCommandQueue::read_completion_queue() --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7aa79aa05253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7aa79db05ac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7aa79db96a84]

The action 'Galaxy Qwen3-32B long context demo tests' has timed out after 20 minutes.

test_qwen_demo_text[wormhole_b0-mesh_device0-device_params0-10-performance-long-8k-b1] KeyError: (0, 'compile_decode')

build_verbosity 2 is not supported for build frontend. Ignoring.

Benchmark data directory 'generated/benchmark_data' does not exist

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Benchmark data directory 'generated/benchmark_data' does not exist

builtin type SwigPyObject has no __module__ attribute

builtin type SwigPyPacked has no __module__ attribute

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Benchmark data directory 'generated/benchmark_data' does not exist

Benchmark data directory 'generated/benchmark_data' does not exist

invalid value encountered in divide

Use load_torch_state_dict instead

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Benchmark data directory 'generated/benchmark_data' does not exist

builtin type SwigPyObject has no __module__ attribute

builtin type SwigPyPacked has no __module__ attribute

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

No files were found with the provided path: /work/generated/test_reports/. No artifacts will be uploaded.

Benchmark data directory 'generated/benchmark_data' does not exist

Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

No files were found with the provided path: /work/generated/test_reports/. No artifacts will be uploaded.

Benchmark data directory 'generated/benchmark_data' does not exist

builtin type SwigPyObject has no __module__ attribute

builtin type SwigPyPacked has no __module__ attribute

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Benchmark data directory 'generated/benchmark_data' does not exist

Use load_torch_state_dict instead

Use load_torch_state_dict instead

builtin type SwigPyObject has no __module__ attribute

builtin type SwigPyPacked has no __module__ attribute

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

No files were found with the provided path: /work/generated/test_reports/. No artifacts will be uploaded.

Benchmark data directory 'generated/benchmark_data' does not exist

Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Benchmark data directory 'generated/benchmark_data' does not exist

builtin type SwigPyObject has no __module__ attribute

builtin type SwigPyPacked has no __module__ attribute

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

No files were found with the provided path: /work/generated/test_reports/. No artifacts will be uploaded.

Benchmark data directory 'generated/benchmark_data' does not exist

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Benchmark data directory 'generated/benchmark_data' does not exist

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Benchmark data directory 'generated/benchmark_data' does not exist

In CPU autocast, but the target dtype is not supported. Disabling autocast. CPU Autocast only supports dtype of torch.bfloat16, torch.float16 currently.

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

builtin type SwigPyObject has no __module__ attribute

builtin type SwigPyPacked has no __module__ attribute

Test functions should return None, but models/demos/tg/falcon7b/demo_tg.py::test_demo_multichip[wormhole_b0-user_input0-32chipTG-perf_mode_2048_stochastic_verify] returned <class 'tuple'>. Did you mean to use `assert` instead of `return`? See https://docs.pytest.org/en/stable/how-to/assert.html#return-not-none for more information.

Test functions should return None, but models/demos/tg/falcon7b/demo_tg.py::test_demo_multichip[wormhole_b0-user_input0-32chipTG-perf_mode_1024_stochastic_verify] returned <class 'tuple'>. Did you mean to use `assert` instead of `return`? See https://docs.pytest.org/en/stable/how-to/assert.html#return-not-none for more information.

Test functions should return None, but models/demos/tg/falcon7b/demo_tg.py::test_demo_multichip[wormhole_b0-user_input0-32chipTG-perf_mode_128_stochastic_verify] returned <class 'tuple'>. Did you mean to use `assert` instead of `return`? See https://docs.pytest.org/en/stable/how-to/assert.html#return-not-none for more information.

Support for class-based `config` is deprecated, use ConfigDict instead.

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

No files were found with the provided path: /work/generated/test_reports/. No artifacts will be uploaded.

Benchmark data directory 'generated/benchmark_data' does not exist

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details