Process completed with exit code 134.

The job has exceeded the maximum execution time of 6h0m0s

The job has exceeded the maximum execution time of 6h0m0s

Process completed with exit code 1.

test_reduce_scatter_async[wormhole_b0-mesh_device0-2-1-1-fabric_ring-mem_config_input0-mem_config_rs0-batch_1_sd35_prompt-check-4links] RuntimeError: TT_FATAL @ /project/tt_metal/fabric/fabric.cpp:152: forwarding_direction.has_value() info: Could not find any forwarding direction from src (M0, D0) to dst (M0, D12) backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x5679e9) [0x7b81717e89e9] --- void tt::tt_fabric::append_fabric_connection_rt_args<tt::tt_metal::Program>(tt::tt_fabric::FabricNodeId const&, tt::tt_fabric::FabricNodeId const&, unsigned int, tt::tt_metal::Program&, tt::xy_pair const&, std::vector<unsigned int, std::allocator<unsigned int> >&, tt::CoreType) --- ttnn::build_ring_reduce_scatter_minimal_async_program_artifacts(tt::tt_metal::Program&, tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, tt::tt_metal::distributed::MeshCoordinate const&, std::optional<tt::tt_metal::distributed::MeshCoordinate> const&, std::optional<tt::tt_metal::distributed::MeshCoordinate> const&, tt::tt_metal::Tensor&, unsigned int, unsigned int, unsigned int, unsigned int, tt::tt_fabric::Topology, std::vector<tt::tt_metal::GlobalSemaphore, std::allocator<tt::tt_metal::GlobalSemaphore> > const&, std::optional<tt::tt_metal::GlobalSemaphore> const&, bool, std::optional const&) --- ttnn::experimental::prim::RingReduceScatterMeshWorkloadFactory::create_at(ttnn::experimental::prim::ReduceScatterMinimalAsyncParams const&, tt::tt_metal::distributed::MeshCoordinate const&, ttnn::experimental::prim::ReduceScatterMinimalAsyncInputs const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&) --- ttnn::experimental::prim::RingReduceScatterMeshWorkloadFactory::create_mesh_workload(ttnn::experimental::prim::ReduceScatterMinimalAsyncParams const&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, ttnn::experimental::prim::ReduceScatterMinimalAsyncInputs const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1793276) [0x7b81739f3276] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterITkNS0_36DeviceOperationWithMeshDeviceAdapterENS0_26MeshDeviceOperationAdapterINS_12experimental4prim40ReduceScatterMinimalAsyncDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x21d) [0x7b81739f10fd] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail6launchITkNS0_22DeviceOperationConceptENS_12experimental4prim40ReduceScatterMinimalAsyncDeviceOperationEEENT_21tensor_return_value_tERKNS6_22operation_attributes_tERKNS6_13tensor_args_tE+0x3f7) [0x7b81739e8867] --- ttnn::prim::reduce_scatter_minimal_async(tt::tt_metal::Tensor const&, std::optional<tt::tt_metal::Tensor> const&, std::optional<tt::tt_metal::Tensor> const&, unsigned int, unsigned int, unsigned int, tt::tt_metal::MemoryConfig, std::optional<tt::tt_metal::MemoryConfig>, tt::tt_fabric::Topology, std::vector<tt::tt_metal::GlobalSemaphore, std::allocator<tt::tt_metal::GlobalSemaphore> >, std::optional<tt::tt_metal::GlobalSemaphore>, bool, std::optional<ttsl::StrongType<unsigned char, tt::tt_metal::SubDeviceIdTag> >, std::optional<unsigned int>, std::optional<unsigned int>, std::optional<unsigned int>, std::optional<unsigned int>) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(_ZN4ttnn10operations12experimental3ccl32ExecuteReduceScatterMinimalAsync6invokeERKN2tt8tt_metal6TensorERKSt8optionalISt6vectorIS6_SaIS6_EEEiRKSA_INS5_15GlobalSemaphoreESaISG_EERKS9_ISG_EjRKS9_INS5_12MemoryConfigEESR_NS4_9tt_fabric8TopologyES9_IN4ttsl10StrongTypeIhNS5_14SubDeviceIdTagEEEES9_IjESZ_SZ_SZ_+0x428) [0x7b81739d8128] --- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x271f03) [0x7b8174d40f03] --- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-lin

build_verbosity 2 is not supported for build frontend. Ignoring.

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.

Support for class-based `config` is deprecated, use ConfigDict instead.

No files were found with the provided path: /work/generated/test_reports/. No artifacts will be uploaded.

No files were found with the provided path: /work/generated/test_reports/. No artifacts will be uploaded.

No files were found with the provided path: /work/generated/test_reports/. No artifacts will be uploaded.

No files were found with the provided path: /work/generated/test_reports/. No artifacts will be uploaded.

Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details

Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details

Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details

Support for class-based `config` is deprecated, use ConfigDict instead.

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

Support for class-based `config` is deprecated, use ConfigDict instead.