(Galaxy) demo tests #1022
Annotations
2 errors and 6 warnings
|
Galaxy Qwen3-32B demo tests
Process completed with exit code 134.
|
|
Galaxy Qwen3-32B demo tests:
models/demos/llama3_70b_galaxy/demo/demo_qwen_decode.py#L712
test_qwen_demo[wormhole_b0-device_params0-mesh_device0-performance-full]
RuntimeError: TT_THROW @ /project/tt_metal/impl/dispatch/system_memory_manager.cpp:572: tt::exception
info:
TIMEOUT: device timeout in fetch queue wait, potential hang detected
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0xb99868) [0x76a3f9a1b868]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x5cf477) [0x76a3f842e477]
--- tt::tt_metal::SystemMemoryManager::fetch_queue_reserve_back(unsigned char)
--- tt::tt_metal::program_dispatch::write_program_command_sequence(tt::tt_metal::ProgramCommandSequence const&, tt::tt_metal::SystemMemoryManager&, unsigned int, tt::CoreType, bool, bool, bool)
--- tt::tt_metal::distributed::FDMeshCommandQueue::write_program_cmds_to_subgrid(tt::tt_metal::distributed::MeshCoordinateRange const&, tt::tt_metal::ProgramCommandSequence&, bool, bool, std::unordered_set<unsigned int, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<unsigned int> >&, unsigned int)
--- tt::tt_metal::distributed::FDMeshCommandQueue::enqueue_mesh_workload(tt::tt_metal::distributed::MeshWorkload&, bool)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29handle_mesh_adapter_cache_hitITkNS0_36DeviceOperationWithMeshDeviceAdapterENS0_26MeshDeviceOperationAdapterINS_4prim22ReshardDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS7_13tensor_args_tERNS7_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceERNSH_13program_cache6detail12ProgramCacheEm+0x154) [0x76a3fa3292c4]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail6launchITkNS0_22DeviceOperationConceptENS_4prim22ReshardDeviceOperationEEENT_21tensor_return_value_tERKNS5_22operation_attributes_tERKNS5_13tensor_args_tE+0x341) [0x76a3fa31bbd1]
--- ttnn::prim::reshard(tt::tt_metal::Tensor const&, tt::tt_metal::MemoryConfig const&, std::optional<tt::tt_metal::Tensor> const&)
--- ttnn::operations::data_movement::ReshardOperation::invoke(tt::tt_metal::Tensor const&, tt::tt_metal::MemoryConfig const&, std::optional<tt::tt_metal::Tensor> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x302f32) [0x76a3fb9d7f32]
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x2471a5) [0x76a3fb91c1a5]
--- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x58d8f427d33d]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x58d8f429225c]
--- /opt/venv/bin/python3(+0x29cd14) [0x58d8f4399d14]
--- /opt/venv/bin/python3(PyObject_Call+0xbb) [0x58d8f4295d3b]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a8f) [0x58d8f42747df]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x58d8f428802c]
--- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x58d8f427d33d]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x58d8f429225c]
--- /opt/venv/bin/python3(+0x29cd14) [0x58d8f4399d14]
--- /opt/venv/bin/python3(_PyObject_MakeTpCall+0x25b) [0x58d8f427e0bb]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x6a59) [0x58d8f42787a9]
--- /opt/venv/bin/python3(+0x198101) [0x58d8f4295101]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x1990) [0x58d8f42736e0]
--- /opt/venv/bin/python3(+0x198101) [0x58d8f4295101]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x1990) [0x58d8f42736e0]
--- /opt/venv/bin/python3(+0x198101) [0x58d8f4295101]
--- /opt/venv/bin/python3(PyObject_Call+0x122) [0x58d8f4295da2]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a8f) [0x58d8f42747df]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x58d8f428802c]
--- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x58d8f427d33d]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0xc1) [0x58d8f42922c1]
--- /opt/venv/bin/python3(+0x29cd14) [0x58d8f4399d14]
--- /opt/venv/bin/python3(_PyObject_MakeTpCall+0x25b) [0x58d8f427e0bb]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x6a59) [0x
|
|
Run tenstorrent/tt-metal/.github/actions/upload-artifact-with-job-uuid@main
No files were found with the provided path: /work/generated/test_reports/. No artifacts will be uploaded.
|
|
Save environment data
Benchmark data directory 'generated/benchmark_data' does not exist
|
|
Galaxy Qwen3-32B demo tests:
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132
Field "model_name" in OpTest has conflict with protected namespace "model_".
You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
|
|
Galaxy Qwen3-32B demo tests:
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
|
Galaxy Qwen3-32B demo tests:
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
|
Galaxy Qwen3-32B demo tests:
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
Loading