(Galaxy) demo tests #1022
Annotations
2 errors and 7 warnings
|
Galaxy Llama3 70B data-parallel demo tests
Process completed with exit code 134.
|
|
Galaxy Llama3 70B data-parallel demo tests:
models/tt_transformers/demo/simple_text_demo.py#L1051
test_demo_text[wormhole_b0-mesh_device0-device_params0-False-performance-ci-b1-DP-4]
RuntimeError: TT_THROW @ /project/tt_metal/impl/dispatch/system_memory_manager.cpp:572: tt::exception
info:
TIMEOUT: device timeout in fetch queue wait, potential hang detected
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0xb99868) [0x7738a0e1b868]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x5cf477) [0x77389f82e477]
--- tt::tt_metal::SystemMemoryManager::fetch_queue_reserve_back(unsigned char)
--- tt::tt_metal::program_dispatch::write_program_command_sequence(tt::tt_metal::ProgramCommandSequence const&, tt::tt_metal::SystemMemoryManager&, unsigned int, tt::CoreType, bool, bool, bool)
--- tt::tt_metal::distributed::FDMeshCommandQueue::write_program_cmds_to_subgrid(tt::tt_metal::distributed::MeshCoordinateRange const&, tt::tt_metal::ProgramCommandSequence&, bool, bool, std::unordered_set<unsigned int, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<unsigned int> >&, unsigned int)
--- tt::tt_metal::distributed::FDMeshCommandQueue::enqueue_mesh_workload(tt::tt_metal::distributed::MeshWorkload&, bool)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail6launchITkNS0_22DeviceOperationConceptENS_12experimental4prim35RotaryEmbeddingLlamaDeviceOperationEEENT_21tensor_return_value_tERKNS6_22operation_attributes_tERKNS6_13tensor_args_tE+0x365) [0x7738a1bb2555]
--- ttnn::prim::rotary_embedding_llama(tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, bool, std::optional<tt::tt_metal::MemoryConfig> const&, std::optional<ttnn::ComputeKernelConfig const> const&)
--- ttnn::operations::experimental::transformer::RotaryEmbeddingLlamaOperation::invoke(tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, bool, std::optional<tt::tt_metal::MemoryConfig> const&, std::optional<ttnn::ComputeKernelConfig const>)
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x4039ce) [0x7738a2ed89ce]
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x2471a5) [0x7738a2d1c1a5]
--- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x62edf65a733d]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x62edf65bc25c]
--- /opt/venv/bin/python3(+0x29cd14) [0x62edf66c3d14]
--- /opt/venv/bin/python3(PyObject_Call+0xbb) [0x62edf65bfd3b]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a8f) [0x62edf659e7df]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x62edf65b202c]
--- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x62edf65a733d]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x62edf65bc25c]
--- /opt/venv/bin/python3(+0x29cd14) [0x62edf66c3d14]
--- /opt/venv/bin/python3(_PyObject_MakeTpCall+0x25b) [0x62edf65a80bb]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x6a59) [0x62edf65a27a9]
--- /opt/venv/bin/python3(+0x198101) [0x62edf65bf101]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x1990) [0x62edf659d6e0]
--- /opt/venv/bin/python3(+0x198101) [0x62edf65bf101]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x1990) [0x62edf659d6e0]
--- /opt/venv/bin/python3(+0x198101) [0x62edf65bf101]
--- /opt/venv/bin/python3(PyObject_Call+0x122) [0x62edf65bfda2]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a8f) [0x62edf659e7df]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x62edf65b202c]
--- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x62edf65a733d]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x62edf65bc25c]
--- /opt/venv/bin/python3(+0x29cd14) [0x62edf66c3d14]
--- /opt/venv/bin/python3(_PyObject_MakeTpCall+0x25b) [0x62edf65a80bb]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x6a59) [0x62edf65a27a9]
--- /opt/venv/bin/python3(+0x198101) [0x62edf65bf101]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x1990) [0
|
|
Run tenstorrent/tt-metal/.github/actions/upload-artifact-with-job-uuid@main
No files were found with the provided path: /work/generated/test_reports/. No artifacts will be uploaded.
|
|
Save environment data
Benchmark data directory 'generated/benchmark_data' does not exist
|
|
Galaxy Llama3 70B data-parallel demo tests:
/work/models/common/llama_models.py#L57
Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details
|
|
Galaxy Llama3 70B data-parallel demo tests:
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132
Field "model_name" in OpTest has conflict with protected namespace "model_".
You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
|
|
Galaxy Llama3 70B data-parallel demo tests:
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
|
Galaxy Llama3 70B data-parallel demo tests:
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
|
Galaxy Llama3 70B data-parallel demo tests:
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
Loading