test_llama_demo[wormhole_b0-device_params0-mesh_device0-performance-full] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:905: tt::exception info: Statically allocated circular buffers in program 742 clash with L1 buffers on core range [(x=1,y=0) - (x=1,y=8)]. L1 buffer allocated at 528448 and static circular buffer region ends at 572384 backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0xe22565) [0x7f3f71fe6565] --- tt::tt_metal::detail::ProgramImpl::validate_circular_buffer_region(tt::tt_metal::IDevice const*) --- tt::tt_metal::distributed::MeshWorkloadImpl::compile(tt::tt_metal::distributed::MeshDevice*) --- tt::tt_metal::distributed::EnqueueMeshWorkload(tt::tt_metal::distributed::MeshCommandQueue&, tt::tt_metal::distributed::MeshWorkload&, bool) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x4636ba) [0x7f3f716276ba] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterITkNS0_36DeviceOperationWithMeshDeviceAdapterENS0_26MeshDeviceOperationAdapterIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS5_6TensorESaIS9_EEEEEEEEvN4ttsl10StrongTypeIhNS_10QueueIdTagEEERKNT_22operation_attributes_tERKNSI_13tensor_args_tERNSI_21tensor_return_value_tEPNS5_11distributed10MeshDeviceE+0x343) [0x7f3f716254f3] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceITkNS0_22DeviceOperationConceptEN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tEN4ttsl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x13d) [0x7f3f71624e0d] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail6invokeITkNS0_22DeviceOperationConceptEN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tEN4ttsl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x169) [0x7f3f71624979] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x4601ff) [0x7f3f716241ff] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, ttsl::StrongType<unsigned char, ttnn::QueueIdTag>) --- ttnn::operations::reduction::ExecuteTopK::invoke(ttsl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, unsigned int, signed char, bool, bool, std::optional<tt::tt_metal::MemoryConfig> const&, std::optional<CoreRangeSet> const&, std::optional<std::tuple<tt::tt_metal::Tensor, tt::tt_metal::Tensor> >) --- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x37176c) [0x7f3f72d5b76c] --- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x371456) [0x7f3f72d5b456] --- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x3683df) [0x7f3f72d523df] --- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x368163) [0x7f3f72d52163] --- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x170b6a) [0x7f3f72b5ab6a] --- /opt/venv/bin/python3(+0x18ae52) [0x5614faaf8e52] --- /opt/venv/bin/python3(_PyObject_MakeTpCall+0x25b) [0x5614faaef77b] --- /opt/venv/bin/python3(+0x198bfb) [0x5614fab06bfb] --- /opt/venv/bin/python3(_PyObject_Call+0x118) [0x5614fab07768] --- /opt/venv/bin/python3(

tg-demo-tests / tg-demo-tests (Galaxy Falcon7b demo tests, wormhole_b0, falcon7b, 30, U05RWH3QUPM)

Process completed with exit code 1.

tg-demo-tests / tg-demo-tests (Galaxy Falcon7b demo tests, wormhole_b0, falcon7b, 30, U05RWH3QUPM): models/demos/tg/falcon7b/demo_tg.py#L97

test_demo_multichip[wormhole_b0-user_input0-32chipTG-perf_mode_128_stochastic_verify] AssertionError: Prefill or decode perf is either lower or higher than {'prefill_t/s': 31500, 'decode_t/s/u': 12.55, 'decode_t/s': 12851.2}. See earlier warnings for more details.

build-artifact / 🐍 Build wheel

build_verbosity 2 is not supported for build frontend. Ignoring.

tg-demo-tests / tg-demo-tests (Galaxy Llama3 8B data-parallel demo tests, wormhole_b0, llama3_8b_dp, 30, U08BH66E...: /opt/venv/lib/python3.10/site-packages/llama_models/llama3/api/datatypes.py#L162