Skip to content
On PR

[D2M/TTNN] allocate cb at address #16101

Sign in to view logs

[D2M/TTNN] allocate cb at address

[D2M/TTNN] allocate cb at address #16101

Triggered via pull request November 25, 2025 20:37
@vtangTTvtangTT
synchronize #5875
vtangTT/generic-cb-w-addr
Status Failure
Total duration 1h 54m 50s
Artifacts 31

on-pr.yml

on: pull_request
prepare-run  /  ignore-files
10s
prepare-run / ignore-files
prepare-run  /  prepare-run
12s
prepare-run / prepare-run
build-image  /  check-if-docker-exist
6s
build-image / check-if-docker-exist
pre-commit  /  pre-commit
23s
pre-commit / pre-commit
pre-commit  /  check-spdx-headers
7s
pre-commit / check-spdx-headers
build-image  /  build-image
0s
build-image / build-image
build-image  /  set-latest-tag
0s
build-image / set-latest-tag
Matrix: release-build / build-ttmlir
debug-build  /  Debug Build
19m 28s
debug-build / Debug Build
lint  /  Lint (clang-tidy)
14m 47s
lint / Lint (clang-tidy)
Matrix: test / run-tests
test  /  test-summary
16s
test / test-summary
Matrix: downstream-checks
check-all-green
5s
check-all-green
Fit to window
Zoom out
Zoom in

Annotations

103 errors and 7 notices
test / run-tests (n150,speedy,0)
Process completed with exit code 1.
test_eltwise.test_unary_op_dram[True-(1024, 2048)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 131072. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(1024, 2048)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 131072. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(1024, 2048)-cos-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 131072. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(1024, 2048)-sin-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 131072. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(2048, 2048)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 262144. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(2048, 2048)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 262144. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(2048, 2048)-cos-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 262144. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(2048, 2048)-sin-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 262144. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(1024, 32)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(1024, 32)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(1024, 32)-cos-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(1024, 32)-sin-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(32, 1024)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(32, 1024)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(32, 1024)-cos-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(32, 1024)-sin-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(64, 1024)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(64, 1024)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(64, 1024)-cos-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(64, 1024)-sin-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(1024, 64)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(1024, 64)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(1024, 64)-cos-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(1024, 64)-sin-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(16, 64, 64)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(16, 64, 64)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(16, 64, 64)-cos-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(16, 64, 64)-sin-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(4, 8, 32, 32)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(4, 8, 32, 32)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(4, 8, 32, 32)-cos-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(4, 8, 32, 32)-sin-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(1024, 2048)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 131072. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(1024, 2048)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 131072. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(1024, 2048)-cos-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 131072. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(1024, 2048)-sin-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 131072. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(2048, 2048)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 262144. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(2048, 2048)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 262144. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(2048, 2048)-cos-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 262144. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(2048, 2048)-sin-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 262144. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(1024, 32)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(1024, 32)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(1024, 32)-cos-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(1024, 32)-sin-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(32, 1024)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(32, 1024)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(32, 1024)-cos-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(32, 1024)-sin-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(64, 1024)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(64, 1024)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test / run-tests (n150,tracy,4)
Process completed with exit code 1.
test_eltwise.test_unary_op_dram[True-(1024, 2048)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 131072. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(1024, 2048)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 131072. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(1024, 2048)-cos-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 131072. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(1024, 2048)-sin-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 131072. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(2048, 2048)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 262144. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(2048, 2048)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 262144. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(2048, 2048)-cos-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 262144. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(2048, 2048)-sin-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 262144. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(1024, 32)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(1024, 32)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(1024, 32)-cos-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(1024, 32)-sin-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(32, 1024)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(32, 1024)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(32, 1024)-cos-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(32, 1024)-sin-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(64, 1024)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(64, 1024)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(64, 1024)-cos-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(64, 1024)-sin-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(1024, 64)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(1024, 64)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(1024, 64)-cos-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(1024, 64)-sin-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(16, 64, 64)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(16, 64, 64)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(16, 64, 64)-cos-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(16, 64, 64)-sin-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(4, 8, 32, 32)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(4, 8, 32, 32)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(4, 8, 32, 32)-cos-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[True-(4, 8, 32, 32)-sin-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(1024, 2048)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 131072. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(1024, 2048)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 131072. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(1024, 2048)-cos-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 131072. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(1024, 2048)-sin-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 131072. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(2048, 2048)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 262144. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(2048, 2048)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 262144. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(2048, 2048)-cos-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 262144. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(2048, 2048)-sin-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 262144. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(1024, 32)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(1024, 32)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(1024, 32)-cos-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(1024, 32)-sin-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(32, 1024)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(32, 1024)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(32, 1024)-cos-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(32, 1024)-sin-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(64, 1024)-abs-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
test_eltwise.test_unary_op_dram[False-(64, 1024)-exp-f32]: test/ttnn-jit/test_eltwise.py#L193
RuntimeError: TT_FATAL @ /__w/tt-mlir/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/buffers/circular_buffer_config.cpp:136: total_size <= this->max_size_ info: Cannot set circular buffer size to 16384. This is larger than the associated dynamically allocated L1 buffer bank size of 4096 B backtrace: --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x75cae8) [0x7fc938e9fae8] --- tt::tt_metal::CircularBufferConfig::set_globally_allocated_address_and_total_size(tt::tt_metal::Buffer const&, unsigned int) --- tt::tt_metal::CircularBufferConfig::CircularBufferConfig(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::CircularBuffer::CircularBuffer(tt::tt_metal::CBDescriptor const&) --- tt::tt_metal::Program::Program(tt::tt_metal::ProgramDescriptor const&) --- ttnn::operations::generic::GenericOpDeviceOperation::GenericProgram::create(tt::tt_metal::ProgramDescriptor const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&, tt::tt_metal::Tensor&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x143889b) [0x7fc939b7b89b] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1437e66) [0x7fc939b7ae66] --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(_ZN4ttnn16device_operation6detail29launch_operation_with_adapterINS0_26MeshDeviceOperationAdapterINS_10operations7generic24GenericOpDeviceOperationEEEEEvRKNT_22operation_attributes_tERKNS8_13tensor_args_tERNS8_21tensor_return_value_tEPN2tt8tt_metal11distributed10MeshDeviceE+0x215) [0x7fc939b682c5] --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_device<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- ttnn::operations::generic::GenericOpDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::generic::GenericOpDeviceOperation>(ttnn::operations::generic::GenericOpDeviceOperation::operation_attributes_t const&, ttnn::operations::generic::GenericOpDeviceOperation::tensor_args_t const&) --- /__w/tt-mlir/tt-mlir/install/lib/_ttnncpp.so(+0x1423e90) [0x7fc939b66e90] --- ttnn::operations::generic::GenericOp::invoke(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, tt::tt_metal::ProgramDescriptor const&) --- /__w/tt-mlir/tt-mlir/install/lib/libTTMLIRRuntime.so(+0x2cd6fb) [0x7fc8e54df6fb] --- tt::runtime::ttnn::operations::generic_op::run(tt::target::ttnn::GenericOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::ProgramExecutor::execute() --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> >&) --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x55be8) [0x7fc90f8ddbe8] --- /__w/tt-mlir/tt-mlir/build/python_packages/ttnn_jit/runtime/_ttmlir_runtime.cpython-311-x86_64-linux-gnu.so(+0x62400) [0x7fc90f8ea400] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Vectorcall+0x35) [0x54b655] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x59) [0x534669] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0xf4) [0x570d04] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(PyObject_Call+0x9d) [0x570cad] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x47c5) [0x541d25] --- /opt/ttmlir-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x173) [0x566703] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xb8) [0x5346c8] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x59) [0x56e8c9] --- /opt/ttmlir-toolchain/venv/bin/python() [0x65a51b] --- /opt/ttmlir-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x28c) [0x52f9dc] --- /opt/ttmlir-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x53dc20] --- /opt/ttmlir-toolchain/venv/bin/python() [0x614584] --- /opt/ttmlir-toolchain/venv/bin/python(PyEval_EvalCode+0x97) [0x613be7] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6349cb] --- /opt/ttmlir-toolchain/venv/bin/python() [0x630c24] --- /opt/ttmlir-toolchain/venv/bin/python() [0x6450c5] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_SimpleFileObject+0x194) [0x6446d4] --- /opt/ttmlir-toolchain/venv/bin/python(_PyRun_AnyFileObject+0x47) [0x6444e7] --- /opt/ttmlir-toolchain/venv/bin/python(Py_RunMain+0x2c9) [0x63f259] --- /opt/ttmlir-toolchain/venv/bin/python(Py_BytesMain+0x2d) [0x604e8d] --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc93c333d90] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fc93c333e40] --- /opt/ttmlir-toolchain/venv/bin/python(_start+0x25) [0x604d15]
check-all-green
Process completed with exit code 1.
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Artifacts

Produced during runtime
Name Size Digest
build-artifacts-speedy
318 MB
sha256:645ae00d2459c2c0c8af5b703a8c996d051b2d61b7336178d03c4ab3ac2a7cdd
build-artifacts-tracy
663 MB
sha256:66d2a079b90e94feb67ddc89101f25739de1e6f0033c3982ba408b864c7a5b80
install-artifacts-speedy
302 MB
sha256:9de151720fb1fc01ff451173db12b29117e98042bd288c895204be3551af2522
install-artifacts-tracy
303 MB
sha256:1159736ebc0d35c6ce47839ac2c888d5b2c9d45da18e708cb6d446e110436d56
llmbox_speedy_56384051668_results
770 Bytes
sha256:e5f995cf567b3b3418cc6e8fbc29d109e3216494a0089a153700b231a15832b5
n150_speedy_56384051648_results
2.46 KB
sha256:6f3e04f49c14968c61076383d71a402604f993320f01618845dd5b56b9aeac87
n150_tracy_56384051657_results
4.86 KB
sha256:175bc89c3307910a054158963cbf9ccd99d99bea687d68673f0993c578724125
n150_tracy_56384051677_results
862 Bytes
sha256:87f51bd13135bd70852930c3574b47beda33c702c29bea1f23fc61973f861193
n300-llmbox_tracy_56384051674_results
256 Bytes
sha256:edb7391dbbc00759da69196942c74b8260e81908da2734c3311440d323273249
n300_speedy_56384051649_results
853 Bytes
sha256:ed6f0484087dc9e489a78df4e3e8ec8175557cb232dd69688d3bdaee75dd10ee
n300_tracy_56384051672_results
624 Bytes
sha256:09be5df667bdfc8ed2b5ea58ca5be534f7d47a961c3d74f3529ec63507f83fc6
p150_tracy_56384051690_results
862 Bytes
sha256:2ce6ba1a043e0b2ae80ca1299f07eb06c1788e0c6f50d5f33fef6203c85010bd
test-durations-llmbox-speedy-56384051668
183 Bytes
sha256:753f3276d2a33c6928dd49f7a797b6614e1d362fd388b31e095599080a0999ac
test-durations-n150-tracy-56384051677
185 Bytes
sha256:dab07d7e56b18359042b8d69ec9d6d41699b5b2b812a80bfdf0839057f39297d
test-durations-n300-llmbox-tracy-56384051674
228 Bytes
sha256:32c74eff297bce50aa1f1b9a02b9e56c3b0ac23f696a5f08a613bbe9d36eac47
test-durations-n300-speedy-56384051649
183 Bytes
sha256:d803211511413fcfe10fa5a4f5500bc640aa93a0b1d0bb047672b989c6558bc0
test-durations-n300-tracy-56384051672
248 Bytes
sha256:d032c1251f9638705ea3e15944e0061e26132a7d4497fedef7cc4b9f31351ba8
test-durations-p150-tracy-56384051690
185 Bytes
sha256:e5b2dbc0f97bcbc4bcd8a8c5fe9f139443dc85adc6a8115cb0c24778d23e1986
test-reports-debug
13.8 KB
sha256:4716b9cfbd181185ae3934758389d675e58540693e606cc7d6a9f5e3edad527c
test-reports-llmbox-speedy-56384051668
14.2 KB
sha256:29b1722b6b82cbcfd339dc7c367841226cc9e60dc8bae390003e09dc165725e0
test-reports-n150-speedy-56384051648
20.3 KB
sha256:9905cd0e6d136880025e42e28a6f3dc95d1b50a57b1a599c2e8571381083c3f0
test-reports-n150-tracy-56384051657
197 KB
sha256:543273a2a975b662ec400142d7c2cfbca782d83cad57529421c52b403f081d65
test-reports-n150-tracy-56384051677
58 KB
sha256:e82821cab0f78bc67ea207a4527e2edccc55671002ee60cfcd7fcffbdfdddaef
test-reports-n300-llmbox-tracy-56384051674
36.4 KB
sha256:d8b497f2934d547a33ae81f6917227ce0d8ae45fe536ff1053493c32dce5cc74
test-reports-n300-speedy-56384051649
14.2 KB
sha256:cb7da23cc949aca064e1fa841acad41dd31c74dba218a507e044429b63c7009b
test-reports-n300-tracy-56384051672
25.4 KB
sha256:a923274ce7dc1d24c6596d48811e6ee7f852a7570f1ace7499bdbd26efe8b6e3
test-reports-p150-tracy-56384051690
57.8 KB
sha256:084614cbd074df972ed4c3bab52385e705b22f0395042791bd4a71be747d67bb
tt-alchemist-whl-speedy
230 MB
sha256:323747a54bd47ab27c2159bf71291c939f8859edd50e5f8bdd8cb06c6ac9fa3e
ttnn-jit-whl-tracy
323 MB
sha256:9f68d5adef2ef605aa883dd678524650b7c8875433543766bb2270ad8576d3d8
ttrt-whl-speedy
184 MB
sha256:73289a791841753f9e6f1bcf51196b573d7a02bedf3749746ea30ed1034d1b5a
ttrt-whl-tracy
187 MB
sha256:81b979961ecac22477ce87c13bbe040b35bcac71453561390cb52e8415d5a549