test_perf_device_bare_metal[128] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/ttnn_mnist --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest tests/ttnn/integration_tests/mnist/test_mnist.py::test_mnist'"]' returned non-zero exit status 4.

device-perf / N300 WH B0 not yet ported device perf: docker-job/tests/ttnn/integration_tests/mnist/test_mnist.py#L42

test_mnist[batch_size=128-device_params={'l1_small_size': 32768}] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/kernel/compute/tilize.cpp:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory 10 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7fe13cbbd168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7fe13a41e80c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fe13a4352b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fe13a43523e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fe13a1c7def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fe293b0bee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7fe13a1c7c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7fe13a43508e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7fe13a3d7d85] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7fe13a3d6890] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7fe13a3d5d78] --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fe1ed6d6253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fe293b06ac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7fe293b97a84] backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7fe13a324ed8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7fe13a304b51] --- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7fe13a3044cb] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fe13a4352b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fe13a43523e] --

device-perf / N300 WH B0 Set 2 device perf

Provided artifact name input during validation is empty

device-perf / N300 WH B0 Set 2 device perf

Process completed with exit code 1.

device-perf / N300 WH B0 Set 2 device perf: docker-job/tests/ttnn/perf_tests/operations/conv/test_conv2d_device_perf.py#L280

test_conv2d_device_perf subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/conv_perf --check-exit-code -a device_kernel_duration --op-support-count 1333 -t 5000 -m 'pytest tests/ttnn/perf_tests/operations/conv/test_conv2d_device_perf.py::test_run_conv2d_ops'"]' returned non-zero exit status 4.

device-perf / N300 WH B0 Set 2 device perf: docker-job/tests/ttnn/perf_tests/operations/conv/test_conv2d_device_perf.py#L209

test_run_conv2d_ops RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for pack_untilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/untilize.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/operations/sliding_window/halo/device/kernels/compute/pack_untilize.cpp:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory 10 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f1515898168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f15130f980c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f15131102b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f151311023e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f1512ea2def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f157c782ee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f1512ea2c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f151311008e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f15130b2d85] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f15130b1890] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f15130b0d78] --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f154e388253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f157c77dac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f157c80ea84] backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f1512fffed8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f1512fdfb51] --- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f1512fdf4cb] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f15131102b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f151311

device-perf / N300 WH B0 Set 2 device perf

Process completed with exit code 1.

device-perf / N300 WH B0 Set 2 device perf: docker-job/models/experimental/efficientnetb0/tests/perf/test_perf.py#L33

test_perf_device_bare_metal_efficientnetb0[1-213.0] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/ttnn_efficientnetb0 --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest models/experimental/efficientnetb0/tests/pcc/test_ttnn_efficientnetb0.py'"]' returned non-zero exit status 4.

device-perf / N300 WH B0 Set 2 device perf: docker-job/models/experimental/efficientnetb0/tests/pcc/test_ttnn_efficientnetb0.py#L28

test_efficientnetb0_model[device_params0] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for transpose_wh_rm TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_unary/eltwise_unary.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/operations/data_movement/transpose/device/kernels/compute/transpose_wh_rm.cpp:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory 10 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7fd87c86b168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7fd87a0cc80c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fd87a0e32b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fd87a0e323e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fd879e75def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fd8e3768ee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7fd879e75c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7fd87a0e308e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7fd87a085d85] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7fd87a084890] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7fd87a083d78] --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fd8b536b253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fd8e3763ac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7fd8e37f4a84] backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7fd879fd2ed8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7fd879fb2b51] --- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7fd879fb24cb] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fd87a0e32b8] --- /opt/venv/lib/python3.10/site-packages/ttn

device-perf / N300 WH B0 Set 2 device perf

Process completed with exit code 1.

device-perf / N300 WH B0 Set 2 device perf: docker-job/models/experimental/vovnet/tests/perf/test_perf.py#L28

test_perf_device_bare_metal_vovnet[1-159.0] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/ttnn_vovnet --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest models/experimental/vovnet/tests/pcc/test_tt_vovnet.py::test_vovnet_model_inference'"]' returned non-zero exit status 4.

device-perf / N300 WH B0 Set 2 device perf: docker-job/models/experimental/vovnet/tests/pcc/test_tt_vovnet.py#L54

test_vovnet_model_inference[device_params0-hf_hub:timm/ese_vovnet19b_dw.ra_in1k] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for conv_bmm_tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/bcast.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/operations/conv/conv2d/device/kernels/conv_bmm_tilize.cpp:8, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory 10 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f112e30d168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f112bb6e80c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f112bb852b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f112bb8523e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f112b917def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f11951fdee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f112b917c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f112bb8508e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f112bb27d85] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f112bb26890] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f112bb25d78] --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f1166e03253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f11951f8ac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f1195289a84] backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f112ba74ed8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f112ba54b51] --- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f112ba544cb] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f112bb852b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/b

device-perf / N300 WH B0 Set 2 device perf

Process completed with exit code 1.

device-perf / N300 WH B0 Set 2 device perf: docker-job/models/experimental/swin_s/tests/perf/test_perf.py#L29

test_perf_device_bare_metal_swin_s[1-Swin_s-11.5] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/Swin_s --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest models/experimental/swin_s/tests/pcc/test_ttnn_swin_transformer.py::test_swin_s_transformer[pretrained_weight_true-0]'"]' returned non-zero exit status 4.

device-perf / N300 WH B0 Set 2 device perf: docker-job/models/experimental/swin_s/tests/pcc/test_ttnn_swin_transformer.py#L198

test_swin_s_transformer[pretrained_weight_true-0] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for transpose_wh_rm TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_unary/eltwise_unary.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/operations/data_movement/transpose/device/kernels/compute/transpose_wh_rm.cpp:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory 10 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7fc876015168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7fc87387680c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fc87388d2b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fc87388d23e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fc87361fdef] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fc8dcf08ee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7fc87361fc49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7fc87388d08e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7fc87382fd85] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7fc87382e890] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7fc87382dd78] --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fc8aeb06253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fc8dcf03ac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7fc8dcf94a84] backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7fc87377ced8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7fc87375cb51] --- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7fc87375c4cb] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fc87388d2b8] --- /opt/venv/lib/python3.10/site-pack

device-perf / N300 WH B0 Set 2 device perf

Process completed with exit code 1.

device-perf / N300 WH B0 Set 2 device perf: docker-job/models/experimental/swin_v2/tests/perf/test_perf.py#L29

test_perf_device_bare_metal_swin_v2[1-Swin_v2-16.6] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/Swin_v2 --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest models/experimental/swin_v2/tests/pcc/test_ttnn_swin_v2_s.py::test_swin_s_transformer'"]' returned non-zero exit status 4.

device-perf / N300 WH B0 Set 2 device perf: docker-job/models/experimental/swin_v2/tests/pcc/test_ttnn_swin_v2_s.py#L165

test_swin_s_transformer[device_params0] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for transpose_xw_tiled TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_unary/eltwise_unary.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/operations/data_movement/permute/device/kernels/compute/transpose_xw_tiled.cpp:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory 10 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f19c7e4f168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f19c56b080c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f19c56c72b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f19c56c723e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f19c5459def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f1a2ed68ee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f19c5459c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f19c56c708e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f19c5669d85] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f19c5668890] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f19c5667d78] --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f1a00953253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f1a2ed63ac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f1a2edf4a84] backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f19c55b6ed8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f19c5596b51] --- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f19c55964cb] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f19c56c72b8] --- /opt/venv/lib/python3.10/site-packages/t

device-perf / N300 WH B0 Set 2 device perf

Process completed with exit code 1.

device-perf / N300 WH B0 Set 2 device perf: docker-job/models/demos/vision/segmentation/vanilla_unet/tests/test_unet_perf.py#L45

test_vanilla_unet_perf_device[1-206.0] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/unet_vanilla --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest models/demos/vision/segmentation/vanilla_unet/tests/test_unet_model.py::test_vanilla_unet_model'"]' returned non-zero exit status 4.

device-perf / N300 WH B0 Set 2 device perf: docker-job/models/demos/vision/segmentation/vanilla_unet/tests/test_unet_model.py#L55

test_vanilla_unet_model[1-3-480-640-device_params0] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for convert_to_hwc TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/pack_untilize.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/operations/experimental/cnn/convert_to_hwc/device/kernels/convert_to_hwc.cpp:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory 10 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7ff39336e168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7ff390bcf80c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7ff390be62b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7ff390be623e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7ff390978def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7ff3fa26eee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7ff390978c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7ff390be608e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7ff390b88d85] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7ff390b87890] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7ff390b86d78] --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7ff3cbe7c253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7ff3fa269ac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7ff3fa2faa84] backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7ff390ad5ed8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7ff390ab5b51] --- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7ff390ab54cb] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7ff390be62b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/buil

device-perf / N300 WH B0 Set 2 device perf

Process completed with exit code 1.

device-perf / N300 WH B0 Set 2 device perf: docker-job/models/demos/vision/classification/mobilenetv2/tests/perf/test_perf_mobilenetv2.py#L28

test_perf_device_mobilenetv2[10-3730] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/ttnn_mobilenetv2 --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest models/demos/vision/classification/mobilenetv2/tests/pcc/test_mobilenetv2.py'"]' returned non-zero exit status 4.

device-perf / N300 WH B0 Set 2 device perf: docker-job/models/demos/vision/classification/mobilenetv2/tests/pcc/test_mobilenetv2.py#L57

test_mobilenetv2[10-pretrained_weight_true-device_params0] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for conv_bmm_tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/bcast.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/operations/conv/conv2d/device/kernels/conv_bmm_tilize.cpp:8, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory 10 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7fb589dfd168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7fb58765e80c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fb5876752b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fb58767523e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fb587407def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fb5f0cf0ee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7fb587407c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7fb58767508e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7fb587617d85] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7fb587616890] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7fb587615d78] --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fb5c28eb253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fb5f0cebac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7fb5f0d7ca84] backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7fb587564ed8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7fb587544b51] --- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7fb5875444cb] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fb5876752b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.s

device-perf / N300 WH B0 Set 2 device perf

Process completed with exit code 1.

device-perf / N300 WH B0 Set 2 device perf: docker-job/models/demos/wormhole/mamba/tests/test_mamba_perf.py#L150

test_mamba_perf_device[32-1.596] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/ttnn_mamba --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest models/demos/wormhole/mamba/tests/test_mamba_model.py::test_device_perf[1]'"]' returned non-zero exit status 4.

device-perf / N300 WH B0 Set 2 device perf: docker-job/models/demos/wormhole/mamba/tests/test_mamba_model.py#L184

test_device_perf[1] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for tilize_wh TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/operations/data_movement/tilize/device/kernels/compute/tilize_wh.cpp:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory 10 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f91487fd168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f914605e80c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f91460752b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f914607523e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f9145e07def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f91af6eaee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f9145e07c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f914607508e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f9146017d85] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f9146016890] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f9146015d78] --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f91812eb253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f91af6e5ac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f91af776a84] backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f9145f64ed8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f9145f44b51] --- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f9145f444cb] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f91460752b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f914607523e] --

device-perf / P150 BH device perf: docker-job/models/experimental/panoptic_deeplab/tests/test_device_perf_pdl.py#L82

test_device_perf_pdl_110_cores[test_panoptic_deeplab_110_cores] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/panoptic_deeplab --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest models/experimental/panoptic_deeplab/tests/pcc/test_tt_model.py::test_model_panoptic_deeplab -k panoptic_deeplab_110_cores'"]' returned non-zero exit status 4.

device-perf / P150 BH device perf: docker-job/models/experimental/panoptic_deeplab/tests/pcc/test_tt_model.py#L145

test_model_panoptic_deeplab[device_params0-panoptic_deeplab_110_cores] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for conv_bmm_tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/bcast.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/operations/conv/conv2d/device/kernels/conv_bmm_tilize.cpp:8, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory 13 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f2eeae44168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f2ee86a580c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2ee86bc2b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2ee86bc23e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2ee844edef] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2f29d1dee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2ee844ec49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2ee86bc08e] --- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&) --- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&) --- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const --- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2ee86bc2b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2ee86bc23e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2ee844edef] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2f29d1dee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2ee844ec49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2ee86bc08e] ---

device-perf / P150 BH device perf

Process completed with exit code 1.

device-perf / P150 BH device perf: docker-job/models/demos/vision/classification/vit/blackhole/tests/test_vit_device_perf.py#L58

test_vit_perf_device[4500-10] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/vit --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest models/demos/vision/classification/vit/blackhole/tests/test_vit_device_perf.py::test_vit_device_ops[10-device_params0]'"]' returned non-zero exit status 4.

device-perf / P150 BH device perf: docker-job/models/demos/vision/classification/vit/blackhole/tests/test_vit_device_perf.py#L37

test_vit_device_ops[10-device_params0] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/kernel/compute/tilize.cpp:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory 13 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7fc6e993a168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7fc6e719b80c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fc6e71b22b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fc6e71b223e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fc6e6f44def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fc72882fee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7fc6e6f44c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7fc6e71b208e] --- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&) --- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&) --- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const --- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fc6e71b22b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fc6e71b223e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fc6e6f44def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fc72882fee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7fc6e6f44c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7fc6e71b208e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0

device-perf / P150 BH device perf

Process completed with exit code 1.

device-perf / P150 BH device perf: docker-job/models/experimental/functional_unet/tests/test_unet_perf.py#L141

test_unet_perf_device[1-4-2875.5] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/unet_shallow --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest models/experimental/functional_unet/tests/test_unet_perf.py::test_unet_model'"]' returned non-zero exit status 4.

device-perf / P150 BH device perf: docker-job/models/experimental/functional_unet/tests/test_unet_perf.py#L39

test_unet_model[device_params0-4-4-1] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for convert_to_hwc TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/pack_untilize.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/operations/experimental/cnn/convert_to_hwc/device/kernels/convert_to_hwc.cpp:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory 13 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7fad7b3b6168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7fad78c1780c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fad78c2e2b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fad78c2e23e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fad789c0def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fadba293ee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7fad789c0c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7fad78c2e08e] --- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&) --- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&) --- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const --- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fad78c2e2b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fad78c2e23e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fad789c0def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fadba293ee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7fad789c0c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7fad78c2e08e] --- /opt/v

device-perf / P150 BH device perf

Process completed with exit code 1.

device-perf / P150 BH device perf: docker-job/models/demos/blackhole/sentence_bert/tests/perf/test_sentence_bert_perf.py#L32

test_perf_device_bare_metal_sentence_bert[8-980.0-sentence_bert] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/ttnn_sentence_bert_model --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest models/demos/blackhole/sentence_bert/tests/pcc/test_ttnn_sentencebert_model.py::test_ttnn_sentence_bert_model'"]' returned non-zero exit status 4.

device-perf / P150 BH device perf: docker-job/models/demos/blackhole/sentence_bert/tests/pcc/test_ttnn_sentencebert_model.py#L54

test_ttnn_sentence_bert_model[device_params0-inputs0] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/operations/data_movement/tilize/device/kernels/compute/tilize.cpp:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory 13 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f95a7756168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f95a4fb780c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f95a4fce2b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f95a4fce23e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f95a4d60def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f95e6635ee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f95a4d60c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f95a4fce08e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f95a4f70d85] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f95a4f6f890] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f95a4f6ed78] --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f95cc240253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f95e6630ac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f95e66c1a84] backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f95a4ebded8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f95a4e9db51] --- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f95a4e9d4cb] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f95a4fce2b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f9

device-perf / P150 BH device perf

Process completed with exit code 1.

device-perf / P150 BH device perf: docker-job/models/demos/vision/segmentation/ufld_v2/blackhole/tests/perf/test_ufld_v2_perf.py#L34

test_perf_device_bare_metal_ufld_v2[1-495-UFLD-v2] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/ttnn_UFLD_v2 --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest models/demos/vision/segmentation/ufld_v2/blackhole/tests/pcc/test_ttnn_ufld_v2.py::test_ufld_v2_model'"]' returned non-zero exit status 4.

device-perf / P150 BH device perf: docker-job/models/demos/vision/segmentation/ufld_v2/blackhole/tests/pcc/test_ttnn_ufld_v2.py#L560

test_ufld_v2_model[device_params0-pretrained_weight_false-1-3-320-800] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for transpose_xw_rm_single_tile_size TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_unary/eltwise_unary.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/operations/data_movement/permute/device/kernels/compute/transpose_xw_rm_single_tile_size.cpp:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory 13 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f65377ce168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f653502f80c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f65350462b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f653504623e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f6534dd8def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f65766baee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f6534dd8c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f653504608e] --- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&) --- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&) --- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const --- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f65350462b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f653504623e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f6534dd8def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f65766baee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f6534dd8c49] --- /opt/venv/lib/python3.10/si

device-perf / P150 BH device perf

Process completed with exit code 1.

device-perf / P150 BH device perf: docker-job/models/demos/vision/classification/resnet50/blackhole/tests/test_perf_device_resnet50.py#L22

test_perf_device[32-True-DataType.BFLOAT8_B-DataType.BFLOAT8_B-MathFidelity.LoFi-32-device_params0-15480.0] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/resnet50 --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest models/demos/vision/classification/resnet50/blackhole/tests/test_resnet50_performant.py::test_run_resnet50_inference[True-DataType.BFLOAT8_B-DataType.BFLOAT8_B-MathFidelity.LoFi-32-device_params0]'"]' returned non-zero exit status 4.

device-perf / P150 BH device perf: docker-job/models/demos/vision/classification/resnet50/blackhole/tests/test_resnet50_performant.py#L28

test_run_resnet50_inference[True-DataType.BFLOAT8_B-DataType.BFLOAT8_B-MathFidelity.LoFi-32-device_params0] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for transpose_wh_rm TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_unary/eltwise_unary.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/operations/data_movement/transpose/device/kernels/compute/transpose_wh_rm.cpp:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory 13 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f5a71675168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f5a6eed680c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f5a6eeed2b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f5a6eeed23e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f5a6ec7fdef] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f5ab055bee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f5a6ec7fc49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f5a6eeed08e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f5a6ee8fd85] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f5a6ee8e890] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f5a6ee8dd78] --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f5a96173253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f5ab0556ac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f5ab05e7a84] backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f5a6eddced8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f5a6edbcb51] --- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f5a6edbc4cb] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f5a6eeed

device-perf / P150 BH device perf: docker-job/models/demos/vision/classification/resnet50/blackhole/tests/test_perf_device_resnet50.py#L22

test_perf_device[16-True-DataType.BFLOAT8_B-DataType.BFLOAT8_B-MathFidelity.LoFi-16-device_params0-10610.0] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/resnet50 --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest models/demos/vision/classification/resnet50/blackhole/tests/test_resnet50_performant.py::test_run_resnet50_inference[True-DataType.BFLOAT8_B-DataType.BFLOAT8_B-MathFidelity.LoFi-16-device_params0]'"]' returned non-zero exit status 4.

device-perf / P150 BH device perf: docker-job/models/demos/vision/classification/resnet50/blackhole/tests/test_resnet50_performant.py#L28

test_run_resnet50_inference[True-DataType.BFLOAT8_B-DataType.BFLOAT8_B-MathFidelity.LoFi-16-device_params0] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for transpose_wh_rm TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_unary/eltwise_unary.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/operations/data_movement/transpose/device/kernels/compute/transpose_wh_rm.cpp:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory 13 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f9d4901d168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f9d4687e80c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f9d468952b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f9d4689523e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f9d46627def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f9d87effee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f9d46627c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f9d4689508e] --- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&) --- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&) --- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const --- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f9d468952b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f9d4689523e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f9d46627def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f9d87effee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f9d46627c49] --- /opt/venv/lib/python3.

device-perf / P150 BH device perf

Process completed with exit code 1.

device-perf / P150 BH device perf: docker-job/models/demos/vision/segmentation/vgg_unet/blackhole/tests/perf/test_perf_vgg_unet.py#L28

test_perf_device_bare_metal_vgg_unet[1-488] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/ttnn_vgg_unet --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest models/demos/vision/segmentation/vgg_unet/blackhole/tests/pcc/test_vgg_unet.py::test_vgg_unet[0-pretrained_weight_false]'"]' returned non-zero exit status 4.

device-perf / P150 BH device perf: docker-job/models/demos/vision/segmentation/vgg_unet/blackhole/tests/pcc/test_vgg_unet.py#L54

test_vgg_unet[0-pretrained_weight_false] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for transpose_xw_rm_single_tile_size TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_unary/eltwise_unary.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/operations/data_movement/permute/device/kernels/compute/transpose_xw_rm_single_tile_size.cpp:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory 13 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f2db226c168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f2dafacd80c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2dafae42b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2dafae423e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2daf876def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2df1152ee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2daf876c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2dafae408e] --- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&) --- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&) --- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const --- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2dafae42b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2dafae423e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2daf876def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2df1152ee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2daf876c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/lib

device-perf / P150 BH device perf

Process completed with exit code 1.

device-perf / P150 BH device perf: docker-job/models/demos/vision/generative/stable_diffusion/blackhole/tests/test_perf.py#L466

test_stable_diffusion_vae_device_perf[4.1] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/ttnn_stable_diffusion_vae --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest models/demos/vision/generative/stable_diffusion/wormhole/tests/vae/test_vae.py::test_decoder[4-64-64-3-512-512-device_params0]'"]' returned non-zero exit status 4.

device-perf / P150 BH device perf: docker-job/models/demos/vision/generative/stable_diffusion/wormhole/tests/vae/test_vae.py#L58

test_decoder[4-64-64-3-512-512-device_params0] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for bmm_large_block_zm_fused_bias_activation TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/matmul.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/operations/matmul/device/kernels/compute/bmm_large_block_zm_fused_bias_activation.cpp:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory 13 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f274ca4c168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f274a2ad80c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f274a2c42b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f274a2c423e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f274a056def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f278b948ee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f274a056c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f274a2c408e] --- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&) --- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&) --- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const --- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f274a2c42b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f274a2c423e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f274a056def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f278b948ee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f274a056c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0

device-perf / P150 BH device perf: docker-job/models/demos/vision/generative/stable_diffusion/blackhole/tests/test_perf.py#L439

test_stable_diffusion_device_perf[21.0] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/ttnn_stable_diffusion --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest models/demos/vision/generative/stable_diffusion/wormhole/tests/test_unet_2d_condition_model.py::test_unet_2d_condition_model_512x512[2-4-64-64-device_params=l1_small_size_24576]'"]' returned non-zero exit status 4.

device-perf / P150 BH device perf: docker-job/models/demos/vision/generative/stable_diffusion/wormhole/tests/test_unet_2d_condition_model.py#L115

test_unet_2d_condition_model_512x512[2-4-64-64-device_params=l1_small_size_24576] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/kernel/compute/tilize.cpp:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory 13 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f88eb0ca168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f88e892b80c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f88e89422b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f88e894223e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f88e86d4def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f8929fafee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f88e86d4c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f88e894208e] --- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&) --- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&) --- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const --- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f88e89422b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f88e894223e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f88e86d4def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f8929fafee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f88e86d4c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f88e894208e] --- /opt/venv/lib/python3.10/sit

device-perf / N300 WH B0 Set 1 device perf

Process completed with exit code 1.

device-perf / N300 WH B0 Set 1 device perf: docker-job/models/demos/vision/classification/resnet50/wormhole/tests/test_perf_device_resnet50.py#L21

test_perf_device[16-True-16-DataType.BFLOAT8_B-DataType.BFLOAT8_B-MathFidelity.LoFi-device_params0-6140.0] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/resnet50 --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest models/demos/vision/classification/resnet50/wormhole/tests/test_resnet50_performant.py::test_run_resnet50_inference[True-16-DataType.BFLOAT8_B-DataType.BFLOAT8_B-MathFidelity.LoFi-device_params0]'"]' returned non-zero exit status 4.

device-perf / N300 WH B0 Set 1 device perf: docker-job/models/demos/vision/classification/resnet50/wormhole/tests/test_resnet50_performant.py#L27

test_run_resnet50_inference[True-16-DataType.BFLOAT8_B-DataType.BFLOAT8_B-MathFidelity.LoFi-device_params0] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for transpose_wh_rm TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_unary/eltwise_unary.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/operations/data_movement/transpose/device/kernels/compute/transpose_wh_rm.cpp:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory 10 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f282ed40168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f282c5a180c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f282c5b82b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f282c5b823e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f282c34adef] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2895c31ee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f282c34ac49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f282c5b808e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f282c55ad85] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f282c559890] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f282c558d78] --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f2867838253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f2895c2cac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f2895cbda84] backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f282c4a7ed8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f282c487b51] --- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f282c4874cb] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8

device-perf / N300 WH B0 Set 1 device perf

Process completed with exit code 1.

device-perf / N300 WH B0 Set 1 device perf: docker-job/models/demos/vision/segmentation/vgg_unet/wormhole/tests/perf/test_perf_vgg_unet.py#L28

test_perf_device_bare_metal_vgg_unet[1-291] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/ttnn_vgg_unet --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest models/demos/vision/segmentation/vgg_unet/wormhole/tests/pcc/test_vgg_unet.py::test_vgg_unet[0-pretrained_weight_false]'"]' returned non-zero exit status 4.

device-perf / N300 WH B0 Set 1 device perf: docker-job/models/demos/vision/segmentation/vgg_unet/wormhole/tests/pcc/test_vgg_unet.py#L54

test_vgg_unet[0-pretrained_weight_false] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for transpose_xw_rm_single_tile_size TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_unary/eltwise_unary.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/operations/data_movement/permute/device/kernels/compute/transpose_xw_rm_single_tile_size.cpp:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory 10 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7effe376d168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7effe0fce80c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7effe0fe52b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7effe0fe523e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7effe0d77def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f004a66eee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7effe0d77c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7effe0fe508e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7effe0f87d85] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7effe0f86890] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7effe0f85d78] --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f001c27c253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f004a669ac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f004a6faa84] backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7effe0ed4ed8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7effe0eb4b51] --- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7effe0eb44cb] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7effe0fe52b8] --- /opt/venv/l

device-perf / N300 WH B0 Set 1 device perf

Process completed with exit code 1.

device-perf / N300 WH B0 Set 1 device perf: docker-job/models/demos/vision/classification/vgg/tests/test_perf_vgg.py#L133

test_perf_device_bare_metal_vgg[1-ttnn_vgg16] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/ttnn_vgg16 --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest tests/ttnn/integration_tests/vgg/test_ttnn_vgg16.py'"]' returned non-zero exit status 4.

device-perf / N300 WH B0 Set 1 device perf: docker-job/tests/ttnn/integration_tests/vgg/test_ttnn_vgg16.py#L63

test_vgg16[batch_size=1-act_dtype=DataType.BFLOAT16-weight_dtype=DataType.BFLOAT16-math_fidelity=MathFidelity.LoFi-device_params={'l1_small_size': 24576}] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for conv_bmm_tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/bcast.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/operations/conv/conv2d/device/kernels/conv_bmm_tilize.cpp:8, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory 10 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f35a319c168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f35a09fd80c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f35a0a142b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f35a0a1423e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f35a07a6def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f360a094ee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f35a07a6c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f35a0a1408e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f35a09b6d85] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f35a09b5890] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f35a09b4d78] --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f35dbc92253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f360a08fac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f360a120a84] backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f35a0903ed8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f35a08e3b51] --- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f35a08e34cb] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7

device-perf / N300 WH B0 Set 1 device perf: docker-job/models/demos/vision/classification/vgg/tests/test_perf_vgg.py#L133

test_perf_device_bare_metal_vgg[1-ttnn_vgg11] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/ttnn_vgg11 --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest tests/ttnn/integration_tests/vgg/test_ttnn_vgg11.py'"]' returned non-zero exit status 4.

device-perf / N300 WH B0 Set 1 device perf: docker-job/tests/ttnn/integration_tests/vgg/test_ttnn_vgg11.py#L61

test_vgg11[batch_size=1-act_dtype=DataType.BFLOAT16-weight_dtype=DataType.BFLOAT16-math_fidelity=MathFidelity.LoFi-device_params={'l1_small_size': 24576}] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for conv_bmm_tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/bcast.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/operations/conv/conv2d/device/kernels/conv_bmm_tilize.cpp:8, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory 10 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7fc9cb11b168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7fc9c897c80c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fc9c89932b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fc9c899323e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fc9c8725def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fca3201bee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7fc9c8725c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7fc9c899308e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7fc9c8935d85] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7fc9c8934890] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7fc9c8933d78] --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fca03c1e253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fca32016ac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7fca320a7a84] backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7fc9c8882ed8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7fc9c8862b51] --- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7fc9c88624cb] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7

device-perf / N300 WH B0 Set 1 device perf

Process completed with exit code 1.

device-perf / N300 WH B0 Set 1 device perf: docker-job/models/demos/vision/generative/stable_diffusion/wormhole/tests/test_perf.py#L466

test_stable_diffusion_vae_device_perf[2.95] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/ttnn_stable_diffusion_vae --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest models/demos/vision/generative/stable_diffusion/wormhole/tests/vae/test_vae.py::test_decoder[4-64-64-3-512-512-device_params0]'"]' returned non-zero exit status 4.

device-perf / N300 WH B0 Set 1 device perf: docker-job/models/demos/vision/generative/stable_diffusion/wormhole/tests/vae/test_vae.py#L58

test_decoder[4-64-64-3-512-512-device_params0] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for bmm_large_block_zm_fused_bias_activation TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/matmul.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/operations/matmul/device/kernels/compute/bmm_large_block_zm_fused_bias_activation.cpp:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory 10 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f5292044168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f528f8a580c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f528f8bc2b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f528f8bc23e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f528f64edef] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f52f8f42ee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f528f64ec49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f528f8bc08e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f528f85ed85] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f528f85d890] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f528f85cd78] --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f52cab53253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f52f8f3dac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f52f8fcea84] backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f528f7abed8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f528f78bb51] --- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f528f78b4cb] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f528f8bc2b8] --- /opt/venv/lib/python3.10/

device-perf / N300 WH B0 Set 1 device perf: docker-job/models/demos/vision/generative/stable_diffusion/wormhole/tests/test_perf.py#L439

test_stable_diffusion_device_perf[13.2] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/ttnn_stable_diffusion --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest models/demos/vision/generative/stable_diffusion/wormhole/tests/test_unet_2d_condition_model.py::test_unet_2d_condition_model_512x512[2-4-64-64-device_params=l1_small_size_24576]'"]' returned non-zero exit status 4.

device-perf / N300 WH B0 Set 1 device perf: docker-job/models/demos/vision/generative/stable_diffusion/wormhole/tests/test_unet_2d_condition_model.py#L115

test_unet_2d_condition_model_512x512[2-4-64-64-device_params=l1_small_size_24576] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/kernel/compute/tilize.cpp:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory 10 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7fdaaf00a168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7fdaac86b80c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fdaac8822b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fdaac88223e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fdaac614def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fdb15f20ee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7fdaac614c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7fdaac88208e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7fdaac824d85] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7fdaac823890] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7fdaac822d78] --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fdae7b06253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fdb15f1bac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7fdb15faca84] backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7fdaac771ed8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7fdaac751b51] --- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7fdaac7514cb] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fdaac8822b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7

device-perf / N300 WH B0 Set 1 device perf: docker-job/models/experimental/stable_diffusion_xl_base/tests/test_sdxl_perf.py#L169

test_sdxl_perf_device[test_sdxl_clip_encoder_1] subprocess.CalledProcessError: Command '['python3 -m tracy -p -r -o /work/generated/profiler/sdxl_clip_encoder_1 --check-exit-code --op-support-count 1333 -t 5000 -m \'pytest models/experimental/stable_diffusion_xl_base/tests/pcc/test_sdxl_clip_encoders.py::test_clip_encoder -k \'"\'"\'encoder_1\'"\'"\'\'']' returned non-zero exit status 4.

device-perf / N300 WH B0 Set 1 device perf: docker-job/models/experimental/stable_diffusion_xl_base/tests/pcc/test_sdxl_clip_encoders.py#L155

test_clip_encoder[wormhole_b0-device_params0-n150-encoder_1] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/kernel/compute/tilize.cpp:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory 10 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f38c6d52168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f38c45b380c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f38c45ca2b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f38c45ca23e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f38c435cdef] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f392dc4cee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f38c435cc49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f38c45ca08e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f38c456cd85] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f38c456b890] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f38c456ad78] --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f38ff853253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f392dc47ac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f392dcd8a84] backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f38c44b9ed8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f38c4499b51] --- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f38c44994cb] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f38c45ca2b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f38c45ca23e] --- /op

device-perf / N300 WH B0 Set 1 device perf: docker-job/models/experimental/stable_diffusion_xl_base/tests/test_sdxl_perf.py#L169

test_sdxl_perf_device[test_sdxl_vae_encode] subprocess.CalledProcessError: Command '['python3 -m tracy -p -r -o /work/generated/profiler/sdxl_vae --check-exit-code --op-support-count 1333 -t 5000 -m \'pytest models/experimental/stable_diffusion_xl_base/vae/tests/pcc/test_module_tt_autoencoder_kl.py::test_vae -k \'"\'"\'test_encode\'"\'"\'\'']' returned non-zero exit status 4.

device-perf / N300 WH B0 Set 1 device perf: docker-job/models/experimental/stable_diffusion_xl_base/vae/tests/pcc/test_module_tt_autoencoder_kl.py#L74

test_vae[device_params0-test_encode] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for pack_untilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/untilize.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/operations/sliding_window/halo/device/kernels/compute/pack_untilize.cpp:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory 10 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f5d27f81168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f5d257e280c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f5d257f92b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f5d257f923e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f5d2558bdef] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f5d8ee81ee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f5d2558bc49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f5d257f908e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f5d2579bd85] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f5d2579a890] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f5d25799d78] --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f5d60a88253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f5d8ee7cac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f5d8ef0da84] backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f5d256e8ed8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f5d256c8b51] --- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f5d256c84cb] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f5d257f92b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7d

device-perf / N300 WH B0 Set 1 device perf: docker-job/models/experimental/stable_diffusion_xl_base/tests/test_sdxl_perf.py#L169

test_sdxl_perf_device[test_sdxl_vae_decode] subprocess.CalledProcessError: Command '['python3 -m tracy -p -r -o /work/generated/profiler/sdxl_vae --check-exit-code --op-support-count 1333 -t 5000 -m \'pytest models/experimental/stable_diffusion_xl_base/vae/tests/pcc/test_module_tt_autoencoder_kl.py::test_vae -k \'"\'"\'test_decode\'"\'"\'\'']' returned non-zero exit status 4.

device-perf / N300 WH B0 Set 1 device perf: docker-job/models/experimental/stable_diffusion_xl_base/vae/tests/pcc/test_module_tt_autoencoder_kl.py#L69

test_vae[device_params0-test_decode] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for transpose_xw_tiled TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_unary/eltwise_unary.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/operations/data_movement/permute/device/kernels/compute/transpose_xw_tiled.cpp:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory 10 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7fbdeb44b168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7fbde8cac80c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fbde8cc32b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fbde8cc323e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fbde8a55def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fbe52334ee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7fbde8a55c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7fbde8cc308e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7fbde8c65d85] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7fbde8c64890] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7fbde8c63d78] --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fbe23f38253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fbe5232fac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7fbe523c0a84] backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7fbde8bb2ed8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7fbde8b92b51] --- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7fbde8b924cb] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fbde8cc32b8] --- /opt/venv/lib/python3.10/site-packages/ttnn

device-perf / N300 WH B0 Set 1 device perf: docker-job/models/experimental/stable_diffusion_xl_base/tests/test_sdxl_perf.py#L169

test_sdxl_perf_device[test_sdxl_refiner_unet] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/sdxl_refiner_unet --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest models/experimental/stable_diffusion_xl_base/tests/test_sdxl_perf.py::test_refiner_unet'"]' returned non-zero exit status 4.

device-perf / N300 WH B0 Set 1 device perf: docker-job/models/experimental/stable_diffusion_xl_base/tests/test_sdxl_perf.py#L76

test_refiner_unet[1-device_params0-input_shape0-timestep_shape0-encoder_shape0-temb_shape0-time_ids_shape0] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for transpose_xw_tiled TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_unary/eltwise_unary.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/operations/data_movement/permute/device/kernels/compute/transpose_xw_tiled.cpp:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory 10 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7fcc9b147168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7fcc989a880c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fcc989bf2b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fcc989bf23e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fcc98751def] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fcd0203fee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7fcc98751c49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7fcc989bf08e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7fcc98961d85] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7fcc98960890] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7fcc9895fd78] --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fccd3c53253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fcd0203aac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7fcd020cba84] backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7fcc988aeed8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7fcc9888eb51] --- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7fcc9888e4cb] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7d

device-perf / N300 WH B0 Set 1 device perf: docker-job/models/experimental/stable_diffusion_xl_base/tests/test_sdxl_perf.py#L169

test_sdxl_perf_device[test_sdxl_unet] subprocess.CalledProcessError: Command '["python3 -m tracy -p -r -o /work/generated/profiler/sdxl_unet --check-exit-code --op-support-count 1333 -t 5000 -m 'pytest models/experimental/stable_diffusion_xl_base/tests/test_sdxl_perf.py::test_unet'"]' returned non-zero exit status 4.

device-perf / N300 WH B0 Set 1 device perf: docker-job/models/experimental/stable_diffusion_xl_base/tests/test_sdxl_perf.py#L40

test_unet[1-device_params0-input_shape0-timestep_shape0-encoder_shape0-temb_shape0-time_ids_shape0] RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception info: Failed to generate binaries for transpose_xw_tiled TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception info: trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_unary/eltwise_unary.h:7, from /__w/tt-metal/tt-metal/docker-job/ttnn/cpp/ttnn/operations/data_movement/permute/device/kernels/compute/transpose_xw_tiled.cpp:7, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21, from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13: /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory 10 | #include "vconst_verifier.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f0e61a11168] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f0e5f27280c] --- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f0e5f2892b8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f0e5f28923e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f0e5f01bdef] --- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f0ec890eee8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f0e5f01bc49] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f0e5f28908e] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f0e5f22bd85] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f0e5f22a890] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f0e5f229d78] --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f0e9a506253] --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f0ec8909ac3] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f0ec899aa84] backtrace: --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f0e5f178ed8] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f0e5f158b51] --- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&) --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f0e5f1584cb] --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0

device-perf / N300 WH B0 not yet ported device perf

Benchmark data directory 'generated/benchmark_data' does not exist

device-perf / N300 WH B0 not yet ported device perf: docker-job/models/demos/vision/classification/mnist/reference/mnist.py#L30

Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.

device-perf / N300 WH B0 not yet ported device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / N300 WH B0 not yet ported device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / N300 WH B0 not yet ported device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / N300 WH B0 Set 2 device perf

Benchmark data directory 'generated/benchmark_data' does not exist

device-perf / N300 WH B0 Set 2 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / N300 WH B0 Set 2 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / N300 WH B0 Set 2 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / N300 WH B0 Set 2 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / N300 WH B0 Set 2 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / N300 WH B0 Set 2 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / N300 WH B0 Set 2 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / N300 WH B0 Set 2 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / N300 WH B0 Set 2 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / N300 WH B0 Set 2 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / N300 WH B0 Set 2 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / N300 WH B0 Set 2 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / N300 WH B0 Set 2 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / N300 WH B0 Set 2 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / N300 WH B0 Set 2 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / N300 WH B0 Set 2 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / N300 WH B0 Set 2 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / N300 WH B0 Set 2 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / N300 WH B0 Set 2 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / N300 WH B0 Set 2 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / N300 WH B0 Set 2 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / P150 BH device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / P150 BH device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / P150 BH device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / P150 BH device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / P150 BH device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / P150 BH device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / P150 BH device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / P150 BH device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / P150 BH device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / P150 BH device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / P150 BH device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / P150 BH device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / P150 BH device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / P150 BH device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / P150 BH device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / P150 BH device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / P150 BH device perf: /opt/venv/lib/python3.10/site-packages/torchvision/models/_utils.py#L223

Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG19_Weights.IMAGENET1K_V1`. You can also use `weights=VGG19_Weights.DEFAULT` to get the most up-to-date weights.

device-perf / P150 BH device perf: /opt/venv/lib/python3.10/site-packages/torchvision/models/_utils.py#L208

The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.

device-perf / P150 BH device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / P150 BH device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / P150 BH device perf: docker-job/models/demos/vision/generative/stable_diffusion/wormhole/custom_preprocessing.py#L33

The use of `x.T` on tensors of dimension other than 2 to reverse their shape is deprecated and it will throw an error in a future release. Consider `x.mT` to transpose batches of matrices or `x.permute(*torch.arange(x.ndim - 1, -1, -1))` to reverse the dimensions of a tensor. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4413.)

device-perf / P150 BH device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / P150 BH device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / N300 WH B0 Set 1 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / N300 WH B0 Set 1 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / N300 WH B0 Set 1 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / N300 WH B0 Set 1 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / N300 WH B0 Set 1 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / N300 WH B0 Set 1 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / N300 WH B0 Set 1 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / N300 WH B0 Set 1 device perf: /opt/venv/lib/python3.10/site-packages/torchvision/models/_utils.py#L223

device-perf / N300 WH B0 Set 1 device perf: /opt/venv/lib/python3.10/site-packages/torchvision/models/_utils.py#L208

The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.

device-perf / N300 WH B0 Set 1 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / N300 WH B0 Set 1 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / N300 WH B0 Set 1 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / N300 WH B0 Set 1 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / N300 WH B0 Set 1 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / N300 WH B0 Set 1 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / N300 WH B0 Set 1 device perf: docker-job/models/demos/vision/generative/stable_diffusion/wormhole/custom_preprocessing.py#L33

device-perf / N300 WH B0 Set 1 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / N300 WH B0 Set 1 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / N300 WH B0 Set 1 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / N300 WH B0 Set 1 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / N300 WH B0 Set 1 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / N300 WH B0 Set 1 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

device-perf / N300 WH B0 Set 1 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132

Field "model_name" in OpTest has conflict with protected namespace "model_". You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.

device-perf / N300 WH B0 Set 1 device perf: /opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291

Support for class-based `config` is deprecated, use ConfigDict instead.

Artifacts

Produced during runtime

Name	Size	Digest
test_reports_0428899e-e52f-4c5f-ba8b-1ac2ede1592a	40.7 KB	`sha256:6be37d304d258d828fe892d893535d04f1c90f78098311a6a821cd67ee4014c3`
test_reports_38911a46-df08-4a66-bec2-0fc5ffeaf545	59.3 KB	`sha256:12f93a919141d7c0ae71e1ab2c40dff1f143fe676c15692d2a9d2ef6031823c6`
test_reports_73541511-f979-4b90-844b-5efbc2f9a470	4.89 KB	`sha256:d358d99bf25d28220236b5e849ff70b83aa6a38012c7587b0ce54c8e45376119`
test_reports_7ad79d33-b8a4-45d9-acbd-875df57181e0	71.7 KB	`sha256:49032afe7bfd887ffd21b039ad16245018bb75718fb6f0964bd190add37cdd79`

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(Single-card) Device perf regressions #11108

Summary

(Single-card) Device perf regressions #11108

Uh oh!

perf-device-models.yaml

Annotations

Artifacts