All post-commit tests #52457
fvranicTTall-post-commit-workflows.yaml
on: workflow_dispatch
build-artifact
/
download-artifacts
tests-to-run
4s
build-artifact
/
parse-platform
3s
build-artifact
/
...
/
check-docker-images
22s
build-artifact
/
...
/
🐳️ Build Ubuntu images
build-artifact
/
...
/
🐳️ Build ManyLinux image
build-artifact
/
...
/
🔄 Update latest tag
build-artifact
/
🛠️ Build Release ubuntu 22.04
5m 31s
build-artifact
/
...
/
🐍 Build wheel (Python 3.10)
6m 25s
t3000-apc-fast-tests
/
T3000 fast tests
29m 28s
Matrix: models-unit-tests
Matrix: ops-unit-tests
Matrix: run-profiler-regression
Matrix: tt-train-cpp-unit-tests
Matrix: ttnn-unit-tests
Annotations
51 errors and 10 warnings
|
models-unit-tests (wormhole_b0, N300) / model wormhole_b0 N300:
models/demos/deepseek_v3/tests/unit/test_matmul.py#L368
test_matmul_dram_sharded_mesh_device[device_params0-False-bf16_bf4b_bf16-32x7168x2304_mlp_w1_style-mesh_device0]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for bmm_large_block_zm_fused_bias_activation TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/matmul.h:7,
from /work/ttnn/cpp/ttnn/operations/matmul/device/kernels/compute/bmm_large_block_zm_fused_bias_activation.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f2b99cc7168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f2b9752880c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2b9753f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2b9753f23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2b972d1def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2bde573ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2b972d1c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2b9753f08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f2b974e1d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f2b974e0890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f2b974dfd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f2bdbc73253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f2bde56eac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f2bde5ffa84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f2b9742eed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f2b9740eb51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f2b9740e4cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2b97
|
|
models-unit-tests (wormhole_b0, N300) / model wormhole_b0 N300:
models/demos/deepseek_v3/tests/unit/test_matmul.py#L216
test_matmul_dram_sharded_single_device[bf16_bf4b_bf16-32x7168x4064_lm_head_style]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for bmm_large_block_zm_fused_bias_activation TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/matmul.h:7,
from /work/ttnn/cpp/ttnn/operations/matmul/device/kernels/compute/bmm_large_block_zm_fused_bias_activation.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f2b99cc7168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f2b9752880c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2b9753f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2b9753f23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2b972d1def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2bde573ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2b972d1c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2b9753f08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f2b974e1d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f2b974e0890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f2b974dfd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f2bdbc73253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f2bde56eac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f2bde5ffa84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f2b9742eed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f2b9740eb51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f2b9740e4cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2b9753f2b8]
--- /opt/venv/lib/pyth
|
|
models-unit-tests (wormhole_b0, N300) / model wormhole_b0 N300:
models/demos/deepseek_v3/tests/unit/test_matmul.py#L216
test_matmul_dram_sharded_single_device[bf16_bf4b_bf16-32x256x7168_small_input]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for bmm_large_block_zm_fused_bias_activation TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/matmul.h:7,
from /work/ttnn/cpp/ttnn/operations/matmul/device/kernels/compute/bmm_large_block_zm_fused_bias_activation.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f2b99cc7168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f2b9752880c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2b9753f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2b9753f23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2b972d1def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2bde573ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2b972d1c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2b9753f08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f2b974e1d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f2b974e0890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f2b974dfd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f2bdbc73253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f2bde56eac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f2bde5ffa84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f2b9742eed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f2b9740eb51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f2b9740e4cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2b9753f2b8]
--- /opt/venv/lib/python3
|
|
models-unit-tests (wormhole_b0, N300) / model wormhole_b0 N300:
models/demos/deepseek_v3/tests/unit/test_matmul.py#L216
test_matmul_dram_sharded_single_device[bf16_bf4b_bf16-32x7168x256_small_output]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for bmm_large_block_zm_fused_bias_activation TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/matmul.h:7,
from /work/ttnn/cpp/ttnn/operations/matmul/device/kernels/compute/bmm_large_block_zm_fused_bias_activation.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f2b99cc7168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f2b9752880c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2b9753f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2b9753f23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2b972d1def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2bde573ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2b972d1c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2b9753f08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f2b974e1d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f2b974e0890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f2b974dfd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f2bdbc73253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f2bde56eac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f2bde5ffa84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f2b9742eed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f2b9740eb51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f2b9740e4cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2b9753f2b8]
--- /opt/venv/lib/python
|
|
models-unit-tests (wormhole_b0, N300) / model wormhole_b0 N300:
models/demos/deepseek_v3/tests/unit/test_matmul.py#L216
test_matmul_dram_sharded_single_device[bf16_bf4b_bf16-32x2304x7168_mlp_w2_style]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for bmm_large_block_zm_fused_bias_activation TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/matmul.h:7,
from /work/ttnn/cpp/ttnn/operations/matmul/device/kernels/compute/bmm_large_block_zm_fused_bias_activation.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f2b99cc7168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f2b9752880c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2b9753f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2b9753f23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2b972d1def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2bde573ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2b972d1c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2b9753f08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f2b974e1d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f2b974e0890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f2b974dfd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f2bdbc73253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f2bde56eac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f2bde5ffa84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f2b9742eed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f2b9740eb51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f2b9740e4cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2b9753f2b8]
--- /opt/venv/lib/pytho
|
|
models-unit-tests (wormhole_b0, N300) / model wormhole_b0 N300:
models/demos/deepseek_v3/tests/unit/test_matmul.py#L216
test_matmul_dram_sharded_single_device[bf16_bf4b_bf16-32x7168x2304_mlp_w1_style]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for bmm_large_block_zm_fused_bias_activation TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/matmul.h:7,
from /work/ttnn/cpp/ttnn/operations/matmul/device/kernels/compute/bmm_large_block_zm_fused_bias_activation.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f2b99cc7168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f2b9752880c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2b9753f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2b9753f23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2b972d1def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2bde573ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2b972d1c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2b9753f08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f2b974e1d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f2b974e0890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f2b974dfd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f2bdbc73253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f2bde56eac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f2bde5ffa84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f2b9742eed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f2b9740eb51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f2b9740e4cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2b9753f2b8]
--- /opt/venv/lib/pytho
|
|
models-unit-tests (wormhole_b0, N300) / model wormhole_b0 N300:
models/demos/deepseek_v3/tests/unit/test_embedding.py#L56
test_embedding[False-Layout.TILE-mem_config0-DataType.BFLOAT16-DataType.UINT32-shape_pair1-device_params0]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7,
from /work/ttnn/cpp/ttnn/operations/data_movement/tilize/device/kernels/compute/tilize.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f2b99cc7168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f2b9752880c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2b9753f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2b9753f23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2b972d1def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2bde573ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2b972d1c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2b9753f08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f2b974e1d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f2b974e0890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f2b974dfd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f2bdbc73253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f2bde56eac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f2bde5ffa84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f2b9742eed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f2b9740eb51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f2b9740e4cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2b9753f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/bui
|
|
models-unit-tests (wormhole_b0, N300) / model wormhole_b0 N300:
models/demos/deepseek_v3/tests/unit/test_embedding.py#L56
test_embedding[False-Layout.TILE-mem_config0-DataType.BFLOAT16-DataType.UINT32-shape_pair0-device_params0]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7,
from /work/ttnn/cpp/ttnn/operations/data_movement/tilize/device/kernels/compute/tilize.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f2b99cc7168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f2b9752880c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2b9753f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2b9753f23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2b972d1def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2bde573ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2b972d1c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2b9753f08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f2b974e1d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f2b974e0890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f2b974dfd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f2bdbc73253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f2bde56eac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f2bde5ffa84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f2b9742eed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f2b9740eb51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f2b9740e4cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2b9753f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/bui
|
|
models-unit-tests (wormhole_b0, N300) / model wormhole_b0 N300:
models/demos/deepseek_v3/tests/unit/test_embedding.py#L56
test_embedding[True-Layout.TILE-mem_config0-DataType.BFLOAT16-DataType.UINT32-shape_pair1-device_params0]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7,
from /work/ttnn/cpp/ttnn/operations/data_movement/tilize/device/kernels/compute/tilize.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f2b99cc7168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f2b9752880c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2b9753f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2b9753f23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2b972d1def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2bde573ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2b972d1c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2b9753f08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f2b974e1d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f2b974e0890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f2b974dfd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f2bdbc73253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f2bde56eac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f2bde5ffa84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f2b9742eed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f2b9740eb51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f2b9740e4cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2b9753f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/buil
|
|
models-unit-tests (wormhole_b0, N300) / model wormhole_b0 N300:
models/demos/deepseek_v3/tests/unit/test_embedding.py#L56
test_embedding[True-Layout.TILE-mem_config0-DataType.BFLOAT16-DataType.UINT32-shape_pair0-device_params0]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7,
from /work/ttnn/cpp/ttnn/operations/data_movement/tilize/device/kernels/compute/tilize.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f2b99cc7168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f2b9752880c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2b9753f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2b9753f23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2b972d1def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2bde573ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2b972d1c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2b9753f08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f2b974e1d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f2b974e0890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f2b974dfd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f2bdbc73253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f2bde56eac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f2bde5ffa84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f2b9742eed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f2b9740eb51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f2b9740e4cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2b9753f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/buil
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn pool group wormhole_b0 N300
Process completed with exit code 1.
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn pool group wormhole_b0 N300:
tests/ttnn/unit_tests/operations/pool/test_adaptive_pool2d.py#L66
test_adaptive_pool2d[pool_type=max-dtype=DataType.BFLOAT16-output_size=(1, 1)-input_shape=(1, 512, 28, 28)-device_params={'l1_small_size': 24576}]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for compute_pool_2d TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/pack_untilize.h:7,
from /work/ttnn/cpp/ttnn/operations/pool/generic/device/kernels/compute/compute_pool_2d.cpp:6,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7ff9a3ae3168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7ff9a134480c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7ff9a135b2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7ff9a135b23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7ff9a10eddef]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7ff9e8404ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7ff9a10edc49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7ff9a135b08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7ff9a12fdd85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7ff9a12fc890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7ff9a12fbd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7ff9e5b4d253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7ff9e83ffac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7ff9e8490a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7ff9a124aed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7ff9a122ab51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7ff9a122a4cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7ff9a135b
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn eltwise group 1 wormhole_b0 N300
Process completed with exit code 1.
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn eltwise group 1 wormhole_b0 N300:
tests/ttnn/unit_tests/operations/eltwise/test_activation.py#L34
test_hardtanh[w=128-h=64]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for eltwise_sfpu TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc0 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/llk_lib/llk_math_eltwise_unary_sfpu.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_sfpu/llk_math_eltwise_unary_sfpu_init.h:8,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_sfpu/llk_math_eltwise_unary_sfpu_rpow.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_unary/rpow.h:8,
from /work/ttnn/cpp/ttnn/operations/eltwise/unary/device/kernels/compute//eltwise_sfpu.cpp:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:37,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f45579cb168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f455522c80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f45552432b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f455524323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f4554fd5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f459c270ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f4554fd5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f455524308e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f45551e5d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f45551e4890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f45551e3d78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f45999ea253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f459c26bac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f459c2fca84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f4555132ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f4555112b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f45551124cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f45552432b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f455524323e]
--- /opt/venv/lib/python3.10/site-packa
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn matmul group wormhole_b0 N300
Process completed with exit code 1.
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn matmul group wormhole_b0 N300:
tests/ttnn/unit_tests/operations/matmul/test_addmm.py#L44
test_addmm_square_matrices[matrix_size=4-dtype=DataType.BFLOAT16]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for bmm_large_block_zm_fused_bias_activation TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/matmul.h:7,
from /work/ttnn/cpp/ttnn/operations/matmul/device/kernels/compute/bmm_large_block_zm_fused_bias_activation.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f7dbfa3f168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f7dbd2a080c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f7dbd2b72b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f7dbd2b723e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f7dbd049def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f7e042f4ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f7dbd049c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f7dbd2b708e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f7dbd259d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f7dbd258890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f7dbd257d78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f7e01a3d253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f7e042efac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f7e04380a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f7dbd1a6ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f7dbd186b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f7dbd1864cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f7dbd2b72b8]
--- /opt/venv/lib/python3.10/site-pack
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn eltwise group 4 wormhole_b0 N300
Process completed with exit code 1.
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn eltwise group 4 wormhole_b0 N300:
tests/ttnn/unit_tests/operations/eltwise/test_eltwise_typecast.py#L241
test_typecast_bfp8_b_to_bf16
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for eltwise_typecast TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /work/ttnn/cpp/ttnn/operations/copy/typecast/device/kernels/compute/eltwise_typecast.cpp:5,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7fd47ebc3168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7fd47c42480c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fd47c43b2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fd47c43b23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fd47c1cddef]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fd4c34c5ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7fd47c1cdc49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7fd47c43b08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7fd47c3ddd85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7fd47c3dc890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7fd47c3dbd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fd4c0c0e253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fd4c34c0ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7fd4c3551a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7fd47c32aed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7fd47c30ab51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7fd47c30a4cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fd47c43b2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fd47c43b23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fd47c1cddef]
--- /lib/x86_64-li
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn fused group wormhole_b0 N300
Process completed with exit code 1.
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn fused group wormhole_b0 N300:
tests/ttnn/unit_tests/operations/fused/test_batch_norm.py#L72
test_batch_norm_tests[testing_dtype=float32-momentum=0.0-eps=1.0-bias=True-training=True-weight=True-check_mean=False-check_var=False-input_shapes=torch.Size([5, 8, 32, 32])]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for reduce_w TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/matmul.h:7,
from /work/ttnn/cpp/ttnn/operations/reduction/generic/device/kernels/compute/reduce_w.cpp:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7ff5c5ed2168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7ff5c373380c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7ff5c374a2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7ff5c374a23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7ff5c34dcdef]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7ff60a680ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7ff5c34dcc49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7ff5c374a08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7ff5c36ecd85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7ff5c36eb890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7ff5c36ead78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7ff607dc9253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7ff60a67bac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7ff60a70ca84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7ff5c3639ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7ff5c3619b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7ff5c36194cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8)
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn eltwise group 2 wormhole_b0 N300
Process completed with exit code 1.
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn eltwise group 2 wormhole_b0 N300:
tests/ttnn/unit_tests/operations/eltwise/test_binary_int32.py#L96
test_binary_int32[use_legacy=True-ttnn_op=ttnn.subtract-low_a=-2147483647-high_a=-2000000000.0-low_b=-2077000000-high_b=-2000000000.0-input_shapes=torch.Size([1, 1, 320, 384])]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for eltwise_binary_sfpu_kernel TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_binary.h:7,
from /work/ttnn/cpp/ttnn/operations/eltwise/binary/device/kernels/compute/eltwise_binary_sfpu_kernel.cpp:6,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7fb5ae7c3168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7fb5ac02480c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fb5ac03b2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fb5ac03b23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fb5abdcddef]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fb5f3116ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7fb5abdcdc49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7fb5ac03b08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7fb5abfddd85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7fb5abfdc890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7fb5abfdbd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fb5f0890253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fb5f3111ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7fb5f31a2a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7fb5abf2aed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7fb5abf0ab51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7fb5abf0a4cb]
--- /opt/venv/lib/python3.10/site-package
|
|
ttnn-unit-tests (wormhole_b0, N300) / core ttnn unit test group wormhole_b0 N300
Process completed with exit code 1.
|
|
ttnn-unit-tests (wormhole_b0, N300) / core ttnn unit test group wormhole_b0 N300:
tests/ttnn/unit_tests/base_functionality/test_async.py#L23
test_add_1D_tensor_and_scalar[size=64-scalar=3]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for eltwise_binary_scalar TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_binary.h:7,
from /work/ttnn/cpp/ttnn/operations/eltwise/binary_ng/device/kernels/compute/eltwise_binary_scalar.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f0e140d2168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f0e1193380c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f0e1194a2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f0e1194a23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f0e116dcdef]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f0e5888cee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f0e116dcc49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f0e1194a08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f0e118ecd85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f0e118eb890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f0e118ead78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f0e55fd5253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f0e58887ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f0e58918a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f0e11839ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f0e11819b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f0e118194cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f0e1194a2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn eltwise group 5 wormhole_b0 N300
Process completed with exit code 1.
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn eltwise group 5 wormhole_b0 N300:
tests/ttnn/unit_tests/operations/eltwise/test_ternary_sharding.py#L1622
test_where_ttt_identical_mixed_strategy[false_sharded=False-true_sharded=True-predicate_sharded=True-out_strategy=height-config_permutation=hbw-dtype_pt=torch.bfloat16-dtype_tt=DataType.BFLOAT16]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for ternary_sfpu_no_bcast_ttt TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/ttnn/cpp/ttnn/operations/eltwise/binary_ng/device/kernels/compute/eltwise_utils_sfpu.hpp:7,
from /work/ttnn/cpp/ttnn/operations/eltwise/ternary/device/kernels/compute/ternary_sfpu_no_bcast_ttt.cpp:8,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7fefefee3168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7fefed74480c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fefed75b2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fefed75b23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fefed4eddef]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7ff0347ebee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7fefed4edc49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7fefed75b08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7fefed6fdd85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7fefed6fc890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7fefed6fbd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7ff031f65253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7ff0347e6ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7ff034877a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7fefed64aed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7fefed62ab51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4c
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn conv group wormhole_b0 N300
Process completed with exit code 1.
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn conv group wormhole_b0 N300:
tests/ttnn/unit_tests/operations/conv/data_movement/test_convert_to_chw.py#L41
test_convert_to_chw[HW=32-core_grid=ttnn.CoreGrid(x=1, y=1)-C=1-input_data_type=DataType.BFLOAT16]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for convert_to_chw TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/pack_untilize.h:7,
from /work/ttnn/cpp/ttnn/operations/experimental/cnn/convert_to_chw/device/kernels/convert_to_chw.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f2d9798e168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f2d951ef80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2d952062b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2d9520623e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2d94f98def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2ddc1a5ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2d94f98c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2d9520608e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f2d951a8d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f2d951a7890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f2d951a6d78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f2dd98ee253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f2ddc1a0ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f2ddc231a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f2d950f5ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f2d950d5b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f2d950d54cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2d952062b8]
--- /opt/venv/lib/python3.10/site
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn transformers group wormhole_b0 N300
Process completed with exit code 1.
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn transformers group wormhole_b0 N300:
tests/ttnn/unit_tests/operations/transformers/test_paged_cache_mask.py#L149
test_update_cache[silicon_arch_name=wormhole_b0-dtype=DataType.BFLOAT16-cache_shape=(32, 1, 32, 128)-1x2_grid]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for update_cache TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /work/ttnn/cpp/ttnn/operations/experimental/paged_cache/device/kernels/compute/update_cache.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f6f3caae168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f6f3a30f80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f6f3a3262b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f6f3a32623e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f6f3a0b8def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f6f81263ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f6f3a0b8c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f6f3a32608e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f6f3a2c8d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f6f3a2c7890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f6f3a2c6d78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f6f7e9ac253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f6f8125eac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f6f812efa84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f6f3a215ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f6f3a1f5b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f6f3a1f54cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f6f3a3262b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f6f3a32623e]
--- /opt/venv/lib/python3.10/site-p
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn sdpa group wormhole_b0 N300
Process completed with exit code 1.
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn sdpa group wormhole_b0 N300:
tests/ttnn/unit_tests/operations/sdpa/test_mla_decode.py#L60
test_flash_mla_decode[reuse_k=True-block_size=64-use_paged_attention=True-q_dtype=DataType.BFLOAT16-dtype=DataType.BFLOAT8_B-batch=4-seq_len=1024-nh=128-nkv=1-kv_lora_rank=512-d_rope=64-q_num_cores=64]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for sdpa_flash_decode TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/compute_kernel_api.h:20,
from /work/ttnn/cpp/ttnn/operations/transformer/sdpa_decode/device/kernels/compute/sdpa_flash_decode.cpp:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7fc27e2e3168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7fc27bb4480c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fc27bb5b2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fc27bb5b23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fc27b8eddef]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fc2c2c51ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7fc27b8edc49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7fc27bb5b08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7fc27bafdd85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7fc27bafc890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7fc27bafbd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fc2c039a253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fc2c2c4cac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7fc2c2cdda84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7fc27ba4aed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7fc27ba2ab51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7fc27ba2a4cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fc27bb5b2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fc27bb5b23e]
--- /opt/venv/lib/python3.10/site-pa
|
|
ops-unit-tests (wormhole_b0, N300) / wormhole_b0 sdxl op tests wormhole_b0 N300
Process completed with exit code 1.
|
|
ops-unit-tests (wormhole_b0, N300) / wormhole_b0 sdxl op tests wormhole_b0 N300:
models/experimental/stable_diffusion_xl_base/tests/test_sdxl_op_unit_test_perf.py#L126
test_dram_group_norm_vae_welford_reciprocal_performance
subprocess.CalledProcessError: Command '['python3 -m tracy -p -r -o /work/generated/profiler/dram_group_norm_vae_welford_reciprocal_perf --check-exit-code -a device_kernel_duration --op-support-count 1333 -t 5000 -m \'pytest "models/experimental/stable_diffusion_xl_base/tests/test_sdxl_op_unit_test_perf.py::test_dram_group_norm_welford_reciprocal_vae" -v\'']' returned non-zero exit status 4.
|
|
ops-unit-tests (wormhole_b0, N300) / wormhole_b0 sdxl op tests wormhole_b0 N300:
models/experimental/stable_diffusion_xl_base/tests/test_sdxl_op_unit_test_perf.py#L18
test_dram_group_norm_welford_reciprocal_vae[device_params0]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7,
from /work/ttnn/cpp/ttnn/kernel/compute/tilize.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f124a283168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f1247ae480c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f1247afb2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f1247afb23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f124788ddef]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f12b1178ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f124788dc49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f1247afb08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f1247a9dd85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f1247a9c890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f1247a9bd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f1282d77253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f12b1173ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f12b1204a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f12479eaed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f12479cab51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f12479ca4cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f1247afb2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f1247afb23e]
--- /opt/venv/lib/python3.10/site-pa
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn reduce and misc ops group wormhole_b0 N300
Process completed with exit code 1.
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn reduce and misc ops group wormhole_b0 N300:
tests/ttnn/unit_tests/operations/reduce/test_cumprod.py#L56
test_cumprod_normal[dtypes=(torch.float32, None)-shape=[2]-dim=0]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for transpose_xw_tiled TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_unary/eltwise_unary.h:7,
from /work/ttnn/cpp/ttnn/operations/data_movement/permute/device/kernels/compute/transpose_xw_tiled.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f0883ac7168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f088132880c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f088133f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f088133f23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f08810d1def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f08c8387ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f08810d1c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f088133f08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f08812e1d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f08812e0890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f08812dfd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f08c5b01253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f08c8382ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f08c8413a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f088122eed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f088120eb51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f088120e4cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f088133f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttn
|
|
run-profiler-regression (wormhole_b0, N300) / profiler-regression:
tests/tt_metal/tools/profiler/test_device_profiler.py#L1149
test_timestamped_events
assert 34304 == 0
|
|
run-profiler-regression (wormhole_b0, N300) / profiler-regression:
tests/tt_metal/tools/profiler/test_device_profiler.py#L1098
test_profiler_host_device_sync
assert 256 == 0
|
|
run-profiler-regression (wormhole_b0, N300) / profiler-regression:
tests/ttnn/tracy/test_profiler_sync.py#L23
test_with_ops
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for bmm_large_block_zm_fused_bias_activation TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/matmul.h:7,
from /work/ttnn/cpp/ttnn/operations/matmul/device/kernels/compute/bmm_large_block_zm_fused_bias_activation.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f3beaaae168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f3be830f80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f3be83262b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f3be832623e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f3be80b8def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f3c2f24fee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f3be80b8c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f3be832608e]
--- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&)
--- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&)
--- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const
--- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f3be83262b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f3be832623e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f3be80b8def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f3c2f24fee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f3be80b8c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f3be832608e]
--- /opt/venv/lib/py
|
|
run-profiler-regression (wormhole_b0, N300) / profiler-regression:
tests/tt_metal/tools/profiler/test_device_profiler.py#L967
test_dispatch_cores
assert 34304 == 0
|
|
run-profiler-regression (wormhole_b0, N300) / profiler-regression:
tests/tt_metal/tools/profiler/test_device_profiler.py#L800
test_device_trace_run
assert 256 == 0
|
|
run-profiler-regression (wormhole_b0, N300) / profiler-regression:
tests/ttnn/tracy/test_trace_runs.py#L30
test_with_ops[device_params={'trace_region_size': 1996800, 'dispatch_core_type': DispatchCoreType.WORKER}]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7,
from /work/ttnn/cpp/ttnn/kernel/compute/tilize.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f8df8efe168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f8df675f80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f8df67762b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f8df677623e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f8df6508def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f8e3d6a3ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f8df6508c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f8df677608e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f8df6718d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f8df6717890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f8df6716d78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f8e3ae11253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f8e3d69eac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f8e3d72fa84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f8df6665ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f8df6645b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f8df66454cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f8df67762b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f8d
|
|
|
|
run-profiler-regression (wormhole_b0, N300) / profiler-regression:
tests/ttnn/tracy/test_trace_runs.py#L103
test_with_ops_multiple_trace_ids[device_params={'trace_region_size': 1996800, 'dispatch_core_type': DispatchCoreType.WORKER}]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7,
from /work/ttnn/cpp/ttnn/kernel/compute/tilize.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7fbf2b08c168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7fbf288ed80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fbf289042b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fbf2890423e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fbf28696def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fbf6f92cee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7fbf28696c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7fbf2890408e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7fbf288a6d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7fbf288a5890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7fbf288a4d78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fbf6d09a253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fbf6f927ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7fbf6f9b8a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7fbf287f3ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7fbf287d3b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7fbf287d34cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fbf289042b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so
|
|
run-profiler-regression (wormhole_b0, N300) / profiler-regression:
tests/tt_metal/tools/profiler/test_device_profiler.py#L270
test_custom_cycle_count
assert 34304 == 0
|
|
run-profiler-regression (wormhole_b0, N300) / profiler-regression:
tests/tt_metal/tools/profiler/test_device_profiler.py#L248
test_custom_cycle_count_slow_dispatch
assert 34304 == 0
|
|
tt-train-cpp-unit-tests (wormhole_b0, N300) / tt-train wormhole_b0 N300
Process completed with exit code 8.
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn eltwise group 3 wormhole_b0 N300
Process completed with exit code 1.
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn eltwise group 3 wormhole_b0 N300:
tests/ttnn/unit_tests/operations/eltwise/test_binary_ng_typecast.py#L989
test_edgecase_dims_eltwise_broadcast_logical[layout=Layout.TILE-ttnn_fn=gt-memory_config=MemoryConfig(memory_layout=TensorMemoryLayout::INTERLEAVED,buffer_type=BufferType::DRAM,shard_spec=std::nullopt,nd_shard_spec=std::nullopt,created_with_nd_shard_spec=0)-input_shapes=((1, 7, 1, 1), (7, 7, 33, 33))]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for eltwise_binary TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_exp.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/cmath_common.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_unary/comp.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_unary/sfpu_split_includes.h:148,
from /work/ttnn/cpp/ttnn/operations/eltwise/binary_ng/device/kernels/compute/eltwise_binary.cpp:6,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/wormhole_b0/metal/common/chlkc_list.h:21,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h:10:10: fatal error: vconst_verifier.h: No such file or directory
10 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7fc9c92e3168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7fc9c6b4480c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fc9c6b5b2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fc9c6b5b23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fc9c68eddef]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fca0dc4dee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7fc9c68edc49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7fc9c6b5b08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7fc9c6afdd85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7fc9c6afc890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7fc9c6afbd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fca0b396253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fca0dc48ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7fca0dcd9a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7fc9c6a4aed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7fc9c6a2ab51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /
|
|
t3000-apc-fast-tests / T3000 fast tests
The action 'Run t3k ttnn multiprocess tests' has timed out after 15 minutes.
|
|
models-unit-tests (wormhole_b0, N300) / model wormhole_b0 N300
No files were found with the provided path: /work/generated/test_reports/. No artifacts will be uploaded.
|
|
models-unit-tests (wormhole_b0, N300) / model wormhole_b0 N300:
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn transformers group wormhole_b0 N300:
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
|
ttnn-unit-tests (wormhole_b0, N300) / ttnn sdpa group wormhole_b0 N300:
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
|
ops-unit-tests (wormhole_b0, N300) / wormhole_b0 sdxl op tests wormhole_b0 N300:
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132
Field "model_name" in OpTest has conflict with protected namespace "model_".
You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
|
|
ops-unit-tests (wormhole_b0, N300) / wormhole_b0 sdxl op tests wormhole_b0 N300:
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
|
ops-unit-tests (wormhole_b0, N300) / wormhole_b0 sdxl op tests wormhole_b0 N300:
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
|
ops-unit-tests (wormhole_b0, N300) / wormhole_b0 sdxl op tests wormhole_b0 N300:
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132
Field "model_name" in OpTest has conflict with protected namespace "model_".
You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
|
|
ops-unit-tests (wormhole_b0, N300) / wormhole_b0 sdxl op tests wormhole_b0 N300:
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
|
ops-unit-tests (wormhole_b0, N300) / wormhole_b0 sdxl op tests wormhole_b0 N300:
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
Artifacts
Produced during runtime
| Name | Size | Digest | |
|---|---|---|---|
|
test_reports_14488493-0f21-490b-938c-db2082f94ef2
|
2.42 KB |
sha256:3b04b33fdc9670fa463681d17a95e3b4a9d66c27a303cc53718b571f3f10c600
|
|
|
test_reports_1d067229-01aa-4f31-8409-21c0fac574c3
|
3.95 KB |
sha256:f10da4e8904725fb17562bcbbd3dcb24d117eed53d4725f84e485779dada8442
|
|
|
test_reports_1ec45287-6ac9-447c-9090-c43ff03490d3
|
3.61 KB |
sha256:4e63a8b52960037a205c4b59f56ca7047963413ec7f80893a0bb40db9ff41f9f
|
|
|
test_reports_495bc73b-0924-4fd8-8218-1c4881113436
|
2.63 KB |
sha256:c7b639d8cdf300dc8ca973af97ff9b526474d5519358d9c765eeff5abbb557de
|
|
|
test_reports_4de5edf3-ad3b-4b85-b7ac-155201b88257
|
4.48 KB |
sha256:25aeed1779989b6dad9e9e4e990050dcba3a21eb1fdfa33d57e27bc79d8f19bb
|
|
|
test_reports_59037ddd-5dcd-48db-8465-79d52a7fef01
|
3.71 KB |
sha256:a69a088544e56b3177633e580225a64beb67543e9a6402eda061866f48f88eaf
|
|
|
test_reports_5a1534b0-41a4-430a-b443-d175e8b0379d
|
2.98 KB |
sha256:98d298a3595b6c196ac499d425410c8b9b72d819e893610d46fa2d048d80025a
|
|
|
test_reports_5a518251-7429-434d-a449-55ff072bd174
|
3.31 KB |
sha256:07e0449fe717c5a961ee5430e698a92895f86acd1f7f3f31932f984a38a358ab
|
|
|
test_reports_5c72ddea-6984-463e-a163-17a56c42443a
|
2.82 KB |
sha256:5ef973bdbe365e553f93c0c6be13844d6e82a906591c936ab565ef09cb80f7a4
|
|
|
test_reports_63a78bd4-b651-4c67-a397-bd95d3871bf4
|
12.4 KB |
sha256:d21f7bf4f56e10800cf69258964803ce044dcb9c8a8b5a9e601c916024b4ffd6
|
|
|
test_reports_732b64bc-697f-405a-a137-41dc09ee5269
|
2.61 KB |
sha256:7f87f99e8a79ffaade3e989069af7d0660ea1d5cae23d2c15340979c84f98dc0
|
|
|
test_reports_8c53ed80-181d-46b7-a3ee-7610659e268d
|
2.64 KB |
sha256:157e1a97ac3fa7bfb1a05f811a4e4826f1c93c165fee9b865e90a8f7880f8230
|
|
|
test_reports_aba54658-28f0-4e42-a9ee-b93c09889c1e
|
2.32 KB |
sha256:aa382b36bd5c508cc1b6a1a69d3f9080fe7be221f36337ab97cde2a32337bdbc
|
|
|
test_reports_b03f483c-97f3-4eb0-be7f-7944e85acf29
|
3.32 KB |
sha256:440bdb841e955f7f61e28f75a1ed8671f09365400ff90fead4056fd98a86a5c6
|
|
|
test_reports_b361c4d1-4f71-4641-bfd6-98aeef6fdbf8
|
3.57 KB |
sha256:26c0b21eee92114ab62328aec3f8a1459b700f3ce416f8d76d98d0a885da95e7
|
|
|
test_reports_d1315eaa-adc4-479b-99ee-0f8faf8d5b9a
|
203 KB |
sha256:73d58bd427d81b9ba2994917f2d4b1e3fdad90e00b7a9c3034417ff8bb0b60dd
|
|
|
test_reports_dd596565-c89b-4163-9751-9290efbea7b9
|
21.5 KB |
sha256:4e6fc825f173ef2d70e841569a17e4afd39c02389057c793ac0e7c3acc7b7d32
|
|