Blackhole post-commit tests #14975
fvranicTTblackhole-post-commit.yaml
on: workflow_dispatch
generate-matrix
3s
build-artifact
/
download-artifacts
build-artifact
/
parse-platform
6s
build-artifact
/
...
/
check-docker-images
14s
build-artifact
/
...
/
🐳️ Build Ubuntu images
build-artifact
/
...
/
🐳️ Build ManyLinux image
build-artifact
/
...
/
🔄 Update latest tag
build-artifact
/
🛠️ Build Release ubuntu 22.04
6m 54s
build-artifact
/
...
/
🐍 Build wheel (Python 3.10)
5m 41s
Matrix: blackhole-multi-card-fast-unit-tests
Matrix: metalium-basic-tests
Matrix: metalium-smoke-tests
Matrix: models-unit-tests
Matrix: ops-unit-tests
Matrix: run-profiler-regression
Matrix: tt-cnn-unit-tests
Matrix: ttnn-basic-tests
Matrix: ttnn-smoke-tests
Matrix: ttnn-unit-tests
Matrix: umd-unit-tests
Annotations
105 errors and 29 warnings
|
models-unit-tests (P150) / model blackhole P150
Process completed with exit code 1.
|
|
models-unit-tests (P150) / model blackhole P150:
models/demos/vision/generative/stable_diffusion/blackhole/tests/vae/test_vae_upsample.py#L66
test_upsample[256-256-256-256-512-512-2-device_params0]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for untilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/untilize.h:7,
from /work/ttnn/cpp/ttnn/operations/data_movement/untilize/device/kernels/compute/untilize.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7faac83c7168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7faac5c2880c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7faac5c3f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7faac5c3f23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7faac59d1def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fab02d88ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7faac59d1c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7faac5c3f08e]
--- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&)
--- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&)
--- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const
--- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7faac5c3f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7faac5c3f23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7faac59d1def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fab02d88ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7faac59d1c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7faac5c3f08e]
--- /opt/venv/lib/python3.10/site-pack
|
|
models-unit-tests (P150) / model blackhole P150:
models/demos/vision/generative/stable_diffusion/blackhole/tests/vae/test_vae_upsample.py#L66
test_upsample[512-128-128-512-256-256-1-device_params0]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for untilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/untilize.h:7,
from /work/ttnn/cpp/ttnn/operations/data_movement/untilize/device/kernels/compute/untilize.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7faac83c7168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7faac5c2880c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7faac5c3f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7faac5c3f23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7faac59d1def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fab02d88ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7faac59d1c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7faac5c3f08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7faac5be1d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7faac5be0890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7faac5bdfd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fab004d1253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fab02d83ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7fab02e14a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7faac5b2eed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7faac5b0eb51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7faac5b0e4cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7faac5c3f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7faac5c3f23e]
--- /o
|
|
models-unit-tests (P150) / model blackhole P150:
models/demos/vision/generative/stable_diffusion/blackhole/tests/vae/test_vae_upsample.py#L66
test_upsample[512-64-64-512-128-128-0-device_params0]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for untilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/untilize.h:7,
from /work/ttnn/cpp/ttnn/operations/data_movement/untilize/device/kernels/compute/untilize.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7faac83c7168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7faac5c2880c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7faac5c3f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7faac5c3f23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7faac59d1def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fab02d88ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7faac59d1c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7faac5c3f08e]
--- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&)
--- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&)
--- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const
--- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7faac5c3f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7faac5c3f23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7faac59d1def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fab02d88ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7faac59d1c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7faac5c3f08e]
--- auto tt::tt_metal::detail::async<std
|
|
models-unit-tests (P150) / model blackhole P150:
models/demos/vision/generative/stable_diffusion/blackhole/tests/vae/test_vae_attention.py#L45
test_vae_attention[input_shape0]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for eltwise_typecast TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /work/ttnn/cpp/ttnn/operations/copy/typecast/device/kernels/compute/eltwise_typecast.cpp:5,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7faac83c7168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7faac5c2880c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7faac5c3f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7faac5c3f23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7faac59d1def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fab02d88ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7faac59d1c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7faac5c3f08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7faac5be1d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7faac5be0890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7faac5bdfd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fab004d1253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fab02d83ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7fab02e14a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7faac5b2eed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7faac5b0eb51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7faac5b0e4cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7faac5c3f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7faac5c3f23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7faac59d1def]
--- /lib/x86_64-linux-gnu/li
|
|
models-unit-tests (P150) / model blackhole P150:
models/demos/vision/generative/stable_diffusion/blackhole/tests/test_unet_2d_condition_model.py#L115
test_unet_2d_condition_model_512x512[2-4-64-64-device_params=l1_small_size_24576]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7,
from /work/ttnn/cpp/ttnn/kernel/compute/tilize.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7faac83c7168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7faac5c2880c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7faac5c3f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7faac5c3f23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7faac59d1def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fab02d88ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7faac59d1c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7faac5c3f08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7faac5be1d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7faac5be0890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7faac5bdfd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fab004d1253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fab02d83ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7fab02e14a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7faac5b2eed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7faac5b0eb51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7faac5b0e4cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7faac5c3f2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7faac5c3f23e]
--- /opt/venv/lib/python3.10
|
|
blackhole-multi-card-fast-unit-tests (P300-viommu) / blackhole P300-viommu ttnn ops fast unit tests (ccl, DRAM prefetcher)
Process completed with exit code 139.
|
|
blackhole-multi-card-fast-unit-tests (P300-viommu) / blackhole P300-viommu ttnn ops fast unit tests (ccl, DRAM prefetcher):
tests/ttnn/unit_tests/operations/ccl/blackhole_CI/box/all_post_commit/test_reduce_scatter_apc.py#L100
test_rs_row_2d[silicon_arch_name=blackhole-silicon_arch_blackhole=True-num_buffers_per_channel=8-num_workers_per_link=2-chunks_per_sync=2-fabric_linear-trace-dram_only-bfloat_8-2_device-2_links]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for line_reduction TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_binary.h:7,
from /work/ttnn/cpp/ttnn/operations/experimental/ccl/reduce_scatter_minimal_async/device/kernels/line_reduction.cpp:6,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f0a344e3168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f0a31d4480c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f0a31d5b2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f0a31d5b23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f0a31aeddef]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f0a78df4ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f0a31aedc49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f0a31d5b08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f0a31cfdd85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f0a31cfc890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f0a31cfbd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f0a76330253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f0a78defac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f0a78e80a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f0a31c4aed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f0a31c2ab51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f0a31c2a4cb]
--- /opt/venv/lib/python3.10/site-pack
|
|
blackhole-multi-card-fast-unit-tests (P300-viommu) / blackhole P300-viommu ttnn ops fast unit tests (ccl, DRAM prefetcher):
tests/ttnn/unit_tests/operations/ccl/blackhole_CI/box/all_post_commit/test_reduce_scatter_apc.py#L100
test_rs_row_2d[silicon_arch_name=blackhole-silicon_arch_blackhole=True-num_buffers_per_channel=8-num_workers_per_link=2-chunks_per_sync=2-fabric_linear-trace-dram_only-float_16-2_device-2_links]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for line_reduction TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_binary.h:7,
from /work/ttnn/cpp/ttnn/operations/experimental/ccl/reduce_scatter_minimal_async/device/kernels/line_reduction.cpp:6,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f0a344e3168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f0a31d4480c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f0a31d5b2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f0a31d5b23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f0a31aeddef]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f0a78df4ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f0a31aedc49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f0a31d5b08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f0a31cfdd85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f0a31cfc890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f0a31cfbd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f0a76330253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f0a78defac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f0a78e80a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f0a31c4aed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f0a31c2ab51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f0a31c2a4cb]
--- /opt/venv/lib/python3.10/site-pack
|
|
ttnn-unit-tests (P150b) / ttnn eltwise group 3 blackhole P150b
Process completed with exit code 1.
|
|
ttnn-unit-tests (P150b) / ttnn eltwise group 3 blackhole P150b:
tests/ttnn/unit_tests/operations/eltwise/test_binary_ng_typecast.py#L989
test_edgecase_dims_eltwise_broadcast_logical[layout=Layout.TILE-ttnn_fn=gt-memory_config=MemoryConfig(memory_layout=TensorMemoryLayout::INTERLEAVED,buffer_type=BufferType::DRAM,shard_spec=std::nullopt,nd_shard_spec=std::nullopt,created_with_nd_shard_spec=0)-input_shapes=((1, 7, 1, 1), (7, 7, 33, 33))]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for eltwise_binary TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_unary/comp.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_unary/sfpu_split_includes.h:148,
from /work/ttnn/cpp/ttnn/operations/eltwise/binary_ng/device/kernels/compute/eltwise_binary.cpp:6,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f7e3ac24168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f7e3848580c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f7e3849c2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f7e3849c23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f7e3822edef]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f7e6b4d7ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f7e3822ec49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f7e3849c08e]
--- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&)
--- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&)
--- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const
--- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f7e3849c2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f7e3849c23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f7e3822edef]
--- /lib/x86_
|
|
metalium-basic-tests (P150b-viommu) / metalium-basic
The process '/usr/bin/git' failed with exit code 128
|
|
metalium-basic-tests (P150b-viommu) / metalium-basic
Process completed with exit code 139.
|
|
ttnn-smoke-tests (P150b-viommu) / smoke
No test report files were found
|
|
ttnn-smoke-tests (P150b-viommu) / smoke
Process completed with exit code 139.
|
|
ttnn-unit-tests (P150b) / ttnn sdpa group blackhole P150b
Process completed with exit code 1.
|
|
ttnn-unit-tests (P150b) / ttnn sdpa group blackhole P150b:
tests/ttnn/unit_tests/operations/sdpa/test_mla_decode.py#L60
test_flash_mla_decode[reuse_k=True-block_size=64-use_paged_attention=True-q_dtype=DataType.BFLOAT16-dtype=DataType.BFLOAT8_B-batch=4-seq_len=1024-nh=128-nkv=1-kv_lora_rank=512-d_rope=64-q_num_cores=64]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for sdpa_flash_decode TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/compute_kernel_api.h:20,
from /work/ttnn/cpp/ttnn/operations/transformer/sdpa_decode/device/kernels/compute/sdpa_flash_decode.cpp:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f9a292eb168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f9a26b4c80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f9a26b632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f9a26b6323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f9a268f5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f9a59befee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f9a268f5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f9a26b6308e]
--- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&)
--- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&)
--- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const
--- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f9a26b632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f9a26b6323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f9a268f5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f9a59befee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f9a268f5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f9a26b6308e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85
|
|
ttnn-unit-tests (P150b) / ttnn fused group blackhole P150b
Process completed with exit code 1.
|
|
ttnn-unit-tests (P150b) / ttnn fused group blackhole P150b:
tests/ttnn/unit_tests/operations/fused/test_batch_norm.py#L72
test_batch_norm_tests[testing_dtype=float32-momentum=0.0-eps=1.0-bias=True-training=True-weight=True-check_mean=False-check_var=False-input_shapes=torch.Size([5, 8, 32, 32])]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for reduce_w TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/matmul.h:7,
from /work/ttnn/cpp/ttnn/operations/reduction/generic/device/kernels/compute/reduce_w.cpp:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f1585a09168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f158326a80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f15832812b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f158328123e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f1583013def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f15b62afee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f1583013c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f158328108e]
--- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&)
--- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&)
--- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const
--- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f15832812b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f158328123e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f1583013def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f15b62afee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f1583013c49]
--- /opt/venv/lib/python3.1
|
|
blackhole-multi-card-fast-unit-tests (BH-LLMBox) / blackhole BH-LLMBox ttnn ops fast unit tests (ccl, DRAM prefetcher)
Process completed with exit code 139.
|
|
blackhole-multi-card-fast-unit-tests (BH-LLMBox) / blackhole BH-LLMBox ttnn ops fast unit tests (ccl, DRAM prefetcher):
tests/ttnn/unit_tests/operations/ccl/blackhole_CI/box/all_post_commit/test_reduce_scatter_apc.py#L100
test_rs_row_2d[silicon_arch_name=blackhole-silicon_arch_blackhole=True-num_buffers_per_channel=8-num_workers_per_link=2-chunks_per_sync=2-fabric_linear-trace-dram_only-float_16-4_device-2_links]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for line_reduction TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_binary.h:7,
from /work/ttnn/cpp/ttnn/operations/experimental/ccl/reduce_scatter_minimal_async/device/kernels/line_reduction.cpp:6,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f0ea19b5168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f0e9f21680c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f0e9f22d2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f0e9f22d23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f0e9efbfdef]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f0f09307ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f0e9efbfc49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f0e9f22d08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f0e9f1cfd85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f0e9f1ce890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f0e9f1cdd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f0f06a7b253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f0f09302ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f0f09393a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f0e9f11ced8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f0e9f0fcb51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f0e9f0fc4cb]
--- /opt/venv/lib/python3.10/site-pack
|
|
blackhole-multi-card-fast-unit-tests (BH-LLMBox) / blackhole BH-LLMBox ttnn ops fast unit tests (ccl, DRAM prefetcher):
tests/ttnn/unit_tests/operations/ccl/blackhole_CI/box/all_post_commit/test_new_all_reduce.py#L431
test_all_reduce_loopback[silicon_arch_name=blackhole-silicon_arch_blackhole=True-device_params={'trace_region_size': 23887872, 'dispatch_core_axis': DispatchCoreAxis.COL, 'fabric_config': FabricConfig.FABRIC_1D}-trace_mode=True-num_iters=100-warmup_iters=10-input_dtype=DataType.BFLOAT8_B-output_shape=[1, 1, 32, 3584]-cluster_axis=0-num_links=1-input_num_cores=24-input_core_range_set={[(x=6,y=6) - (x=6,y=6)], [(x=6,y=7) - (x=6,y=7)], [(x=6,y=9) - (x=6,y=9)], [(x=6,y=0) - (x=6,y=0)], [(x=6,y=1) - (x=6,y=1)], [(x=6,y=2) - (x=6,y=2)], [(x=6,y=4) - (x=6,y=4)], [(x=6,y=5) - (x=6,y=5)], [(x=5,y=5) - (x=5,y=5)], [(x=5,y=6) - (x=5,y=6)], [(x=5,y=7) - (x=5,y=7)], [(x=5,y=9) - (x=5,y=9)], [(x=5,y=0) - (x=5,y=0)], [(x=5,y=1) - (x=5,y=1)], [(x=5,y=2) - (x=5,y=2)], [(x=5,y=4) - (x=5,y=4)], [(x=1,y=4) - (x=1,y=4)], [(x=1,y=5) - (x=1,y=5)], [(x=1,y=9) - (x=1,y=9)], [(x=1,y=0) - (x=1,y=0)], [(x=2,y=0) - (x=2,y=0)], [(x=2,y=4) - (x=2,y=4)], [(x=2,y=5) - (x=2,y=5)], [(x=2,y=9) - (x=2,y=9)]}-output_num_cores=28-output_core_range_set={[(x=1,y=0) - (x=3,y=8)], [(x=1,y=9) - (x=1,y=9)]}]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for reduction TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_binary.h:7,
from /work/ttnn/cpp/ttnn/operations/experimental/ccl/all_reduce_async/device/kernels/compute/reduction.cpp:6,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f0ea19b5168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f0e9f21680c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f0e9f22d2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f0e9f22d23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f0e9efbfdef]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f0f09307ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f0e9efbfc49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f0e9f22d08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd
|
|
blackhole-multi-card-fast-unit-tests (BH-LLMBox) / blackhole BH-LLMBox ttnn ops fast unit tests (ccl, DRAM prefetcher):
tests/ttnn/unit_tests/operations/ccl/blackhole_CI/box/all_post_commit/test_new_all_reduce.py#L358
test_all_reduce[silicon_arch_name=blackhole-silicon_arch_blackhole=True-device_params={'trace_region_size': 23887872, 'dispatch_core_axis': DispatchCoreAxis.COL, 'fabric_config': FabricConfig.FABRIC_1D}-trace_mode=True-num_iters=100-warmup_iters=10-input_dtype=DataType.BFLOAT16-output_shape=[1, 1, 32, 1280]-cluster_axis=0-num_links=2-input_num_cores=24-input_core_range_set={[(x=6,y=6) - (x=6,y=6)], [(x=6,y=7) - (x=6,y=7)], [(x=6,y=9) - (x=6,y=9)], [(x=6,y=0) - (x=6,y=0)], [(x=6,y=1) - (x=6,y=1)], [(x=6,y=2) - (x=6,y=2)], [(x=6,y=4) - (x=6,y=4)], [(x=6,y=5) - (x=6,y=5)], [(x=5,y=5) - (x=5,y=5)], [(x=5,y=6) - (x=5,y=6)], [(x=5,y=7) - (x=5,y=7)], [(x=5,y=9) - (x=5,y=9)], [(x=5,y=0) - (x=5,y=0)], [(x=5,y=1) - (x=5,y=1)], [(x=5,y=2) - (x=5,y=2)], [(x=5,y=4) - (x=5,y=4)], [(x=1,y=4) - (x=1,y=4)], [(x=1,y=5) - (x=1,y=5)], [(x=1,y=9) - (x=1,y=9)], [(x=1,y=0) - (x=1,y=0)], [(x=2,y=0) - (x=2,y=0)], [(x=2,y=4) - (x=2,y=4)], [(x=2,y=5) - (x=2,y=5)], [(x=2,y=9) - (x=2,y=9)]}-output_num_cores=10-output_core_range_set={[(x=1,y=0) - (x=3,y=2)], [(x=1,y=3) - (x=1,y=3)]}]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for reduction TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_binary.h:7,
from /work/ttnn/cpp/ttnn/operations/experimental/ccl/all_reduce_async/device/kernels/compute/reduction.cpp:6,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f0ea19b5168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f0e9f21680c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f0e9f22d2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f0e9f22d23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f0e9efbfdef]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f0f09307ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f0e9efbfc49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f0e9f22d08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f0
|
|
blackhole-multi-card-fast-unit-tests (BH-LLMBox) / blackhole BH-LLMBox ttnn ops fast unit tests (ccl, DRAM prefetcher):
tests/ttnn/unit_tests/operations/ccl/blackhole_CI/box/all_post_commit/test_all_gather_apc.py#L595
test_all_gather_failing_shapes[silicon_arch_name=blackhole-silicon_arch_blackhole=True-New all gather impl-num_buffers_per_channel=2-num_workers_per_link=2-chunks_per_sync=20-fabric_linear-non-trace-dram_only-float_16-tile-gemma dim 3-2_links]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for pack_untilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/untilize.h:7,
from /work/ttnn/cpp/ttnn/operations/data_movement/untilize/device/kernels/compute/pack_untilize.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f0ea19b5168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f0e9f21680c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f0e9f22d2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f0e9f22d23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f0e9efbfdef]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f0f09307ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f0e9efbfc49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f0e9f22d08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f0e9f1cfd85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f0e9f1ce890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f0e9f1cdd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f0f06a7b253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f0f09302ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f0f09393a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f0e9f11ced8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f0e9f0fcb51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f0e9f0fc4cb]
--- /opt/ven
|
|
ttnn-unit-tests (P150b) / ttnn eltwise group 1 blackhole P150b
Process completed with exit code 1.
|
|
ttnn-unit-tests (P150b) / ttnn eltwise group 1 blackhole P150b:
tests/ttnn/unit_tests/operations/eltwise/test_activation.py#L34
test_hardtanh[w=128-h=64]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for eltwise_sfpu TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc0 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/llk_lib/llk_math_eltwise_unary_sfpu.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_sfpu/llk_math_eltwise_unary_sfpu_init.h:8,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_sfpu/llk_math_eltwise_unary_sfpu_rpow.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_unary/rpow.h:8,
from /work/ttnn/cpp/ttnn/operations/eltwise/unary/device/kernels/compute//eltwise_sfpu.cpp:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:42,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f20b3bda168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f20b143b80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f20b14522b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f20b145223e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f20b11e4def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f20e438aee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f20b11e4c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f20b145208e]
--- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&)
--- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&)
--- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const
--- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f20b14522b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f20b145223e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f20b11e4def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f20e438aee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f20b11e4c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f20b145208e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x
|
|
ttnn-unit-tests (P150b) / ttnn pool group blackhole P150b
Process completed with exit code 1.
|
|
ttnn-unit-tests (P150b) / ttnn pool group blackhole P150b:
tests/ttnn/unit_tests/operations/pool/test_adaptive_pool2d.py#L66
test_adaptive_pool2d[pool_type=max-dtype=DataType.BFLOAT16-output_size=(1, 1)-input_shape=(1, 512, 28, 28)-device_params={'l1_small_size': 24576}]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for compute_pool_2d TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/pack_untilize.h:7,
from /work/ttnn/cpp/ttnn/operations/pool/generic/device/kernels/compute/compute_pool_2d.cpp:6,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f8004ceb168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f800254c80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f80025632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f800256323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f80022f5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f803565eee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f80022f5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f800256308e]
--- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&)
--- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&)
--- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const
--- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f80025632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f800256323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f80022f5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f803565eee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f80022f5c49]
--- /opt/venv/lib/python3.10/site-packag
|
|
ttnn-unit-tests (P150b) / ttnn matmul group blackhole P150b
Process completed with exit code 1.
|
|
ttnn-unit-tests (P150b) / ttnn matmul group blackhole P150b:
tests/ttnn/unit_tests/operations/matmul/test_addmm.py#L44
test_addmm_square_matrices[matrix_size=4-dtype=DataType.BFLOAT16]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for bmm_large_block_zm_fused_bias_activation TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/matmul.h:7,
from /work/ttnn/cpp/ttnn/operations/matmul/device/kernels/compute/bmm_large_block_zm_fused_bias_activation.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f7cbe8eb168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f7cbc14c80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f7cbc1632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f7cbc16323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f7cbbef5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f7cef1f8ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f7cbbef5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f7cbc16308e]
--- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&)
--- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&)
--- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const
--- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f7cbc1632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f7cbc16323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f7cbbef5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f7cef1f8ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f7cbbef5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e)
|
|
ttnn-unit-tests (P150b) / ttnn eltwise group 4 blackhole P150b
Process completed with exit code 1.
|
|
ttnn-unit-tests (P150b) / ttnn eltwise group 4 blackhole P150b:
tests/ttnn/unit_tests/operations/eltwise/test_eltwise_typecast.py#L241
test_typecast_bfp8_b_to_bf16
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for eltwise_typecast TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /work/ttnn/cpp/ttnn/operations/copy/typecast/device/kernels/compute/eltwise_typecast.cpp:5,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f2e060eb168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f2e0394c80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2e039632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2e0396323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2e036f5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2e36a52ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2e036f5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2e0396308e]
--- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&)
--- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&)
--- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const
--- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2e039632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2e0396323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2e036f5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2e36a52ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2e036f5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2e0396308e]
--- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&)
--- tt::tt_metal::launch_build_step(std::function<void ()> const&,
|
|
blackhole-multi-card-fast-unit-tests (BH-LoudBox) / blackhole BH-LoudBox Sockets fast unit test:
models/demos/deepseek_v3_b1/tests/unit_tests/test_host_io.py#L139
test_host_io_loopback_with_embedding[blackhole-256-4-64-14336-H2DMode.HOST_PUSH]
RuntimeError: TT_THROW @ /project/tt_metal/third_party/umd/device/pcie/pci_device.cpp:668: tt::exception
info:
Failed to pin pages for DMA buffer at virtual address 0x7f9400043000 with size 0x1d000 and flags 0x1: Invalid argument
backtrace:
--- tt::umd::PCIDevice::map_for_dma(void*, unsigned long)
--- tt::umd::SysmemBuffer::SysmemBuffer(tt::umd::TLBManager*, void*, unsigned long, bool)
--- tt::umd::SiliconSysmemManager::map_sysmem_buffer(void*, unsigned long, bool)
--- tt::Cluster::map_sysmem_buffer(int, void*, unsigned long, bool) const
--- tt::tt_metal::experimental::PinnedMemoryImpl::initialize_from_devices(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemoryImpl::PinnedMemoryImpl(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::PinnedMemory(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::Create(tt::tt_metal::distributed::MeshDevice&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, tt::tt_metal::HostBuffer&, bool)
--- tt::tt_metal::distributed::D2HSocket::init_host_buffer(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, unsigned int)
--- tt::tt_metal::distributed::D2HSocket::D2HSocket(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoreCoord const&, unsigned int)
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x33794b) [0x7f94e96ac94b]
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x23ee65) [0x7f94e95b3e65]
--- /opt/venv/bin/python3(+0x2a5b30) [0x55e638ea6b30]
--- /opt/venv/bin/python3(+0x2a7125) [0x55e638ea8125]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x12f3e) [0x55e638eda64e]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x36749) [0x55e638efde59]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x55e638ecfc17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(+0x273350) [0x55e638e74350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x55e638f6fc65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x55e638fa3b46]
--- /opt/venv/bin/python3(+0x2fd2a2) [0x55e638efe2a2]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x3eef) [0x55e638ecb5ff]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8924) [0x55e638ed0034]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x55e638ecfc17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(+0x273350) [0x55e638e74350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x55e638f6fc65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x55e638fa3b46]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa16c) [0x55e638ed187c]
--- /opt/ven
|
|
blackhole-multi-card-fast-unit-tests (BH-LoudBox) / blackhole BH-LoudBox Sockets fast unit test:
models/demos/deepseek_v3_b1/tests/unit_tests/test_host_io.py#L139
test_host_io_loopback_with_embedding[blackhole-128-2-512-1792-H2DMode.DEVICE_PULL]
RuntimeError: TT_THROW @ /project/tt_metal/third_party/umd/device/pcie/pci_device.cpp:668: tt::exception
info:
Failed to pin pages for DMA buffer at virtual address 0x7f940005e000 with size 0x2000 and flags 0x1: Invalid argument
backtrace:
--- tt::umd::PCIDevice::map_for_dma(void*, unsigned long)
--- tt::umd::SysmemBuffer::SysmemBuffer(tt::umd::TLBManager*, void*, unsigned long, bool)
--- tt::umd::SiliconSysmemManager::map_sysmem_buffer(void*, unsigned long, bool)
--- tt::Cluster::map_sysmem_buffer(int, void*, unsigned long, bool) const
--- tt::tt_metal::experimental::PinnedMemoryImpl::initialize_from_devices(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemoryImpl::PinnedMemoryImpl(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::PinnedMemory(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::Create(tt::tt_metal::distributed::MeshDevice&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, tt::tt_metal::HostBuffer&, bool)
--- tt::tt_metal::distributed::D2HSocket::init_host_buffer(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, unsigned int)
--- tt::tt_metal::distributed::D2HSocket::D2HSocket(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoreCoord const&, unsigned int)
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x33794b) [0x7f94e96ac94b]
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x23ee65) [0x7f94e95b3e65]
--- /opt/venv/bin/python3(+0x2a5b30) [0x55e638ea6b30]
--- /opt/venv/bin/python3(+0x2a7125) [0x55e638ea8125]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x12f3e) [0x55e638eda64e]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x36749) [0x55e638efde59]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x55e638ecfc17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(+0x273350) [0x55e638e74350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x55e638f6fc65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x55e638fa3b46]
--- /opt/venv/bin/python3(+0x2fd2a2) [0x55e638efe2a2]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x3eef) [0x55e638ecb5ff]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8924) [0x55e638ed0034]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x55e638ecfc17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(+0x273350) [0x55e638e74350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x55e638f6fc65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x55e638fa3b46]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa16c) [0x55e638ed187c]
--- /opt/ve
|
|
blackhole-multi-card-fast-unit-tests (BH-LoudBox) / blackhole BH-LoudBox Sockets fast unit test:
models/demos/deepseek_v3_b1/tests/unit_tests/test_host_io.py#L139
test_host_io_loopback_with_embedding[blackhole-128-2-512-1792-H2DMode.HOST_PUSH]
RuntimeError: TT_THROW @ /project/tt_metal/third_party/umd/device/pcie/pci_device.cpp:668: tt::exception
info:
Failed to pin pages for DMA buffer at virtual address 0x7f940005d000 with size 0x2000 and flags 0x1: Invalid argument
backtrace:
--- tt::umd::PCIDevice::map_for_dma(void*, unsigned long)
--- tt::umd::SysmemBuffer::SysmemBuffer(tt::umd::TLBManager*, void*, unsigned long, bool)
--- tt::umd::SiliconSysmemManager::map_sysmem_buffer(void*, unsigned long, bool)
--- tt::Cluster::map_sysmem_buffer(int, void*, unsigned long, bool) const
--- tt::tt_metal::experimental::PinnedMemoryImpl::initialize_from_devices(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemoryImpl::PinnedMemoryImpl(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::PinnedMemory(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::Create(tt::tt_metal::distributed::MeshDevice&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, tt::tt_metal::HostBuffer&, bool)
--- tt::tt_metal::distributed::D2HSocket::init_host_buffer(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, unsigned int)
--- tt::tt_metal::distributed::D2HSocket::D2HSocket(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoreCoord const&, unsigned int)
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x33794b) [0x7f94e96ac94b]
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x23ee65) [0x7f94e95b3e65]
--- /opt/venv/bin/python3(+0x2a5b30) [0x55e638ea6b30]
--- /opt/venv/bin/python3(+0x2a7125) [0x55e638ea8125]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x12f3e) [0x55e638eda64e]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x36749) [0x55e638efde59]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x55e638ecfc17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(+0x273350) [0x55e638e74350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x55e638f6fc65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x55e638fa3b46]
--- /opt/venv/bin/python3(+0x2fd2a2) [0x55e638efe2a2]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x3eef) [0x55e638ecb5ff]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8924) [0x55e638ed0034]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x55e638ecfc17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(+0x273350) [0x55e638e74350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x55e638f6fc65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x55e638fa3b46]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa16c) [0x55e638ed187c]
--- /opt/venv
|
|
blackhole-multi-card-fast-unit-tests (BH-LoudBox) / blackhole BH-LoudBox Sockets fast unit test:
models/demos/deepseek_v3_b1/tests/unit_tests/test_host_io.py#L50
test_host_io_loopback[blackhole-H2DMode.HOST_PUSH-32768-65536-128]
RuntimeError: TT_THROW @ /project/tt_metal/third_party/umd/device/pcie/pci_device.cpp:668: tt::exception
info:
Failed to pin pages for DMA buffer at virtual address 0x7f940004e000 with size 0x11000 and flags 0x1: Invalid argument
backtrace:
--- tt::umd::PCIDevice::map_for_dma(void*, unsigned long)
--- tt::umd::SysmemBuffer::SysmemBuffer(tt::umd::TLBManager*, void*, unsigned long, bool)
--- tt::umd::SiliconSysmemManager::map_sysmem_buffer(void*, unsigned long, bool)
--- tt::Cluster::map_sysmem_buffer(int, void*, unsigned long, bool) const
--- tt::tt_metal::experimental::PinnedMemoryImpl::initialize_from_devices(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemoryImpl::PinnedMemoryImpl(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::PinnedMemory(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::Create(tt::tt_metal::distributed::MeshDevice&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, tt::tt_metal::HostBuffer&, bool)
--- tt::tt_metal::distributed::D2HSocket::init_host_buffer(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, unsigned int)
--- tt::tt_metal::distributed::D2HSocket::D2HSocket(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoreCoord const&, unsigned int)
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x33794b) [0x7f94e96ac94b]
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x23ee65) [0x7f94e95b3e65]
--- /opt/venv/bin/python3(+0x2a5b30) [0x55e638ea6b30]
--- /opt/venv/bin/python3(+0x2a7125) [0x55e638ea8125]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x12f3e) [0x55e638eda64e]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x36749) [0x55e638efde59]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x55e638ecfc17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(+0x273350) [0x55e638e74350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x55e638f6fc65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x55e638fa3b46]
--- /opt/venv/bin/python3(+0x2fd2a2) [0x55e638efe2a2]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x3eef) [0x55e638ecb5ff]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8924) [0x55e638ed0034]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x55e638ecfc17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(+0x273350) [0x55e638e74350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x55e638f6fc65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x55e638fa3b46]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa16c) [0x55e638ed187c]
--- /opt/venv/bin/python3(
|
|
blackhole-multi-card-fast-unit-tests (BH-LoudBox) / blackhole BH-LoudBox Sockets fast unit test:
models/demos/deepseek_v3_b1/tests/unit_tests/test_host_io.py#L139
test_host_io_loopback_with_embedding[blackhole-128-2-256-3584-H2DMode.DEVICE_PULL]
RuntimeError: TT_THROW @ /project/tt_metal/third_party/umd/device/pcie/pci_device.cpp:668: tt::exception
info:
Failed to pin pages for DMA buffer at virtual address 0x7f940005c000 with size 0x4000 and flags 0x1: Invalid argument
backtrace:
--- tt::umd::PCIDevice::map_for_dma(void*, unsigned long)
--- tt::umd::SysmemBuffer::SysmemBuffer(tt::umd::TLBManager*, void*, unsigned long, bool)
--- tt::umd::SiliconSysmemManager::map_sysmem_buffer(void*, unsigned long, bool)
--- tt::Cluster::map_sysmem_buffer(int, void*, unsigned long, bool) const
--- tt::tt_metal::experimental::PinnedMemoryImpl::initialize_from_devices(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemoryImpl::PinnedMemoryImpl(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::PinnedMemory(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::Create(tt::tt_metal::distributed::MeshDevice&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, tt::tt_metal::HostBuffer&, bool)
--- tt::tt_metal::distributed::D2HSocket::init_host_buffer(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, unsigned int)
--- tt::tt_metal::distributed::D2HSocket::D2HSocket(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoreCoord const&, unsigned int)
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x33794b) [0x7f94e96ac94b]
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x23ee65) [0x7f94e95b3e65]
--- /opt/venv/bin/python3(+0x2a5b30) [0x55e638ea6b30]
--- /opt/venv/bin/python3(+0x2a7125) [0x55e638ea8125]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x12f3e) [0x55e638eda64e]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x36749) [0x55e638efde59]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x55e638ecfc17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(+0x273350) [0x55e638e74350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x55e638f6fc65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x55e638fa3b46]
--- /opt/venv/bin/python3(+0x2fd2a2) [0x55e638efe2a2]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x3eef) [0x55e638ecb5ff]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8924) [0x55e638ed0034]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x55e638ecfc17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(+0x273350) [0x55e638e74350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x55e638f6fc65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x55e638fa3b46]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa16c) [0x55e638ed187c]
--- /opt/ve
|
|
blackhole-multi-card-fast-unit-tests (BH-LoudBox) / blackhole BH-LoudBox Sockets fast unit test:
models/demos/deepseek_v3_b1/tests/unit_tests/test_host_io.py#L139
test_host_io_loopback_with_embedding[blackhole-128-2-256-3584-H2DMode.HOST_PUSH]
RuntimeError: TT_THROW @ /project/tt_metal/third_party/umd/device/pcie/pci_device.cpp:668: tt::exception
info:
Failed to pin pages for DMA buffer at virtual address 0x7f940015c000 with size 0x4000 and flags 0x1: Invalid argument
backtrace:
--- tt::umd::PCIDevice::map_for_dma(void*, unsigned long)
--- tt::umd::SysmemBuffer::SysmemBuffer(tt::umd::TLBManager*, void*, unsigned long, bool)
--- tt::umd::SiliconSysmemManager::map_sysmem_buffer(void*, unsigned long, bool)
--- tt::Cluster::map_sysmem_buffer(int, void*, unsigned long, bool) const
--- tt::tt_metal::experimental::PinnedMemoryImpl::initialize_from_devices(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemoryImpl::PinnedMemoryImpl(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::PinnedMemory(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::Create(tt::tt_metal::distributed::MeshDevice&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, tt::tt_metal::HostBuffer&, bool)
--- tt::tt_metal::distributed::D2HSocket::init_host_buffer(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, unsigned int)
--- tt::tt_metal::distributed::D2HSocket::D2HSocket(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoreCoord const&, unsigned int)
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x33794b) [0x7f94e96ac94b]
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x23ee65) [0x7f94e95b3e65]
--- /opt/venv/bin/python3(+0x2a5b30) [0x55e638ea6b30]
--- /opt/venv/bin/python3(+0x2a7125) [0x55e638ea8125]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x12f3e) [0x55e638eda64e]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x36749) [0x55e638efde59]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x55e638ecfc17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(+0x273350) [0x55e638e74350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x55e638f6fc65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x55e638fa3b46]
--- /opt/venv/bin/python3(+0x2fd2a2) [0x55e638efe2a2]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x3eef) [0x55e638ecb5ff]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8924) [0x55e638ed0034]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x55e638ecfc17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(+0x273350) [0x55e638e74350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x55e638f6fc65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x55e638fa3b46]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa16c) [0x55e638ed187c]
--- /opt/venv
|
|
blackhole-multi-card-fast-unit-tests (BH-LoudBox) / blackhole BH-LoudBox Sockets fast unit test:
models/demos/deepseek_v3_b1/tests/unit_tests/test_host_io.py#L139
test_host_io_loopback_with_embedding[blackhole-128-2-128-7168-H2DMode.DEVICE_PULL]
RuntimeError: TT_THROW @ /project/tt_metal/third_party/umd/device/pcie/pci_device.cpp:668: tt::exception
info:
Failed to pin pages for DMA buffer at virtual address 0x7f9400158000 with size 0x8000 and flags 0x1: Invalid argument
backtrace:
--- tt::umd::PCIDevice::map_for_dma(void*, unsigned long)
--- tt::umd::SysmemBuffer::SysmemBuffer(tt::umd::TLBManager*, void*, unsigned long, bool)
--- tt::umd::SiliconSysmemManager::map_sysmem_buffer(void*, unsigned long, bool)
--- tt::Cluster::map_sysmem_buffer(int, void*, unsigned long, bool) const
--- tt::tt_metal::experimental::PinnedMemoryImpl::initialize_from_devices(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemoryImpl::PinnedMemoryImpl(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::PinnedMemory(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::Create(tt::tt_metal::distributed::MeshDevice&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, tt::tt_metal::HostBuffer&, bool)
--- tt::tt_metal::distributed::D2HSocket::init_host_buffer(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, unsigned int)
--- tt::tt_metal::distributed::D2HSocket::D2HSocket(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoreCoord const&, unsigned int)
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x33794b) [0x7f94e96ac94b]
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x23ee65) [0x7f94e95b3e65]
--- /opt/venv/bin/python3(+0x2a5b30) [0x55e638ea6b30]
--- /opt/venv/bin/python3(+0x2a7125) [0x55e638ea8125]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x12f3e) [0x55e638eda64e]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x36749) [0x55e638efde59]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x55e638ecfc17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(+0x273350) [0x55e638e74350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x55e638f6fc65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x55e638fa3b46]
--- /opt/venv/bin/python3(+0x2fd2a2) [0x55e638efe2a2]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x3eef) [0x55e638ecb5ff]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8924) [0x55e638ed0034]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x55e638ecfc17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(+0x273350) [0x55e638e74350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x55e638f6fc65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x55e638fa3b46]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa16c) [0x55e638ed187c]
--- /opt/ve
|
|
blackhole-multi-card-fast-unit-tests (BH-LoudBox) / blackhole BH-LoudBox Sockets fast unit test:
models/demos/deepseek_v3_b1/tests/unit_tests/test_host_io.py#L139
test_host_io_loopback_with_embedding[blackhole-128-2-128-7168-H2DMode.HOST_PUSH]
RuntimeError: TT_THROW @ /project/tt_metal/third_party/umd/device/pcie/pci_device.cpp:668: tt::exception
info:
Failed to pin pages for DMA buffer at virtual address 0x7f9400158000 with size 0x8000 and flags 0x1: Invalid argument
backtrace:
--- tt::umd::PCIDevice::map_for_dma(void*, unsigned long)
--- tt::umd::SysmemBuffer::SysmemBuffer(tt::umd::TLBManager*, void*, unsigned long, bool)
--- tt::umd::SiliconSysmemManager::map_sysmem_buffer(void*, unsigned long, bool)
--- tt::Cluster::map_sysmem_buffer(int, void*, unsigned long, bool) const
--- tt::tt_metal::experimental::PinnedMemoryImpl::initialize_from_devices(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemoryImpl::PinnedMemoryImpl(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::PinnedMemory(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::Create(tt::tt_metal::distributed::MeshDevice&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, tt::tt_metal::HostBuffer&, bool)
--- tt::tt_metal::distributed::D2HSocket::init_host_buffer(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, unsigned int)
--- tt::tt_metal::distributed::D2HSocket::D2HSocket(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoreCoord const&, unsigned int)
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x33794b) [0x7f94e96ac94b]
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x23ee65) [0x7f94e95b3e65]
--- /opt/venv/bin/python3(+0x2a5b30) [0x55e638ea6b30]
--- /opt/venv/bin/python3(+0x2a7125) [0x55e638ea8125]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x12f3e) [0x55e638eda64e]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x36749) [0x55e638efde59]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x55e638ecfc17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(+0x273350) [0x55e638e74350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x55e638f6fc65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x55e638fa3b46]
--- /opt/venv/bin/python3(+0x2fd2a2) [0x55e638efe2a2]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x3eef) [0x55e638ecb5ff]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8924) [0x55e638ed0034]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x55e638ecfc17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(+0x273350) [0x55e638e74350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x55e638f6fc65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x55e638fa3b46]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa16c) [0x55e638ed187c]
--- /opt/venv
|
|
blackhole-multi-card-fast-unit-tests (BH-LoudBox) / blackhole BH-LoudBox Sockets fast unit test:
models/demos/deepseek_v3_b1/tests/unit_tests/test_host_io.py#L139
test_host_io_loopback_with_embedding[blackhole-128-2-64-14336-H2DMode.DEVICE_PULL]
RuntimeError: TT_THROW @ /project/tt_metal/third_party/umd/device/pcie/pci_device.cpp:668: tt::exception
info:
Failed to pin pages for DMA buffer at virtual address 0x7f9400151000 with size 0xf000 and flags 0x1: Invalid argument
backtrace:
--- tt::umd::PCIDevice::map_for_dma(void*, unsigned long)
--- tt::umd::SysmemBuffer::SysmemBuffer(tt::umd::TLBManager*, void*, unsigned long, bool)
--- tt::umd::SiliconSysmemManager::map_sysmem_buffer(void*, unsigned long, bool)
--- tt::Cluster::map_sysmem_buffer(int, void*, unsigned long, bool) const
--- tt::tt_metal::experimental::PinnedMemoryImpl::initialize_from_devices(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemoryImpl::PinnedMemoryImpl(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::PinnedMemory(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::Create(tt::tt_metal::distributed::MeshDevice&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, tt::tt_metal::HostBuffer&, bool)
--- tt::tt_metal::distributed::D2HSocket::init_host_buffer(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, unsigned int)
--- tt::tt_metal::distributed::D2HSocket::D2HSocket(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoreCoord const&, unsigned int)
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x33794b) [0x7f94e96ac94b]
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x23ee65) [0x7f94e95b3e65]
--- /opt/venv/bin/python3(+0x2a5b30) [0x55e638ea6b30]
--- /opt/venv/bin/python3(+0x2a7125) [0x55e638ea8125]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x12f3e) [0x55e638eda64e]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x36749) [0x55e638efde59]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x55e638ecfc17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(+0x273350) [0x55e638e74350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x55e638f6fc65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x55e638fa3b46]
--- /opt/venv/bin/python3(+0x2fd2a2) [0x55e638efe2a2]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x3eef) [0x55e638ecb5ff]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8924) [0x55e638ed0034]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x55e638ecfc17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(+0x273350) [0x55e638e74350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x55e638f6fc65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x55e638fa3b46]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa16c) [0x55e638ed187c]
--- /opt/ve
|
|
blackhole-multi-card-fast-unit-tests (BH-LoudBox) / blackhole BH-LoudBox Sockets fast unit test:
models/demos/deepseek_v3_b1/tests/unit_tests/test_host_io.py#L139
test_host_io_loopback_with_embedding[blackhole-128-2-64-14336-H2DMode.HOST_PUSH]
RuntimeError: TT_THROW @ /project/tt_metal/third_party/umd/device/pcie/pci_device.cpp:668: tt::exception
info:
Failed to pin pages for DMA buffer at virtual address 0x7f9400151000 with size 0xf000 and flags 0x1: Invalid argument
backtrace:
--- tt::umd::PCIDevice::map_for_dma(void*, unsigned long)
--- tt::umd::SysmemBuffer::SysmemBuffer(tt::umd::TLBManager*, void*, unsigned long, bool)
--- tt::umd::SiliconSysmemManager::map_sysmem_buffer(void*, unsigned long, bool)
--- tt::Cluster::map_sysmem_buffer(int, void*, unsigned long, bool) const
--- tt::tt_metal::experimental::PinnedMemoryImpl::initialize_from_devices(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemoryImpl::PinnedMemoryImpl(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::PinnedMemory(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::Create(tt::tt_metal::distributed::MeshDevice&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, tt::tt_metal::HostBuffer&, bool)
--- tt::tt_metal::distributed::D2HSocket::init_host_buffer(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, unsigned int)
--- tt::tt_metal::distributed::D2HSocket::D2HSocket(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoreCoord const&, unsigned int)
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x33794b) [0x7f94e96ac94b]
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x23ee65) [0x7f94e95b3e65]
--- /opt/venv/bin/python3(+0x2a5b30) [0x55e638ea6b30]
--- /opt/venv/bin/python3(+0x2a7125) [0x55e638ea8125]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x12f3e) [0x55e638eda64e]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x36749) [0x55e638efde59]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x55e638ecfc17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(+0x273350) [0x55e638e74350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x55e638f6fc65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x55e638fa3b46]
--- /opt/venv/bin/python3(+0x2fd2a2) [0x55e638efe2a2]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x3eef) [0x55e638ecb5ff]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8924) [0x55e638ed0034]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x55e638ecfc17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(+0x273350) [0x55e638e74350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x55e638ed1aef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x55e638e72352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x55e638f6fc65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x55e638fa3b46]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa16c) [0x55e638ed187c]
--- /opt/venv
|
|
ttnn-unit-tests (P150b) / ttnn eltwise group 5 blackhole P150b
Process completed with exit code 1.
|
|
ttnn-unit-tests (P150b) / ttnn eltwise group 5 blackhole P150b:
tests/ttnn/unit_tests/operations/eltwise/test_ternary_sharding.py#L1622
test_where_ttt_identical_mixed_strategy[false_sharded=False-true_sharded=True-predicate_sharded=True-out_strategy=height-config_permutation=hbw-dtype_pt=torch.bfloat16-dtype_tt=DataType.BFLOAT16]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for ternary_sfpu_no_bcast_ttt TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/ttnn/cpp/ttnn/operations/eltwise/binary_ng/device/kernels/compute/eltwise_utils_sfpu.hpp:7,
from /work/ttnn/cpp/ttnn/operations/eltwise/ternary/device/kernels/compute/ternary_sfpu_no_bcast_ttt.cpp:8,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f09084eb168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f0905d4c80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f0905d632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f0905d6323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f0905af5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f0938e22ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f0905af5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f0905d6308e]
--- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&)
--- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&)
--- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const
--- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f0905d632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f0905d6323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f0905af5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f0938e22ee8]
--- /opt/venv/lib/python3
|
|
ttnn-unit-tests (P150b) / core ttnn unit test group blackhole P150b
Process completed with exit code 1.
|
|
ttnn-unit-tests (P150b) / core ttnn unit test group blackhole P150b:
tests/ttnn/unit_tests/base_functionality/test_async.py#L23
test_add_1D_tensor_and_scalar[size=64-scalar=3]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for eltwise_binary_scalar TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_binary.h:7,
from /work/ttnn/cpp/ttnn/operations/eltwise/binary_ng/device/kernels/compute/eltwise_binary_scalar.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7fdec5447168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7fdec2ca880c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fdec2cbf2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fdec2cbf23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fdec2a51def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fdef5cffee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7fdec2a51c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7fdec2cbf08e]
--- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&)
--- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&)
--- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const
--- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fdec2cbf2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fdec2cbf23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fdec2a51def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fdef5cffee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7fdec2a51c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7fdec2cbf08e]
--- /opt/venv/lib/p
|
|
ttnn-basic-tests (P150b-viommu) / metalium-basic
The process '/usr/bin/git' failed with exit code 128
|
|
ttnn-basic-tests (P150b-viommu) / metalium-basic
Process completed with exit code 139.
|
|
ttnn-unit-tests (P150b) / ttnn conv group blackhole P150b
Process completed with exit code 1.
|
|
ttnn-unit-tests (P150b) / ttnn conv group blackhole P150b:
tests/ttnn/unit_tests/operations/conv/data_movement/test_convert_to_chw.py#L41
test_convert_to_chw[HW=32-core_grid=ttnn.CoreGrid(x=1, y=1)-C=1-input_data_type=DataType.BFLOAT16]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for convert_to_chw TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/pack_untilize.h:7,
from /work/ttnn/cpp/ttnn/operations/experimental/cnn/convert_to_chw/device/kernels/convert_to_chw.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7fc7e3aeb168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7fc7e134c80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fc7e13632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fc7e136323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fc7e10f5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fc81446aee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7fc7e10f5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7fc7e136308e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7fc7e1305d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7fc7e1304890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7fc7e1303d78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fc811be4253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fc814465ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7fc8144f6a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7fc7e1252ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7fc7e1232b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7fc7e12324cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fc7e13632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn
|
|
ttnn-unit-tests (P150b) / ttnn transformers group blackhole P150b
Process completed with exit code 1.
|
|
ttnn-unit-tests (P150b) / ttnn transformers group blackhole P150b:
tests/ttnn/unit_tests/operations/transformers/test_prefetcher_BH.py#L644
test_prefetcher_ring_matmul_BH[silicon_arch_name=blackhole-silicon_arch_blackhole=True-device_params={'dispatch_core_axis': DispatchCoreAxis.COL, 'trace_region_size': 23887872}-mesh_device=1]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for bmm_large_block_zm_fused_bias_activation_gathered TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/matmul.h:7,
from /work/ttnn/cpp/ttnn/operations/matmul/device/kernels/compute/bmm_large_block_zm_fused_bias_activation_gathered.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7fbdaf4eb168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7fbdacd4c80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fbdacd632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fbdacd6323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fbdacaf5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fbddfde9ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7fbdacaf5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7fbdacd6308e]
--- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&)
--- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&)
--- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const
--- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fbdacd632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fbdacd6323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fbdacaf5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fbddfde9ee8]
--- /opt/venv/lib/python3.10/site-packages
|
|
ttnn-unit-tests (P150b) / ttnn eltwise group 2 blackhole P150b
Process completed with exit code 1.
|
|
ttnn-unit-tests (P150b) / ttnn eltwise group 2 blackhole P150b:
tests/ttnn/unit_tests/operations/eltwise/test_binary_int32.py#L96
test_binary_int32[use_legacy=True-ttnn_op=ttnn.subtract-low_a=-2147483647-high_a=-2000000000.0-low_b=-2077000000-high_b=-2000000000.0-input_shapes=torch.Size([1, 1, 320, 384])]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for eltwise_binary_sfpu_kernel TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_binary.h:7,
from /work/ttnn/cpp/ttnn/operations/eltwise/binary/device/kernels/compute/eltwise_binary_sfpu_kernel.cpp:6,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7fc34d196168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7fc34a9f780c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fc34aa0e2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fc34aa0e23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fc34a7a0def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fc37d937ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7fc34a7a0c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7fc34aa0e08e]
--- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&)
--- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&)
--- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const
--- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7fc34aa0e2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7fc34aa0e23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7fc34a7a0def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fc37d937ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x
|
|
ops-unit-tests (P150b) / blackhole deepseek blitz op tests blackhole P150b
Process completed with exit code 139.
|
|
ops-unit-tests (P150b) / blackhole deepseek blitz op tests blackhole P150b:
models/demos/deepseek_v3_b1/tests/unit_tests/test_create_q_heads.py#L175
test_create_q_heads[qnope_shard_shape1-qrope_shard_shape1-NOC.NOC_1]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for create_q_heads_kernel TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7,
from /work/models/demos/deepseek_v3_b1/micro_ops/create_q_heads/kernels/../../../unified_kernels/create_q_heads.hpp:14,
from /work/models/demos/deepseek_v3_b1/micro_ops/create_q_heads/kernels/create_q_heads_kernel.cpp:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f8560d62168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f855e5c380c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f855e5da2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f855e5da23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f855e36cdef]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f8591509ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f855e36cc49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f855e5da08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f855e57cd85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f855e57b890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f855e57ad78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f858ec35253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f8591504ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f8591595a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f855e4c9ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f855e4a9b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f855e4a94cb]
--- /opt/venv/lib/python3.10/site-packages/tt
|
|
ops-unit-tests (P150b) / blackhole deepseek blitz op tests blackhole P150b:
models/demos/deepseek_v3_b1/tests/unit_tests/test_create_q_heads.py#L175
test_create_q_heads[qnope_shard_shape0-qrope_shard_shape0-NOC.NOC_0]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for create_q_heads_kernel TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7,
from /work/models/demos/deepseek_v3_b1/micro_ops/create_q_heads/kernels/../../../unified_kernels/create_q_heads.hpp:14,
from /work/models/demos/deepseek_v3_b1/micro_ops/create_q_heads/kernels/create_q_heads_kernel.cpp:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f8560d62168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f855e5c380c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f855e5da2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f855e5da23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f855e36cdef]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f8591509ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f855e36cc49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f855e5da08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f855e57cd85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f855e57b890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f855e57ad78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f858ec35253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f8591504ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f8591595a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f855e4c9ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f855e4a9b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f855e4a94cb]
--- /opt/venv/lib/python3.10/site-packages/tt
|
|
ops-unit-tests (P150b) / blackhole deepseek blitz op tests blackhole P150b:
models/demos/deepseek_v3_b1/tests/unit_tests/test_broadcast_rms_single_device.py#L113
test_broadcast_rms_single_device[blackhole-mesh_device0-False-1e-06-DataType.BFLOAT16-Layout.TILE-output_shape0-input_shard_shape0-TensorMemoryLayout.WIDTH_SHARDED]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for broadcast_rms_kernel TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/compute_kernel_api.h:20,
from /work/models/demos/deepseek_v3_b1/fused_ops/broadcast_rms/kernels/../../../unified_kernels/rmsnorm.hpp:16,
from /work/models/demos/deepseek_v3_b1/fused_ops/broadcast_rms/kernels/broadcast_rms_kernel.cpp:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f8560d62168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f855e5c380c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f855e5da2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f855e5da23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f855e36cdef]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f8591509ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f855e36cc49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f855e5da08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f855e57cd85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f855e57b890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f855e57ad78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f858ec35253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f8591504ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f8591595a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f855e4c9ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f855e4a9b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f855e4a94cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f855e5da2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_me
|
|
tt-cnn-unit-tests (P150b) / tt-cnn builder unit tests blackhole P150b:
models/tt_cnn/tests/test_builder.py#L168
test_conv2d[sharding_strategy0-kernel_config1-1-channel_config0-input_size1-device_params0]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for conv_bmm_tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/bcast.h:7,
from /work/ttnn/cpp/ttnn/operations/conv/conv2d/device/kernels/conv_bmm_tilize.cpp:8,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f2a21aeb168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f2a1f34c80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2a1f3632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2a1f36323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2a1f0f5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2a523f8ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2a1f0f5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2a1f36308e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f2a1f305d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f2a1f304890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f2a1f303d78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f2a4fb72253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f2a523f3ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f2a52484a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f2a1f252ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f2a1f232b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f2a1f2324cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2a1f3632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2
|
|
tt-cnn-unit-tests (P150b) / tt-cnn builder unit tests blackhole P150b:
models/tt_cnn/tests/test_builder.py#L168
test_conv2d[sharding_strategy0-kernel_config1-1-channel_config0-input_size0-device_params0]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for conv_bmm_tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/bcast.h:7,
from /work/ttnn/cpp/ttnn/operations/conv/conv2d/device/kernels/conv_bmm_tilize.cpp:8,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f2a21aeb168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f2a1f34c80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2a1f3632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2a1f36323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2a1f0f5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2a523f8ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2a1f0f5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2a1f36308e]
--- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&)
--- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&)
--- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const
--- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2a1f3632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2a1f36323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2a1f0f5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2a523f8ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2a1f0f5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2a1f36308e]
--- auto t
|
|
tt-cnn-unit-tests (P150b) / tt-cnn builder unit tests blackhole P150b:
models/tt_cnn/tests/test_builder.py#L168
test_conv2d[sharding_strategy0-kernel_config0-2-channel_config1-input_size1-device_params0]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for conv_bmm_tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/bcast.h:7,
from /work/ttnn/cpp/ttnn/operations/conv/conv2d/device/kernels/conv_bmm_tilize.cpp:8,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f2a21aeb168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f2a1f34c80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2a1f3632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2a1f36323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2a1f0f5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2a523f8ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2a1f0f5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2a1f36308e]
--- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&)
--- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&)
--- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const
--- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2a1f3632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2a1f36323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2a1f0f5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2a523f8ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2a1f0f5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2a1f36308e]
--- auto t
|
|
tt-cnn-unit-tests (P150b) / tt-cnn builder unit tests blackhole P150b:
models/tt_cnn/tests/test_builder.py#L168
test_conv2d[sharding_strategy0-kernel_config0-2-channel_config1-input_size0-device_params0]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for conv_bmm_tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/bcast.h:7,
from /work/ttnn/cpp/ttnn/operations/conv/conv2d/device/kernels/conv_bmm_tilize.cpp:8,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f2a21aeb168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f2a1f34c80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2a1f3632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2a1f36323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2a1f0f5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2a523f8ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2a1f0f5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2a1f36308e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f2a1f305d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f2a1f304890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f2a1f303d78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f2a4fb72253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f2a523f3ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f2a52484a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f2a1f252ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f2a1f232b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f2a1f2324cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2a1f3632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2
|
|
tt-cnn-unit-tests (P150b) / tt-cnn builder unit tests blackhole P150b:
models/tt_cnn/tests/test_builder.py#L168
test_conv2d[sharding_strategy0-kernel_config0-2-channel_config0-input_size1-device_params0]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for conv_bmm_tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/bcast.h:7,
from /work/ttnn/cpp/ttnn/operations/conv/conv2d/device/kernels/conv_bmm_tilize.cpp:8,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f2a21aeb168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f2a1f34c80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2a1f3632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2a1f36323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2a1f0f5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2a523f8ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2a1f0f5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2a1f36308e]
--- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&)
--- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&)
--- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const
--- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2a1f3632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2a1f36323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2a1f0f5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2a523f8ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2a1f0f5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2a1f36308e]
--- /opt/v
|
|
tt-cnn-unit-tests (P150b) / tt-cnn builder unit tests blackhole P150b:
models/tt_cnn/tests/test_builder.py#L168
test_conv2d[sharding_strategy0-kernel_config0-2-channel_config0-input_size0-device_params0]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for conv_bmm_tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/bcast.h:7,
from /work/ttnn/cpp/ttnn/operations/conv/conv2d/device/kernels/conv_bmm_tilize.cpp:8,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f2a21aeb168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f2a1f34c80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2a1f3632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2a1f36323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2a1f0f5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2a523f8ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2a1f0f5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2a1f36308e]
--- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&)
--- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&)
--- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const
--- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2a1f3632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2a1f36323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2a1f0f5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2a523f8ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2a1f0f5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2a1f36308e]
--- /opt/v
|
|
tt-cnn-unit-tests (P150b) / tt-cnn builder unit tests blackhole P150b:
models/tt_cnn/tests/test_builder.py#L168
test_conv2d[sharding_strategy0-kernel_config0-1-channel_config1-input_size1-device_params0]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for conv_bmm_tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/bcast.h:7,
from /work/ttnn/cpp/ttnn/operations/conv/conv2d/device/kernels/conv_bmm_tilize.cpp:8,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f2a21aeb168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f2a1f34c80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2a1f3632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2a1f36323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2a1f0f5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2a523f8ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2a1f0f5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2a1f36308e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f2a1f305d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f2a1f304890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f2a1f303d78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f2a4fb72253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f2a523f3ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f2a52484a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f2a1f252ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f2a1f232b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f2a1f2324cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2a1f3632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2
|
|
tt-cnn-unit-tests (P150b) / tt-cnn builder unit tests blackhole P150b:
models/tt_cnn/tests/test_builder.py#L168
test_conv2d[sharding_strategy0-kernel_config0-1-channel_config1-input_size0-device_params0]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for conv_bmm_tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/bcast.h:7,
from /work/ttnn/cpp/ttnn/operations/conv/conv2d/device/kernels/conv_bmm_tilize.cpp:8,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f2a21aeb168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f2a1f34c80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2a1f3632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2a1f36323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2a1f0f5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2a523f8ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2a1f0f5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2a1f36308e]
--- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&)
--- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&)
--- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const
--- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2a1f3632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2a1f36323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2a1f0f5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2a523f8ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2a1f0f5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2a1f36308e]
--- auto t
|
|
tt-cnn-unit-tests (P150b) / tt-cnn builder unit tests blackhole P150b:
models/tt_cnn/tests/test_builder.py#L168
test_conv2d[sharding_strategy0-kernel_config0-1-channel_config0-input_size1-device_params0]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for conv_bmm_tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/bcast.h:7,
from /work/ttnn/cpp/ttnn/operations/conv/conv2d/device/kernels/conv_bmm_tilize.cpp:8,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f2a21aeb168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f2a1f34c80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2a1f3632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2a1f36323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2a1f0f5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2a523f8ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2a1f0f5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2a1f36308e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f2a1f305d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f2a1f304890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f2a1f303d78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f2a4fb72253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f2a523f3ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f2a52484a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f2a1f252ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f2a1f232b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f2a1f2324cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2a1f3632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2
|
|
tt-cnn-unit-tests (P150b) / tt-cnn builder unit tests blackhole P150b:
models/tt_cnn/tests/test_builder.py#L168
test_conv2d[sharding_strategy0-kernel_config0-1-channel_config0-input_size0-device_params0]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for conv_bmm_tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/bcast.h:7,
from /work/ttnn/cpp/ttnn/operations/conv/conv2d/device/kernels/conv_bmm_tilize.cpp:8,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f2a21aeb168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f2a1f34c80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2a1f3632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2a1f36323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2a1f0f5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2a523f8ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2a1f0f5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2a1f36308e]
--- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&)
--- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&)
--- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const
--- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f2a1f3632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f2a1f36323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f2a1f0f5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f2a523f8ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f2a1f0f5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f2a1f36308e]
--- auto t
|
|
ttnn-unit-tests (P150b) / ttnn reduce and misc ops group blackhole P150b
Process completed with exit code 1.
|
|
ttnn-unit-tests (P150b) / ttnn reduce and misc ops group blackhole P150b:
tests/ttnn/unit_tests/operations/reduce/test_cumprod.py#L56
test_cumprod_normal[dtypes=(torch.float32, None)-shape=[2]-dim=0]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for transpose_xw_tiled TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/eltwise_unary/eltwise_unary.h:7,
from /work/ttnn/cpp/ttnn/operations/data_movement/permute/device/kernels/compute/transpose_xw_tiled.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f36136eb168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f3610f4c80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f3610f632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f3610f6323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f3610cf5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f3644059ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f3610cf5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f3610f6308e]
--- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&)
--- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&)
--- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const
--- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f3610f632b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f3610f6323e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f3610cf5def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f3644059ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f3610cf5c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f361
|
|
tt-cnn-unit-tests (P150b) / tt-cnn pipeline unit tests blackhole P150b:
models/tt_cnn/tests/test_pipeline.py#L257
test_executor_single_runs[device_params0-relu_model-shape_(1, 1, 512, 64)-MultiCQTracedModelPipelinedIOExecutor]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7,
from /work/ttnn/cpp/ttnn/kernel/compute/tilize.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f8ff8069168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f8ff58ca80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f8ff58e12b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f8ff58e123e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f8ff5673def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f902892dee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f8ff5673c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f8ff58e108e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f8ff5883d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f8ff5882890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f8ff5881d78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f90260a7253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f9028928ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f90289b9a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f8ff57d0ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f8ff57b0b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f8ff57b04cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f8ff58e12b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f8ff58e123e
|
|
tt-cnn-unit-tests (P150b) / tt-cnn pipeline unit tests blackhole P150b:
models/tt_cnn/tests/test_pipeline.py#L257
test_executor_single_runs[device_params0-relu_model-shape_(1, 1, 512, 64)-MultiCQTracedModelOverlappedInputExecutor]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7,
from /work/ttnn/cpp/ttnn/kernel/compute/tilize.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f8ff8069168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f8ff58ca80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f8ff58e12b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f8ff58e123e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f8ff5673def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f902892dee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f8ff5673c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f8ff58e108e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f8ff5883d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f8ff5882890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f8ff5881d78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f90260a7253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f9028928ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f90289b9a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f8ff57d0ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f8ff57b0b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f8ff57b04cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f8ff58e12b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f8ff58e
|
|
tt-cnn-unit-tests (P150b) / tt-cnn pipeline unit tests blackhole P150b:
models/tt_cnn/tests/test_pipeline.py#L257
test_executor_single_runs[device_params0-relu_model-shape_(1, 1, 512, 64)-MultiCQModelOverlappedInputExecutor]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7,
from /work/ttnn/cpp/ttnn/kernel/compute/tilize.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f8ff8069168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f8ff58ca80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f8ff58e12b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f8ff58e123e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f8ff5673def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f902892dee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f8ff5673c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f8ff58e108e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f8ff5883d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f8ff5882890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f8ff5881d78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f90260a7253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f9028928ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f90289b9a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f8ff57d0ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f8ff57b0b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f8ff57b04cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f8ff58e12b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f8ff58e123e]
|
|
tt-cnn-unit-tests (P150b) / tt-cnn pipeline unit tests blackhole P150b:
models/tt_cnn/tests/test_pipeline.py#L257
test_executor_single_runs[device_params0-relu_model-shape_(1, 1, 512, 64)-TracedModelExecutor]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7,
from /work/ttnn/cpp/ttnn/kernel/compute/tilize.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f8ff8069168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f8ff58ca80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f8ff58e12b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f8ff58e123e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f8ff5673def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f902892dee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f8ff5673c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f8ff58e108e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f8ff5883d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f8ff5882890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f8ff5881d78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f90260a7253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f9028928ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f90289b9a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f8ff57d0ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f8ff57b0b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f8ff57b04cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f8ff58e12b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f8ff58e123e]
--- /opt/venv/l
|
|
tt-cnn-unit-tests (P150b) / tt-cnn pipeline unit tests blackhole P150b:
models/tt_cnn/tests/test_pipeline.py#L257
test_executor_single_runs[device_params0-relu_model-shape_(1, 1, 512, 64)-ModelExecutor]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7,
from /work/ttnn/cpp/ttnn/kernel/compute/tilize.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f8ff8069168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f8ff58ca80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f8ff58e12b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f8ff58e123e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f8ff5673def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f902892dee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f8ff5673c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f8ff58e108e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f8ff5883d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f8ff5882890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f8ff5881d78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f90260a7253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f9028928ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f90289b9a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f8ff57d0ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f8ff57b0b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f8ff57b04cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f8ff58e12b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f8ff58e123e]
--- /opt/venv/lib/pyt
|
|
tt-cnn-unit-tests (P150b) / tt-cnn pipeline unit tests blackhole P150b:
models/tt_cnn/tests/test_pipeline.py#L257
test_executor_single_runs[device_params0-relu_model-shape_(1, 1, 32, 32)-MultiCQTracedModelPipelinedIOExecutor]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7,
from /work/ttnn/cpp/ttnn/kernel/compute/tilize.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f8ff8069168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f8ff58ca80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f8ff58e12b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f8ff58e123e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f8ff5673def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f902892dee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f8ff5673c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f8ff58e108e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f8ff5883d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f8ff5882890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f8ff5881d78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f90260a7253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f9028928ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f90289b9a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f8ff57d0ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f8ff57b0b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f8ff57b04cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f8ff58e12b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f8ff58e123e]
|
|
tt-cnn-unit-tests (P150b) / tt-cnn pipeline unit tests blackhole P150b:
models/tt_cnn/tests/test_pipeline.py#L257
test_executor_single_runs[device_params0-relu_model-shape_(1, 1, 32, 32)-MultiCQTracedModelOverlappedInputExecutor]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7,
from /work/ttnn/cpp/ttnn/kernel/compute/tilize.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f8ff8069168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f8ff58ca80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f8ff58e12b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f8ff58e123e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f8ff5673def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f902892dee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f8ff5673c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f8ff58e108e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f8ff5883d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f8ff5882890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f8ff5881d78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f90260a7253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f9028928ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f90289b9a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f8ff57d0ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f8ff57b0b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f8ff57b04cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f8ff58e12b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f8ff58e1
|
|
tt-cnn-unit-tests (P150b) / tt-cnn pipeline unit tests blackhole P150b:
models/tt_cnn/tests/test_pipeline.py#L257
test_executor_single_runs[device_params0-relu_model-shape_(1, 1, 32, 32)-MultiCQModelOverlappedInputExecutor]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7,
from /work/ttnn/cpp/ttnn/kernel/compute/tilize.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f8ff8069168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f8ff58ca80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f8ff58e12b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f8ff58e123e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f8ff5673def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f902892dee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f8ff5673c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f8ff58e108e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f8ff5883d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f8ff5882890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f8ff5881d78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f90260a7253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f9028928ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f90289b9a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f8ff57d0ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f8ff57b0b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f8ff57b04cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f8ff58e12b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f8ff58e123e]
|
|
tt-cnn-unit-tests (P150b) / tt-cnn pipeline unit tests blackhole P150b:
models/tt_cnn/tests/test_pipeline.py#L257
test_executor_single_runs[device_params0-relu_model-shape_(1, 1, 32, 32)-TracedModelExecutor]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7,
from /work/ttnn/cpp/ttnn/kernel/compute/tilize.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f8ff8069168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f8ff58ca80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f8ff58e12b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f8ff58e123e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f8ff5673def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f902892dee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f8ff5673c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f8ff58e108e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f8ff5883d85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f8ff5882890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f8ff5881d78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f90260a7253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f9028928ac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f90289b9a84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f8ff57d0ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f8ff57b0b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f8ff57b04cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f8ff58e12b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f8ff58e123e]
--- /opt/venv/li
|
|
tt-cnn-unit-tests (P150b) / tt-cnn pipeline unit tests blackhole P150b:
models/tt_cnn/tests/test_pipeline.py#L257
test_executor_single_runs[device_params0-relu_model-shape_(1, 1, 32, 32)-ModelExecutor]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7,
from /work/ttnn/cpp/ttnn/kernel/compute/tilize.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f8ff8069168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f8ff58ca80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f8ff58e12b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f8ff58e123e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f8ff5673def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f902892dee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f8ff5673c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f8ff58e108e]
--- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&)
--- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&)
--- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const
--- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f8ff58e12b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f8ff58e123e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f8ff5673def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f902892dee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f8ff5673c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f8ff58e108e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/
|
|
metalium-smoke-tests (P150b-viommu) / smoke
Failed tests were found and 'fail-on-error' option is set to true
|
|
metalium-smoke-tests (P150b-viommu) / smoke
Process completed with exit code 1.
|
|
blackhole-multi-card-fast-unit-tests (BH-LoudBox) / blackhole BH-LoudBox ttnn ops fast unit tests (ccl, DRAM prefetcher)
Process completed with exit code 139.
|
|
blackhole-multi-card-fast-unit-tests (BH-LoudBox) / blackhole BH-LoudBox ttnn ops fast unit tests (ccl, DRAM prefetcher):
models/demos/deepseek_v3_b1/tests/unit_tests/test_reduce_to_one_b1.py#L340
test_reduce_to_one_2d[silicon_arch_name=blackhole-silicon_arch_blackhole=True-fabric_2d]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for reduce_to_one_kernel TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/compute_kernel_api.h:20,
from /work/models/demos/deepseek_v3_b1/micro_ops/reduce_to_one_b1/kernels/../../../unified_kernels/reduce_to_one_b1.hpp:37,
from /work/models/demos/deepseek_v3_b1/micro_ops/reduce_to_one_b1/kernels/reduce_to_one_kernel.cpp:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7bea5cad2168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7bea5a33380c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7bea5a34a2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7bea5a34a23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7bea5a0dcdef]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7beb1dcf1ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7bea5a0dcc49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7bea5a34a08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7bea5a2ecd85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7bea5a2eb890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7bea5a2ead78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7beb1b22d253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7beb1dcecac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7beb1dd7da84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7bea5a239ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7bea5a219b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7bea5a2194cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7bea5a34a2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7bea5a34a23e]
--- /opt/venv/lib/python3
|
|
blackhole-multi-card-fast-unit-tests (BH-LoudBox) / blackhole BH-LoudBox ttnn ops fast unit tests (ccl, DRAM prefetcher):
tests/ttnn/unit_tests/operations/ccl/blackhole_CI/box/all_post_commit/test_all_gather_apc.py#L595
test_all_gather_failing_shapes[silicon_arch_name=blackhole-silicon_arch_blackhole=True-New all gather impl-num_buffers_per_channel=2-num_workers_per_link=2-chunks_per_sync=20-fabric_linear-non-trace-dram_only-float_16-tile-gemma dim 3-2_links]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for pack_untilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/untilize.h:7,
from /work/ttnn/cpp/ttnn/operations/data_movement/untilize/device/kernels/compute/pack_untilize.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7bea5cad2168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7bea5a33380c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7bea5a34a2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7bea5a34a23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7bea5a0dcdef]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7beb1dcf1ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7bea5a0dcc49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7bea5a34a08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7bea5a2ecd85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7bea5a2eb890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7bea5a2ead78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7beb1b22d253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7beb1dcecac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7beb1dd7da84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7bea5a239ed8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7bea5a219b51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7bea5a2194cb]
--- /opt/ven
|
|
run-profiler-regression (P150b) / profiler-regression:
tests/tt_metal/tools/profiler/test_device_profiler.py#L1511
test_get_programs_perf_data
subprocess.CalledProcessError: Command 'cd /work && TT_METAL_PROFILER_MID_RUN_DUMP=1 TT_METAL_PROFILER_CPP_POST_PROCESS=1 ./build/test/ttnn/tracy/test_get_programs_perf_data --gtest_filter=GetProgramsPerfDataFixture.TestGetProgramsPerfDataBeforeReadMeshDeviceProfilerResultsCall' returned non-zero exit status 139.
|
|
run-profiler-regression (P150b) / profiler-regression:
tests/tt_metal/tools/profiler/test_device_profiler.py#L1149
test_timestamped_events
assert 34304 == 0
|
|
run-profiler-regression (P150b) / profiler-regression:
tests/tt_metal/tools/profiler/test_device_profiler.py#L1098
test_profiler_host_device_sync
assert 256 == 0
|
|
run-profiler-regression (P150b) / profiler-regression:
tests/ttnn/tracy/test_profiler_sync.py#L23
test_with_ops
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for bmm_large_block_zm_fused_bias_activation TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/matmul.h:7,
from /work/ttnn/cpp/ttnn/operations/matmul/device/kernels/compute/bmm_large_block_zm_fused_bias_activation.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f4e161f5168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f4e13a5680c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f4e13a6d2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f4e13a6d23e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f4e137ffdef]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f4e46a91ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f4e137ffc49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f4e13a6d08e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77dd85) [0x7f4e13a0fd85]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77c890) [0x7f4e13a0e890]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x77bd78) [0x7f4e13a0dd78]
--- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f4e441ff253]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f4e46a8cac3]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f4e46b1da84]
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6caed8) [0x7f4e1395ced8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aab51) [0x7f4e1393cb51]
--- tt::tt_metal::JitBuildCache::build_once(unsigned long, std::function<void ()> const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x6aa4cb) [0x7f4e1393c4cb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f4e13a6d2b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f4e13a6d23e]
--
|
|
run-profiler-regression (P150b) / profiler-regression:
tests/tt_metal/tools/profiler/test_device_profiler.py#L800
test_device_trace_run
assert 256 == 0
|
|
run-profiler-regression (P150b) / profiler-regression:
tests/ttnn/tracy/test_trace_runs.py#L30
test_with_ops[device_params={'trace_region_size': 1996800, 'dispatch_core_type': DispatchCoreType.WORKER}]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7,
from /work/ttnn/cpp/ttnn/kernel/compute/tilize.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f0b2bada168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f0b2933b80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f0b293522b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f0b2935223e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f0b290e4def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f0b5c278ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f0b290e4c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f0b2935208e]
--- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&)
--- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&)
--- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const
--- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f0b293522b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f0b2935223e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f0b290e4def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f0b5c278ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f0b290e4c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f0b2935208e]
--- /opt/venv/lib/python3.10/site-p
|
|
run-profiler-regression (P150b) / profiler-regression:
tests/tt_metal/tools/profiler/test_device_profiler.py#L780
test_trace_run
assert 256 == 0
|
|
run-profiler-regression (P150b) / profiler-regression:
tests/ttnn/tracy/test_trace_runs.py#L103
test_with_ops_multiple_trace_ids[device_params={'trace_region_size': 1996800, 'dispatch_core_type': DispatchCoreType.WORKER}]
RuntimeError: TT_THROW @ /project/tt_metal/impl/program/program.cpp:155: tt::exception
info:
Failed to generate binaries for tilize TT_THROW @ /project/tt_metal/jit_build/build.cpp:58: tt::exception
info:
trisc1 build failed. Log: In file included from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_exp.h:11,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_activations.h:13,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel_sfpu.h:14,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/cmath_common.h:12,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_math_common_api.h:10,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common_globals.h:16,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/common.h:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/inc/api/compute/tilize.h:7,
from /work/ttnn/cpp/ttnn/kernel/compute/tilize.cpp:7,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/ckernels/blackhole/metal/common/chlkc_list.h:22,
from /opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/hw/firmware/src/tt-1xx/trisck.cc:13:
/opt/venv/lib/python3.10/site-packages/ttnn/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/sfpu/ckernel_sfpu_recip.h:13:10: fatal error: vconst_verifier.h: No such file or directory
13 | #include "vconst_verifier.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
backtrace:
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/_ttnncpp.so(+0x1f16168) [0x7f5bfcedd168]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7c480c) [0x7f5bfa73e80c]
--- tt::tt_metal::JitBuildState::compile_one(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*, unsigned long) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f5bfa7552b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f5bfa75523e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f5bfa4e7def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f5c2d782ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f5bfa4e7c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f5bfa75508e]
--- auto tt::tt_metal::detail::async<std::function<void ()> const&>(std::function<void ()> const&)
--- tt::tt_metal::launch_build_step(std::function<void ()> const&, std::vector<std::shared_future<void>, std::allocator<std::shared_future<void> > >&)
--- tt::tt_metal::JitBuildState::compile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tt::tt_metal::JitBuildSettings const*) const
--- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db2b8) [0x7f5bfa7552b8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db23e) [0x7f5bfa75523e]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56ddef) [0x7f5bfa4e7def]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f5c2d782ee8]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x56dc49) [0x7f5bfa4e7c49]
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x7db08e) [0x7f5bfa75508e]
--- /opt/venv/li
|
|
run-profiler-regression (P150b) / profiler-regression:
tests/tt_metal/tools/profiler/test_device_profiler.py#L270
test_custom_cycle_count
assert 34304 == 0
|
|
run-profiler-regression (P150b) / profiler-regression:
tests/tt_metal/tools/profiler/test_device_profiler.py#L248
test_custom_cycle_count_slow_dispatch
assert 34304 == 0
|
|
blackhole-multi-card-fast-unit-tests (BH-LLMBox) / blackhole BH-LLMBox Sockets fast unit test:
models/demos/deepseek_v3_b1/tests/unit_tests/test_host_io.py#L139
test_host_io_loopback_with_embedding[blackhole-256-4-64-14336-H2DMode.HOST_PUSH]
RuntimeError: TT_THROW @ /project/tt_metal/third_party/umd/device/pcie/pci_device.cpp:668: tt::exception
info:
Failed to pin pages for DMA buffer at virtual address 0x7f2ae017c000 with size 0x1d000 and flags 0x1: Invalid argument
backtrace:
--- tt::umd::PCIDevice::map_for_dma(void*, unsigned long)
--- tt::umd::SysmemBuffer::SysmemBuffer(tt::umd::TLBManager*, void*, unsigned long, bool)
--- tt::umd::SiliconSysmemManager::map_sysmem_buffer(void*, unsigned long, bool)
--- tt::Cluster::map_sysmem_buffer(int, void*, unsigned long, bool) const
--- tt::tt_metal::experimental::PinnedMemoryImpl::initialize_from_devices(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemoryImpl::PinnedMemoryImpl(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::PinnedMemory(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::Create(tt::tt_metal::distributed::MeshDevice&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, tt::tt_metal::HostBuffer&, bool)
--- tt::tt_metal::distributed::D2HSocket::init_host_buffer(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, unsigned int)
--- tt::tt_metal::distributed::D2HSocket::D2HSocket(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoreCoord const&, unsigned int)
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x33794b) [0x7f2b744cc94b]
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x23ee65) [0x7f2b743d3e65]
--- /opt/venv/bin/python3(+0x2a5b30) [0x56274b5afb30]
--- /opt/venv/bin/python3(+0x2a7125) [0x56274b5b1125]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x12f3e) [0x56274b5e364e]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x36749) [0x56274b606e59]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x56274b5d8c17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(+0x273350) [0x56274b57d350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x56274b678c65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x56274b6acb46]
--- /opt/venv/bin/python3(+0x2fd2a2) [0x56274b6072a2]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x3eef) [0x56274b5d45ff]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8924) [0x56274b5d9034]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x56274b5d8c17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(+0x273350) [0x56274b57d350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x56274b678c65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x56274b6acb46]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa16c) [0x56274b5da87c]
--- /opt/ven
|
|
blackhole-multi-card-fast-unit-tests (BH-LLMBox) / blackhole BH-LLMBox Sockets fast unit test:
models/demos/deepseek_v3_b1/tests/unit_tests/test_host_io.py#L139
test_host_io_loopback_with_embedding[blackhole-128-2-512-1792-H2DMode.DEVICE_PULL]
RuntimeError: TT_THROW @ /project/tt_metal/third_party/umd/device/pcie/pci_device.cpp:668: tt::exception
info:
Failed to pin pages for DMA buffer at virtual address 0x7f2b52a05000 with size 0x2000 and flags 0x1: Invalid argument
backtrace:
--- tt::umd::PCIDevice::map_for_dma(void*, unsigned long)
--- tt::umd::SysmemBuffer::SysmemBuffer(tt::umd::TLBManager*, void*, unsigned long, bool)
--- tt::umd::SiliconSysmemManager::map_sysmem_buffer(void*, unsigned long, bool)
--- tt::Cluster::map_sysmem_buffer(int, void*, unsigned long, bool) const
--- tt::tt_metal::experimental::PinnedMemoryImpl::initialize_from_devices(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemoryImpl::PinnedMemoryImpl(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::PinnedMemory(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::Create(tt::tt_metal::distributed::MeshDevice&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, tt::tt_metal::HostBuffer&, bool)
--- tt::tt_metal::distributed::D2HSocket::init_host_buffer(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, unsigned int)
--- tt::tt_metal::distributed::D2HSocket::D2HSocket(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoreCoord const&, unsigned int)
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x33794b) [0x7f2b744cc94b]
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x23ee65) [0x7f2b743d3e65]
--- /opt/venv/bin/python3(+0x2a5b30) [0x56274b5afb30]
--- /opt/venv/bin/python3(+0x2a7125) [0x56274b5b1125]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x12f3e) [0x56274b5e364e]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x36749) [0x56274b606e59]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x56274b5d8c17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(+0x273350) [0x56274b57d350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x56274b678c65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x56274b6acb46]
--- /opt/venv/bin/python3(+0x2fd2a2) [0x56274b6072a2]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x3eef) [0x56274b5d45ff]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8924) [0x56274b5d9034]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x56274b5d8c17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(+0x273350) [0x56274b57d350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x56274b678c65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x56274b6acb46]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa16c) [0x56274b5da87c]
--- /opt/ve
|
|
blackhole-multi-card-fast-unit-tests (BH-LLMBox) / blackhole BH-LLMBox Sockets fast unit test:
models/demos/deepseek_v3_b1/tests/unit_tests/test_host_io.py#L139
test_host_io_loopback_with_embedding[blackhole-128-2-512-1792-H2DMode.HOST_PUSH]
RuntimeError: TT_THROW @ /project/tt_metal/third_party/umd/device/pcie/pci_device.cpp:668: tt::exception
info:
Failed to pin pages for DMA buffer at virtual address 0x7f2b50000000 with size 0x2000 and flags 0x1: Invalid argument
backtrace:
--- tt::umd::PCIDevice::map_for_dma(void*, unsigned long)
--- tt::umd::SysmemBuffer::SysmemBuffer(tt::umd::TLBManager*, void*, unsigned long, bool)
--- tt::umd::SiliconSysmemManager::map_sysmem_buffer(void*, unsigned long, bool)
--- tt::Cluster::map_sysmem_buffer(int, void*, unsigned long, bool) const
--- tt::tt_metal::experimental::PinnedMemoryImpl::initialize_from_devices(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemoryImpl::PinnedMemoryImpl(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::PinnedMemory(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::Create(tt::tt_metal::distributed::MeshDevice&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, tt::tt_metal::HostBuffer&, bool)
--- tt::tt_metal::distributed::D2HSocket::init_host_buffer(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, unsigned int)
--- tt::tt_metal::distributed::D2HSocket::D2HSocket(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoreCoord const&, unsigned int)
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x33794b) [0x7f2b744cc94b]
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x23ee65) [0x7f2b743d3e65]
--- /opt/venv/bin/python3(+0x2a5b30) [0x56274b5afb30]
--- /opt/venv/bin/python3(+0x2a7125) [0x56274b5b1125]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x12f3e) [0x56274b5e364e]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x36749) [0x56274b606e59]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x56274b5d8c17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(+0x273350) [0x56274b57d350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x56274b678c65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x56274b6acb46]
--- /opt/venv/bin/python3(+0x2fd2a2) [0x56274b6072a2]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x3eef) [0x56274b5d45ff]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8924) [0x56274b5d9034]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x56274b5d8c17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(+0x273350) [0x56274b57d350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x56274b678c65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x56274b6acb46]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa16c) [0x56274b5da87c]
--- /opt/venv
|
|
blackhole-multi-card-fast-unit-tests (BH-LLMBox) / blackhole BH-LLMBox Sockets fast unit test:
models/demos/deepseek_v3_b1/tests/unit_tests/test_host_io.py#L50
test_host_io_loopback[blackhole-H2DMode.HOST_PUSH-32768-65536-128]
RuntimeError: TT_THROW @ /project/tt_metal/third_party/umd/device/pcie/pci_device.cpp:668: tt::exception
info:
Failed to pin pages for DMA buffer at virtual address 0x7f2ae0188000 with size 0x11000 and flags 0x1: Invalid argument
backtrace:
--- tt::umd::PCIDevice::map_for_dma(void*, unsigned long)
--- tt::umd::SysmemBuffer::SysmemBuffer(tt::umd::TLBManager*, void*, unsigned long, bool)
--- tt::umd::SiliconSysmemManager::map_sysmem_buffer(void*, unsigned long, bool)
--- tt::Cluster::map_sysmem_buffer(int, void*, unsigned long, bool) const
--- tt::tt_metal::experimental::PinnedMemoryImpl::initialize_from_devices(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemoryImpl::PinnedMemoryImpl(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::PinnedMemory(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::Create(tt::tt_metal::distributed::MeshDevice&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, tt::tt_metal::HostBuffer&, bool)
--- tt::tt_metal::distributed::D2HSocket::init_host_buffer(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, unsigned int)
--- tt::tt_metal::distributed::D2HSocket::D2HSocket(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoreCoord const&, unsigned int)
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x33794b) [0x7f2b744cc94b]
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x23ee65) [0x7f2b743d3e65]
--- /opt/venv/bin/python3(+0x2a5b30) [0x56274b5afb30]
--- /opt/venv/bin/python3(+0x2a7125) [0x56274b5b1125]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x12f3e) [0x56274b5e364e]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x36749) [0x56274b606e59]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x56274b5d8c17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(+0x273350) [0x56274b57d350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x56274b678c65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x56274b6acb46]
--- /opt/venv/bin/python3(+0x2fd2a2) [0x56274b6072a2]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x3eef) [0x56274b5d45ff]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8924) [0x56274b5d9034]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x56274b5d8c17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(+0x273350) [0x56274b57d350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x56274b678c65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x56274b6acb46]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa16c) [0x56274b5da87c]
--- /opt/venv/bin/python3(
|
|
blackhole-multi-card-fast-unit-tests (BH-LLMBox) / blackhole BH-LLMBox Sockets fast unit test:
models/demos/deepseek_v3_b1/tests/unit_tests/test_host_io.py#L139
test_host_io_loopback_with_embedding[blackhole-128-2-256-3584-H2DMode.DEVICE_PULL]
RuntimeError: TT_THROW @ /project/tt_metal/third_party/umd/device/pcie/pci_device.cpp:668: tt::exception
info:
Failed to pin pages for DMA buffer at virtual address 0x7f2b4aa46000 with size 0x4000 and flags 0x1: Invalid argument
backtrace:
--- tt::umd::PCIDevice::map_for_dma(void*, unsigned long)
--- tt::umd::SysmemBuffer::SysmemBuffer(tt::umd::TLBManager*, void*, unsigned long, bool)
--- tt::umd::SiliconSysmemManager::map_sysmem_buffer(void*, unsigned long, bool)
--- tt::Cluster::map_sysmem_buffer(int, void*, unsigned long, bool) const
--- tt::tt_metal::experimental::PinnedMemoryImpl::initialize_from_devices(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemoryImpl::PinnedMemoryImpl(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::PinnedMemory(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::Create(tt::tt_metal::distributed::MeshDevice&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, tt::tt_metal::HostBuffer&, bool)
--- tt::tt_metal::distributed::D2HSocket::init_host_buffer(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, unsigned int)
--- tt::tt_metal::distributed::D2HSocket::D2HSocket(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoreCoord const&, unsigned int)
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x33794b) [0x7f2b744cc94b]
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x23ee65) [0x7f2b743d3e65]
--- /opt/venv/bin/python3(+0x2a5b30) [0x56274b5afb30]
--- /opt/venv/bin/python3(+0x2a7125) [0x56274b5b1125]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x12f3e) [0x56274b5e364e]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x36749) [0x56274b606e59]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x56274b5d8c17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(+0x273350) [0x56274b57d350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x56274b678c65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x56274b6acb46]
--- /opt/venv/bin/python3(+0x2fd2a2) [0x56274b6072a2]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x3eef) [0x56274b5d45ff]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8924) [0x56274b5d9034]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x56274b5d8c17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(+0x273350) [0x56274b57d350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x56274b678c65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x56274b6acb46]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa16c) [0x56274b5da87c]
--- /opt/ve
|
|
blackhole-multi-card-fast-unit-tests (BH-LLMBox) / blackhole BH-LLMBox Sockets fast unit test:
models/demos/deepseek_v3_b1/tests/unit_tests/test_host_io.py#L139
test_host_io_loopback_with_embedding[blackhole-128-2-256-3584-H2DMode.HOST_PUSH]
RuntimeError: TT_THROW @ /project/tt_metal/third_party/umd/device/pcie/pci_device.cpp:668: tt::exception
info:
Failed to pin pages for DMA buffer at virtual address 0x7f2b4aa46000 with size 0x4000 and flags 0x1: Invalid argument
backtrace:
--- tt::umd::PCIDevice::map_for_dma(void*, unsigned long)
--- tt::umd::SysmemBuffer::SysmemBuffer(tt::umd::TLBManager*, void*, unsigned long, bool)
--- tt::umd::SiliconSysmemManager::map_sysmem_buffer(void*, unsigned long, bool)
--- tt::Cluster::map_sysmem_buffer(int, void*, unsigned long, bool) const
--- tt::tt_metal::experimental::PinnedMemoryImpl::initialize_from_devices(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemoryImpl::PinnedMemoryImpl(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::PinnedMemory(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::Create(tt::tt_metal::distributed::MeshDevice&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, tt::tt_metal::HostBuffer&, bool)
--- tt::tt_metal::distributed::D2HSocket::init_host_buffer(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, unsigned int)
--- tt::tt_metal::distributed::D2HSocket::D2HSocket(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoreCoord const&, unsigned int)
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x33794b) [0x7f2b744cc94b]
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x23ee65) [0x7f2b743d3e65]
--- /opt/venv/bin/python3(+0x2a5b30) [0x56274b5afb30]
--- /opt/venv/bin/python3(+0x2a7125) [0x56274b5b1125]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x12f3e) [0x56274b5e364e]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x36749) [0x56274b606e59]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x56274b5d8c17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(+0x273350) [0x56274b57d350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x56274b678c65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x56274b6acb46]
--- /opt/venv/bin/python3(+0x2fd2a2) [0x56274b6072a2]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x3eef) [0x56274b5d45ff]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8924) [0x56274b5d9034]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x56274b5d8c17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(+0x273350) [0x56274b57d350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x56274b678c65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x56274b6acb46]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa16c) [0x56274b5da87c]
--- /opt/venv
|
|
blackhole-multi-card-fast-unit-tests (BH-LLMBox) / blackhole BH-LLMBox Sockets fast unit test:
models/demos/deepseek_v3_b1/tests/unit_tests/test_host_io.py#L139
test_host_io_loopback_with_embedding[blackhole-128-2-128-7168-H2DMode.DEVICE_PULL]
RuntimeError: TT_THROW @ /project/tt_metal/third_party/umd/device/pcie/pci_device.cpp:668: tt::exception
info:
Failed to pin pages for DMA buffer at virtual address 0x7f2b4aa42000 with size 0x8000 and flags 0x1: Invalid argument
backtrace:
--- tt::umd::PCIDevice::map_for_dma(void*, unsigned long)
--- tt::umd::SysmemBuffer::SysmemBuffer(tt::umd::TLBManager*, void*, unsigned long, bool)
--- tt::umd::SiliconSysmemManager::map_sysmem_buffer(void*, unsigned long, bool)
--- tt::Cluster::map_sysmem_buffer(int, void*, unsigned long, bool) const
--- tt::tt_metal::experimental::PinnedMemoryImpl::initialize_from_devices(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemoryImpl::PinnedMemoryImpl(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::PinnedMemory(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::Create(tt::tt_metal::distributed::MeshDevice&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, tt::tt_metal::HostBuffer&, bool)
--- tt::tt_metal::distributed::D2HSocket::init_host_buffer(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, unsigned int)
--- tt::tt_metal::distributed::D2HSocket::D2HSocket(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoreCoord const&, unsigned int)
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x33794b) [0x7f2b744cc94b]
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x23ee65) [0x7f2b743d3e65]
--- /opt/venv/bin/python3(+0x2a5b30) [0x56274b5afb30]
--- /opt/venv/bin/python3(+0x2a7125) [0x56274b5b1125]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x12f3e) [0x56274b5e364e]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x36749) [0x56274b606e59]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x56274b5d8c17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(+0x273350) [0x56274b57d350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x56274b678c65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x56274b6acb46]
--- /opt/venv/bin/python3(+0x2fd2a2) [0x56274b6072a2]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x3eef) [0x56274b5d45ff]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8924) [0x56274b5d9034]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x56274b5d8c17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(+0x273350) [0x56274b57d350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x56274b678c65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x56274b6acb46]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa16c) [0x56274b5da87c]
--- /opt/ve
|
|
blackhole-multi-card-fast-unit-tests (BH-LLMBox) / blackhole BH-LLMBox Sockets fast unit test:
models/demos/deepseek_v3_b1/tests/unit_tests/test_host_io.py#L139
test_host_io_loopback_with_embedding[blackhole-128-2-128-7168-H2DMode.HOST_PUSH]
RuntimeError: TT_THROW @ /project/tt_metal/third_party/umd/device/pcie/pci_device.cpp:668: tt::exception
info:
Failed to pin pages for DMA buffer at virtual address 0x7f2b4aa42000 with size 0x8000 and flags 0x1: Invalid argument
backtrace:
--- tt::umd::PCIDevice::map_for_dma(void*, unsigned long)
--- tt::umd::SysmemBuffer::SysmemBuffer(tt::umd::TLBManager*, void*, unsigned long, bool)
--- tt::umd::SiliconSysmemManager::map_sysmem_buffer(void*, unsigned long, bool)
--- tt::Cluster::map_sysmem_buffer(int, void*, unsigned long, bool) const
--- tt::tt_metal::experimental::PinnedMemoryImpl::initialize_from_devices(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemoryImpl::PinnedMemoryImpl(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::PinnedMemory(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::Create(tt::tt_metal::distributed::MeshDevice&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, tt::tt_metal::HostBuffer&, bool)
--- tt::tt_metal::distributed::D2HSocket::init_host_buffer(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, unsigned int)
--- tt::tt_metal::distributed::D2HSocket::D2HSocket(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoreCoord const&, unsigned int)
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x33794b) [0x7f2b744cc94b]
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x23ee65) [0x7f2b743d3e65]
--- /opt/venv/bin/python3(+0x2a5b30) [0x56274b5afb30]
--- /opt/venv/bin/python3(+0x2a7125) [0x56274b5b1125]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x12f3e) [0x56274b5e364e]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x36749) [0x56274b606e59]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x56274b5d8c17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(+0x273350) [0x56274b57d350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x56274b678c65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x56274b6acb46]
--- /opt/venv/bin/python3(+0x2fd2a2) [0x56274b6072a2]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x3eef) [0x56274b5d45ff]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8924) [0x56274b5d9034]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x56274b5d8c17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(+0x273350) [0x56274b57d350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x56274b678c65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x56274b6acb46]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa16c) [0x56274b5da87c]
--- /opt/venv
|
|
blackhole-multi-card-fast-unit-tests (BH-LLMBox) / blackhole BH-LLMBox Sockets fast unit test:
models/demos/deepseek_v3_b1/tests/unit_tests/test_host_io.py#L139
test_host_io_loopback_with_embedding[blackhole-128-2-64-14336-H2DMode.DEVICE_PULL]
RuntimeError: TT_THROW @ /project/tt_metal/third_party/umd/device/pcie/pci_device.cpp:668: tt::exception
info:
Failed to pin pages for DMA buffer at virtual address 0x7f2ae018a000 with size 0xf000 and flags 0x1: Invalid argument
backtrace:
--- tt::umd::PCIDevice::map_for_dma(void*, unsigned long)
--- tt::umd::SysmemBuffer::SysmemBuffer(tt::umd::TLBManager*, void*, unsigned long, bool)
--- tt::umd::SiliconSysmemManager::map_sysmem_buffer(void*, unsigned long, bool)
--- tt::Cluster::map_sysmem_buffer(int, void*, unsigned long, bool) const
--- tt::tt_metal::experimental::PinnedMemoryImpl::initialize_from_devices(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemoryImpl::PinnedMemoryImpl(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::PinnedMemory(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::Create(tt::tt_metal::distributed::MeshDevice&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, tt::tt_metal::HostBuffer&, bool)
--- tt::tt_metal::distributed::D2HSocket::init_host_buffer(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, unsigned int)
--- tt::tt_metal::distributed::D2HSocket::D2HSocket(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoreCoord const&, unsigned int)
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x33794b) [0x7f2b744cc94b]
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x23ee65) [0x7f2b743d3e65]
--- /opt/venv/bin/python3(+0x2a5b30) [0x56274b5afb30]
--- /opt/venv/bin/python3(+0x2a7125) [0x56274b5b1125]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x12f3e) [0x56274b5e364e]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x36749) [0x56274b606e59]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x56274b5d8c17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(+0x273350) [0x56274b57d350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x56274b678c65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x56274b6acb46]
--- /opt/venv/bin/python3(+0x2fd2a2) [0x56274b6072a2]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x3eef) [0x56274b5d45ff]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8924) [0x56274b5d9034]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x56274b5d8c17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(+0x273350) [0x56274b57d350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x56274b678c65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x56274b6acb46]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa16c) [0x56274b5da87c]
--- /opt/ve
|
|
blackhole-multi-card-fast-unit-tests (BH-LLMBox) / blackhole BH-LLMBox Sockets fast unit test:
models/demos/deepseek_v3_b1/tests/unit_tests/test_host_io.py#L139
test_host_io_loopback_with_embedding[blackhole-128-2-64-14336-H2DMode.HOST_PUSH]
RuntimeError: TT_THROW @ /project/tt_metal/third_party/umd/device/pcie/pci_device.cpp:668: tt::exception
info:
Failed to pin pages for DMA buffer at virtual address 0x7f2ae018a000 with size 0xf000 and flags 0x1: Invalid argument
backtrace:
--- tt::umd::PCIDevice::map_for_dma(void*, unsigned long)
--- tt::umd::SysmemBuffer::SysmemBuffer(tt::umd::TLBManager*, void*, unsigned long, bool)
--- tt::umd::SiliconSysmemManager::map_sysmem_buffer(void*, unsigned long, bool)
--- tt::Cluster::map_sysmem_buffer(int, void*, unsigned long, bool) const
--- tt::tt_metal::experimental::PinnedMemoryImpl::initialize_from_devices(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemoryImpl::PinnedMemoryImpl(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::PinnedMemory(std::vector<tt::tt_metal::IDevice*, std::allocator<tt::tt_metal::IDevice*> > const&, void*, unsigned long, bool)
--- tt::tt_metal::experimental::PinnedMemory::Create(tt::tt_metal::distributed::MeshDevice&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, tt::tt_metal::HostBuffer&, bool)
--- tt::tt_metal::distributed::D2HSocket::init_host_buffer(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoordinateRangeSet const&, unsigned int)
--- tt::tt_metal::distributed::D2HSocket::D2HSocket(std::shared_ptr<tt::tt_metal::distributed::MeshDevice> const&, tt::tt_metal::distributed::MeshCoreCoord const&, unsigned int)
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x33794b) [0x7f2b744cc94b]
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x23ee65) [0x7f2b743d3e65]
--- /opt/venv/bin/python3(+0x2a5b30) [0x56274b5afb30]
--- /opt/venv/bin/python3(+0x2a7125) [0x56274b5b1125]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x12f3e) [0x56274b5e364e]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x36749) [0x56274b606e59]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x56274b5d8c17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(+0x273350) [0x56274b57d350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x56274b678c65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x56274b6acb46]
--- /opt/venv/bin/python3(+0x2fd2a2) [0x56274b6072a2]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x3eef) [0x56274b5d45ff]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8924) [0x56274b5d9034]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x8507) [0x56274b5d8c17]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(+0x273350) [0x56274b57d350]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa3df) [0x56274b5daaef]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x202) [0x56274b57b352]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x115) [0x56274b678c65]
--- /opt/venv/bin/python3(+0x3a2b46) [0x56274b6acb46]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa16c) [0x56274b5da87c]
--- /opt/venv
|
|
models-unit-tests (P150) / model blackhole P150:
/work/models/demos/vision/generative/stable_diffusion/wormhole/custom_preprocessing.py#L33
The use of `x.T` on tensors of dimension other than 2 to reverse their shape is deprecated and it will throw an error in a future release. Consider `x.mT` to transpose batches of matrices or `x.permute(*torch.arange(x.ndim - 1, -1, -1))` to reverse the dimensions of a tensor. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4413.)
|
|
hugepages-service-not-found-startup
Hugepages service not found. Using old rc.local method
|
|
mlperf-mount-service-not-found-startup
MLPerf mount service not found.
|
|
blackhole-multi-card-fast-unit-tests (P300-viommu) / blackhole P300-viommu ttnn ops fast unit tests (ccl, DRAM prefetcher):
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
|
blackhole-multi-card-fast-unit-tests (P300-viommu) / blackhole P300-viommu ttnn ops fast unit tests (ccl, DRAM prefetcher):
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132
Field "model_name" in OpTest has conflict with protected namespace "model_".
You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
|
|
blackhole-multi-card-fast-unit-tests (P300-viommu) / blackhole P300-viommu ttnn ops fast unit tests (ccl, DRAM prefetcher):
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
|
blackhole-multi-card-fast-unit-tests (P300-viommu) / blackhole P300-viommu ttnn ops fast unit tests (ccl, DRAM prefetcher):
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
|
mlperf-mount-service-not-found-startup
MLPerf mount service not found.
|
|
metalium-basic-tests (P150b-viommu) / metalium-basic
No files were found with the provided path: /work/test-reports/. No artifacts will be uploaded.
|
|
ttnn-smoke-tests (P150b-viommu) / smoke
No files were found with the provided path: /work/test-reports/. No artifacts will be uploaded.
|
|
ttnn-smoke-tests (P150b-viommu) / smoke
No file matches path /work/test-reports/*.xml
|
|
ttnn-unit-tests (P150b) / ttnn sdpa group blackhole P150b:
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
|
blackhole-multi-card-fast-unit-tests (BH-LLMBox) / blackhole BH-LLMBox ttnn ops fast unit tests (ccl, DRAM prefetcher):
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
|
blackhole-multi-card-fast-unit-tests (BH-LLMBox) / blackhole BH-LLMBox ttnn ops fast unit tests (ccl, DRAM prefetcher):
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132
Field "model_name" in OpTest has conflict with protected namespace "model_".
You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
|
|
blackhole-multi-card-fast-unit-tests (BH-LLMBox) / blackhole BH-LLMBox ttnn ops fast unit tests (ccl, DRAM prefetcher):
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
|
blackhole-multi-card-fast-unit-tests (BH-LLMBox) / blackhole BH-LLMBox ttnn ops fast unit tests (ccl, DRAM prefetcher):
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
|
ttnn-basic-tests (P150b-viommu) / metalium-basic
No files were found with the provided path: /work/test-reports/. No artifacts will be uploaded.
|
|
ttnn-unit-tests (P150b) / ttnn transformers group blackhole P150b:
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
|
ops-unit-tests (P150b) / blackhole deepseek blitz op tests blackhole P150b
No files were found with the provided path: /work/generated/test_reports/. No artifacts will be uploaded.
|
|
ops-unit-tests (P150b) / blackhole deepseek blitz op tests blackhole P150b:
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132
Field "model_name" in OpTest has conflict with protected namespace "model_".
You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
|
|
ops-unit-tests (P150b) / blackhole deepseek blitz op tests blackhole P150b:
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
|
ops-unit-tests (P150b) / blackhole deepseek blitz op tests blackhole P150b:
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
|
tt-cnn-unit-tests (P150b) / tt-cnn builder unit tests blackhole P150b
No files were found with the provided path: /work/generated/test_reports/. No artifacts will be uploaded.
|
|
tt-cnn-unit-tests (P150b) / tt-cnn pipeline unit tests blackhole P150b:
/work/models/tt_cnn/tests/test_pipeline.py#L27
cannot collect test class 'TestShapeConfig' because it has a __init__ constructor (from: models/tt_cnn/tests/test_pipeline.py)
|
|
blackhole-multi-card-fast-unit-tests (BH-LoudBox) / blackhole BH-LoudBox ttnn ops fast unit tests (ccl, DRAM prefetcher):
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
|
blackhole-multi-card-fast-unit-tests (BH-LoudBox) / blackhole BH-LoudBox ttnn ops fast unit tests (ccl, DRAM prefetcher):
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_fields.py#L132
Field "model_name" in OpTest has conflict with protected namespace "model_".
You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
|
|
blackhole-multi-card-fast-unit-tests (BH-LoudBox) / blackhole BH-LoudBox ttnn ops fast unit tests (ccl, DRAM prefetcher):
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
|
blackhole-multi-card-fast-unit-tests (BH-LoudBox) / blackhole BH-LoudBox ttnn ops fast unit tests (ccl, DRAM prefetcher):
/opt/venv/lib/python3.10/site-packages/pydantic/_internal/_config.py#L291
Support for class-based `config` is deprecated, use ConfigDict instead.
|
|
mlperf-mount-service-not-found-startup
MLPerf mount service not found.
|
Artifacts
Produced during runtime
| Name | Size | Digest | |
|---|---|---|---|
|
test_reports_16b6a197-fac1-4d88-9cb1-4e4735e16ffb
|
3.05 KB |
sha256:771c1ac03f461a3a570c3487720def7033720ee9124d41c7c8bae14ad25c9c3a
|
|
|
test_reports_24c2a34f-a80b-4417-bde1-c766a17b645d
|
2.64 KB |
sha256:5074ad41d4f01c6c70f7ba3bfc75974da43a88cd254c642b7080a6846d27d642
|
|
|
test_reports_267c5664-6a73-4d2e-81c1-5299a2292ba4
|
12.5 KB |
sha256:63aade2e25d30319d2a2087fc9291812b306151b02415ce72360bc0f59174c25
|
|
|
test_reports_3abe2a58-9687-4674-a4f7-43c60463ff50
|
2.45 KB |
sha256:232c39c52e00bb084693ef0ba5ec61820f2778b3049992116efbce9736d17cfc
|
|
|
test_reports_3ba81260-f1b9-4ec6-9369-e60ecb9b41e6
|
483 Bytes |
sha256:5abb5d24dea525162c9df694ff7d87c13e4639b5583f9190096e90f40c499e53
|
|
|
test_reports_3d045822-e7b4-4e46-8d3c-15d7b261da06
|
3.73 KB |
sha256:0a3281232e2f7357a45897e56548463dd2574b44fbab2651b985eeb8391b3f74
|
|
|
test_reports_48d76e92-9b15-42aa-a5d1-16eb11591636
|
3.75 KB |
sha256:525c44a806a8ba1fe125104c3915a54099ab6f92f46d3181ab075ba5a14e46fa
|
|
|
test_reports_64a42033-9295-4452-a64c-e767a53475bb
|
4.08 KB |
sha256:6890d1b02147e9155b9504943c78ce9e4c42391c2de0a496a220f49496a499c0
|
|
|
test_reports_6a70da82-0ad1-4305-861f-bd52d875b555
|
18.2 KB |
sha256:bf4048bdca20e1fd622758e70870135ee5df93f9863dbf16054c14f3a69eb25f
|
|
|
test_reports_7e93ac09-4ba2-4985-81c4-03876e28f0d0
|
11.1 KB |
sha256:f7cefa33e91f5f898b237c38e176e45de94f17ad58a32267f71e8f577ca720e3
|
|
|
test_reports_7fd7a629-179b-44fa-87ba-a34183164434
|
3.41 KB |
sha256:dfe32df8a81e32593c4faec218a2ae78f27a25c6045215c4cf1489a14ce0ff56
|
|
|
test_reports_8b9fa385-6914-46c0-b0bd-17abd0bf4eb7
|
2.74 KB |
sha256:1c18c7e5373ac539380f41118ed4a83a24e5e3143702df6fa24572c388b153b5
|
|
|
test_reports_9102f3ee-d3a0-4680-897f-bc8733db755d
|
4.62 KB |
sha256:5d3486831fce5f8f0bc21e6d1d7b4df4661cb24076ad89f966c7331e57fd0cb2
|
|
|
test_reports_9be0e2df-a787-41fc-bdfb-77c1fa0e2bbe
|
11.2 KB |
sha256:84c1726baaf5a336381de73120c48ed73c046686c01477bf7986bf2f17f0279b
|
|
|
test_reports_a0364ad6-2c53-4d56-b636-84e7ba7a3424
|
2.55 KB |
sha256:71ca37009f754c586f1529281fb34f122e9fa1a336caf38ff49bcf6df2f69f51
|
|
|
test_reports_a6796100-b3ae-4483-a709-3efeea0f1ad1
|
488 Bytes |
sha256:396a75746038eee295fe0edacd0f4147ac2d6c3b97a30c22d4e63cca189dad22
|
|
|
test_reports_a9bae28b-9cbe-4bb5-98c0-5ae83109b245
|
3.12 KB |
sha256:b23061fc7832f78a15981886d8719793544e23a29560de26ada397b069030617
|
|
|
test_reports_aa9e7df6-dc6d-469d-9bee-a71a3841f46e
|
5.06 KB |
sha256:428744874a3bf947e1887704461668aa37603e89286bfca06d8cd058f21ce7aa
|
|
|
test_reports_b2fd4c69-57ce-41f1-b174-186b670ea90c
|
4.03 KB |
sha256:f177c7403ee952909fbbd564e4036bd01e60b0ca2eb8886702bca4030ff2f767
|
|
|
test_reports_e757ec49-5f45-484b-9a4a-16f0517beaa0
|
581 Bytes |
sha256:fab4f0d32cfd5f833a471ab06f3ab2f0cfe43346b2e5b81cd54c4afaade99ea4
|
|
|
test_reports_ec4ded75-b79c-461b-85a9-ca46eb9fc95d
|
3.85 KB |
sha256:6f70ea6550c1a3397c7460a670cc4b1acd202889c30b4bc1ae355d6d92b2d4e6
|
|
|
test_reports_f9baa84b-3949-4997-960f-3886bdce58cc
|
21.4 KB |
sha256:8ef45b405b4cdbeb3571477e224376fd315dde4b8baca562b7766febed7f1f71
|
|