Software Stack
TT-Metallium
OS Version
Linux sandisk-HBF 6.8.0-117-generic #117~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu May 7 22:17:46 UTC x86_64 x86_64 x86_64 GNU/Linux
Hardware Platform
Blackhole Card - p100a
Serial Number
000004323191B063
Symptom
We followed the installation instructions from the Tenstorrent installer repository:
GitHub - tenstorrent/tt-installer: Install the tenstorrent stack with one command
After completing the installation of all required packages, we were able to successfully activate the tenstorrent-venv environment. At that point, running tt-smi showed the device with 16 GB DRAM, but the memory remained in an untrained state.
To proceed with DRAM training, we assumed that the issue might be related to an incorrect or outdated firmware version. Based on that assumption, we attempted to flash a newer firmware package on the device.
However, after performing the firmware update, we started encountering the following error:
ARC FW has not yet booted
Device fails to reinitialize properly after reset
tt-flash waits for PCI re-enumeration and does not recover cleanly
Despite multiple attempts (including force flashing both board-specific and full firmware bundles), the firmware write and verification stages consistently complete successfully. However, the device does not come back online cleanly after the reset phase, and ARC firmware does not boot.
Error in detecting devices!
No existing board type for board ID: 0x0
Location: /project/device/api/umd/device/types/cluster_descriptor_types.hpp:234
tt::umd::TTDevice::get_chip_info()
tt::umd::BlackholeTTDevice::get_chip_info()
tt::umd::TTDevice::construct_soc_descriptor(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)
tt::umd::TopologyDiscovery::get_connected_devices()
tt::umd::TopologyDiscovery::create_ethernet_map()
tt::umd::TopologyDiscovery::discover(tt::umd::TopologyDiscoveryOptions const&, tt::umd::IODeviceType, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)
/home/sandisk/.tenstorrent-venv/lib/python3.10/site-packages/tt_umd/tt_umd.cpython-310-x86_64-linux-gnu.so(+0x186951) [0x7d7766f86951]
/home/sandisk/.tenstorrent-venv/lib/python3.10/site-packages/tt_umd/tt_umd.cpython-310-x86_64-linux-gnu.so(+0x1a0833) [0x7d7766fa0833]
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
/home/sandisk/.tenstorrent-venv/bin/python3(+0x259d36) [0x57f1123fad36]
PyEval_EvalCode
/home/sandisk/.tenstorrent-venv/bin/python3(+0x280678) [0x57f112421678]
/home/sandisk/.tenstorrent-venv/bin/python3(+0x27accf) [0x57f11241bccf]
/home/sandisk/.tenstorrent-venv/bin/python3(+0x280415) [0x57f112421415]
_PyRun_SimpleFileObject
_PyRun_AnyFileObject
Py_RunMain
Py_BytesMain
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7d7768229d90]
__libc_start_main
_start
Exiting...
Mohit Gupta
Today 12:46 PM
[ 8.674109] systemd[1]: Mounting Mount hugepages at /dev/hugepages-1G for Tenstorrent ASICs...
[ 8.708960] systemd[1]: Starting Script that configures hugepages for Tenstorrent ASICs...
[ 8.710845] systemd[1]: Mounted Mount hugepages at /dev/hugepages-1G for Tenstorrent ASICs.
[ 9.872162] tenstorrent: loading out-of-tree module taints kernel.
[ 9.872169] tenstorrent: module verification failed: signature and/or required key missing - tainting kernel
[ 9.945959] tenstorrent: Loading Tenstorrent AI driver module v2.9.1-pre
[ 9.946187] tenstorrent 0000:95:00.0: Found a Tenstorrent Blackhole device
[ 9.946216] tenstorrent 0000:95:00.0: enabling device (0140 -> 0142)
[ 9.949165] tenstorrent 0000:95:00.0: Telemetry not available
even with v2.9.1-pre seeing the same issue
we were expecting to see 28GB DDR along with power we were expecting around Watt. for reference i am attaching the snap of telemetry which shows just 150W
Software Stack
TT-Metallium
OS Version
Linux sandisk-HBF 6.8.0-117-generic #117~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu May 7 22:17:46 UTC x86_64 x86_64 x86_64 GNU/Linux
Hardware Platform
Blackhole Card - p100a
Serial Number
000004323191B063
Symptom
We followed the installation instructions from the Tenstorrent installer repository:
GitHub - tenstorrent/tt-installer: Install the tenstorrent stack with one command
After completing the installation of all required packages, we were able to successfully activate the tenstorrent-venv environment. At that point, running tt-smi showed the device with 16 GB DRAM, but the memory remained in an untrained state.
To proceed with DRAM training, we assumed that the issue might be related to an incorrect or outdated firmware version. Based on that assumption, we attempted to flash a newer firmware package on the device.
However, after performing the firmware update, we started encountering the following error:
ARC FW has not yet booted
Device fails to reinitialize properly after reset
tt-flash waits for PCI re-enumeration and does not recover cleanly
Despite multiple attempts (including force flashing both board-specific and full firmware bundles), the firmware write and verification stages consistently complete successfully. However, the device does not come back online cleanly after the reset phase, and ARC firmware does not boot.
Error in detecting devices!
No existing board type for board ID: 0x0
Location: /project/device/api/umd/device/types/cluster_descriptor_types.hpp:234
tt::umd::TTDevice::get_chip_info()
tt::umd::BlackholeTTDevice::get_chip_info()
tt::umd::TTDevice::construct_soc_descriptor(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)
tt::umd::TopologyDiscovery::get_connected_devices()
tt::umd::TopologyDiscovery::create_ethernet_map()
tt::umd::TopologyDiscovery::discover(tt::umd::TopologyDiscoveryOptions const&, tt::umd::IODeviceType, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)
/home/sandisk/.tenstorrent-venv/lib/python3.10/site-packages/tt_umd/tt_umd.cpython-310-x86_64-linux-gnu.so(+0x186951) [0x7d7766f86951]
/home/sandisk/.tenstorrent-venv/lib/python3.10/site-packages/tt_umd/tt_umd.cpython-310-x86_64-linux-gnu.so(+0x1a0833) [0x7d7766fa0833]
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
/home/sandisk/.tenstorrent-venv/bin/python3(+0x259d36) [0x57f1123fad36]
PyEval_EvalCode
/home/sandisk/.tenstorrent-venv/bin/python3(+0x280678) [0x57f112421678]
/home/sandisk/.tenstorrent-venv/bin/python3(+0x27accf) [0x57f11241bccf]
/home/sandisk/.tenstorrent-venv/bin/python3(+0x280415) [0x57f112421415]
_PyRun_SimpleFileObject
_PyRun_AnyFileObject
Py_RunMain
Py_BytesMain
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7d7768229d90]
__libc_start_main
_start
Exiting...
Mohit Gupta
Today 12:46 PM
[ 8.674109] systemd[1]: Mounting Mount hugepages at /dev/hugepages-1G for Tenstorrent ASICs...
[ 8.708960] systemd[1]: Starting Script that configures hugepages for Tenstorrent ASICs...
[ 8.710845] systemd[1]: Mounted Mount hugepages at /dev/hugepages-1G for Tenstorrent ASICs.
[ 9.872162] tenstorrent: loading out-of-tree module taints kernel.
[ 9.872169] tenstorrent: module verification failed: signature and/or required key missing - tainting kernel
[ 9.945959] tenstorrent: Loading Tenstorrent AI driver module v2.9.1-pre
[ 9.946187] tenstorrent 0000:95:00.0: Found a Tenstorrent Blackhole device
[ 9.946216] tenstorrent 0000:95:00.0: enabling device (0140 -> 0142)
[ 9.949165] tenstorrent 0000:95:00.0: Telemetry not available
even with v2.9.1-pre seeing the same issue
we were expecting to see 28GB DDR along with power we were expecting around Watt. for reference i am attaching the snap of telemetry which shows just 150W