Issue bringing up Goya cards

[Szalacinski]

Ubuntu 24.04
Kernel version 6.8.0-51-generic

[ 2.039046] habanalabs: loading driver
[ 2.039128] habanalabs 0000:0d:00.0: habanalabs device found [1da3:0001] (rev 1)
[ 3.044926] habanalabs 0000:0d:00.0: Device boot progress - Invalid or unexpected status code 2
[ 3.044936] habanalabs 0000:0d:00.0: CPU boot ready timeout (status = 2)
[ 3.656600] habanalabs: Failed to initialize accel0. Device 0000:0d:00.0 is NOT usable!
[ 3.656609] habanalabs 0000:0d:00.0: Fatal error during habanalabs device init

Happens with both the upstream and latest DKMS drivers.

[Szalacinski]

Using the 1.1.1 packages on 20.04 I get a different result. It seems to load the firmware but it complains about the firmware being corrupt.

[ 451.808445] habanalabs_en: module verification failed: signature and/or required key missing - tainting kernel
[ 451.809012] habanalabs_en: loading driver, version: 1.1.1-0ca2d0bc5
[ 451.923985] habanalabs: loading driver, version: 1.1.1-0ca2d0bc5
[ 451.924072] habanalabs 0000:0d:00.0: habanalabs device found [1da3:0001] (rev 1)
[ 451.924386] habanalabs hl0: BTL version 0b2b7d3f
[ 451.924388] habanalabs hl0: preboot version 3.0.1
[ 451.933733] habanalabs hl0: Going to wait for device boot (up to 15s)
[ 452.940528] habanalabs hl0: boot-fit version 3.0.1
[ 452.940533] habanalabs hl0: Loading firmware to device, may take some time...
[ 467.951435] habanalabs hl0: Device reports FIT image is corrupt
[ 467.951566] habanalabs hl0: failed to initialize CPU
[ 467.951670] habanalabs hl0: failed to initialize the H/W
[ 467.951948] habanalabs 0000:0d:00.0: Failed to initialize hl0. Device is NOT usable !

Attached is the hardware revision info

[Szalacinski]

Has anybody managed to take a look at this?

[Szalacinski]

Here is a potential solution from https://www.linkedin.com/in/jpluska/
I have yet to test it:

🚀 "Houston, we have a problem" – Habana Goya .
A few months ago, I acquired several AI accelerators – Habana Labs Goya, the first generation of inference-dedicated TPC (Tensor Processing Core) devices.
These HL-100 PCIe cards are built around a single Goya processor and designed to accelerate a wide range of AI inference workloads.
However, upon installation, the cards failed to initialise properly. After some digging, I managed to bring them back to life — and here’s how I did it, in case it helps others working with legacy or specialised hardware.

🧰 Requirements
Ubuntu 22.04 LTS (x86-64)
Free PCIe slot (I used a Dell T3600 as a test system)

🛠️ Initial Setup
After physically installing the Goya card and booting the system, you may encounter firmware errors due to missing components.
The device uses U-Boot and an embedded Linux environment.
Go to Habana’s APT repository:
🔗 https://lnkd.in/d5d9ugch
Download these packages using wget:
habanalabs-dkms v1.1.1-94
habanalabs-firmware v1.1.1-94
habanalabs-firmware-odm v1.16.0-526
habana-firmware-tools v1.1.1-94
habanalabs-think 1.1.1-94
habanatools 1.1.1-94
Once installed, reboot your system but do not load the drivers yet via modprobe.

🧯 The Fix: Firmware Recovery
In my case, the Goya cards shipped with corrupted firmware, causing the following kernel log errors:
habanalabs_en: module verification failed: signature and/or required key missing - tainting kernel
habanalabs_en: loading driver, version: 1.1.1-0ca2d0bc5
habanalabs: loading driver, version: 1.1.1-0ca2d0bc5
habanalabs 0000:0d:00.0: habanalabs device found [1da3:0001] (rev 1)
habanalabs hl0: BTL version 0b2b7d3f
habanalabs hl0: preboot version 3.0.1
habanalabs hl0: Going to wait for device boot (up to 15s)
habanalabs hl0: boot-fit version 3.0.1
habanalabs hl0: Loading firmware to device, may take some time...
habanalabs hl0: Device reports FIT image is corrupt
habanalabs hl0: failed to initialize CPU
habanalabs hl0: failed to initialize the H/W
habanalabs 0000:0d:00.0: Failed to initialize hl0. Device is NOT usable !

🔨 To recover:

1. Switch to root: sudo su
2. Navigate to /usr/sbin/ and run:
   hl-fw-loader -r
   This re-flashes the firmware (ignore warnings if any).
3. Reboot the system again.
   Now, load the drivers in the following order:
   sudo modprobe habanalabs_compat
   sudo modprobe habanalabs
   sudo modprobe habanalabs_cn
   sudo modprobe habanalabs_en
   sudo modprobe habanalabs_ib
   Run hl-smi to verify — and you’re done! ✅