Driver not working anymore since Linux kernel 5.4 on certain hardware

[othiman]

Hallo,

we have some strange problems with the driver and the MadiFX card plugged into a HP Proliant DL380 Gen9 server hardware since Linux kernel 5.4. Today I tried many old Ubuntu 18.04.x images to narrow down the kernel version since when the problem occurred. I used a simple speaker-test command like this:

speaker-test -c2 -Dsysdefault:CARD=MADIFXtest

On Ubuntu 18.04.4 with kernel 5.3 the output works fine and speaker-test toggles between "0 - Front Left" and "1 - Front Right". With Ubuntu 18.04.6, which uses kernel 5.4, the speaker-test output just stops at "0 - Front Left".

It is worth mentioning that the same applies to the HDSPe AES and Madi cards which we also tried with the buildin drivers of the kernel. In addition we found that the HDSPe AES card plugged in a Supermicro X13SAQ mainboard does have the same issues. On many other mainboards the cards all work just fine, even with the 6.1 kernel.

We can provide remote access to the server if you wish to find out more details regarding the issue. Any help would be greatly appreciated.

Best regards,
Thomas

[adiknoth]

It could be that some user-level audio daemon (e.g., pulseaudio) gets in the way, especially on those old Ubuntu distributions.

If you're sure that it is the kernel, my next guess would be interrupts and ACPI. You can check cat /proc/interrupts to see if the card issues interrupts if you start the playback.

Also, there's /proc/asound/cardX/hdpsm (or madifx), which is worth a look for what the hardware does while you play.

Maybe there are some PCI interrupt settings in your BIOS that you could tweak (MSI? Though I think PCIe has still all by default), or kernel cmdline parameters. But before going down that rabbit hole, I'd bring in the latest mainline kernel that you can find (6.1 for instance). I'm fairly sure this is entirely unrelated to the HDSPM driver.

[othiman]

Hi,

First of all, thank you very much for your help. It is really appreciated.

I used the old Ubuntu versions only to narrow down the problem, but it still persists in Ubuntu 23.04 with kernel 6.1 and also in our own build Yocto Linux with kernel 6.1 which only utilizes alsa and jackd (no pulseaudio or Pipewire).

However, thanks to your answer, I had a look at /proc/interrupts and indeed on the newer kernels (>5.3) it shows all 0 for the HDSPe and MadiFX drivers:

16: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC 16-fasteoi snd_hdspm, snd_hdspm, madifx

while other devices still have interrupts, e.g.

18: 0 0 0 38 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC 18-fasteoi ehci_hcd:usb1, ehci_hcd:usb3, i801_smbus

On kernel 5.3 cat /proc/interrupts |grep madi does show:

16: 27 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 IR-IO-APIC 16-fasteoi snd_hdspm, snd_hdspm, madifx

after startup of the kernel module and rising numbers while playing audio via speaker-test.

So in summary there seem to be a problem with the IRQ handling. I found that the APIC is related to the XEON E5 v4 CPU:

lspci -knn |grep APIC
00:05.4 PIC [0800]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D I/O APIC [8086:6f2c] (rev 01)
	Subsystem: Hewlett-Packard Company Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D I/O APIC [103c:21ea]
80:05.4 PIC [0800]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D I/O APIC [8086:6f2c] (rev 01)
	Subsystem: Hewlett-Packard Company Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D I/O APIC [103c:21ea]

and my idea would be to diff the relevant parts of kernel 5.3 to 5.4 to see what has changed and causes the issue, but I do not know where to look at precisely. Do you have an idea where I have to look for releant changes or could you give me a hint to narrow down the problem?

Thanks again and best regards,
Thomas

[othiman]

I just tried to add the kernel boot option noapic and with that it is working fine, even with the new kernels (>5.4). So something must be broken in the combination of the HDSPe driver and the IO-APIC driver that is used. Via https://kernelnewbies.org/Linux_5.4 I found this patchset https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c5f12fdb8bd873aa3ffdb79512e6bdac92b257b0 which is new in kernel 5.4. Maybe it has something to do with this.

What would be the best way to report this bug? A mail to the kernel mailinglist?

[othiman]

I had access to the PC with the Supermicro X13SAQ mainboard again and there the interrupts do not work, too. However, in this case the noapic kernel option does not help. With the noapic option we get from /proc/interrups:

           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       CPU8       CPU9       CPU10      CPU11      
  2:          0          0          0          0          0          0          0          0          0          0          0          0    XT-PIC      cascade
  5:          0          0          0          0          0          0          0          0          0          0          0          0    XT-PIC      i801_smbus, snd_hdspm
  8:          0          0          0          0          0          0          0          0          0          0          0          0    XT-PIC      rtc0
  9:          0          0          0          0          0          0          0          0          0          0          0          0    XT-PIC      acpi
 20:          0          0          0          0          0         30          0          0          0          0          0          0   PCI-MSI 524288-edge      nvme0q0
 21:          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI 376832-edge      ahci[0000:00:17.0]
 22:       5976          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI 524289-edge      nvme0q1
 23:          0       3110          0          0          0          0          0          0          0          0          0          0   PCI-MSI 524290-edge      nvme0q2
 24:          0          0       7193          0          0          0          0          0          0          0          0          0   PCI-MSI 524291-edge      nvme0q3
...

without noapic it looks like this:

            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       CPU8       CPU9       CPU10      CPU11      
   8:          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC    8-edge      rtc0
   9:          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC    9-fasteoi   acpi
  17:          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC   17-fasteoi   snd_hdspm
  18:          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC   18-fasteoi   i801_smbus
  27:          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC   27-fasteoi   idma64.0, i2c_designware.0
  40:          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC   40-fasteoi   idma64.1, i2c_designware.1
 109:          0          0          0          0          0          0        119          0          0          0          0          0  IR-IO-APIC  109-fasteoi 
 120:          0          0          0          0          0          0          0          0          0          0          0          0  DMAR-MSI    0-edge      dmar0
 121:          0          0          0          0          0          0          0          0          0          0          0          0  DMAR-MSI    1-edge      dmar1
 126:          0          0          0          0          0          0          0          0          0         30          0          0  IR-PCI-MSI 524288-edge      nvme0q0
 127:          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 376832-edge      ahci[0000:00:17.0]
 128:       1462          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 524289-edge      nvme0q1
 129:          0       1265          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 524290-edge      nvme0q2
 130:          0          0       2702          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 524291-edge      nvme0q3

[othiman]

Small Update: The Supermicro X13SAQ mainboard has PCIe slots which are connected to the Intel Q670E chipset and one that is connected directly to the CPU (Intel Core i5-12600). If we plug the soundcard to the PCIe slot that is directly connected to the CPU it works (noapic does not need to be added to the kernel command line).

[trummerschlunk]

@adiknoth my madifx is kind of sleeping in an old pc these days... if you need it for testing or so... always happy to see linux pro audio driver development :)