xCAT IPMI does not work anymore after server firmware update

[paddg]

Hello,

Very strange, after the firmware update the xCAT IPMI commands no longer work.

node209: old firmware
node207: new firmware

# rpower node209 status
node209: on

# rpower node207 status
node207: [ul-dcsm-x01]: Error: ERROR: Incorrect password provided

I swear, the password is correct! The xCAT tables haven't changed either. And after firmware roll-back, it worked again.

But the strange thing is that ipmitool-xcat has no problems with the new firmware:

# ipmitool-xcat -H 10.10.20.209 -I lanplus -U $USER -P $PASSWORD chassis power status
Chassis Power is on

# ipmitool-xcat -H 10.10.20.207 -I lanplus -U $USER -P $PASSWORD chassis power status
Chassis Power is on

Well, I saw in the code that rpower isn't using ipmitool-xcat, but communicates directly with the IPMI endpoint. Therefore, I suspect the bug lies there. After all, ipmitool and ipmitool-xcat don't seem to be have any problems with the new firmware.

Do you perhaps have any ideas on how I could narrow it down further?

[Obihoernchen]

Which version of xCAT are you using?
Maybe the supported cipher suites changed with the firmare update.

Compare: ipmitool lan print 1 | grep "Cipher"

xCAT doesn't use ipmitool for r* commands, but unses it's own ipmi implementation in perl.

[paddg]

Which version of xCAT are you using?

2.17.0

Compare: ipmitool lan print 1 | grep "Cipher"

There is nothing on channel 1 (Invalid channel: 1) but on 8:

# ipmitool lan print 8| grep Cipher 
RMCP+ Cipher Suites     : 
Cipher Suite Priv Max   : XXXaXXXXXXXaXXX
                        :     X=Cipher Suite Unused

On both firmware versions the same result.

For the record, here the FW versions from ipmitool mc info:

The failing FW:

Firmware Revision: 7.80

The working FW:

Firmware Revision: 7.52

The hardware:

Lenovo SR635 (ThinkSystem)

And as said, only the xCAT Perl IPMI implementation has this problem, while ipmitool and ipmitool-xcat work fine. Perhaps it's a compatibility issue between xCAT Perl IPMI and the new BMC firmware.

[samveen]

The magic is in xCAT-server/lib/perl/xCAT/IPMI.pm , but I think the only person who could reliably figure out the issue might be Jarod (@jjohnson42 ), who wrote it.

This might be a good point to consider a migration to confluent i.e. https://xcat2.github.io/confluent-docs/#installation

[paddg]

This might be a good point to consider a migration to confluent

Indeed! Unfortunately, we're not quite ready for that step yet. (And it might not even solve the problem.)

Here are two compressed PCAPs showing the communication of a rpower node20# command. The first PCAP (node207) shows the error (communication stops after the 3rd response packet), and the second shows the successful communication with the old firmware of the node209 with nine query/response packets.

1. ipmi-node207.pcap.gz
2. ipmi-node209.pcap.gz

[viniciusferrao]

@paddg do you still have access to the system? We've made some changes and referenced them in #7511

[viniciusferrao]

Same bug as #7511. Newer XCC firmware needs the name-only lookup bit (bit 4) in RAKP1's privilege byte. xCAT sent 0x04, ipmitool sends 0x14. That bit is part of the HMAC calculation, so without it the hash never matches and you get "Incorrect password provided" even with the right credentials. Fix is in the PR linked from #7511. @Obihoernchen can be closed as duplicate.

[Obihoernchen]

Should be fixed with #7555. Please reopen if this is not the case.

[paddg]

Thanks for the update! I applied the patch. Unfortunately the error didn't disappear:

cd /opt/xcat/lib/perl/xCAT
patch -p0 IPMI.pm < 7555.patch
patching file IPMI.pm
patching file IPMI.pm
patching file IPMI.pm
patching file IPMI.pm

rpower node207 status
node207: [ul-dcsm-x01]: Error: ERROR: Incorrect password provided

[viniciusferrao]

Hi @paddg I have the 2U version of the same server. Not sure if it is the same generation, yours is V3?

Product Name: ThinkSystem SR645 V3
BMC Version
2.81 (Build ID: KAX330B)
UEFI Version
4.11 (Build ID: KAE118M)
LXPM Version
4.12 (Build ID: GNL114G)

Can you confirm which server version you do have?

[viniciusferrao]

Another question @paddg. Can you try:

ipmitool -H 10.10.20.207 -U test -P $PASSWORD -L USER chassis power status

And...

ipmitool -H 10.10.20.207 -U test -P $PASSWORD -L ADMINISTRATOR chassis power status

Are you using any role attributes or something like that? I think the issue may be related to it. Maybe the user isn't full administrator?

[paddg]

Can you confirm which server version you do have?

It's a 1U "Thinksystem SR635" (V1 probable). Machine Type/Model: 7Y99CTO1WW.

BMC Version 	
V7.80 (BUILD ID:AMBT60F)
UEFI Version 	
V7.71 (BUILD ID:CFE150F)
LXPM Version 	
V2.44 (BUILD ID:l4y786b)

Can you try:

It's not working as you suggested:

# ipmitool -H 10.10.20.207 -U $USER -P $PASSWORD -L USER chassis power status
Authentication type NONE not supported
Error: Unable to establish LAN session
Error: Unable to establish IPMI v1.5 / RMCP session

# ipmitool -H 10.10.20.207 -U $USER -P $PASSWORD -L ADMINISTRATOR chassis power status
Authentication type NONE not supported
Error: Unable to establish LAN session
Error: Unable to establish IPMI v1.5 / RMCP session

But it works like this with ipmitool as well as with ipmitool-xcat:

# ipmitool -H 10.10.20.207 -I lanplus -U $USER -P $PASSWORD -L USER chassis power status
Chassis Power is on
# ipmitool -H 10.10.20.207 -I lanplus -U $USER -P $PASSWORD -L ADMINISTRATOR chassis power status
Chassis Power is on

# ipmitool-xcat -H 10.10.20.207 -I lanplus -U $USER -P $PASSWORD -L USER chassis power status
Chassis Power is on
# ipmitool-xcat -H 10.10.20.207 -I lanplus -U $USER -P $PASSWORD -L ADMINISTRATOR chassis power status
Chassis Power is on

rpower isn't using ipmitool or ipmitool-xcat, but communicates directly with the IPMI endpoint using IPMI.pm. Therefore, I suspect the probmel lies there. After all, ipmitool and ipmitool-xcat don't seem to be have any problems with the new firmware.