ipmikvm stopped working, now just getting Retry =s

[quite]

I was happily (as can be :) accessing my Supermicro using ipmikvm, mounting and booting my .iso files for some weeks, but suddenly it stopped working. Now I'm getting

+ exec java -Djava.library.path=/home/quite/.cache/ipmikvm/iKVM__V1.69.40.0x0 -cp /home/quite/.cache/ipmikvm/iKVM__V1.69.40.0x0/iKVM__V1.69.40.0x0.jar tw.com.aten.ikvm.KVMMain [..]
connect failed sd:19
Retry =1
Retry =2
Retry =3
Retry =4
Retry =5
Retry =6
Retry =7
Retry =8
Retry =9
Retry =10
a singal 17 is raised
a singal 17 is raised

I have tried resetting IKVM and rebooting IPMI from withint the web BMC interface, but no change.

Anything you have run into?

[wdoekes]

I'm thinking expired KVM cert.

https://www.osso.nl/blog/2025/supermicro-ikvm-expired-certificate/#faking-the-time

[quite]

I'm thinking expired KVM cert.

https://www.osso.nl/blog/2025/supermicro-ikvm-expired-certificate/#faking-the-time

Oh, thank you! easycert on port 5900 gives that it expires in -1. Investigating...

[quite]

No luck faking the time unfortunately, tried various diffs, with that faketime.so and also libfaketime. The BMC firmware on this server (which is only 1-2 years old) also looks to be the latest, from what I see is available for the motherboard on Supermicros web. Ouch

[quite]

I filed a case with Supermicro and they answered: The certificate can be renewed by reflashing the IPMI firmware but unchecking the “Preserve SSL certificate” checkbox before, finishing the reflash. -- now should i dare...

[quite]

I ran it, but it did not regenerate the cert on port 5900, only that on 443 (i think it did that). As I think you suspected in your blog post. Ugh.

[wdoekes]

If easycert says it expired 1 day ago, then it's really strange if your firmware is latest / only two years old.

You'll want to double check that.

Get the exact certs using openssl -connect and look at them more thoroughly.

Over here:

$ easycert -T 10.x.x.158 5900
Certificate chain
 0 s: {95:F7:F4:83:AB:7C:DA:D9:08:80:D7:E1:86:2E:D1:9E:01:60:65:44} [a8c7e6cd] C = US, ST = California, L = San Jose, O = Super Micro Computer, OU = Software, CN = IPMI, emailAddress = support@supermicro.com
   i: {95:F7:F4:83:AB:7C:DA:D9:08:80:D7:E1:86:2E:D1:9E:01:60:65:44} [a8c7e6cd] C = US, ST = California, L = San Jose, O = Super Micro Computer, OU = Software, CN = IPMI, emailAddress = support@supermicro.com
---
Expires in 3191 days
(10.x.x.158 NOT IN (IPMI, emailAddress = support@supermicro.com )) 

Then:

$ openssl s_client -connect 10.x.x.158:5900 -showcerts </dev/null
...
Certificate chain
 0 s:C = US, ST = California, L = San Jose, O = Super Micro Computer, OU = Software, CN = IPMI, emailAddress = support@supermicro.com
   i:C = US, ST = California, L = San Jose, O = Super Micro Computer, OU = Software, CN = IPMI, emailAddress = support@supermicro.com
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Feb 13 00:00:00 2025 GMT; NotAfter: Feb 13 00:00:00 2035 GMT
-----BEGIN CERTIFICATE-----
MIIE3TCCA8WgAwIBAgIUefhNBuhEgC59xBUISppcDJ1JAWgwDQYJKoZIhvcNAQEL
BQAwgZ0xCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMREwDwYDVQQH
EwhTYW4gSm9zZTEdMBsGA1UEChMUU3VwZXIgTWljcm8gQ29tcHV0ZXIxETAPBgNV
BAsTCFNvZnR3YXJlMQ0wCwYDVQQDEwRJUE1JMSUwIwYJKoZIhvcNAQkBFhZzdXBw
b3J0QHN1cGVybWljcm8uY29tMB4XDTI1MDIxMzAwMDAwMFoXDTM1MDIxMzAwMDAw
MFowgZ0xCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMREwDwYDVQQH
EwhTYW4gSm9zZTEdMBsGA1UEChMUU3VwZXIgTWljcm8gQ29tcHV0ZXIxETAPBgNV
BAsTCFNvZnR3YXJlMQ0wCwYDVQQDEwRJUE1JMSUwIwYJKoZIhvcNAQkBFhZzdXBw
b3J0QHN1cGVybWljcm8uY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
AQEA4eG950BRE9oHZALz3MssndeDY2EOZMZ9hj6+rXWqB5A0YR0wfRnnckpEF++p
itQfyhObFDZR8+wlJCQ/cfR9P5/Mbdx0AxmepCj6vMXgF8OvEbbsxGBT85ShgQp2
5k0oXR3xa8SKrg2B7C25TU6cFjsGtBtmmkGy76HcMypQb9eQgtAkwotS1Z3Q6Lxi
dPFedCBsZgm/jbiWiKMkb9Rj5biDhfnpxuroY1OwLwI2vKoOVSfrjZNS57cmpVPj
mf7dg1eBlCHmDY+3fKD37pKnXr/v2+MSqyyQE62AiNp+ZW3RLzWV8qVRyA4yHE4u
d4JPf4sMFG7OzId65PgC1i+XVQIDAQABo4IBETCCAQ0wHQYDVR0OBBYEFJX39IOr
fNrZCIDX4YYu0Z4BYGVEMIHdBgNVHSMEgdUwgdKAFJX39IOrfNrZCIDX4YYu0Z4B
YGVEoYGjpIGgMIGdMQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTER
MA8GA1UEBxMIU2FuIEpvc2UxHTAbBgNVBAoTFFN1cGVyIE1pY3JvIENvbXB1dGVy
MREwDwYDVQQLEwhTb2Z0d2FyZTENMAsGA1UEAxMESVBNSTElMCMGCSqGSIb3DQEJ
ARYWc3VwcG9ydEBzdXBlcm1pY3JvLmNvbYIUefhNBuhEgC59xBUISppcDJ1JAWgw
DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAJ1ysEqHO937RN7q3UR1A
TOMYxex9G8BeMSKDlgfQwiZI13CCIdvwTtO+aI2jJ8/72Pm1JC4B+M9Owq9mq36U
R7UmQO/2guyxAXef+banW9JNp37Kij63iruUeStb6r4yojzYBRhop0DcfpjI505a
s72O2gkn49ExW/EDMz6QokJkLlpjRf5Sef7tTynjayK/DzfSQhF6NFWMFMfnfJ+U
NhkyzJ8KN7BZ7XzjKYkUyqcZDy3zR9wh2GsSrmcmGqSFgwEuEfF9H9mzJuCanl/F
2PwjMscsxqqbFkShQ8g9SwUQZMcwjl623uO6J2tXiLKpVCsoCIPBZcV7RmadubGe
EQ==
-----END CERTIFICATE-----
...

Dump that in a file: sm.crt

$ openssl x509 -in sm.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            79:f8:4d:06:e8:44:80:2e:7d:c4:15:08:4a:9a:5c:0c:9d:49:01:68
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, ST = California, L = San Jose, O = Super Micro Computer, OU = Software, CN = IPMI, emailAddress = support@supermicro.com
        Validity
            Not Before: Feb 13 00:00:00 2025 GMT
            Not After : Feb 13 00:00:00 2035 GMT
...

If the cert is (a lot) older, I suspect you don't have the latest firmware. (Or your hardware is EOL since a long time.)

In all cases, I think you should be able to use faketime BUT, maybe you need to forge the time on the server as well.
See this discussion: https://www.osso.nl/blog/2021/supermicro-ikvm-sslv3-alert-bad-certificate/
If you fake the time 2 years in the past, you might need to set the server BMC time 2 years in the past as well.

[quite]

The ipmi web claims:

Firmware Revision: 04.04
Firmware Build Time: 06/04/2025
BIOS Version: 2.3
BIOS Build Time: 04/14/2025

This is also what is available for download on the motherboard's download page: https://www.supermicro.com/en/support/resources/downloadcenter/firmware/MBD-A2SDi-4C-HLN4F/BMC

And this is the IPMI firmware that I reinstalled, with "preserve SSL certificate" unchecked.

This is the cert on port 5900, actually very similar validity dates to what you had in the blog post:

$ cert-ssl-info st-demo-bmc:5900 | grep Validity -A3
        Validity
            Not Before: May 19 09:44:10 2016 GMT
            Not After : May 17 09:44:10 2026 GMT
        Subject: C=US, ST=California, L=San Jose, O=Super Micro Computer, OU=Software, CN=IPMI, emailAddress=Email

It does feel weird that it expires that soon, given how new the firmware claims to be.

The time in the IPMI is correct. I'll see about trying to fake that too hm

[wdoekes]

Bad news:

~/Junk/A2SDi-xC-HLN4F_2.3_AS4.04_SUM2.14.0-p9/A2SDi-xC-HLN4F_2.3_AS4.04_SUM2.14.0-p9/bmc/BMC_X10AST2400-C001MS_20250604_04.04_STD
$ binwalk BMC_X10AST2400-C001MS_20250604_04.04_STD.bin 

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
103360        0x193C0         CRC32 polynomial table, little endian
4194304       0x400000        CramFS filesystem, little endian, size: 15302656, version 2, sorted_dirs, CRC 0xA3EA5F43, edition 0, 8662 blocks, 1102 files
20971520      0x1400000       uImage header, header size: 64 bytes, header CRC: 0xE0BD5E06, created: 2025-06-04 04:27:43, image size: 1537695 bytes, Data Address: 0x40008000, Entry Point: 0x40008000, data CRC: 0xEBECDD97, OS: Linux, CPU: ARM, image type: OS Kernel Image, compression type: gzip, image name: "21400000"
20971584      0x1400040       gzip compressed data, maximum compression, has original file name: "linux.bin", from Unix, last modified: 2025-06-04 03:09:22
24117248      0x1700000       CramFS filesystem, little endian, size: 7491584, version 2, sorted_dirs, CRC 0x23D5C47A, edition 0, 3116 blocks, 458 files

dd if=BMC_X10AST2400-C001MS_20250604_04.04_STD.bin of=rootfs1.cramfs bs=1 skip=4194304 count=15302656

sudo mount -t cramfs -o loop,ro rootfs1.cramfs /mnt/cram1

$ easycert -T /mnt/cram1/etc/stunnel/server.crt
Certificate chain
 0 s: {8C:B9:7E:4B:33:2B:3D:03:74:98:D5:DE:43:34:3D:7A:05:FD:58:6B} [6e6e5643] C = US, ST = California, L = San Jose, O = Super Micro Computer, OU = Software, CN = IPMI, emailAddress = Email
   i: {8C:B9:7E:4B:33:2B:3D:03:74:98:D5:DE:43:34:3D:7A:05:FD:58:6B} [6e6e5643] C = US, ST = California, L = San Jose, O = Super Micro Computer, OU = Software, CN = IPMI, emailAddress = Email
---
Expires in -2 days

I think they forgot(?) to update the cert when they built a new BMC image in 2025?

[quite]

Oh oh, great bin digging. That sounds very plausible, given that this IKVM 5900 cert can't be set by user at all. I'll be typing in my case with Supermicro 🫰🫰

[quite]

And setting the clock 1 year back in IPMI and on client worked.