ochami discover static --log-level=debug -f yaml -d @/opt/ohpc/admin/nodes/nodes.yaml in the OpenHPC repository fails

[adrianreber]

Running ochami discover static --log-level=debug -f yaml -d @/opt/ohpc/admin/nodes/nodes.yaml as mentioned in the OpenHPC PR does not seem to work correctly:

[root@ohpc-lenovo-sms ~]# ochami discover static --log-level=debug  -f yaml -d @/opt/ohpc/admin/nodes/nodes.yaml
2025-07-19T10:02:25Z DBG lib.go:102 > logging has been initialized
2025-07-19T10:02:25Z DBG lib.go:287 > using base URI from default cluster c
2025-07-19T10:02:25Z DBG lib.go:367 > Determining token from environment variable based on cluster in config file
2025-07-19T10:02:25Z DBG lib.go:373 > --cluster not specified, using default-cluster: c
2025-07-19T10:02:25Z DBG lib.go:384 > Reading token from environment variable: C_ACCESS_TOKEN
2025-07-19T10:02:25Z DBG lib.go:386 > Token found from environment variable: C_ACCESS_TOKEN=eyJhbGciOiJSUzI1NiIsImtpZCI6IjhhNTRiYjBhLTVmZDQtNGYyZC1iNWU1LWIyMTZkZTVhYTY1MSIsInR5cCI6IkpXVCJ9.eyJhdWQiOltdLCJjbGllbnRfaWQiOiJkOGNiOGU5NS1lOTk3LTRlOGMtYTRjOS04OWMxOThlOWU2YjAiLCJleHAiOjE3NTI5MjI1NjQsImV4dCI6e30sImlhdCI6MTc1MjkxODk2NCwiaXNzIjoiaHR0cHM6Ly9vaHBjLWxlbm92by1zbXMiLCJqdGkiOiI0ZDNhNWUwMC02N2UxLTRjYWEtOTYyNy05NmUyZGNhMDFmMGIiLCJuYmYiOjE3NTI5MTg5NjQsInNjcCI6WyJvcGVuaWQiLCJzbWQucmVhZCJdLCJzdWIiOiJkOGNiOGU5NS1lOTk3LTRlOGMtYTRjOS04OWMxOThlOWU2YjAifQ.Fvb-P7GsWH9yh8XZK2VK6eo9ubyGvvw4j2ZAbd6j9EKzn4JrlX7V3nDevVXo0VdHPpPRHw_OTQes-7CDahVlPRaLsjqwnhcc3wgRkJFysrQq8S4PmyxaR1AVm4cifyuvTFS3BWpwCqZMXoQjNhUYvbp4Uv9KsUed_sqdnuzot0RAq6WiNICSmiBisz9_CamGFlbeeFQmIo3WjtE65hjr1p6eePhH_kWxJzr74lc4VpRysyZmfSBVx2gleh9dWfM2LFli3-qU1b00BDoxxXLo7HcZcT4dKsiJTr9xT_Fg873yAYdSzChB5LJTPrEL6Pa5_NMhS7haWmv01ODI94NZmG2skriD5BH_Z5gJ8sTHNaIrZUvAjWhkRjBkw9cqnZ24mSbBFDf-kZZ8K4PQBzc1beI09XcFioPJruT5Yf5TQOUiuOrf73yr7BEns0QBWnZbeZeOIGpVy4PWrD_MvVIgyQblCJPDVILE6blF31rZ_M3Jq9ww8k-w06DHp5XzgiQaHiVJl4rAuNTnIekBwhWXmFLlkkruO0T-cG9nnNhDHBsRUS1AbIKDVDNqBgEROv0RAoM6duScE-_LWTWApx9U2u96yOSmAA0XiWpCKXHLNGNaUMk_ddjkaFHSwQZ5lZKxjafdbwDmURMKxqA5MukdxiIKc2SWXLGKjgempT6BDYI
2025-07-19T10:02:25Z DBG client.go:389 > payload file: /opt/ohpc/admin/nodes/nodes.yaml
2025-07-19T10:02:25Z DBG client.go:390 > payload file format: yaml
2025-07-19T10:02:25Z DBG client.go:406 > bytes read: "nodes:\n- name: c1\n  nid: 1\n  xname: x1000c0s0b1\n  bmc_ip: 10.241.58.139\n  bmc_mac: 02:68:b3:29:da:98\n  group: compute\n  interfaces:\n  - mac_addr: f4:c7:aa:44:41:4a\n    ip_addrs:\n    - name: management\n      ip_addr: 10.241.58.132\n- name: c2\n  nid: 2\n  xname: x1000c0s0b2\n  bmc_ip: 10.241.58.138\n  bmc_mac: 02:68:b3:29:da:98\n  group: compute\n  interfaces:\n  - mac_addr: f4:c7:aa:44:40:f0\n    ip_addrs:\n    - name: management\n      ip_addr: 10.241.58.133\n"
2025-07-19T10:02:25Z DBG discover-static.go:89 > read 2 nodes
2025-07-19T10:02:25Z DBG discover-static.go:90 > nodes: [node0={name="c1" nid=1 xname=x1000c0s0b1 bmc_mac=02:68:b3:29:da:98 bmc_ip=10.241.58.139 interfaces=[iface0={ip_addrs=[ip0={network="" ip_addr=10.241.58.132}]}]} node1={name="c2" nid=2 xname=x1000c0s0b2 bmc_mac=02:68:b3:29:da:98 bmc_ip=10.241.58.138 interfaces=[iface0={ip_addrs=[ip0={network="" ip_addr=10.241.58.133}]}]}]
2025-07-19T10:02:25Z DBG discover-static.go:93 > generating redfish structures to send to SMD
2025-07-19T10:02:25Z DBG discover.go:117 > generating component structure for node with xname x1000c0s0b1
2025-07-19T10:02:25Z DBG discover.go:126 > adding component {x1000c0s0b1 Node On true   1}
2025-07-19T10:02:25Z DBG discover.go:133 > generating redfish structure for node with xname x1000c0s0b1
2025-07-19T10:02:25Z DBG discover.go:157 > node x1000c0s0b1: generating fake BMC System
2025-07-19T10:02:25Z DBG discover.go:197 > node x1000c0s0b1: generated system: {https://ohpc-lenovo-sms:8443/redfish/v1/Systems/x1000c0s0b1 2bcb2c63-a323-4b6a-a6d9-baf4a99c25cc c1 [{ f4:c7:aa:44:41:4a 10.241.58.132 x1000c0s0b1 Interface 0 for c1 false}]}
2025-07-19T10:02:25Z DBG discover.go:206 > BMC x1000c0s0b1: generating fake BMC Manager
2025-07-19T10:02:25Z DBG discover.go:234 > BMC x1000c0s0b1: generated manager: {{https://ohpc-lenovo-sms:8443/redfish/v1/Managers/x1000c0s0b1 3b18bc82-7b28-481e-929e-f37c0d6c4279 x1000c0s0b1 [{ 02:68:b3:29:da:98 10.241.58.139 x1000c0s0b1 Interface for BMC x1000c0s0b1 false}]}  NodeBMC}
2025-07-19T10:02:25Z DBG discover.go:117 > generating component structure for node with xname x1000c0s0b2
2025-07-19T10:02:25Z DBG discover.go:126 > adding component {x1000c0s0b2 Node On true   2}
2025-07-19T10:02:25Z DBG discover.go:133 > generating redfish structure for node with xname x1000c0s0b2
2025-07-19T10:02:25Z DBG discover.go:157 > node x1000c0s0b2: generating fake BMC System
2025-07-19T10:02:25Z DBG discover.go:197 > node x1000c0s0b2: generated system: {https://ohpc-lenovo-sms:8443/redfish/v1/Systems/x1000c0s0b2 85462674-d641-40ff-8a94-4fc4080a335c c2 [{ f4:c7:aa:44:40:f0 10.241.58.133 x1000c0s0b2 Interface 0 for c2 false}]}
2025-07-19T10:02:25Z DBG discover.go:206 > BMC x1000c0s0b2: generating fake BMC Manager
2025-07-19T10:02:25Z DBG discover.go:234 > BMC x1000c0s0b2: generated manager: {{https://ohpc-lenovo-sms:8443/redfish/v1/Managers/x1000c0s0b2 07d4b572-d279-4e8e-8c38-423ab61835d2 x1000c0s0b2 [{ 02:68:b3:29:da:98 10.241.58.138 x1000c0s0b2 Interface for BMC x1000c0s0b2 false}]}  NodeBMC}
2025-07-19T10:02:25Z DBG discover-static.go:100 > generated redfish structures: [{{x1000c0s0b1 NodeBMC c1    false  3b18bc82-7b28-481e-929e-f37c0d6c4279   false false 02:68:b3:29:da:98 10.241.58.139 false  {0001-01-01 00:00:00 +0000 UTC  }} 1 [{https://ohpc-lenovo-sms:8443/redfish/v1/Systems/x1000c0s0b1 2bcb2c63-a323-4b6a-a6d9-baf4a99c25cc c1 [{ f4:c7:aa:44:41:4a 10.241.58.132 x1000c0s0b1 Interface 0 for c1 false}]}] [{{https://ohpc-lenovo-sms:8443/redfish/v1/Managers/x1000c0s0b1 3b18bc82-7b28-481e-929e-f37c0d6c4279 x1000c0s0b1 [{ 02:68:b3:29:da:98 10.241.58.139 x1000c0s0b1 Interface for BMC x1000c0s0b1 false}]}  NodeBMC}]} {{x1000c0s0b2 NodeBMC c2    false  07d4b572-d279-4e8e-8c38-423ab61835d2   false false 02:68:b3:29:da:98 10.241.58.138 false  {0001-01-01 00:00:00 +0000 UTC  }} 1 [{https://ohpc-lenovo-sms:8443/redfish/v1/Systems/x1000c0s0b2 85462674-d641-40ff-8a94-4fc4080a335c c2 [{ f4:c7:aa:44:40:f0 10.241.58.133 x1000c0s0b2 Interface 0 for c2 false}]}] [{{https://ohpc-lenovo-sms:8443/redfish/v1/Managers/x1000c0s0b2 07d4b572-d279-4e8e-8c38-423ab61835d2 x1000c0s0b2 [{ 02:68:b3:29:da:98 10.241.58.138 x1000c0s0b2 Interface for BMC x1000c0s0b2 false}]}  NodeBMC}]}]
2025-07-19T10:02:25Z DBG client.go:245 > POST: https://ohpc-lenovo-sms:8443/hsm/v2/State/Components
2025-07-19T10:02:25Z DBG client.go:266 > Request headers:
2025-07-19T10:02:25Z DBG client.go:268 >   User-Agent: [ochami/0.3.4]
2025-07-19T10:02:25Z DBG client.go:268 >   Authorization: [Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjhhNTRiYjBhLTVmZDQtNGYyZC1iNWU1LWIyMTZkZTVhYTY1MSIsInR5cCI6IkpXVCJ9.eyJhdWQiOltdLCJjbGllbnRfaWQiOiJkOGNiOGU5NS1lOTk3LTRlOGMtYTRjOS04OWMxOThlOWU2YjAiLCJleHAiOjE3NTI5MjI1NjQsImV4dCI6e30sImlhdCI6MTc1MjkxODk2NCwiaXNzIjoiaHR0cHM6Ly9vaHBjLWxlbm92by1zbXMiLCJqdGkiOiI0ZDNhNWUwMC02N2UxLTRjYWEtOTYyNy05NmUyZGNhMDFmMGIiLCJuYmYiOjE3NTI5MTg5NjQsInNjcCI6WyJvcGVuaWQiLCJzbWQucmVhZCJdLCJzdWIiOiJkOGNiOGU5NS1lOTk3LTRlOGMtYTRjOS04OWMxOThlOWU2YjAifQ.Fvb-P7GsWH9yh8XZK2VK6eo9ubyGvvw4j2ZAbd6j9EKzn4JrlX7V3nDevVXo0VdHPpPRHw_OTQes-7CDahVlPRaLsjqwnhcc3wgRkJFysrQq8S4PmyxaR1AVm4cifyuvTFS3BWpwCqZMXoQjNhUYvbp4Uv9KsUed_sqdnuzot0RAq6WiNICSmiBisz9_CamGFlbeeFQmIo3WjtE65hjr1p6eePhH_kWxJzr74lc4VpRysyZmfSBVx2gleh9dWfM2LFli3-qU1b00BDoxxXLo7HcZcT4dKsiJTr9xT_Fg873yAYdSzChB5LJTPrEL6Pa5_NMhS7haWmv01ODI94NZmG2skriD5BH_Z5gJ8sTHNaIrZUvAjWhkRjBkw9cqnZ24mSbBFDf-kZZ8K4PQBzc1beI09XcFioPJruT5Yf5TQOUiuOrf73yr7BEns0QBWnZbeZeOIGpVy4PWrD_MvVIgyQblCJPDVILE6blF31rZ_M3Jq9ww8k-w06DHp5XzgiQaHiVJl4rAuNTnIekBwhWXmFLlkkruO0T-cG9nnNhDHBsRUS1AbIKDVDNqBgEROv0RAoM6duScE-_LWTWApx9U2u96yOSmAA0XiWpCKXHLNGNaUMk_ddjkaFHSwQZ5lZKxjafdbwDmURMKxqA5MukdxiIKc2SWXLGKjgempT6BDYI]
2025-07-19T10:02:25Z DBG client.go:274 > Request body:
2025-07-19T10:02:25Z DBG client.go:275 > {"Components":[{"ID":"x1000c0s0b1","Type":"Node","State":"On","Enabled":true,"NID":1},{"ID":"x1000c0s0b2","Type":"Node","State":"On","Enabled":true,"NID":2}]}
2025-07-19T10:02:25Z DBG client.go:288 > Response status: 500 Internal Server Error
2025-07-19T10:02:25Z DBG client.go:290 > Response headers:
2025-07-19T10:02:25Z DBG client.go:292 >   Date: [Sat, 19 Jul 2025 10:02:25 GMT]
2025-07-19T10:02:25Z DBG client.go:292 >   Content-Length: [0]
2025-07-19T10:02:25Z DBG client.go:309 > No body in response
2025-07-19T10:02:25Z ERR discover-static.go:145 > SMD component request yielded unsuccessful HTTP response error="PostComponents(): failed to POST component(s) to SMD: unsuccessful HTTP status: HTTP/2.0 500 Internal Server Error"
2025-07-19T10:02:25Z DBG client.go:245 > POST: https://ohpc-lenovo-sms:8443/hsm/v2/Inventory/RedfishEndpoints
2025-07-19T10:02:25Z DBG client.go:266 > Request headers:
2025-07-19T10:02:25Z DBG client.go:268 >   Authorization: [Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjhhNTRiYjBhLTVmZDQtNGYyZC1iNWU1LWIyMTZkZTVhYTY1MSIsInR5cCI6IkpXVCJ9.eyJhdWQiOltdLCJjbGllbnRfaWQiOiJkOGNiOGU5NS1lOTk3LTRlOGMtYTRjOS04OWMxOThlOWU2YjAiLCJleHAiOjE3NTI5MjI1NjQsImV4dCI6e30sImlhdCI6MTc1MjkxODk2NCwiaXNzIjoiaHR0cHM6Ly9vaHBjLWxlbm92by1zbXMiLCJqdGkiOiI0ZDNhNWUwMC02N2UxLTRjYWEtOTYyNy05NmUyZGNhMDFmMGIiLCJuYmYiOjE3NTI5MTg5NjQsInNjcCI6WyJvcGVuaWQiLCJzbWQucmVhZCJdLCJzdWIiOiJkOGNiOGU5NS1lOTk3LTRlOGMtYTRjOS04OWMxOThlOWU2YjAifQ.Fvb-P7GsWH9yh8XZK2VK6eo9ubyGvvw4j2ZAbd6j9EKzn4JrlX7V3nDevVXo0VdHPpPRHw_OTQes-7CDahVlPRaLsjqwnhcc3wgRkJFysrQq8S4PmyxaR1AVm4cifyuvTFS3BWpwCqZMXoQjNhUYvbp4Uv9KsUed_sqdnuzot0RAq6WiNICSmiBisz9_CamGFlbeeFQmIo3WjtE65hjr1p6eePhH_kWxJzr74lc4VpRysyZmfSBVx2gleh9dWfM2LFli3-qU1b00BDoxxXLo7HcZcT4dKsiJTr9xT_Fg873yAYdSzChB5LJTPrEL6Pa5_NMhS7haWmv01ODI94NZmG2skriD5BH_Z5gJ8sTHNaIrZUvAjWhkRjBkw9cqnZ24mSbBFDf-kZZ8K4PQBzc1beI09XcFioPJruT5Yf5TQOUiuOrf73yr7BEns0QBWnZbeZeOIGpVy4PWrD_MvVIgyQblCJPDVILE6blF31rZ_M3Jq9ww8k-w06DHp5XzgiQaHiVJl4rAuNTnIekBwhWXmFLlkkruO0T-cG9nnNhDHBsRUS1AbIKDVDNqBgEROv0RAoM6duScE-_LWTWApx9U2u96yOSmAA0XiWpCKXHLNGNaUMk_ddjkaFHSwQZ5lZKxjafdbwDmURMKxqA5MukdxiIKc2SWXLGKjgempT6BDYI]
2025-07-19T10:02:25Z DBG client.go:268 >   User-Agent: [ochami/0.3.4]
2025-07-19T10:02:25Z DBG client.go:274 > Request body:
2025-07-19T10:02:25Z DBG client.go:275 > {"ID":"x1000c0s0b1","Type":"NodeBMC","Name":"c1","UUID":"3b18bc82-7b28-481e-929e-f37c0d6c4279","MACAddr":"02:68:b3:29:da:98","IPAddress":"10.241.58.139","DiscoveryInfo":{"LastAttempt":"0001-01-01T00:00:00Z"},"SchemaVersion":1,"Systems":[{"uri":"https://ohpc-lenovo-sms:8443/redfish/v1/Systems/x1000c0s0b1","uuid":"2bcb2c63-a323-4b6a-a6d9-baf4a99c25cc","name":"c1","ethernet_interfaces":[{"mac":"f4:c7:aa:44:41:4a","ip":"10.241.58.132","name":"x1000c0s0b1","description":"Interface 0 for c1"}]}],"Managers":[{"uri":"https://ohpc-lenovo-sms:8443/redfish/v1/Managers/x1000c0s0b1","uuid":"3b18bc82-7b28-481e-929e-f37c0d6c4279","name":"x1000c0s0b1","ethernet_interfaces":[{"mac":"02:68:b3:29:da:98","ip":"10.241.58.139","name":"x1000c0s0b1","description":"Interface for BMC x1000c0s0b1"}],"description":"","type":"NodeBMC"}]}
2025-07-19T10:02:26Z DBG client.go:288 > Response status: 500 Internal Server Error
2025-07-19T10:02:26Z DBG client.go:290 > Response headers:
2025-07-19T10:02:26Z DBG client.go:292 >   Date: [Sat, 19 Jul 2025 10:02:26 GMT]
2025-07-19T10:02:26Z DBG client.go:292 >   Content-Length: [0]
2025-07-19T10:02:26Z DBG client.go:309 > No body in response
2025-07-19T10:02:26Z DBG client.go:245 > POST: https://ohpc-lenovo-sms:8443/hsm/v2/Inventory/RedfishEndpoints
2025-07-19T10:02:26Z DBG client.go:266 > Request headers:
2025-07-19T10:02:26Z DBG client.go:268 >   Authorization: [Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjhhNTRiYjBhLTVmZDQtNGYyZC1iNWU1LWIyMTZkZTVhYTY1MSIsInR5cCI6IkpXVCJ9.eyJhdWQiOltdLCJjbGllbnRfaWQiOiJkOGNiOGU5NS1lOTk3LTRlOGMtYTRjOS04OWMxOThlOWU2YjAiLCJleHAiOjE3NTI5MjI1NjQsImV4dCI6e30sImlhdCI6MTc1MjkxODk2NCwiaXNzIjoiaHR0cHM6Ly9vaHBjLWxlbm92by1zbXMiLCJqdGkiOiI0ZDNhNWUwMC02N2UxLTRjYWEtOTYyNy05NmUyZGNhMDFmMGIiLCJuYmYiOjE3NTI5MTg5NjQsInNjcCI6WyJvcGVuaWQiLCJzbWQucmVhZCJdLCJzdWIiOiJkOGNiOGU5NS1lOTk3LTRlOGMtYTRjOS04OWMxOThlOWU2YjAifQ.Fvb-P7GsWH9yh8XZK2VK6eo9ubyGvvw4j2ZAbd6j9EKzn4JrlX7V3nDevVXo0VdHPpPRHw_OTQes-7CDahVlPRaLsjqwnhcc3wgRkJFysrQq8S4PmyxaR1AVm4cifyuvTFS3BWpwCqZMXoQjNhUYvbp4Uv9KsUed_sqdnuzot0RAq6WiNICSmiBisz9_CamGFlbeeFQmIo3WjtE65hjr1p6eePhH_kWxJzr74lc4VpRysyZmfSBVx2gleh9dWfM2LFli3-qU1b00BDoxxXLo7HcZcT4dKsiJTr9xT_Fg873yAYdSzChB5LJTPrEL6Pa5_NMhS7haWmv01ODI94NZmG2skriD5BH_Z5gJ8sTHNaIrZUvAjWhkRjBkw9cqnZ24mSbBFDf-kZZ8K4PQBzc1beI09XcFioPJruT5Yf5TQOUiuOrf73yr7BEns0QBWnZbeZeOIGpVy4PWrD_MvVIgyQblCJPDVILE6blF31rZ_M3Jq9ww8k-w06DHp5XzgiQaHiVJl4rAuNTnIekBwhWXmFLlkkruO0T-cG9nnNhDHBsRUS1AbIKDVDNqBgEROv0RAoM6duScE-_LWTWApx9U2u96yOSmAA0XiWpCKXHLNGNaUMk_ddjkaFHSwQZ5lZKxjafdbwDmURMKxqA5MukdxiIKc2SWXLGKjgempT6BDYI]
2025-07-19T10:02:26Z DBG client.go:268 >   User-Agent: [ochami/0.3.4]
2025-07-19T10:02:26Z DBG client.go:274 > Request body:
2025-07-19T10:02:26Z DBG client.go:275 > {"ID":"x1000c0s0b2","Type":"NodeBMC","Name":"c2","UUID":"07d4b572-d279-4e8e-8c38-423ab61835d2","MACAddr":"02:68:b3:29:da:98","IPAddress":"10.241.58.138","DiscoveryInfo":{"LastAttempt":"0001-01-01T00:00:00Z"},"SchemaVersion":1,"Systems":[{"uri":"https://ohpc-lenovo-sms:8443/redfish/v1/Systems/x1000c0s0b2","uuid":"85462674-d641-40ff-8a94-4fc4080a335c","name":"c2","ethernet_interfaces":[{"mac":"f4:c7:aa:44:40:f0","ip":"10.241.58.133","name":"x1000c0s0b2","description":"Interface 0 for c2"}]}],"Managers":[{"uri":"https://ohpc-lenovo-sms:8443/redfish/v1/Managers/x1000c0s0b2","uuid":"07d4b572-d279-4e8e-8c38-423ab61835d2","name":"x1000c0s0b2","ethernet_interfaces":[{"mac":"02:68:b3:29:da:98","ip":"10.241.58.138","name":"x1000c0s0b2","description":"Interface for BMC x1000c0s0b2"}],"description":"","type":"NodeBMC"}]}
2025-07-19T10:02:26Z DBG client.go:288 > Response status: 500 Internal Server Error
2025-07-19T10:02:26Z DBG client.go:290 > Response headers:
2025-07-19T10:02:26Z DBG client.go:292 >   Date: [Sat, 19 Jul 2025 10:02:26 GMT]
2025-07-19T10:02:26Z DBG client.go:292 >   Content-Length: [0]
2025-07-19T10:02:26Z DBG client.go:309 > No body in response
2025-07-19T10:02:26Z ERR discover-static.go:234 > SMD redfish endpoint request yielded unsuccessful HTTP response error="PostRedfishEndpointsV2(): failed to POST redfish endpoint to SMD: unsuccessful HTTP status: HTTP/2.0 500 Internal Server Error"
2025-07-19T10:02:26Z ERR discover-static.go:234 > SMD redfish endpoint request yielded unsuccessful HTTP response error="PostRedfishEndpointsV2(): failed to POST redfish endpoint to SMD: unsuccessful HTTP status: HTTP/2.0 500 Internal Server Error"
2025-07-19T10:02:26Z DBG client.go:245 > POST: https://ohpc-lenovo-sms:8443/hsm/v2/Inventory/EthernetInterfaces
2025-07-19T10:02:26Z DBG client.go:266 > Request headers:
2025-07-19T10:02:26Z DBG client.go:268 >   User-Agent: [ochami/0.3.4]
2025-07-19T10:02:26Z DBG client.go:268 >   Authorization: [Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjhhNTRiYjBhLTVmZDQtNGYyZC1iNWU1LWIyMTZkZTVhYTY1MSIsInR5cCI6IkpXVCJ9.eyJhdWQiOltdLCJjbGllbnRfaWQiOiJkOGNiOGU5NS1lOTk3LTRlOGMtYTRjOS04OWMxOThlOWU2YjAiLCJleHAiOjE3NTI5MjI1NjQsImV4dCI6e30sImlhdCI6MTc1MjkxODk2NCwiaXNzIjoiaHR0cHM6Ly9vaHBjLWxlbm92by1zbXMiLCJqdGkiOiI0ZDNhNWUwMC02N2UxLTRjYWEtOTYyNy05NmUyZGNhMDFmMGIiLCJuYmYiOjE3NTI5MTg5NjQsInNjcCI6WyJvcGVuaWQiLCJzbWQucmVhZCJdLCJzdWIiOiJkOGNiOGU5NS1lOTk3LTRlOGMtYTRjOS04OWMxOThlOWU2YjAifQ.Fvb-P7GsWH9yh8XZK2VK6eo9ubyGvvw4j2ZAbd6j9EKzn4JrlX7V3nDevVXo0VdHPpPRHw_OTQes-7CDahVlPRaLsjqwnhcc3wgRkJFysrQq8S4PmyxaR1AVm4cifyuvTFS3BWpwCqZMXoQjNhUYvbp4Uv9KsUed_sqdnuzot0RAq6WiNICSmiBisz9_CamGFlbeeFQmIo3WjtE65hjr1p6eePhH_kWxJzr74lc4VpRysyZmfSBVx2gleh9dWfM2LFli3-qU1b00BDoxxXLo7HcZcT4dKsiJTr9xT_Fg873yAYdSzChB5LJTPrEL6Pa5_NMhS7haWmv01ODI94NZmG2skriD5BH_Z5gJ8sTHNaIrZUvAjWhkRjBkw9cqnZ24mSbBFDf-kZZ8K4PQBzc1beI09XcFioPJruT5Yf5TQOUiuOrf73yr7BEns0QBWnZbeZeOIGpVy4PWrD_MvVIgyQblCJPDVILE6blF31rZ_M3Jq9ww8k-w06DHp5XzgiQaHiVJl4rAuNTnIekBwhWXmFLlkkruO0T-cG9nnNhDHBsRUS1AbIKDVDNqBgEROv0RAoM6duScE-_LWTWApx9U2u96yOSmAA0XiWpCKXHLNGNaUMk_ddjkaFHSwQZ5lZKxjafdbwDmURMKxqA5MukdxiIKc2SWXLGKjgempT6BDYI]
2025-07-19T10:02:26Z DBG client.go:274 > Request body:
2025-07-19T10:02:26Z DBG client.go:275 > {"ID":"","ComponentID":"x1000c0s0b1","Type":"Node","Description":"Interface 0 for c1","MACAddress":"f4:c7:aa:44:41:4a","IPAddresses":[{"IPAddress":"10.241.58.132","Network":""}]}
2025-07-19T10:02:26Z DBG client.go:288 > Response status: 500 Internal Server Error
2025-07-19T10:02:26Z DBG client.go:290 > Response headers:
2025-07-19T10:02:26Z DBG client.go:292 >   Date: [Sat, 19 Jul 2025 10:02:26 GMT]
2025-07-19T10:02:26Z DBG client.go:292 >   Content-Length: [0]
2025-07-19T10:02:26Z DBG client.go:309 > No body in response
2025-07-19T10:02:26Z DBG client.go:245 > POST: https://ohpc-lenovo-sms:8443/hsm/v2/Inventory/EthernetInterfaces
2025-07-19T10:02:26Z DBG client.go:266 > Request headers:
2025-07-19T10:02:26Z DBG client.go:268 >   User-Agent: [ochami/0.3.4]
2025-07-19T10:02:26Z DBG client.go:268 >   Authorization: [Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjhhNTRiYjBhLTVmZDQtNGYyZC1iNWU1LWIyMTZkZTVhYTY1MSIsInR5cCI6IkpXVCJ9.eyJhdWQiOltdLCJjbGllbnRfaWQiOiJkOGNiOGU5NS1lOTk3LTRlOGMtYTRjOS04OWMxOThlOWU2YjAiLCJleHAiOjE3NTI5MjI1NjQsImV4dCI6e30sImlhdCI6MTc1MjkxODk2NCwiaXNzIjoiaHR0cHM6Ly9vaHBjLWxlbm92by1zbXMiLCJqdGkiOiI0ZDNhNWUwMC02N2UxLTRjYWEtOTYyNy05NmUyZGNhMDFmMGIiLCJuYmYiOjE3NTI5MTg5NjQsInNjcCI6WyJvcGVuaWQiLCJzbWQucmVhZCJdLCJzdWIiOiJkOGNiOGU5NS1lOTk3LTRlOGMtYTRjOS04OWMxOThlOWU2YjAifQ.Fvb-P7GsWH9yh8XZK2VK6eo9ubyGvvw4j2ZAbd6j9EKzn4JrlX7V3nDevVXo0VdHPpPRHw_OTQes-7CDahVlPRaLsjqwnhcc3wgRkJFysrQq8S4PmyxaR1AVm4cifyuvTFS3BWpwCqZMXoQjNhUYvbp4Uv9KsUed_sqdnuzot0RAq6WiNICSmiBisz9_CamGFlbeeFQmIo3WjtE65hjr1p6eePhH_kWxJzr74lc4VpRysyZmfSBVx2gleh9dWfM2LFli3-qU1b00BDoxxXLo7HcZcT4dKsiJTr9xT_Fg873yAYdSzChB5LJTPrEL6Pa5_NMhS7haWmv01ODI94NZmG2skriD5BH_Z5gJ8sTHNaIrZUvAjWhkRjBkw9cqnZ24mSbBFDf-kZZ8K4PQBzc1beI09XcFioPJruT5Yf5TQOUiuOrf73yr7BEns0QBWnZbeZeOIGpVy4PWrD_MvVIgyQblCJPDVILE6blF31rZ_M3Jq9ww8k-w06DHp5XzgiQaHiVJl4rAuNTnIekBwhWXmFLlkkruO0T-cG9nnNhDHBsRUS1AbIKDVDNqBgEROv0RAoM6duScE-_LWTWApx9U2u96yOSmAA0XiWpCKXHLNGNaUMk_ddjkaFHSwQZ5lZKxjafdbwDmURMKxqA5MukdxiIKc2SWXLGKjgempT6BDYI]
2025-07-19T10:02:26Z DBG client.go:274 > Request body:
2025-07-19T10:02:26Z DBG client.go:275 > {"ID":"","ComponentID":"x1000c0s0b2","Type":"Node","Description":"Interface 0 for c2","MACAddress":"f4:c7:aa:44:40:f0","IPAddresses":[{"IPAddress":"10.241.58.133","Network":""}]}
2025-07-19T10:02:26Z DBG client.go:288 > Response status: 500 Internal Server Error
2025-07-19T10:02:26Z DBG client.go:290 > Response headers:
2025-07-19T10:02:26Z DBG client.go:292 >   Date: [Sat, 19 Jul 2025 10:02:26 GMT]
2025-07-19T10:02:26Z DBG client.go:292 >   Content-Length: [0]
2025-07-19T10:02:26Z DBG client.go:309 > No body in response
2025-07-19T10:02:26Z ERR discover-static.go:317 > SMD ethernet interface request yielded unsuccessful HTTP response error="PostEthernetInterfaces(): failed to POST ethernet interface(s) to SMD: unsuccessful HTTP status: HTTP/2.0 500 Internal Server Error"
2025-07-19T10:02:26Z ERR discover-static.go:317 > SMD ethernet interface request yielded unsuccessful HTTP response error="PostEthernetInterfaces(): failed to POST ethernet interface(s) to SMD: unsuccessful HTTP status: HTTP/2.0 500 Internal Server Error"
2025-07-19T10:02:26Z DBG client.go:245 > POST: https://ohpc-lenovo-sms:8443/hsm/v2/groups
2025-07-19T10:02:26Z DBG client.go:266 > Request headers:
2025-07-19T10:02:26Z DBG client.go:268 >   User-Agent: [ochami/0.3.4]
2025-07-19T10:02:26Z DBG client.go:268 >   Authorization: [Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjhhNTRiYjBhLTVmZDQtNGYyZC1iNWU1LWIyMTZkZTVhYTY1MSIsInR5cCI6IkpXVCJ9.eyJhdWQiOltdLCJjbGllbnRfaWQiOiJkOGNiOGU5NS1lOTk3LTRlOGMtYTRjOS04OWMxOThlOWU2YjAiLCJleHAiOjE3NTI5MjI1NjQsImV4dCI6e30sImlhdCI6MTc1MjkxODk2NCwiaXNzIjoiaHR0cHM6Ly9vaHBjLWxlbm92by1zbXMiLCJqdGkiOiI0ZDNhNWUwMC02N2UxLTRjYWEtOTYyNy05NmUyZGNhMDFmMGIiLCJuYmYiOjE3NTI5MTg5NjQsInNjcCI6WyJvcGVuaWQiLCJzbWQucmVhZCJdLCJzdWIiOiJkOGNiOGU5NS1lOTk3LTRlOGMtYTRjOS04OWMxOThlOWU2YjAifQ.Fvb-P7GsWH9yh8XZK2VK6eo9ubyGvvw4j2ZAbd6j9EKzn4JrlX7V3nDevVXo0VdHPpPRHw_OTQes-7CDahVlPRaLsjqwnhcc3wgRkJFysrQq8S4PmyxaR1AVm4cifyuvTFS3BWpwCqZMXoQjNhUYvbp4Uv9KsUed_sqdnuzot0RAq6WiNICSmiBisz9_CamGFlbeeFQmIo3WjtE65hjr1p6eePhH_kWxJzr74lc4VpRysyZmfSBVx2gleh9dWfM2LFli3-qU1b00BDoxxXLo7HcZcT4dKsiJTr9xT_Fg873yAYdSzChB5LJTPrEL6Pa5_NMhS7haWmv01ODI94NZmG2skriD5BH_Z5gJ8sTHNaIrZUvAjWhkRjBkw9cqnZ24mSbBFDf-kZZ8K4PQBzc1beI09XcFioPJruT5Yf5TQOUiuOrf73yr7BEns0QBWnZbeZeOIGpVy4PWrD_MvVIgyQblCJPDVILE6blF31rZ_M3Jq9ww8k-w06DHp5XzgiQaHiVJl4rAuNTnIekBwhWXmFLlkkruO0T-cG9nnNhDHBsRUS1AbIKDVDNqBgEROv0RAoM6duScE-_LWTWApx9U2u96yOSmAA0XiWpCKXHLNGNaUMk_ddjkaFHSwQZ5lZKxjafdbwDmURMKxqA5MukdxiIKc2SWXLGKjgempT6BDYI]
2025-07-19T10:02:26Z DBG client.go:274 > Request body:
2025-07-19T10:02:26Z DBG client.go:275 > {"label":"compute","description":"The compute group","members":{"ids":["x1000c0s0b1","x1000c0s0b2"]}}
2025-07-19T10:02:26Z DBG client.go:288 > Response status: 500 Internal Server Error
2025-07-19T10:02:26Z DBG client.go:290 > Response headers:
2025-07-19T10:02:26Z DBG client.go:292 >   Date: [Sat, 19 Jul 2025 10:02:26 GMT]
2025-07-19T10:02:26Z DBG client.go:292 >   Content-Length: [0]
2025-07-19T10:02:26Z DBG client.go:309 > No body in response
2025-07-19T10:02:26Z ERR discover-static.go:422 > SMD groups request yielded unsuccessful HTTP response error="PostGroups(): failed to POST group to SMD: unsuccessful HTTP status: HTTP/2.0 500 Internal Server Error"
2025-07-19T10:02:26Z ERR lib.go:433 > see 'ochami discover static --help' for long command help
2025-07-19T10:02:26Z WRN discover-static.go:434 > component requests completed with errors
2025-07-19T10:02:26Z WRN discover-static.go:438 > redfish endpoint requests completed with errors
2025-07-19T10:02:26Z WRN discover-static.go:442 > ethernet interface requests completed with errors
2025-07-19T10:02:26Z WRN discover-static.go:446 > group requests completed with errors

Looking at the SMD server I see:

Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]: 2025/07/19 10:02:26 [smd/pRtPgAHuJv-000029] "POST http://ohpc-lenovo-sms:8443/hsm/v2/Inventory/EthernetInterfaces HTTP/1.1" from 10.241.58.134 - 500 0B in 1.551187ms
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]: 10:02AM INF Request bytes_in=178 bytes_out=0 duration=0.063776 method=POST remote_addr=10.241.58.134 request_id=smd/pRtPgAHuJv-000030 request_uri=/hsm/v2/Inventory/EthernetInterfaces status= status_code=0 user_agent=ochami/0.3.4
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]: 
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:  panic: runtime error: invalid memory address or nil pointer dereference
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:  
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:  -> github.com/OpenCHAMI/jwtauth/v5.(*JWTAuth).parse
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:  ->   /home/runner/go/pkg/mod/github.com/!open!c!h!a!m!i/jwtauth/v5@v5.0.0-20240321222802-e6cb468a2a18/jwtauth.go:170
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]: 
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     github.com/OpenCHAMI/jwtauth/v5.(*JWTAuth).Decode
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/!open!c!h!a!m!i/jwtauth/v5@v5.0.0-20240321222802-e6cb468a2a18/jwtauth.go:157
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     github.com/OpenCHAMI/jwtauth/v5.VerifyToken
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/!open!c!h!a!m!i/jwtauth/v5@v5.0.0-20240321222802-e6cb468a2a18/jwtauth.go:122
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     github.com/OpenCHAMI/jwtauth/v5.VerifyRequest
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/!open!c!h!a!m!i/jwtauth/v5@v5.0.0-20240321222802-e6cb468a2a18/jwtauth.go:117
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     main.(*SmD).NewRouter.func2.Verifier.Verify.3.1
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/!open!c!h!a!m!i/jwtauth/v5@v5.0.0-20240321222802-e6cb468a2a18/jwtauth.go:93
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     net/http.HandlerFunc.ServeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /opt/hostedtoolcache/go/1.23.5/x64/src/net/http/server.go:2220
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     github.com/go-chi/chi/v5.(*ChainHandler).ServeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/go-chi/chi/v5@v5.1.0/chain.go:31
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     github.com/go-chi/chi/v5.(*Mux).routeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/go-chi/chi/v5@v5.1.0/mux.go:459
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     net/http.HandlerFunc.ServeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /opt/hostedtoolcache/go/1.23.5/x64/src/net/http/server.go:2220
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     main.(*SmD).NewRouter.Timeout.func4.1
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/go-chi/chi/v5@v5.1.0/middleware/timeout.go:44
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     net/http.HandlerFunc.ServeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /opt/hostedtoolcache/go/1.23.5/x64/src/net/http/server.go:2220
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     main.(*SmD).NewRouter.OpenCHAMILogger.func3.1
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/openchami/chi-middleware/log@v0.0.0-20240812224658-b16b83c70700/middleware.go:45
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     net/http.HandlerFunc.ServeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /opt/hostedtoolcache/go/1.23.5/x64/src/net/http/server.go:2220
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     github.com/go-chi/chi/v5/middleware.StripSlashes.func1
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/go-chi/chi/v5@v5.1.0/middleware/strip.go:30
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     net/http.HandlerFunc.ServeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /opt/hostedtoolcache/go/1.23.5/x64/src/net/http/server.go:2220
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     github.com/go-chi/chi/v5/middleware.Recoverer.func1
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/go-chi/chi/v5@v5.1.0/middleware/recoverer.go:45
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     net/http.HandlerFunc.ServeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /opt/hostedtoolcache/go/1.23.5/x64/src/net/http/server.go:2220
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     github.com/go-chi/chi/v5/middleware.init.0.RequestLogger.func1.1
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/go-chi/chi/v5@v5.1.0/middleware/logger.go:55
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     net/http.HandlerFunc.ServeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /opt/hostedtoolcache/go/1.23.5/x64/src/net/http/server.go:2220
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     github.com/go-chi/chi/v5/middleware.RealIP.func1
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/go-chi/chi/v5@v5.1.0/middleware/realip.go:36
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     net/http.HandlerFunc.ServeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /opt/hostedtoolcache/go/1.23.5/x64/src/net/http/server.go:2220
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     github.com/go-chi/chi/v5/middleware.RequestID.func1
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/go-chi/chi/v5@v5.1.0/middleware/request_id.go:76
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     net/http.HandlerFunc.ServeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /opt/hostedtoolcache/go/1.23.5/x64/src/net/http/server.go:2220
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     github.com/go-chi/chi/v5.(*Mux).ServeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/go-chi/chi/v5@v5.1.0/mux.go:90
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     net/http.serverHandler.ServeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /opt/hostedtoolcache/go/1.23.5/x64/src/net/http/server.go:3210
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     net/http.(*conn).serve
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /opt/hostedtoolcache/go/1.23.5/x64/src/net/http/server.go:2092
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     created by net/http.(*Server).Serve in goroutine 1
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /opt/hostedtoolcache/go/1.23.5/x64/src/net/http/server.go:3360
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]: 2025/07/19 10:02:26 [smd/pRtPgAHuJv-000030] "POST http://ohpc-lenovo-sms:8443/hsm/v2/Inventory/EthernetInterfaces HTTP/1.1" from 10.241.58.134 - 500 0B in 1.602901ms
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]: 10:02AM INF Request bytes_in=101 bytes_out=0 duration=0.063512 method=POST remote_addr=10.241.58.134 request_id=smd/pRtPgAHuJv-000031 request_uri=/hsm/v2/groups status= status_code=0 user_agent=ochami/0.3.4
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]: 2025/07/19 10:02:26 [smd/pRtPgAHuJv-000031] "POST http://ohpc-lenovo-sms:8443/hsm/v2/groups HTTP/1.1" from 10.241.58.134 - 500 0B in 1.514799ms
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]: 
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:  panic: runtime error: invalid memory address or nil pointer dereference
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:  
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:  -> github.com/OpenCHAMI/jwtauth/v5.(*JWTAuth).parse
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:  ->   /home/runner/go/pkg/mod/github.com/!open!c!h!a!m!i/jwtauth/v5@v5.0.0-20240321222802-e6cb468a2a18/jwtauth.go:170
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]: 
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     github.com/OpenCHAMI/jwtauth/v5.(*JWTAuth).Decode
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/!open!c!h!a!m!i/jwtauth/v5@v5.0.0-20240321222802-e6cb468a2a18/jwtauth.go:157
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     github.com/OpenCHAMI/jwtauth/v5.VerifyToken
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/!open!c!h!a!m!i/jwtauth/v5@v5.0.0-20240321222802-e6cb468a2a18/jwtauth.go:122
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     github.com/OpenCHAMI/jwtauth/v5.VerifyRequest
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/!open!c!h!a!m!i/jwtauth/v5@v5.0.0-20240321222802-e6cb468a2a18/jwtauth.go:117
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     main.(*SmD).NewRouter.func2.Verifier.Verify.3.1
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/!open!c!h!a!m!i/jwtauth/v5@v5.0.0-20240321222802-e6cb468a2a18/jwtauth.go:93
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     net/http.HandlerFunc.ServeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /opt/hostedtoolcache/go/1.23.5/x64/src/net/http/server.go:2220
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     github.com/go-chi/chi/v5.(*ChainHandler).ServeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/go-chi/chi/v5@v5.1.0/chain.go:31
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     github.com/go-chi/chi/v5.(*Mux).routeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/go-chi/chi/v5@v5.1.0/mux.go:459
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     net/http.HandlerFunc.ServeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /opt/hostedtoolcache/go/1.23.5/x64/src/net/http/server.go:2220
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     main.(*SmD).NewRouter.Timeout.func4.1
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/go-chi/chi/v5@v5.1.0/middleware/timeout.go:44
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     net/http.HandlerFunc.ServeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /opt/hostedtoolcache/go/1.23.5/x64/src/net/http/server.go:2220
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     main.(*SmD).NewRouter.OpenCHAMILogger.func3.1
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/openchami/chi-middleware/log@v0.0.0-20240812224658-b16b83c70700/middleware.go:45
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     net/http.HandlerFunc.ServeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /opt/hostedtoolcache/go/1.23.5/x64/src/net/http/server.go:2220
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     github.com/go-chi/chi/v5/middleware.StripSlashes.func1
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/go-chi/chi/v5@v5.1.0/middleware/strip.go:30
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     net/http.HandlerFunc.ServeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /opt/hostedtoolcache/go/1.23.5/x64/src/net/http/server.go:2220
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     github.com/go-chi/chi/v5/middleware.Recoverer.func1
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/go-chi/chi/v5@v5.1.0/middleware/recoverer.go:45
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     net/http.HandlerFunc.ServeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /opt/hostedtoolcache/go/1.23.5/x64/src/net/http/server.go:2220
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     github.com/go-chi/chi/v5/middleware.init.0.RequestLogger.func1.1
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/go-chi/chi/v5@v5.1.0/middleware/logger.go:55
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     net/http.HandlerFunc.ServeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /opt/hostedtoolcache/go/1.23.5/x64/src/net/http/server.go:2220
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     github.com/go-chi/chi/v5/middleware.RealIP.func1
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/go-chi/chi/v5@v5.1.0/middleware/realip.go:36
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     net/http.HandlerFunc.ServeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /opt/hostedtoolcache/go/1.23.5/x64/src/net/http/server.go:2220
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     github.com/go-chi/chi/v5/middleware.RequestID.func1
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/go-chi/chi/v5@v5.1.0/middleware/request_id.go:76
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     net/http.HandlerFunc.ServeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /opt/hostedtoolcache/go/1.23.5/x64/src/net/http/server.go:2220
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     github.com/go-chi/chi/v5.(*Mux).ServeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /home/runner/go/pkg/mod/github.com/go-chi/chi/v5@v5.1.0/mux.go:90
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     net/http.serverHandler.ServeHTTP
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /opt/hostedtoolcache/go/1.23.5/x64/src/net/http/server.go:3210
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     net/http.(*conn).serve
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /opt/hostedtoolcache/go/1.23.5/x64/src/net/http/server.go:2092
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     created by net/http.(*Server).Serve in goroutine 1
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:       /opt/hostedtoolcache/go/1.23.5/x64/src/net/http/server.go:3360
Jul 19 10:02:26 ohpc-lenovo-sms smd[381713]:     
Jul 19 10:02:56 ohpc-lenovo-sms smd[381713]: 10.89.1.8 - - [19/Jul/2025:10:02:56 +0000] "GET /hsm/v2/Inventory/EthernetInterfaces/ HTTP/1.1" 200 3 "" "Go-http-client/1.1"
Jul 19 10:02:56 ohpc-lenovo-sms smd[381713]: 2025/07/19 10:02:56 [smd/pRtPgAHuJv-000032] "GET http://smd:27779/hsm/v2/Inventory/EthernetInterfaces/ HTTP/1.1" from 10.89.1.8:42016 - 200 3B in 922.877µs
Jul 19 10:02:56 ohpc-lenovo-sms smd[381713]: 10:02AM INF Request bytes_in=0 bytes_out=3 duration=0.788222 method=GET remote_addr=10.89.1.8:42016 request_id=smd/pRtPgAHuJv-000032 request_uri=/hsm/v2/Inventory/EthernetInterfaces/ status=OK status_code=200 user_agent=Go-http-client/1.1
Jul 19 10:03:56 ohpc-lenovo-sms smd[381713]: 10.89.1.8 - - [19/Jul/2025:10:03:56 +0000] "GET /hsm/v2/Inventory/EthernetInterfaces/ HTTP/1.1" 200 3 "" "Go-http-client/1.1"
Jul 19 10:03:56 ohpc-lenovo-sms smd[381713]: 2025/07/19 10:03:56 [smd/pRtPgAHuJv-000033] "GET http://smd:27779/hsm/v2/Inventory/EthernetInterfaces/ HTTP/1.1" from 10.89.1.8:55202 - 200 3B in 894.117µs
Jul 19 10:03:56 ohpc-lenovo-sms smd[381713]: 10:03AM INF Request bytes_in=0 bytes_out=3 duration=0.746083 method=GET remote_addr=10.89.1.8:55202 request_id=smd/pRtPgAHuJv-000033 request_uri=/hsm/v2/Inventory/EthernetInterfaces/ status=OK status_code=200 user_agent=Go-http-client/1.1

[synackd]

This appears to be an issue with SMD. Can we move this issue to the SMD repo?

[synackd]

Thanks @alexlovelltroy .

Upon further inspection, the issue looks to be with the JWT decoder in the jwtauth repo here:

https://github.com/OpenCHAMI/jwtauth/blob/e6cb468a2a18bad529acaa3dd17fdaf9dbd2696a/jwtauth.go#L157

Looks to be a lack of input validation in the parse function at first glance. Interesting that even though I see the JWT being presented in the request headers, parse sees it as nil.

[synackd]

Let's keep it here until we can figure out what exactly is causing the issue, though it looks like input validation could be a part of it.

[alexlovelltroy]

Agreed. jwtauth is actually an upstream project that we only maintain a fork of because we found and fixed a bug that they haven't accepted.

[synackd]

@davidallendj, would you be able to take a look at this?

[davidallendj]

Digging a little deeper, I see that there's an error check in the v3 version of the Parse and the fork is still v2 which does not have it. We can try updating the fork to use v3 and see if that fixes it.

jwtauth is actually an upstream project that we only maintain a fork of because we found and fixed a bug that they haven't accepted.

We need to maintain the fork until upstream merges the JWKSKeySet function to verify key sets. The PR is still open here.

[alexlovelltroy]

I'm suspicious that this isn't the real problem. A failure in Parse means that the JWT is invalid. We should update things for better error reporting, but I'm definitely curious about why the jwt doesn't parse.

[travisbcotton]

This happens when SMD fails to get a token from whatever SMD_JWKS_URL is set to. In this case it is failing to find the opaal instance:

failed to initialize auth token: failed to fetch "http://opaal:3333/keys": Get "http://opaal:3333/keys": dial tcp: lookup opaal on 10.89.3.1:53: no such host

SMD should handle this case better, alerting or failing to start with a clearer message. Instead it continues to run and whenever requests are made that require a token to be validated we get the invalid memory address or nil pointer dereference panic