Commands for ipmi and bmc die with timeout

[mekanix]

I have the following in my ipmi-exporter.yml

modules:
  default:
    collectors:
    - bmc
    - ipmi
    - chassis
    - sel
    - sel-events
    driver: "LAN_2_0"
    pass: "mypass"
    privilege: "user"
    user: "myuser"

This is part of my ipmi-targets.yml

- labels:
    job: ipmi_exporter
  targets:
  - host1
  - host2
  . . .

My prometheus.yml portion for this exporter is this

- file_sd_configs:
  - files:
    - /etc/prometheus/ipmi-targets.yml
    refresh_interval: 10m
  job_name: ipmi-exporter
  metrics_path: /ipmi
  params:
    module:
    - default

When I do ps ax | grep ipmi-sel I see the following

ipmi-sel --quiet-cache --comma-separated-output --no-header-output --sdr-cache-recreate --output-event-state --interpret-oem-data --entity-sensor-names --config-file /tmp/ipmi_exporter-cc2225c5535b995b1eb9a53327724295 -h host1

I thought that config file will contain user/pass, but no, that file is always zero in length. What am I doing wrong?

[bitfehler]

This is working as intended. The config is not a regular file, it is a named pipe. Hence, it can only be read once, and it will be read by the exporter itself (edit: actually, by ipmi-sel, executed by the exporter) immediately after creation.

Please describe the actual problems you are observing, e.g. any errors you get in the logs or such.

[mekanix]

The ipmi-sel process is just stuck and dies with timeout. If I use the same command I posted in previous post and add -u <user> -P the command works. I didn't realize the config file is a pipe, so it makes sense it's zero length, thank you for explaining that.

[mekanix]

Is there an easy(ish) way to get data that was written to the config file (the pipe)? I'd like to manually run ipmi-sel with the same args and config that the exporter does and see what I get. Ideally, send the PR or somehow change the config once I realize what's wrong.

[bitfehler]

Strange. And the SEL collector the only one that is not working?

[bitfehler]

Also, can you post the error message from ipmi-sel that should show up in the exporter log?

[mekanix]

source=collector.go:120 msg="Collector failed" name=sel-events error="error running ipmi-sel: exit status 1: ipmi-sel: connection timeout\n"
source=collector.go:120 msg="Collector failed" name=bmc error="error running bmc-info: exit status 1: bmc-info: connection timeout\n"
source=collector.go:120 msg="Collector failed" name=ipmi error="error running ipmimonitoring: exit status 1: /usr/sbin/ipmi-sensors: connection timeout\n"

[bitfehler]

Ok, this already looks quite different then. Looking at your config in the initial post, this looks a bit odd, actually. Are you running an exporter on every host you're trying to scrape? Or do you have one central exporter instance for all hosts?

[mekanix]

I have a central exporter instance.

[mekanix]

If we assume I have a centralized ipmi-exporter and prometheus instance on the same server, what should the config look like? The reason for centralized node is that we have JBODs with IPMI so we can't install exporter on them. Thank you!

[bitfehler]

Does your prometheus config not have a relabeling rule for setting __address__ and __param_target? Or did you just omit those?

[mekanix]

Sorry, just my bad copy/paste. This is the whole section

- file_sd_configs:
  - files:
    - /etc/prometheus/ipmi-targets.yml
    refresh_interval: 10m
  job_name: ipmi-exporter
  metrics_path: /ipmi
  params:
    module:
    - default
  relabel_configs:
  - action: replace
    regex: (.*)
    replacement: ${1}
    separator: ;
    source_labels:
    - __address__
    target_label: __param_target
  - action: replace
    regex: (.*)
    replacement: ${1}
    separator: ;
    source_labels:
    - __param_target
    target_label: instance
  - action: replace
    regex: .*
    replacement: localhost:9290
    separator: ;
    target_label: __address__
  scheme: http
  scrape_interval: 5m
  scrape_timeout: 5m

[bitfehler]

Ok, that looks about right then. And is the exporter running in Docker or native?

[mekanix]

Native

[bitfehler]

Ok, then this is a bit strange, indeed. I just noticed one thing in the config you posted, but not really sure that this is the issue:

My prometheus.yml portion for this exporter is this

- file_sd_configs:
  - files:
    - /etc/prometheus/ipmi-targets.yml
    refresh_interval: 10m
  job_name: ipmi-exporter
  metrics_path: /ipmi
  params:
    module:
    - default

The last part should probably be:

  params:
    - module: "default"

But again, not sure this really makes a difference.

Now, given your initially posted config snippet, the config file that gets generated for FreeIPMI would look like this:

driver-type LAN_2_0
privilege-level user
username myuser
password mypass

Can you create this files with the appropriate user/password values and run the ipmi-sel command from your initial post, just replacing the value for --config-file to point to this file?

Please make sure to to run this command on the host that the exporter runs on and also as the user that the exporter runs as.