feat: add a Prometheus exporter for proxmox backup server

[33Fraise33]

There are 2 instances of proxmox backup server which can be gathered with Prometheus netbox SD, they are virtual machines with the tag PBS.

Please deploy a pbs exporter, add those 2 targets. Also add alerts for failed backups. Of the exporter is not reachable the default "up" metric should already fire indicating an issue with reachibility or the service being up.

[33Fraise33]

/oc plan look into the above requirements. If anything needs to be cleared up, let me know. Look to the existing Prometheus role

[opencode-agent]

Repository findings

• Exporters are deployed centrally by roles/prometheus, attached to the monitoring Docker network, and enabled through prometheus.features.*.
• beokpdcovm01 is the primary Prometheus instance: it discovers targets globally through NetBox and is connected to Alertmanager.
• The existing generic alert up{offline!="true"} == 0 already covers an unreachable PBS exporter or failed scrape after 10 minutes.
• Both PBS machines can be dynamically discovered with the NetBox virtual-machine endpoint using status=active&tag=pbs; their addresses should not be hardcoded.
• No PBS exporter credentials currently exist in the repository.

Clarifications needed

1. Which PBS exporter should be used? The issue does not identify an image/project, and exporters differ in configuration, multi-target support, and metric names. If there is no preference, please confirm that a maintained multi-target exporter may be selected.

2. Authentication: Please confirm whether both PBS instances will use:

   • one common API user/password or identically named token, or
   • separate API token IDs/secrets.

   The credentials will need to be supplied as encrypted inventory variables.

3. TLS: Do the PBS API endpoints have certificates trusted by the exporter, or should certificate verification be disabled?

4. Alert semantics: I propose a critical alert when the exporter's “latest backup task succeeded” metric reports failure for 10 minutes. Please confirm this, as the exact expression depends on the selected exporter.

5. Deployment scope: I recommend deploying and scraping this only from beokpdcovm01, since it performs global discovery and sends alerts. Confirm that the secondary otgkit01vm01 instance should not also scrape PBS directly.

Concrete implementation plan

1. Add roles/prometheus/tasks/pbs_exporter.yml:

   • Validate required credentials without logging their values.
   • Render a mode 0600 exporter configuration when required by the selected image.
   • Start the exporter with community.docker.docker_container.
   • Attach it to monitoring, disable Traefik exposure, declare restart policy and health state, and use the image’s non-root user where supported.
   • Use a major image tag and matching WUD constraint when available; otherwise use latest without WUD labels.

2. Update roles/prometheus/tasks/main.yml:

   • Import the new task behind prometheus.features.pbs | default(false).
   • Add pbs, pbs_exporter, and pbs-exporter tags.

3. Add encrypted, inventory-scoped exporter credentials for the primary monitoring host rather than embedding secrets in the role. Support separate credential profiles if the two PBS servers cannot share credentials.

4. Update playbooks/host_files/beokpdcovm01/templates/prometheus/prometheus.yml.j2:

   • Add a PBS scrape job using NetBox HTTP SD:
     virtual-machines/?status=active&tag=pbs.
   • Preserve node_name, site, tenant, and device_role labels.
   • Pass each discovered primary IP as the exporter target and replace the scrape address with the PBS exporter container.
   • Retain IPv4/IPv6 compatibility.
   • Do not add a separate reachability alert because the generic up alert already covers it.

5. Add playbooks/host_files/beokpdcovm01/files/prometheus/alerts.d/pbs.rules.yml:

   • Alert on the selected exporter’s failed-backup metric.
   • Include the PBS node, datastore/job where available, severity, and a useful failure summary.
   • Avoid alerting indefinitely on historical failures if the exporter exposes a latest-task or timestamp metric.

6. Add a focused Molecule scenario for the Prometheus PBS exporter:

   • Prepare the Docker network and secret-free test credentials.
   • Converge twice for idempotency.
   • Pause after convergence and inspect exporter logs.
   • Assert the container is running/healthy, configuration is mounted read-only, credentials are not exposed in task output, and the metrics endpoint responds.
   • Validate the generated Prometheus configuration and alert rules with promtool.

7. Validate with pre-commit, yamllint, ansible-lint, ansible-playbook --syntax-check playbooks/monitoring.yml, and the new Molecule scenario.

github run

[33Fraise33]

/oc plan please provide me with the possibilities for exporters. The natrontech one is the only I know.

I will use a shared read-only API token.
TLS should accept self signed.

There should be alerts for failed backup but also failed verify and failed sync.
Only beokpdcovm01 should have pbs_exporter feature enabled

[opencode-agent]

Exporter options

Option	Suitability
Natron Tech pbs-exporter	Best fit. Purpose-built for PBS, supports API-token authentication, self-signed TLS, multi-target probing, and task metrics needed for backup, verification, and synchronization alerts.
PBS native metric server → InfluxDB/Graphite bridge	Suitable for resource and datastore statistics, but introduces another component and does not provide the required task-failure alerting as cleanly.
Telegraf/custom API collector	Can query /api2/json/admin/tasks and provide exact alert semantics, but requires maintaining metric mapping, pagination, authentication, and compatibility code.
Prometheus PVE exporters	Generally intended for Proxmox VE rather than Proxmox Backup Server. They should not be selected unless PBS task support is explicitly demonstrated.

Recommendation: use Natron Tech. It is the simplest option matching all requirements without maintaining a custom collector.

Implementation plan

1. Add the exporter deployment

   • Create roles/prometheus/tasks/pbs_exporter.yml.
   • Render a mode 0600 configuration containing:
     • Shared API token ID.
     • Vault-encrypted token secret.
     • TLS verification disabled for the self-signed PBS certificates.
   • Validate required variables without displaying their values.
   • Deploy the exporter with community.docker.docker_container.
   • Attach it only to the existing monitoring network.
   • Bind-mount configuration read-only, disable Traefik exposure, declare restart policy, health state, and the supported non-root UID/GID.
   • Use a major image tag if Natron Tech publishes one; otherwise use latest without WUD constraints.

2. Add feature gating

   • Import the task from roles/prometheus/tasks/main.yml when:

     prometheus.features.pbs_exporter | default(false)

   • Add pbs, pbs_exporter, and pbs-exporter tags.
   • Enable this feature only for beokpdcovm01; leave otgkit01vm01 unchanged.
   • Preserve any existing inventory-provided prometheus.features values when adding the host-specific setting.

3. Store shared credentials

   • Define one API token ID and one encrypted token secret for beokpdcovm01.
   • Keep the secret out of templates rendered into Prometheus itself; only the exporter receives it.
   • Use the same credentials for both dynamically discovered PBS targets.

4. Discover both PBS instances

   • Extend only:
     playbooks/host_files/beokpdcovm01/templates/prometheus/prometheus.yml.j2
   • Add a multi-target scrape job using:

     /api/plugins/prometheus-sd/virtual-machines/?status=active&tag=pbs
     

   • Pass each discovered primary IP to the exporter and rewrite the scrape address to its monitoring-network container name.
   • Preserve instance, node_name, site, tenant, and device_role.
   • Handle IPv4 and IPv6 targets consistently with existing jobs.

5. Add PBS task alerts

   • Create:
     playbooks/host_files/beokpdcovm01/files/prometheus/alerts.d/pbs.rules.yml
   • Add separate alerts for:
     • Failed backup task.
     • Failed verification task.
     • Failed synchronization task.
   • Use Natron Tech’s actual task type/status/timestamp labels.
   • Ensure alerts represent recent/latest failures rather than firing forever for historical failed tasks.
   • Include PBS node, datastore, task/job identifier, failure type, and status where exported.
   • Assign critical severity because all three operations protect backup availability or integrity.

6. Retain existing reachability coverage

   • Do not add a duplicate exporter-down rule.
   • The existing up{offline!="true"} == 0 rule already alerts after ten minutes when the exporter is unavailable or a PBS target cannot be scraped.

7. Testing

   • Add a top-level Molecule scenario for the Prometheus PBS component.
   • Prepare the Docker monitoring network and a mock PBS API using test-only credentials and self-signed TLS.
   • Test convergence and idempotency.
   • Pause after convergence for exporter-log inspection.
   • Verify:
     • Only the enabled host deploys the container.
     • Container is running/healthy.
     • Configuration is mounted read-only with restrictive permissions.
     • Shared token authentication and disabled certificate verification work.
     • Both targets produce metrics.
     • Backup, verification, and synchronization failure fixtures trigger their respective rules.
     • Exporter secrets do not appear in Ansible output.
   • Validate the rendered scrape configuration and rules with promtool.

8. Validation

   • Run secret-free pre-commit, yamllint, and ansible-lint.
   • Syntax-check playbooks/monitoring.yml.
   • Run the new Molecule scenario without using the external inventory or Vault decryption.

github run

[33Fraise33]

/oc build implement the plan, use the natrontech image

[opencode-agent]

Created PR #17

github run