Add PBS exporter and failure alerts - #17

/oc plan what changes are required on pbs (mainly which permissions does the api key need) , how can I use 1 secret for two hosts as they seem to be autogenerated

PBS-side requirements

On each PBS server:

1. Create a dedicated user, for example prometheus@pbs.
2. Create an API token such as prometheus@pbs!exporter.
3. Grant the built-in Audit role on / with propagation enabled.
   • This provides the read-only system, datastore, task, remote, and verification information needed by the exporter.
   • DatastoreAudit alone is likely insufficient because the exporter also reads node and task information.
4. With token privilege separation enabled, grant Audit to both the user and token. PBS calculates token permissions as the intersection of both.

Equivalent commands:

proxmox-backup-manager user create prometheus@pbs
proxmox-backup-manager acl update / Audit \
  --auth-id prometheus@pbs

proxmox-backup-manager user generate-token \
  prometheus@pbs exporter

proxmox-backup-manager acl update / Audit \
  --auth-id 'prometheus@pbs!exporter'

The exporter also needs network access from beokpdcovm01 to TCP port 8007 on each PBS. The existing tag_pbs.yml already permits port 8007, although it could optionally be restricted to the monitoring host/network.

Using one secret for two PBS servers

The token ID can be identical on both servers:

prometheus@pbs!exporter

However, the token secret is generated and stored independently by each PBS server. Creating the same token name on two servers produces two different secrets. PBS does not provide a supported way to request the same token secret on both independent servers.

Therefore, use:

• One common token ID, if desired.
• One distinct vaulted token secret per PBS server.

Copying PBS token storage between servers to force a shared secret would be unsupported and would unnecessarily increase the impact of credential compromise.

Required repository changes for two PBS hosts

The current implementation starts one exporter with one global credential pair:

PBS_TOKEN_ID: "{{ pbs_exporter_api_token_id }}"
PBS_TOKEN_SECRET: "{{ pbs_exporter_api_token_secret }}"

Meanwhile, NetBox discovery can return both PBS hosts. The second host will receive the first host’s credentials and fail authentication.

Implementation plan

1. Replace the singular credentials with an instance list in playbooks/host_vars/beokpdcovm01.yml:

   pbs_exporter_instances:
     - name: pbs-dco
       netbox_name: beokpdco...
       api_token_id: prometheus@pbs!exporter
       api_token_secret: !vault |
         ...
     - name: pbs-home
       netbox_name: bejbegia...
       api_token_id: prometheus@pbs!exporter
       api_token_secret: !vault |
         ...

2. Update roles/prometheus/tasks/pbs_exporter.yml to:

   • Validate every instance.
   • Ensure names and NetBox host selectors are unique.
   • Start one exporter container per PBS credential.
   • Keep all credential-bearing tasks under no_log: true.

3. Update prometheus.yml.j2 to create one PBS scrape job per exporter instance:

   • Continue using NetBox service discovery.
   • Keep only the matching PBS __meta_netbox_name.
   • Route that target to its corresponding exporter container.

4. Correct the alert identifiers in pbs.rules.yml to valid Prometheus names such as:

   • PBSBackupFailed
   • PBSVerificationFailed
   • PBSSyncFailed

5. Verify the actual metric and label names emitted by the exporter, then run secret-free lint, syntax, and container configuration tests.

No PBS configuration needs to be managed by Ansible for this PR unless automatic user/token provisioning is specifically desired.

github run

/oc plan

Is there a way for me to set the API secret somewhere in netbox so it can be inserted automatically? Or maybe even in 1password and requested at Ansible-playbook runtime. (Less preferred, same issue as below)

As now I would have to rerun the ansible playbook after adding a new pbs host in netbox.

Short answer

NetBox can expose a custom field or config-context value to Ansible, but it is not an appropriate secret store. The value could also appear in API responses and Prometheus service-discovery labels.

1Password lookup at playbook runtime is feasible, but neither NetBox nor 1Password can update an already-running exporter. Each PBS requires its own generated token secret, and the current exporter container accepts only one credential pair. Some form of reconciliation must run whenever a PBS host is added.

The safest approach is:

• NetBox stores only a non-secret 1Password item reference.
• 1Password stores one token secret per PBS host.
• A NetBox webhook triggers a focused Ansible job automatically.

This removes the manual rerun, although Ansible still runs behind the scenes.

Concrete implementation plan

1. Define a NetBox custom field

   • Add pbs_exporter_credential_ref to PBS virtual machines.
   • Store a 1Password item UUID or title, never the token secret.
   • Continue using the existing pbs tag for discovery.

2. Store credentials in 1Password

   • Create one item per PBS host containing:
     • token_id, normally prometheus@pbs!exporter
     • token_secret, unique to that PBS
   • Use a dedicated 1Password vault and predictable field names.

3. Build exporter instances from NetBox inventory

   • Update roles/prometheus/tasks/pbs_exporter.yml to iterate over groups['tag_pbs'].
   • Read each host’s NetBox name, primary IP, and credential reference from hostvars.
   • Resolve the two credential fields using community.general.onepassword on the Ansible controller.
   • Permit an injected credential list for secret-free Molecule testing.
   • Validate missing references, duplicate names, and invalid Docker container names.
   • Apply no_log: true only to credential lookup and container operations.

4. Run one exporter per PBS

   • Create containers named predictably, such as:
     • prometheus-pbs-exporter-beokpd...
     • prometheus-pbs-exporter-bejbegia...
   • Give each container only its corresponding token.
   • Label managed containers so stale exporters can be identified and removed when a PBS is removed from NetBox.

5. Generate matching Prometheus scrape jobs

   • Update playbooks/host_files/beokpdcovm01/templates/prometheus/prometheus.yml.j2.
   • Generate one job per PBS instance.
   • Retain NetBox HTTP service discovery, but keep only the exact matching __meta_netbox_name.
   • Route that target to the corresponding exporter container.
   • Preserve NetBox labels such as site, tenant, role, and node name.

6. Add a focused reconciliation playbook

   • Add playbooks/pbs_exporters.yml targeting beokpdcovm01.
   • Run only PBS exporter reconciliation and Prometheus configuration/reload tasks, rather than all monitoring roles.

7. Connect NetBox automation

   • Configure a NetBox webhook for PBS-tagged VM creation, update, and deletion.
   • Have the automation endpoint run the focused playbook after the 1Password item reference has been populated.
   • This is what eliminates the manual playbook invocation.

8. Testing

   • Add a Prometheus/PBS-exporter Molecule scenario with two fake instances and injected secrets.
   • Verify:
     • Two containers are created with separate credentials.
     • The generated Prometheus configuration maps each NetBox name to the correct exporter.
     • Removed instances are cleaned up.
     • Convergence is idempotent.
     • Exporter logs contain no runtime warnings or credential output.
   • Run lint, syntax checks, and Molecule without accessing 1Password or Vault.

If absolutely no reconciliation run is acceptable, the architecture would need to change—for example, deploying an exporter directly on every PBS host with locally stored credentials. The current central exporter model cannot dynamically acquire a newly generated per-host token from NetBox service discovery alone.

github run

/oc plan go with the initial idea of using a variable, this is easier and pbs instances are not changing that often

Implementation plan

1. Define PBS instances in host variables

   • Add pbs_exporter_instances to playbooks/host_vars/beokpdcovm01.yml.
   • Each entry will contain:
     • A unique container/job-safe name.
     • The exact NetBox netbox_name.
     • api_token_id.
     • A distinct vaulted api_token_secret.
   • Use the same token ID on both PBS servers where applicable, but separate secrets.

2. Deploy one exporter per instance

   • Update roles/prometheus/tasks/pbs_exporter.yml to:
     • Validate that the list is non-empty and every required field is present.
     • Reject duplicate names and NetBox names.
     • Create one container named prometheus-pbs-exporter-<name> per entry.
     • Pass only that instance’s credentials to its container.
     • Retain no_log: true for credential validation and container tasks.
     • Explicitly remove the legacy singular prometheus-pbs_exporter container to prevent stale deployment.

3. Generate one Prometheus scrape job per instance

   • Update prometheus.yml.j2 to loop over pbs_exporter_instances.
   • Each job will:
     • Continue discovering PBS systems through NetBox.
     • Keep only the target whose __meta_netbox_name matches the configured instance.
     • Route probes to the corresponding exporter container.
     • Preserve the existing instance, node, site, tenant, and role labels.
   • Render these jobs only when the PBS exporter feature is enabled.

4. Correct alert identifiers

   • Change invalid alert names containing spaces to valid Prometheus identifiers:
     • PBSBackupFailed
     • PBSVerificationFailed
     • PBSSyncFailed
   • Preserve the existing critical severity, expressions, and annotations unless runtime exporter metrics show they require adjustment.

5. Validation

   • Test with two injected, non-secret PBS instances.
   • Confirm two independently configured containers and two correctly filtered scrape jobs.
   • Check idempotency and ensure credentials do not appear in output.
   • Validate the generated Prometheus configuration and alert rules, then run pre-commit, YAML lint, Ansible lint, and the monitoring playbook syntax check.

Adding or removing a PBS instance will require updating this variable and rerunning the monitoring playbook, which matches the selected low-change approach.

github run

/oc build make the requested changes

Implemented multi-instance PBS exporter support, per-instance NetBox scrape routing, legacy exporter cleanup, secret-safe validation, and valid Prometheus alert identifiers.

github run

/oc plan rework so there is only 1 exporter but use a target file with an API secret per host. Spinning up an exporter per host is not the pattern we want to have. Also for alerts, use the naming convention as shown with other alerts

Implementation plan

1. Replace per-host containers with one exporter

   • Rework roles/prometheus/tasks/pbs_exporter.yml to create a single prometheus-pbs_exporter container.
   • Remove the per-instance container loop and clean up existing prometheus-pbs-exporter-* containers from the current implementation.
   • Keep the exporter on the monitoring network with its existing restart policy and image.

2. Generate a secret target configuration

   • Add roles/prometheus/templates/pbs_targets.yml.j2 using the exporter’s supported multi-target schema.
   • Define one target per PBS host, including its endpoint, token ID, token secret, and TLS setting.
   • Create a dedicated host configuration directory and render the target file with mode 0600.
   • Mount it read-only into the single exporter and configure the exporter to use it.
   • Apply no_log: true to validation, template rendering, and other tasks that process credentials.

3. Simplify inventory variables

   • Replace pbs_exporter_instances with a target collection such as pbs_exporter_targets in playbooks/host_vars/beokpdcovm01.yml.
   • Each entry will identify the corresponding NetBox host/target and contain its independently vaulted api_token_secret; token IDs may be shared.
   • Validate required fields and reject duplicate target selectors.

4. Use one Prometheus scrape job

   • Replace the loop in prometheus.yml.j2 with one pbs_exporter job.
   • Continue discovering all active PBS-tagged VMs through NetBox.
   • Pass each discovered host’s target identifier through __param_target.
   • Rewrite __address__ to the single prometheus-pbs_exporter:9569 container.
   • Preserve the current instance, node_name, site, tenant, and device_role labels.

5. Match existing alert naming

   • Update pbs.rules.yml from compact identifiers to the repository’s human-readable convention:
     • PBS - Backup Failed
     • PBS - Verification Failed
     • PBS - Sync Failed
   • Retain the existing severity, timing, expressions, and annotations.

6. Validation

   • Render a two-host target file with injected test secrets and confirm only one exporter container is declared.
   • Verify secrets are absent from Ansible output and the generated Prometheus configuration.
   • Validate Prometheus rules/configuration, idempotency, and exporter logs.
   • Run pre-commit, YAML lint, Ansible lint, and syntax-check playbooks/monitoring.yml; add Molecule coverage for the single-container, multi-target configuration.

github run

/oc build implement

Implemented single-container PBS exporter configuration with per-target credential file, updated scrape routing, legacy exporter cleanup, and human-readable PBS alert names.

github run