[BUG-NOK] Alerts - Host alerts - CPU Temperature, Disk usage alerts appear using API calls but not in the browser UI.

[ss5829]

Bug Description

Alerts like CPU Temperature Exceeds threshold, Disk Usage Exceeds Threshold can be retrieved using API calls however they are not listed in the browser UI, Alerts section.

it can also be observed that the receivers for these alerts the project ids prefixes generation doesn't happen as expected.

System Setup

Orchestrator - v 3.1.3

Reproducible Steps

please refer bug description

Root Cause Analysis

No response

[guruzen]

https://github.com/open-edge-platform/o11y-alerting-monitor/blob/d90f6f09e94e1a55af7c093191d8c69be42e6f90/deployments/alerting-monitor/files/rules/rules.yaml#L4

rules:
- alert: "DiskUsageExceedsThreshold"
expr: 'count by (tenant_id, hostGuid, host) (avg_over_time(disk_used_percent{}[{{ $hostaggregation }}]) >= [[ .Threshold ]])'
for: {{ $duration }}
labels:
threshold: "85"
duration: {{ $duration }}
alert_category: performance
alert_context: host
host_uuid: "{{{{$labels.hostGuid}}}}"

    annotations:
      summary: "{{`{{$value}}`}} disks have usage over the threshold on host {{`{{$labels.hostGuid}}`}}."
      description: "{{`{{$value}}`}} disks have usage over the threshold on host {{`{{$labels.hostGuid}}`}}."
      display_name: "Host Disk Usage Exceeds Threshold"

      am_uuid: 0f956dac-7476-4c13-8d65-b09e4e32f8de
      am_duration: {{ $duration }}
      am_duration_min: {{ $minduration }}
      am_duration_max: {{ $maxduration }}
      am_threshold: 85
      am_threshold_min: 1
      am_threshold_max: 100
      am_definition_type: integer
      am_enabled: true

• name: 53c1ba6b-66e4-4c80-9006-92fa4f20f5e1
  interval: {{ $interval | quote }}
  rules:

  • alert: "HostCPUTemperatureExceedsThreshold"
    expr: 'count by (tenant_id, hostGuid, host) (avg_over_time(temp_temp{sensor=~"coretemp_core_.+"}[{{ $hostaggregation }}]) >= [[ .Threshold ]])'
    for: {{ $duration }}
    labels:
    threshold: "75"
    duration: {{ $duration }}
    alert_category: performance
    alert_context: host
    host_uuid: "{{{{$labels.hostGuid}}}}"

    annotations:
    summary: "Temperature of {{{{$value}}}} CPUs is over the threshold on host {{{{$labels.hostGuid}}}}."
    description: "Temperature of {{{{$value}}}} CPUs is over the threshold on host {{{{$labels.hostGuid}}}}."
    display_name: "Host CPU Temperature Exceeds Threshold"

    am_uuid: 53c1ba6b-66e4-4c80-9006-92fa4f20f5e1
    am_duration: {{ $duration }}
    am_duration_min: {{ $minduration }}
    am_duration_max: {{ $maxduration }}
    am_threshold: 75
    am_threshold_min: 0
    am_threshold_max: 110
    am_definition_type: integer
    am_enabled: true

tenant_id is wrong. It has to be "projectId".

I will have my team commit this change. we tested this in our local instance and got it working.

[soniabha-intc]

Sure @guruzen. Please go ahead and raise the PR

[amathan]

PR : open-edge-platform/o11y-alerting-monitor#238