Add an alert for high heap memory usage (#503)

* Add an alert for high heap memory usage

- Fix application filtering for existing queries

* Reduce the threshold for "High number of errors"

Based on observation, it is unlikely the threshold of 10 would be
reached, even under serious problems.

Author: mmwinther

.nais/test/klass-api-alerts.yaml

@@ -9,8 +9,21 @@ spec:
   groups:
     - name: dapla-metadata
       rules:
+        - alert: High heap memory usage
+          expr: (100 * sum by (instance) (jvm_memory_used_bytes{application="klass", area="heap"})) / (sum by (instance) (jvm_memory_max_bytes{application="klass", area="heap"}) ) > 70.0
+          for: 3m
+          annotations:
+            title: "High heap memory usage"
+            consequence: "If this increase continues then the app could run out of memory and either lock or crash."
+            action: "Immediate: Restart the app from the Nais console\nShort term: Investigate the cause of high heap usage and either fix the bug or "
+          labels:
+            service: klass-api
+            namespace: dapla-metadata
+            severity: critical
+            environment: test
+
         - alert: High number of errors
-          expr: (100 * sum by (app, namespace) (rate(logback_events_total{app="klass-api",level="error"}[3m])) / sum by (app, namespace) (rate(logback_events_total{app="klass-api"}[3m]))) > 10
+          expr: (100 * sum by (app, namespace) (rate(logback_events_total{application="klass",level="error"}[3m])) / sum by (app, namespace) (rate(logback_events_total{application="klass"}[3m]))) > 1
           for: 3m
           annotations:
             title: "High number of errors logged"
@@ -23,7 +36,7 @@ spec:
             environment: test
 
         - alert: A Klass-api client is unavailable
-          expr: rate(http_client_requests_seconds_count{app="klass-api", status!="200"}[1m]) > 0
+          expr: rate(http_client_requests_seconds_count{application="klass", status!="200"}[1m]) > 0
           for: 1m
           annotations:
             title: "A Klass-api client is unavailable "
@@ -45,4 +58,4 @@ spec:
             service: klass-api
             namespace: dapla-metadata
             severity: critical
-            environment: test
+            environment: test