feat(kyverno): remove generic error pattern to reduce false positives (#26)

Author: filippolmt

CHANGELOG.md

@@ -8,13 +8,27 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
-## [0.13.1] - 2026-02-05
+## [0.14.0] - 2026-02-05
 
-[Compare with previous version](https://github.com/sparkfabrik/terraform-google-services-monitoring/compare/0.13.0...0.13.1)
+[Compare with previous version](https://github.com/sparkfabrik/terraform-google-services-monitoring/compare/0.13.0...0.14.0)
+
+### Breaking change
+
+- **Kyverno log matching now uses `jsonPayload.message` instead of `jsonPayload.error`**. This provides more precise control over which log messages trigger alerts and enables proper exclusion of specific messages.
+  - Error-detail patterns like `"is forbidden"`, `"context deadline exceeded"`, `"timeout"` have been removed as they appear in the `error` field, not the `message` field.
+  - Patterns are now specific (e.g., `"failed to update lock"`) instead of generic (e.g., `"failed to update"`) to avoid overlap when excluding.
+  - To migrate: review your `error_patterns_exclude` configuration and update pattern names if needed.
 
 ### Changed
 
-- Extend `error_patterns_exclude` behavior: excluded patterns now also generate `NOT jsonPayload.message=~"pattern"` conditions, allowing exclusion of logs where the pattern appears in the message field (not just the error field).
+- Add `severity=ERROR` filter condition to ensure only error-level logs trigger alerts.
+- Update Kyverno default patterns to message-based matching:
+  - `"failed to list resources"`, `"failed to watch resource"`, `"failed to start watcher"`
+  - `"failed to sync"`, `"failed to run warmup"`, `"failed to load certificate"`
+  - `"failed to update lock"`, `"failed to process request"`
+  - `"failed to check permissions"`, `"failed to scan resource"`, `"failed to fetch data"`
+  - `"failed to substitute variables"`, `"failed calling webhook"`
+  - `"leader election lost"`, `"dropping request"`, `"panic"`
 
 ## [0.13.0] - 2026-02-04
 

README.md

@@ -56,7 +56,7 @@ Supported services:
 | <a name="input_cert_manager"></a> [cert\_manager](#input\_cert\_manager) | Configuration for cert-manager missing issuer log alert. Allows customization of project, cluster, namespace, notification channels, alert documentation, enablement, extra filters, auto-close timing, and notification rate limiting. | <pre>object({<br/>    enabled                          = optional(bool, true)<br/>    cluster_name                     = optional(string, null)<br/>    project_id                       = optional(string, null)<br/>    namespace                        = optional(string, "cert-manager")<br/>    notification_enabled             = optional(bool, true)<br/>    notification_channels            = optional(list(string), [])<br/>    logmatch_notification_rate_limit = optional(string, "300s")<br/>    alert_documentation              = optional(string, null)<br/>    auto_close_seconds               = optional(number, 3600)<br/>    filter_extra                     = optional(string, "")<br/>  })</pre> | `{}` | no |
 | <a name="input_cloud_sql"></a> [cloud\_sql](#input\_cloud\_sql) | Configuration for Cloud SQL monitoring alerts. Supports customization of project, auto-close timing, notification channels, and per-instance alert thresholds for CPU, memory, and disk utilization. | <pre>object({<br/>    enabled               = optional(bool, true)<br/>    project_id            = optional(string, null)<br/>    auto_close            = optional(string, "86400s") # default 24h<br/>    notification_enabled  = optional(bool, true)<br/>    notification_channels = optional(list(string), [])<br/>    instances = optional(map(object({<br/>      cpu_utilization = optional(list(object({<br/>        severity         = optional(string, "WARNING"),<br/>        threshold        = optional(number, 0.90)<br/>        alignment_period = optional(string, "120s")<br/>        duration         = optional(string, "300s")<br/>        })), [<br/>        {<br/>          threshold = 0.85,<br/>          duration  = "1200s",<br/>        },<br/>        {<br/>          severity         = "CRITICAL",<br/>          threshold        = 1,<br/>          duration         = "300s",<br/>          alignment_period = "60s",<br/>        }<br/>      ])<br/>      memory_utilization = optional(list(object({<br/>        severity         = optional(string, "WARNING"),<br/>        threshold        = optional(number, 0.90)<br/>        alignment_period = optional(string, "300s")<br/>        duration         = optional(string, "300s")<br/>        })), [<br/>        {<br/>          severity = "WARNING",<br/>        },<br/>        {<br/>          severity  = "CRITICAL",<br/>          threshold = 0.95,<br/>        }<br/>      ])<br/>      disk_utilization = optional(list(object({<br/>        severity         = optional(string, "WARNING"),<br/>        threshold        = optional(number, 0.85)<br/>        alignment_period = optional(string, "300s")<br/>        duration         = optional(string, "600s")<br/>        })), [<br/>        {<br/>          severity = "WARNING",<br/>        },<br/>        {<br/>          severity  = "CRITICAL",<br/>          threshold = 0.95,<br/>        }<br/>      ])<br/>    })), {})<br/>  })</pre> | `{}` | no |
 | <a name="input_konnectivity_agent"></a> [konnectivity\_agent](#input\_konnectivity\_agent) | Configuration for Konnectivity agent deployment replica alert in GKE. Triggers when there are no available replicas. | <pre>object({<br/>    enabled               = optional(bool, true)<br/>    cluster_name          = optional(string, null)<br/>    project_id            = optional(string, null)<br/>    namespace             = optional(string, "kube-system")<br/>    deployment_name       = optional(string, "konnectivity-agent")<br/>    duration_seconds      = optional(number, 60)<br/>    auto_close_seconds    = optional(number, 3600)<br/>    notification_enabled  = optional(bool, true)<br/>    notification_channels = optional(list(string), [])<br/>    notification_prompts  = optional(list(string), null)<br/>  })</pre> | `{}` | no |
-| <a name="input_kyverno"></a> [kyverno](#input\_kyverno) | Configuration for Kyverno monitoring alerts. Allows customization of cluster name, project, notification channels, alert documentation, metric thresholds, auto-close timing, enablement, error pattern inclusions/exclusions for jsonPayload.error matching, and namespace. | <pre>object({<br/>    enabled               = optional(bool, true)<br/>    cluster_name          = optional(string, null)<br/>    project_id            = optional(string, null)<br/>    notification_enabled  = optional(bool, true)<br/>    notification_channels = optional(list(string), [])<br/>    # Rate limit for notifications, e.g. "300s" for 5 minutes, used only for log match alerts<br/>    logmatch_notification_rate_limit = optional(string, "300s")<br/>    alert_documentation              = optional(string, null)<br/>    auto_close_seconds               = optional(number, 3600)<br/>    namespace                        = optional(string, "kyverno")<br/>    # List of error patterns to exclude from the default set.<br/>    # Default patterns available for exclusion:<br/>    #   "internal error", "failed calling webhook", "timeout", "client-side throttling",<br/>    #   "failed to run warmup", "schema not found", "failed to list resources",<br/>    #   "failed to watch resource", "context deadline exceeded", "is forbidden",<br/>    #   "cannot list resource", "cannot watch resource", "RBAC.*denied",<br/>    #   "failed to start watcher", "leader election lost", "unable to update .*WebhookConfiguration",<br/>    #   "failed to sync", "dropping request", "failed to load certificate",<br/>    #   "failed to update lock", "the object has been modified", "no matches for kind",<br/>    #   "the server could not find the requested resource", "Too Many Requests", "x509",<br/>    #   "is invalid:", "connection refused", "no agent available", "fatal error", "panic"<br/>    error_patterns_exclude = optional(list(string), [])<br/>    # List of additional regex error patterns to include (added to default set)<br/>    # e.g. ["my custom.*error", "failed to connect.*database"]<br/>    error_patterns_include = optional(list(string), [])<br/>  })</pre> | `{}` | no |
+| <a name="input_kyverno"></a> [kyverno](#input\_kyverno) | Configuration for Kyverno monitoring alerts. Allows customization of cluster name, project, notification channels, alert documentation, metric thresholds, auto-close timing, enablement, message pattern inclusions/exclusions for jsonPayload.message matching, and namespace. | <pre>object({<br/>    enabled               = optional(bool, true)<br/>    cluster_name          = optional(string, null)<br/>    project_id            = optional(string, null)<br/>    notification_enabled  = optional(bool, true)<br/>    notification_channels = optional(list(string), [])<br/>    # Rate limit for notifications, e.g. "300s" for 5 minutes, used only for log match alerts<br/>    logmatch_notification_rate_limit = optional(string, "300s")<br/>    alert_documentation              = optional(string, null)<br/>    auto_close_seconds               = optional(number, 3600)<br/>    namespace                        = optional(string, "kyverno")<br/>    # List of message patterns to exclude from the default set (matches against jsonPayload.message).<br/>    # Default patterns available for exclusion:<br/>    #   "failed to list resources", "failed to watch resource", "failed to start watcher",<br/>    #   "failed to sync", "failed to run warmup", "failed to load certificate",<br/>    #   "failed to update lock", "failed to update lease", "failed to process request",<br/>    #   "failed to check permissions", "failed to scan resource", "failed to fetch data",<br/>    #   "failed to substitute variables", "failed calling webhook",<br/>    #   "leader election lost", "dropping request", "panic"<br/>    error_patterns_exclude = optional(list(string), [])<br/>    # List of additional regex message patterns to include (added to default set)<br/>    # e.g. ["failed to update lease", "failed to connect.*"]<br/>    error_patterns_include = optional(list(string), [])<br/>  })</pre> | `{}` | no |
 | <a name="input_litellm"></a> [litellm](#input\_litellm) | Configuration for LiteLLM monitoring alerts. Supports uptime checks for HTTP endpoints and container-level alerts (pod restarts) in GKE. Each app is identified by its name (map key). | <pre>object({<br/>    enabled               = optional(bool, false)<br/>    project_id            = optional(string, null)<br/>    notification_enabled  = optional(bool, true)<br/>    notification_channels = optional(list(string), [])<br/>    cluster_name          = optional(string, null)<br/><br/>    apps = optional(map(object({<br/>      uptime_check = optional(object({<br/>        enabled = optional(bool, true)<br/>        host    = string<br/>        path    = optional(string, "/health/readiness")<br/>      }), null)<br/><br/>      container_check = optional(object({<br/>        enabled   = optional(bool, true)<br/>        namespace = string<br/>        pod_restart = optional(object({<br/>          threshold            = optional(number, 0)<br/>          alignment_period     = optional(number, 60)<br/>          duration             = optional(number, 180)<br/>          auto_close_seconds   = optional(number, 3600)<br/>          notification_prompts = optional(list(string), null)<br/>        }), {})<br/>      }), null)<br/>    })), {})<br/>  })</pre> | `{}` | no |
 | <a name="input_notification_channels"></a> [notification\_channels](#input\_notification\_channels) | List of notification channel IDs to notify when an alert is triggered | `list(string)` | `[]` | no |
 | <a name="input_project_id"></a> [project\_id](#input\_project\_id) | The Google Cloud project ID where logging exclusions will be created | `string` | n/a | yes |

examples/main.tf

@@ -50,13 +50,13 @@ module "example" {
   kyverno = {
     cluster_name          = "test-cluster"
     notification_channels = []
-    # Exclude specific error patterns from the default set (only affects jsonPayload.error matching)
+    # Exclude specific message patterns from the default set (matches against jsonPayload.message)
     error_patterns_exclude = [
       "failed to start watcher",
       "failed to list resources",
     ]
-    # Add custom regex error patterns to the default set (matched against jsonPayload.error)
-    # Note: These options only support error pattern matching. Arbitrary log filter conditions
+    # Add custom regex message patterns to the default set (matched against jsonPayload.message)
+    # Note: These options only support message pattern matching. Arbitrary log filter conditions
     # (e.g., negative filters like -textPayload:"...") are not supported.
     # error_patterns_include = [
     #   "my custom.*error",

kyverno.tf

@@ -5,76 +5,56 @@ locals {
 
   kyverno_cluster_name = var.kyverno.cluster_name != null ? trimspace(var.kyverno.cluster_name) : ""
 
-  # Default error patterns for Kyverno log matching
-  kyverno_default_error_patterns = [
-    "internal error",
-    "failed calling webhook",
-    "timeout",
-    "client-side throttling",
-    "failed to run warmup",
-    "schema not found",
+  # Default message patterns for Kyverno log matching (matches against jsonPayload.message)
+  kyverno_default_message_patterns = [
     "failed to list resources",
     "failed to watch resource",
-    "context deadline exceeded",
-    "is forbidden",
-    "cannot list resource",
-    "cannot watch resource",
-    "RBAC.*denied",
     "failed to start watcher",
-    "leader election lost",
-    "unable to update .*WebhookConfiguration",
     "failed to sync",
-    "dropping request",
+    "failed to run warmup",
     "failed to load certificate",
     "failed to update lock",
-    "the object has been modified",
-    "no matches for kind",
-    "the server could not find the requested resource",
-    "Too Many Requests",
-    "x509",
-    "is invalid:",
-    "connection refused",
-    "no agent available",
-    "fatal error",
+    "failed to process request",
+    "failed to check permissions",
+    "failed to scan resource",
+    "failed to fetch data",
+    "failed to substitute variables",
+    "failed calling webhook",
+    "leader election lost",
+    "dropping request",
     "panic",
   ]
 
   # Combine default patterns with included patterns, then filter out excluded ones
-  kyverno_all_error_patterns = distinct(concat(
-    local.kyverno_default_error_patterns,
+  kyverno_all_message_patterns = distinct(concat(
+    local.kyverno_default_message_patterns,
     var.kyverno.error_patterns_include
   ))
 
-  kyverno_active_error_patterns = [
-    for pattern in local.kyverno_all_error_patterns :
+  kyverno_active_message_patterns = [
+    for pattern in local.kyverno_all_message_patterns :
     pattern if !contains(var.kyverno.error_patterns_exclude, pattern)
   ]
 
-  # Build the error patterns filter string
-  kyverno_error_patterns_filter = length(local.kyverno_active_error_patterns) > 0 ? join("\n      OR ", [
-    for pattern in local.kyverno_active_error_patterns :
-    "jsonPayload.error=~\"(?i)${pattern}\""
-  ]) : ""
-
-  # Build NOT conditions for excluded patterns on jsonPayload.message
-  kyverno_message_exclusions = length(var.kyverno.error_patterns_exclude) > 0 ? join("\n    ", [
-    for pattern in var.kyverno.error_patterns_exclude :
-    "AND NOT jsonPayload.message=~\"(?i)${pattern}\""
+  # Build the message patterns filter string
+  kyverno_message_patterns_filter = length(local.kyverno_active_message_patterns) > 0 ? join("\n      OR ", [
+    for pattern in local.kyverno_active_message_patterns :
+    "jsonPayload.message=~\"(?i)${pattern}\""
   ]) : ""
 
-  kyverno_log_filter = local.kyverno_cluster_name != "" && length(local.kyverno_active_error_patterns) > 0 ? (<<-EOT
+  kyverno_log_filter = local.kyverno_cluster_name != "" && length(local.kyverno_active_message_patterns) > 0 ? (<<-EOT
     resource.type="k8s_container"
     AND resource.labels.project_id="${local.kyverno_project_id}"
     AND resource.labels.cluster_name="${local.kyverno_cluster_name}"
     AND resource.labels.namespace_name="${var.kyverno.namespace}"
+    AND severity=ERROR
     AND (
       labels."k8s-pod/app_kubernetes_io/component"=~"(admission-controller|background-controller|cleanup-controller|reports-controller)"
       OR resource.labels.pod_name=~"kyverno-(admission|background|cleanup|reports)-controller-.*"
     )
     AND (
-      ${local.kyverno_error_patterns_filter}
+      ${local.kyverno_message_patterns_filter}
     )
-    ${local.kyverno_message_exclusions}
   EOT
   ) : ""
 }
@@ -83,7 +63,7 @@ resource "google_monitoring_alert_policy" "kyverno_logmatch_alert" {
   count = (
     var.kyverno.enabled
     && local.kyverno_cluster_name != ""
-    && length(local.kyverno_active_error_patterns) > 0
+    && length(local.kyverno_active_message_patterns) > 0
   ) ? 1 : 0
 
   display_name = "Kyverno controllers ERROR logs (namespace=${var.kyverno.namespace})"