[RFC] Alerting V2

[toepkerd]

Overview

This RFC proposes building V2 APIs for Alerting to support PPL (Piped Processing Language) and simplify the monitor creation experience.

Problem Statement

Current alerting APIs lack PPL support and have a complex user experience for defining multiple monitor types. Users need a more streamlined approach with enhanced query capabilities.

Goals

1. Implement PPL language support in alerting
2. Simplify monitor creation workflow
3. Provide enhanced trigger condition configuration

Workflow Architecture

1. Create Monitor - Define Monitor name - Configure schedule
2. Configure Alert Condition - Query Editor - Query Input - Dry Run capability - PPL query support
3. Configure Notifications - Define trigger conditions - Set notification rules

API Shapes

1. Create Monitor v2

• name must be unique.
• Only one schedule definition (period or cron) is allowed.

Endpoint

POST /_plugins/_alerting/v2/monitors

{
  "name": "v2-ppl-monitor-1", // (Required) Unique, descriptive name.
  "description" : "this is a ppl based monitor", // (Required) Description of the monitor.
  "enabled": true, // (Required) Activate monitor on creation.
  "query": "source=_internal (log_level=ERROR OR log_level=FATAL OR log_level=CRITICAL) | stats count by log_level", // (Required) PPL query string.
  "schedule": { // (Required) Execution schedule (choose one).
  "frequency": "by_interval", // (Required) "by_interval" or "cron".
  "period": { // (Required if by_interval) Interval settings.
    "interval": 5, // (Required) Interval count.
    "unit": "MINUTES" // (Required) MINUTES|HOURS|DAYS.
   }
 /*
  "cron": { // (Required if cron) Cron schedule.
    "expression": "0 * * * *", // (Required) Cron expression.
    "timezone": "UTC" // (Optional) Timezone.
   }
 */
},
  "labels": { // (Optional) Key/value tags.
    "service": "payment_gateway", // (Required if used) metadata tag.
    "team": "oreo" // (Required if used) metadata tag.
  },
 "triggers": [ // (Optional) triggers.
   {
     "name": "high-number-of-fatal-errors", // (Required) Trigger identifier.
     "mode": "result_set" // "result_set" = action fires once for all results; 
 // "per_result" = action fires for each result individually
 /* Option 1 : Based on number of results */
 "type": "number_of_results",
 "condition": "greater than",
 "threshold": "4",
 /* Option 2 : Custom condition */
 "type" : "custom",
 "condition": "where level='FATAL' | search count > 10", // (Required) PPL (based on language) condition.
 
 "severity": "critical", // (Required) Severity level.
 "suppress": {
    "for" : { // (Optional) Suppression duration.
       "period": {
         "interval": 30, // (Required if used) Interval count.
         "unit": "MINUTES" // (Required if used) MINUTES|HOURS|DAYS.
         }
       }
    "fields": "log_level,host" // (Required if suppression on per_result) Comma‑delimited list of fields to use for suppression.
 }
 "expires": { // (Optional) Expiration duration.
       "period": {
         "interval": 24, // (Required if used) Interval count.
         "unit": "HOURS" // (Required if used) MINUTES|HOURS|DAYS.
         }
       }
 
 "actions": [ // (Required if trigger used) Actions to run.
 {
 "name": "<action-name>", // (Required) Action name.
 "destination_id": "<dest-id>", // (Required) Destination ID.
 "message_template": { // (Required) Message template.
 "source": "<message>", // (Required) Message content.
 "markup": "mustache" // (Required) Template language.
 },
 "subject_template": { // (Optional) Subject template.
 "source": "<subject>", // (Required if subject_template used) Subject content.
 "markup": "mustache" // (Required if subject_template used) Template language.
 }
 }
 ],
 }
 ]
}

Success Response (201 Created)

{
  "monitor_id": "v2-ppl-monitor-1",
  "version": 1,
  "seq_no": 1,
  "primary_term": 1,
  "monitor": { /* full stored monitor object */ }
}

Error Responses

• 400 Bad Request – missing/invalid fields or PPL syntax errors
• 409 Conflict – duplicate monitor name

2. Get Monitor v2

Endpoint

GET /_plugins/_alerting/v2/monitors/{monitor_id}

Success Response (200 OK)

{
  "monitor_id": "v2-ppl-monitor-1",
  "version": 1,
  "monitor": { /* full v2 monitor object */ }
}

Error Response

• 404 Not Found – monitor does not exist

3. Update Monitor v2

• query_language and type are immutable once set and cannot be modified as part of update.

Endpoint

PUT /_plugins/_alerting/v2/monitors/{monitor_id}&if_seq_no={n}&if_primary_term={m}

Request Body

{
  // (Optional) New name
  "name": "v2-ppl-monitor-renamed",               
  // (Optional) Enable or disable
  "enabled": false,                             
  // (Optional) Replaces existing query
  "queries": "source=_internal (log_level=ERROR OR log_level=FATAL OR log_level=CRITICAL) | stats count by log_level",
  // (Optional) New schedule (either period or cron)
  "schedule": {
    "frequency": "cron",
    "cron": {
      "expression": "0 0 * * *",                // (Required) Cron expression.
      "timezone": "UTC"                         // (Optional) Timezone.
    }
  },
  // (Optional) Updated labels - will replace/override existing labels
  "labels": { 
  "service": "billing" 
  },
  // (Optional) Updated triggers
  "triggers": [ /* same format as Create Monitor */ ]
}

Success Response (200 OK)

{
  "monitor_id": "v2-ppl-monitor-1",
  "version": 2,
  "seq_no": 2,
  "primary_term": 1,
  "monitor": { /* updated monitor object */ }
}

Error Responses

• 400 Bad Request – invalid payload or attempt to change immutable fields
• 404 Not Found – monitor does not exist
• 409 Conflict – version conflict

4. Delete Monitor v2

Endpoint

DELETE /_plugins/_alerting/v2/monitors/{monitor_id}

Success Response (204 No Content)
Error Response

• 404 Not Found – monitor does not exist

5. Search Monitors v2

Endpoint

GET /_plugins/_alerting/v2/monitors/_search

Request Body (Optional)

{
  "query": {                                   // (Optional) filter.
    "bool": {
      "filter": [
        { "term": { "name": "v2-ppl-monitor-1" } },
        { "term": { "enabled": true } }
      ]
    }
  }
}

Success Response (200 OK)

{
  "hits": {
    "total": { "value": 2 },
    "hits": [
      {
        "_id": "v2-ppl-monitor-1",
        "_source": { "monitor": { /* full monitor */ } }
      }
    ]
  }
}

Error Response

• 400 Bad Request – malformed search query

[krisfreedain]

Catch All Triage - 1 2