Skip to content

generate-mode.md

Latest commit

 

History

History
224 lines (164 loc) · 8.19 KB

generate-mode.md

File metadata and controls

224 lines (164 loc) · 8.19 KB
Raw

Generate Mode — Design Document

Status: planned / not yet implemented. This document describes the intended design for a future --mode generate capability.

Overview

The current nr-tf-util (default mode) imports resources that already exist in New Relic into Terraform. Generate mode does the opposite: it scans live entities (APM services, infrastructure hosts, Synthetics monitors, Browser apps) and generates new Terraform configuration that creates baseline alert policies, NRQL conditions, and optionally dashboards for them.

The goal is to bootstrap monitoring coverage for entities that have none, without manual configuration.

Invocation

nr-tf-util --mode generate \
  --account-id 2883194 \
  --entity-type apm \
  --entity-type infra \
  --out ./generated-alerts

The flag --mode generate (default: import) switches the entire pipeline from import-block generation to new-resource generation.

Entity types

--entity-type value New Relic entities scanned Resources generated
apm APM applications (domain=APM) Alert policy + NRQL conditions per app
infra Infrastructure hosts (domain=INFRA) Alert policy + NRQL conditions per host
synthetics Synthetics monitors Alert policy + NRQL condition for monitor failures
browser Browser applications (domain=BROWSER) Alert policy + NRQL conditions per browser app

All four may be specified together; they are processed concurrently.

Output structure

generated-alerts/
└── 2883194-my-account/
    ├── provider.tf
    ├── variables.tf
    ├── apm_alerts.tf          # one policy + N conditions per APM app
    ├── infra_alerts.tf
    ├── synthetics_alerts.tf
    └── browser_alerts.tf

Generated resource shape

Each entity produces one newrelic_alert_policy and a set of newrelic_nrql_alert_condition resources. Example for an APM application named "Checkout Service":

resource "newrelic_alert_policy" "checkout_service" {
  name                = "Checkout Service — Baseline Alerts"
  incident_preference = "PER_CONDITION_AND_TARGET"
}

resource "newrelic_nrql_alert_condition" "checkout_service_error_rate" {
  account_id  = var.account_id
  policy_id   = newrelic_alert_policy.checkout_service.id
  name        = "Checkout Service — Error Rate"
  type        = "static"
  enabled     = true

  nrql {
    query = "SELECT percentage(count(*), WHERE error IS TRUE) FROM Transaction WHERE appName = 'Checkout Service'"
  }

  critical {
    operator              = "above"
    threshold             = 5
    threshold_duration    = 300
    threshold_occurrences = "ALL"
  }

  warning {
    operator              = "above"
    threshold             = 2
    threshold_duration    = 300
    threshold_occurrences = "ALL"
  }
}

Default alert thresholds

Defaults are designed to be conservative (low false-positive rate) and can be overridden per entity type via flags.

APM

Condition NRQL Critical Warning
Error rate percentage(count(*), WHERE error IS TRUE) FROM Transaction > 5% > 2%
Response time (p95) percentile(duration, 95) FROM Transaction > 2s > 1s
Throughput drop rate(count(*), 1 minute) FROM Transaction < 10% < 30%

Infrastructure hosts

Condition NRQL Critical Warning
CPU usage average(cpuPercent) FROM SystemSample > 90% > 75%
Memory usage average(memoryUsedPercent) FROM SystemSample > 90% > 80%
Disk usage average(diskUsedPercent) FROM StorageSample > 85% > 70%
Host not reporting count(*) FROM SystemSample < 1 —

Synthetics

Condition NRQL Critical Warning
Monitor failure filter(count(*), WHERE result = 'FAILED') FROM SyntheticCheck ≥ 1 —
Duration average(duration) FROM SyntheticCheck > 10s > 5s

Browser

Condition NRQL Critical Warning
JS error rate percentage(count(*), WHERE errorType IS NOT NULL) FROM JavaScriptError > 5% > 2%
Page load time average(duration) FROM PageView > 5s > 3s

Threshold override flags

nr-tf-util --mode generate \
  --apm-error-rate-critical 10 \
  --apm-response-time-critical 3000 \
  --infra-cpu-critical 95

Each threshold follows the pattern --<entity-type>-<metric>-<level> <value>.

Entity name filtering

Scan only a subset of entities using --entity-filter (supports glob patterns):

# Only APM apps whose names start with "Checkout" or "Payment"
nr-tf-util --mode generate \
  --entity-type apm \
  --entity-filter "Checkout*" \
  --entity-filter "Payment*"

Filtering is applied client-side after the NerdGraph entity search.

Dashboard generation (optional)

Pass --generate-dashboards to also produce one newrelic_one_dashboard per entity type containing the golden signal charts (error rate, throughput, latency, saturation).

nr-tf-util --mode generate --entity-type apm --generate-dashboards

Dashboard resources are written to a separate *_dashboards.tf file.

Notification wiring (optional)

If you have existing notification destinations (emails, PagerDuty, Slack channels) that were previously imported, pass --wire-notifications to auto-generate newrelic_workflow resources that link the generated alert policies to those destinations.

nr-tf-util --mode generate --wire-notifications

The tool will use the notification destinations present in the current --out directory (from a prior import-mode scan) to build the workflows.

Implementation notes

  • Entity scanning uses the NerdGraph entitySearch(query: "domain='APM' AND type='APPLICATION'") API with cursor pagination.
  • Each entity type has its own GenerateFetcher interface analogous to the existing Fetcher interface, but producing *hclwrite.File blocks instead of import blocks.
  • Threshold structs are defined in internal/generate/thresholds.go with per-entity-type defaults and CLI override injection.
  • The NRQL templates live in internal/generate/nrql.go as Go template strings, keyed by entity type + metric name.
  • All generated Terraform references the account's var.account_id and var.api_key, matching the import-mode output so both can coexist in the same directory.

Open questions

  1. Deduplication: if an entity already has an alert policy with the same name, should generate mode skip it, warn, or overwrite?
  2. Naming convention: should the generated policy name include the account name for cross-account legibility?
  3. Tag-based scoping: should generate mode support filtering entities by NR tag (e.g. --entity-tag team=payments)?
  4. Notification destination discovery: auto-wiring notifications requires knowing which destination corresponds to which team — this likely requires additional metadata or a mapping config file.