Skip to content

README.md

Latest commit

 

History

History
602 lines (439 loc) · 25.1 KB

README.md

File metadata and controls

602 lines (439 loc) · 25.1 KB
Raw

nr-tf-util

A New Relic Terraform utility with three modes:

  • audit mode — scan accounts for hardcoded entity IDs that will break after a migration, and flag alert configuration problems (disabled conditions, empty policies, silent notifications, legacy metric conditions). Exports findings to CSV.
  • cleanup mode — read an audit CSV and permanently delete the matched assets, with dry-run preview and typed confirmation before any changes are made.
  • import mode (default) — scan accounts via NerdGraph and generate Terraform import blocks ready for terraform plan -generate-config-out=resources.tf.

Audit mode

Audit mode produces two categories of findings.

Hardcoded ID references — New Relic generates a unique entityGuid for every monitored entity (APM services, infrastructure hosts, dashboards, alerts, etc.). Dashboards and alert conditions often reference entities by these GUIDs or by numeric IDs such as appId or hostId. These IDs are account-specific — they change when resources are migrated to a new account, silently breaking any asset that references them.

Alert configuration health — Beyond migration risk, audit mode also flags structural problems in your alert setup that cause monitoring to be ineffective regardless of migration:

  • Conditions that are disabled and therefore never fire.
  • Policies with no conditions, which can never generate an alert.
  • Policies whose conditions are active but have no notification workflow routing their alerts — violations open and close silently with nobody notified.
  • Legacy APM/Browser/Mobile metric conditions that predate NRQL alerting and should be recreated as NRQL conditions for long-term support and flexibility.

Audit mode:

  1. Fetches dashboards, NRQL alert conditions, workloads, service levels, workflows, alert policies, and legacy metric conditions for each account.
  2. Scans NRQL queries, widget configurations, entity lists, and filter expressions for hardcoded ID references, and checks alert policies and conditions for configuration problems.
  3. Resolves each found entity ID to a name via NerdGraph.
  4. Prints a grouped report and optionally writes a CSV file (--csv).

Cleanup mode

Cleanup mode consumes an audit CSV and permanently deletes the matched assets via NerdGraph.

Workflow:

# 1. Audit and export findings to CSV
nr-tf-util --mode audit --account-id 2883194 --csv findings.csv

# 2. Review the CSV — filter or edit rows as needed

# 3. Dry-run: see exactly what would be deleted
nr-tf-util --mode cleanup --csv findings.csv \
  --asset-type nrql_condition --dry-run

# 4. Execute with typed confirmation
nr-tf-util --mode cleanup --csv findings.csv \
  --asset-type nrql_condition

How cleanup selects rows from the CSV:

  • Only rows with an empty id_type column are eligible. These are config findings (disabled conditions, empty policies, etc.). Hardcoded-ID reference rows — which flag assets that need to be fixed, not deleted — are always skipped.
  • The --asset-type flag further filters by the asset_type column value. You must specify at least one type to prevent accidental mass deletion.
  • Multiple rows for the same asset (e.g. a policy flagged for both empty conditions and missing notification) are deduplicated — each asset is deleted only once.

Confirmation prompt:

Before any deletion, the tool prints the full list of assets to be removed and prompts you to type the exact token "delete N assets" — where N is the count — before proceeding. This prevents accidental execution and makes the scope of the operation explicit.

The following assets will be permanently deleted:

  Account 2883194 "Prod" (2 asset(s)):
    [nrql_condition] "Dormant Alert"    id: 2883194:99
    [nrql_condition] "Off Condition"    id: 2883194:102

Total: 2 asset(s) across 1 account(s)

Type "delete 2 assets" to confirm (Ctrl-C to cancel):

Supported asset types:

--asset-type value Operation What changes NerdGraph mutation
nrql_condition Full delete Removes the NRQL condition entirely alertsNrqlConditionDelete
alert_policy Full delete Removes the policy and all its conditions alertsPolicyDelete
workload_entity Entity removal Removes one entity GUID from a workload's static entity list workloadUpdate
workflow_entity Entity removal Removes one entity GUID from a workflow's issue filter predicates aiWorkflowsUpdateWorkflow
synthetics_monitor Full delete Permanently removes a synthetic monitor syntheticsDeleteMonitor

Warning: Deleting an alert_policy removes the policy and all conditions attached to it. Use --dry-run to review the scope before confirming.

Import mode

Import mode generates Terraform import blocks for all supported New Relic resource types, ready to use with terraform plan -generate-config-out=resources.tf.

  1. Scan — connects to NerdGraph and enumerates every supported resource type across one or more accounts (auto-discovered or explicitly listed).
  2. Generate — writes one .tf file per resource type, plus provider.tf and variables.tf, under a per-account subdirectory of --out.
  3. Import — run terraform init && terraform plan -generate-config-out=resources.tf inside each generated directory so Terraform produces the matching HCL resource blocks from live state.

No Terraform state is created or modified until you explicitly run terraform apply.

Installation

Pre-built binary

Download the binary for your platform from the GitHub Releases page.

# macOS (Apple Silicon)
curl -L https://github.com/asomensari/nr-tf-util/releases/latest/download/nr-tf-util-darwin-arm64 \
  -o nr-tf-util && chmod +x nr-tf-util

# Linux (x86-64)
curl -L https://github.com/asomensari/nr-tf-util/releases/latest/download/nr-tf-util-linux-amd64 \
  -o nr-tf-util && chmod +x nr-tf-util

Build from source

Requires Go 1.21+.

git clone https://github.com/asomensari/nr-tf-util.git
cd nr-tf-util
make build          # produces bin/nr-tf-util

Cross-compile for all platforms:

make dist           # produces dist/ with win/mac/linux binaries

Quick start

export NEW_RELIC_API_KEY="NRAK-..."

# ── Audit mode ──────────────────────────────────────────────────────────────

# Full audit: hardcoded ID refs + alert config health
nr-tf-util --mode audit

# Audit a single account and export findings to CSV
nr-tf-util --mode audit --account-id 2883194 --csv findings.csv

# Skip legacy-condition and uncovered-policy checks
nr-tf-util --mode audit --skip legacy_conditions --skip uncovered_alert_policies

# Skip synthetic monitor scanning (useful if account has many monitors)
nr-tf-util --mode audit --skip synthetics_monitors

# ID-reference checks only (skip all config health checks)
nr-tf-util --mode audit \
  --skip legacy_conditions \
  --skip uncovered_alert_policies

# ── Cleanup mode ────────────────────────────────────────────────────────────

# Dry-run: preview what would be deleted from the audit CSV
nr-tf-util --mode cleanup --csv findings.csv \
  --asset-type nrql_condition --dry-run

# Delete disabled NRQL conditions (requires typed confirmation)
nr-tf-util --mode cleanup --csv findings.csv \
  --asset-type nrql_condition

# Delete both empty policies and disabled conditions
nr-tf-util --mode cleanup --csv findings.csv \
  --asset-type alert_policy --asset-type nrql_condition

# Remove a hardcoded entity GUID from workload static entity lists
nr-tf-util --mode cleanup --csv findings.csv \
  --asset-type workload_entity --dry-run

# Remove a hardcoded entity GUID from workflow issue filter predicates
nr-tf-util --mode cleanup --csv findings.csv \
  --asset-type workflow_entity

# Delete disabled synthetic monitors
nr-tf-util --mode cleanup --csv findings.csv \
  --asset-type synthetics_monitor

# ── Import mode (default) ────────────────────────────────────────────────────

# Generate import blocks for all accessible accounts
nr-tf-util

# Scan specific accounts, EU region, dry-run
nr-tf-util --region EU --account-id 2883194 --account-id 1234567 --dry-run

# Skip dashboards and workloads
nr-tf-util --skip dashboards --skip workloads

# Then for each generated account directory:
cd output/2883194-my-account/
terraform init
terraform plan -generate-config-out=resources.tf
terraform apply

Flags

Shared flags (all modes)

Flag Default Description
--mode import Operation mode: audit, cleanup, or import.
--api-key $NEW_RELIC_API_KEY New Relic User API key (NRAK-...). Falls back to NEW_RELIC_API_KEY env var.
--region US NerdGraph region: US, EU, FedRAMP, or JP.
--rate-limit 25 Max NerdGraph API requests per second.
--max-retries 3 Max retries for failed API calls.
--timeout 15m Per-account scan timeout (Go duration string, e.g. 2m30s).
--verbose false Show additional detail in progress output.
--quiet false Suppress all non-error output.

Audit mode flags

Flag Default Description
--account-id auto-discover Account ID to scan. Repeat for multiple.
--skip — Fetcher/asset type to skip. Repeat for multiple.
--csv — Write findings to a CSV file (e.g. --csv findings.csv).

Cleanup mode flags

Flag Default Description
--csv — Required. Input CSV file produced by audit mode.
--asset-type — Required. Asset type to delete. Repeat for multiple. Supported: nrql_condition, alert_policy, workload_entity, workflow_entity, synthetics_monitor.
--dry-run false Print the deletion plan without deleting anything.

Import mode flags

Flag Default Description
--account-id auto-discover Account ID to scan. Repeat for multiple.
--skip — Fetcher/asset type to skip. Repeat for multiple.
--out ./output Root directory for output. One subdirectory is created per account.
--dry-run false Print what would be written without writing any files.

Audit mode reference

Asset types scanned

Hardcoded ID reference checks

Asset type --skip value What is scanned
Dashboards dashboards NRQL queries in every widget (rawConfiguration)
NRQL alert conditions nrql_conditions Condition NRQL query string
Workloads workloads Static entity GUID list + dynamic entity filter expressions
Service levels service_levels Monitored entity GUID (via entity tag)
Workflows workflows Issue filter predicate values
Synthetic monitors synthetics_monitors Script body of SCRIPT_API and SCRIPT_BROWSER monitors
Legacy metric conditions legacy_conditions Monitored entity IDs (appId, browserId, mobileId) — fetched via REST API v2

Alert configuration health checks

Check --skip value What is flagged Rationale
Disabled NRQL conditions disabled_conditions Conditions with enabled=false A disabled condition never fires. It should be re-enabled if monitoring is still needed, or removed to keep the policy readable.
Disabled synthetic monitors synthetics_monitors Monitors with status=DISABLED (any monitor type) A disabled monitor produces no results and incurs no checks-per-month cost, but it clutters the monitors list and may give a false sense of coverage.
Legacy metric conditions legacy_conditions APM/Browser/Mobile metric conditions (non-NRQL) These predate NRQL alerting and are no longer the recommended format. NRQL conditions offer finer control, support streaming alerts, and will receive future New Relic improvements; legacy conditions will not. Scanned via the REST API v2 (same User API key; no extra credential needed).
Policies with no conditions uncovered_alert_policies Alert policies that have zero conditions An empty policy can never generate an alert. It is dead configuration that clutters the policy list and may give a false sense of coverage.
Policies with no notification uncovered_alert_policies Policies with at least one enabled condition but no aiWorkflow routing their alerts When a condition fires, New Relic creates a violation and opens an issue — but if no workflow routes that issue to a destination, nobody is notified. This is one of the most common causes of silent alerting failures.

Note on notification detection: The tool checks for aiWorkflows that explicitly target a policy via labels.policyIds predicates, or for a catch-all workflow (VIEW_ALL filter type). Workflows that match by policy name or other attributes are not detected as coverage — review those policies manually.

Hardcoded ID patterns detected

ID type Example Why it breaks
entityGuid WHERE entity.guid = 'MTIz...' GUIDs are account-specific; change on migration
appId WHERE appId = 99887766 Numeric APM application ID; reassigned in new account
hostId WHERE hostId = 44556677 Numeric infrastructure host ID; reassigned in new account
entityId WHERE entityId = 12345 Generic numeric entity ID; account-specific
monitorId WHERE monitorId = 'abc-123' Synthetics monitor ID; changes after re-import

IN (...) lists are exploded into individual findings so each value is resolved separately.

Migration suggestions

The audit report includes a per-finding suggestion:

ID type Resolved Suggestion
appId yes Replace with: appName = '<name>'
hostId yes Replace with: hostname = '<name>'
entityGuid yes GUID is account-specific; after migration find '<name>' by name to get the new GUID
monitorId yes Monitor ID changes on migration; after re-import reference '<name>' by name
any no Entity could not be resolved — may have been deleted or is in another account

CSV export

Pass --csv <file> to write all findings to a CSV file alongside the text report. The CSV has 13 columns:

account_id, account_name, asset_type, asset_id, asset_name,
nrql_query, location, id_type, id_value,
resolved, entity_name, entity_type, suggestion

The nrql_query column contains the NRQL query string where the issue was found (populated for nrql_condition and dashboard widget findings; empty otherwise).

Configuration health findings (disabled conditions, empty policies, etc.) appear as rows with blank id_type, id_value, entity_name, and entity_type columns. These are the rows that cleanup mode acts on.

Sample audit output

══════════════════════════════════════════════════════════════
Account: Production (2883194)   findings: 7   assets affected: 5
──────────────────────────────────────────────────────────────

  DASHBOARD  "APM Overview"  (id: MTIzNDU2N3xWS...)
  ├─ page "Overview" · widget: Error Rate
  │    appId 99887766 → Checkout Service (APPLICATION)
  │    suggestion: Replace with: appName = 'Checkout Service'
  │
  └─ page "Overview" · widget: Throughput
       entityGuid MTIzNDU2N3x... → Checkout Service (APPLICATION)
       suggestion: GUID is account-specific; after migration find 'Checkout Service' by name to get the new GUID

  NRQL_CONDITION  "Latency P99 — Deprecated"  (id: 2883194:555)
  └─ condition configuration
       Condition is disabled — enable it or remove it to reduce policy noise

  ALERT_CONDITION  "Response Time"  (id: 2883194:10)
  └─ policy: APM Alerts
       Legacy APM_APP_METRIC condition — recreate as a NRQL alert condition for long-term support

  ALERT_POLICY  "Infra Monitoring"  (id: 77701)
  └─ policy conditions
       Policy has no conditions — it will never generate alerts

  ALERT_POLICY  "Checkout Alerts"  (id: 77702)
  └─ policy notification
       Policy has active conditions but no workflow is routing its alerts — they will fire silently

══════════════════════════════════════════════════════════════
Overall summary
  Accounts           : 1
  Hardcoded ID refs  : 2 total  (2 resolved, 0 unresolved)
  Config issues      : 5
  Assets affected    : 5

Cleanup mode reference

How rows are selected

Cleanup mode reads the audit CSV and applies the following filters before building the deletion list:

  1. Config findings only — rows where id_type is empty. Hardcoded-ID reference rows (where id_type is e.g. appId or entityGuid) flag assets that need to be fixed, not deleted, so they are always excluded.
  2. Asset type filter — only rows whose asset_type matches one of the --asset-type values you specify.
  3. Deduplication — a policy flagged for both "no conditions" and "no notification" produces two CSV rows but is deleted only once.

Dry-run mode

--dry-run prints the full deletion plan — account, asset type, asset name, and ID — then exits without making any API calls or prompting for confirmation. Use it to audit the scope before committing.

nr-tf-util --mode cleanup --csv findings.csv \
  --asset-type nrql_condition --dry-run
The following assets will be permanently deleted:

  Account 2883194 "Prod" (3 asset(s)):
    [nrql_condition]  "Dormant Alert"     id: 2883194:99
    [nrql_condition]  "CPU P99 — Legacy"  id: 2883194:102
    [nrql_condition]  "Off Condition"     id: 2883194:107

Total: 3 asset(s) across 1 account(s)

[DRY-RUN — no assets will be deleted]

Live deletion flow

Without --dry-run, cleanup mode prints the same plan and then prompts you to type the exact confirmation token before proceeding. Typing anything else — or pressing Ctrl-C — aborts with no changes made.

Deletions are executed one at a time. Each result is reported as it completes:

  OK    [nrql_condition] "Dormant Alert"
  OK    [nrql_condition] "CPU P99 — Legacy"
  FAIL  [nrql_condition] "Off Condition": alertsNrqlConditionDelete: permission denied

Deleted: 2   Failed: 1

A non-zero failure count causes the command to exit with a non-zero status, which makes it safe to use in scripts that check $?.

Supported asset types

Full deletions

--asset-type Matches CSV rows where NerdGraph mutation Notes
nrql_condition asset_type=nrql_condition, id_type empty alertsNrqlConditionDelete Deletes the condition. The parent policy is not affected.
alert_policy asset_type=alert_policy, id_type empty alertsPolicyDelete Deletes the policy and all conditions attached to it.
synthetics_monitor asset_type=synthetics_monitor, id_type empty syntheticsDeleteMonitor Permanently deletes the monitor. Applies to all monitor types (SIMPLE, BROWSER, SCRIPT_API, etc.).

Entity removal (read-modify-write)

Entity-removal types select rows where id_type=entityGuid and perform a fetch-then-update instead of a straight delete.

--asset-type Matches CSV rows where NerdGraph operations Notes
workload_entity asset_type=workload, id_type=entityGuid, location=static entity list workload { collection } → workloadUpdate Removes the GUID from the workload's static entity list. Dynamic filter expressions are not touched.
workflow_entity asset_type=workflow, id_type=entityGuid aiWorkflows { workflows } → aiWorkflowsUpdateWorkflow Removes the GUID from every predicate that contains it. Empty predicates are dropped. Returns an error if removing the GUID would leave the workflow with no predicates at all — that case must be handled manually.

Import mode reference

Supported resources

Fetcher name Terraform resource type
alert_policies newrelic_alert_policy
nrql_conditions newrelic_nrql_alert_condition
muting_rules newrelic_alert_muting_rule
notification_destinations newrelic_notification_destination
notification_channels newrelic_notification_channel
workflows newrelic_workflow
synthetics_monitors newrelic_synthetics_monitor (and variants)
dashboards newrelic_one_dashboard
service_levels newrelic_service_level
workloads newrelic_workload

Output structure

output/
└── 2883194-my-account/
    ├── provider.tf              # newrelic provider block + terraform required_providers
    ├── variables.tf             # account_id and api_key input variables
    ├── alert_policies.tf        # import blocks for alert policies
    ├── nrql_conditions.tf
    ├── workflows.tf
    ├── dashboards.tf
    └── ...

Each resource file contains one import block per resource:

import {
  # My Alert Policy
  to = newrelic_alert_policy.my_alert_policy
  id = "12345"
}

Only files with at least one resource are written.

Next steps after generation

cd output/<account-dir>/

# 1. Initialise Terraform (downloads the New Relic provider)
terraform init

# 2. Generate resource configuration from the import blocks
terraform plan -generate-config-out=resources.tf

# 3. Review resources.tf, adjust as needed, then apply
terraform apply

After apply, your New Relic resources are under Terraform management.

Regions

Value NerdGraph endpoint REST API v2 base URL Notes
US https://api.newrelic.com/graphql https://api.newrelic.com Default
EU https://api.eu.newrelic.com/graphql https://api.eu.newrelic.com
FedRAMP https://gov-api.newrelic.com/graphql https://gov-api.newrelic.com Generates nerdgraph_api_url override in provider.tf
JP https://api.jp.newrelic.com/graphql https://api.jp.newrelic.com Japanese datacenter

API key permissions

The tool requires a User API key (NRAK-...).

  • Audit and import modes — read access to the accounts you want to scan. The same key is used for both NerdGraph queries and REST API v2 calls (legacy conditions). No additional credential is needed.
  • Cleanup mode — write access is required in addition to read access, as the tool calls NerdGraph delete mutations.

Rate limiting and retries

  • A token-bucket rate limiter (default 25 req/s) prevents hitting New Relic's NerdGraph API limits.
  • The underlying newrelic-client-go/v2 library retries transient failures automatically using exponential back-off (hashicorp/go-retryablehttp).
  • Adjust --rate-limit if you have many accounts or see throttling errors.
  • Legacy condition scanning makes one REST API v2 call per alert policy. Accounts with many policies may take additional time during the legacy_conditions check. Use --skip legacy_conditions to skip this check if speed is a priority.

Contributing

make test           # run all unit tests
make test-cover     # test with coverage report
make vet            # go vet
make lint           # golangci-lint (must be installed)

Bug reports and pull requests are welcome at github.com/asomensari/nr-tf-util/issues.

Planned modes

Mode Status Description
audit available Detect hardcoded entity IDs and alert config health issues
cleanup available Delete assets identified by an audit CSV
import available Generate Terraform import blocks from existing resources
generate planned Generate baseline alert policies and dashboards for live entities

See docs/generate-mode.md for the generate mode design.

License

MIT — see LICENSE.