New Relic Airflow Metric Exporter

A plugin for Apache Airflow to send metrics to New Relic.

Requirements

Airflow versions >= 1.10 are preferred for ease of use; however, versions >= 1.8 should work.

Using the plugin

To start, the newrelic-airflow-plugin package must be installed. To install via pip:

$ pip install newrelic-airflow-plugin

Configuration

Set the NEW_RELIC_INSERT_KEY environment variable to a valid New Relic insert key

The NEW_RELIC_SERVICE_NAME environment variable can be set to customize the service.name attribute on all generated metrics. The default value is Airflow.

NEW_RELIC_HOST environment variable can be used to set datacenter host. For example, to send metrics to EU data center set this variable to metric-api.eu.newrelic.com By default metrics will be send to US data center.

Airflow Versions >= 2.0

Disable Airflow's lazy plugin loading. This is required for the plugin to properly patch Airflow's Stats engine.

This can be done via environment variable:

AIRFLOW__CORE__LAZY_LOAD_PLUGINS=False

Or can be set in your config file (airflow.cfg):

[core]
lazy_load_plugins = False

Airflow Versions >= 1.10,<2.0

✨ That's it! ✨

Airflow Versions < 1.10

The newrelic_plugin.py file must be copied into the configured plugins_folder directory. This defaults to {AIRFLOW_HOME}/plugins.