Kbauer/internal telemetry nr k8s otel collector

other-examples/collector/internal-telemetry-nr-k8s-otel-collector/.gitignore

@@ -0,0 +1 @@
+k8s/secrets.yaml

other-examples/collector/internal-telemetry-nr-k8s-otel-collector/README.md

@@ -0,0 +1,78 @@
+# Internal Telemetry for nr-k8s-otel-collector Helm Chart
+
+This example demonstrates how to enable comprehensive internal telemetry for the OpenTelemetry collectors deployed via the [newrelic/nr-k8s-otel-collector](https://github.com/newrelic/helm-charts/tree/master/charts/nr-k8s-otel-collector) helm chart using our erecommended configuration](https://github.com/newrelic/nrdot-collector-releases/blob/ed3bd71116742391c40209039e999ebea6ba80f9/examples/internal-telemetry-config.yaml). This configuration enables detailed monitoring of the collectors themselves, including metrics, logs, and optionally traces. It also decorates the telemetry with necessary attributes to drive container-to-service relationships.
+
+Please note that this example overrides [some of the existing internal telemetry configuration](https://github.com/newrelic/helm-charts/blob/96436bef4e6311bf4e1a71031ff536cc7d42625f/charts/nr-k8s-otel-collector/templates/deployment-configmap.yaml#L791-L814). We are working towards embedding the configuration from this example directly in the helm chart to make enabling collector observability easier.
+
+## Requirements
+
+* A Kubernetes cluster with kubectl configured
+* Helm 3.x installed
+* [A New Relic account](https://one.newrelic.com/)
+* [A New Relic license key](https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/#license-key)
+
+## Running the example
+
+1. Copy the template file and update with your values:
+
+    ```shell
+    cp k8s/secrets.yaml.template k8s/secrets.yaml
+    ```
+
+2. Edit `k8s/secrets.yaml` and update the following values:
+    * `NEW_RELIC_LICENSE_KEY`: Your New Relic license key
+    * `NEW_RELIC_OTLP_ENDPOINT`: If your account is based in the EU, change to `https://otlp.eu01.nr-data.net`
+    * `CLUSTER_NAME`: Your cluster name (should match the value passed to `--set cluster=` in the helm install command)
+
+3. Add the New Relic helm repository:
+
+    ```shell
+    helm repo add newrelic https://helm-charts.newrelic.com
+    helm repo update
+    ```
+
+4. Create the namespace and apply the secrets and ConfigMap:
+
+    ```shell
+    kubectl create namespace internal-telemetry-nr-k8s-otel-collector
+    kubectl apply -f k8s/
+    ```
+
+5. Install the helm chart with the custom values (replace `<YOUR_LICENSE_KEY>` and `<YOUR_CLUSTER_NAME>` with your actual values):
+
+    ```shell
+    helm install nr-k8s-otel-collector newrelic/nr-k8s-otel-collector \
+      --namespace internal-telemetry-nr-k8s-otel-collector \
+      -f values.yaml \
+      --set licenseKey=<YOUR_LICENSE_KEY> \
+      --set cluster=<YOUR_CLUSTER_NAME>
+    ```
+
+    * When finished, cleanup all resources by deleting the namespace:
+
+    ```shell
+    kubectl delete namespace internal-telemetry-nr-k8s-otel-collector
+    ```
+
+## Viewing your data
+
+To review your collector internal telemetry in New Relic, navigate to "New Relic -> All Entities" and search for entities with the service names you configured (by default: `nr-k8s-otel-collector-deployment` and `nr-k8s-otel-collector-daemonset`). Click on an entity to view the service summary, including golden metrics and performance data for the collector itself.
+
+## Additional notes
+
+### Customizing telemetry levels
+
+You can customize the telemetry levels by adding additional environment variables in [values.yaml](./values.yaml):
+
+- `INTERNAL_TELEMETRY_METRICS_LEVEL`: `detailed` (default), `normal`, `basic`, or `none`
+- `INTERNAL_TELEMETRY_LOG_LEVEL`: `INFO` (default), `DEBUG`, `WARN`, or `ERROR`
+- `INTERNAL_TELEMETRY_TRACE_LEVEL`: `none` (default) or `basic` (experimental)
+- `INTERNAL_TELEMETRY_TRACE_SAMPLE_RATIO`: `0.01` (default, 1% sampling)
+
+### Two collectors
+
+The nr-k8s-otel-collector chart deploys two collectors:
+- A **deployment** collector for cluster-level metrics
+- A **daemonset** collector that runs on each node for host-level metrics
+
+This example configures internal telemetry for both, with distinct service names to differentiate them in New Relic.

other-examples/collector/internal-telemetry-nr-k8s-otel-collector/k8s/internal-telemetry-config.yaml

@@ -0,0 +1,82 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: otel-internal-telemetry-config
+  namespace: internal-telemetry-nr-k8s-otel-collector
+data:
+  config.yaml: |
+    ##### Example configuration for internal telemetry
+    # This configuration is intended to be used in conjunction with a configuration of components and pipelines. The
+    # collector supports config merging on startup.
+    # Source: https://github.com/newrelic/nrdot-collector-releases/blob/ed3bd71116742391c40209039e999ebea6ba80f9/examples/internal-telemetry-config.yaml
+    ##### Requirements
+    # - nrdot-collector (any distro) >= 1.3.0 or collector core version >= v1.35.0 / v0.129.0
+    ##### Configuration via environment variables
+    # For official documentation, see: https://opentelemetry.io/docs/collector/internal-telemetry/
+    ## Required
+    # - INTERNAL_TELEMETRY_NEW_RELIC_LICENSE_KEY
+    ## Optional
+    # - INTERNAL_TELEMETRY_SERVICE_NAME: defaults to 'otel-collector'; determines entity name in New Relic
+    # - INTERNAL_TELEMETRY_OTLP_ENDPOINT: defaults to 'https://otlp.nr-data.net'; see https://docs.newrelic.com/docs/opentelemetry/best-practices/opentelemetry-otlp/ and https://docs.newrelic.com/docs/opentelemetry/best-practices/opentelemetry-otlp-troubleshooting/
+    # - INTERNAL_TELEMETRY_METRICS_LEVEL: defaults to 'detailed'; other values are 'normal', 'basic', 'none'
+    # - INTERNAL_TELEMETRY_LOG_LEVEL: defaults to INFO; other values are DEBUG, WARN, ERROR
+    # - INTERNAL_TELEMETRY_TRACE_LEVEL: defaults to 'none' (traces disabled); other value is 'basic'
+    # - INTERNAL_TELEMETRY_TRACE_SAMPLE_RATIO: defaults to 0.01, i.e. 1% sampling; has no effect if TRACE_LEVEL is 'none'
+    service:
+      telemetry:
+        metrics:
+          level: "${env:INTERNAL_TELEMETRY_METRICS_LEVEL:-detailed}"
+          readers:
+            - periodic:
+                exporter:
+                  otlp:
+                    protocol: http/protobuf
+                    endpoint: "${env:INTERNAL_TELEMETRY_OTLP_ENDPOINT:-https://otlp.nr-data.net}"
+                    headers:
+                      - name: api-key
+                        value: "${env:INTERNAL_TELEMETRY_NEW_RELIC_LICENSE_KEY}"
+        logs:
+          level: "${env:INTERNAL_TELEMETRY_LOG_LEVEL:-INFO}"
+          # default sampling config for reference to simplify overwrites even if not exposed via env var, e.g. --config=yaml:service::telemetry::logs::sampling::enabled::false
+          sampling:
+            enabled: true
+            # The interval in seconds that the logger applies to each sampling.
+            tick: 10s
+            # The number of messages logged at the start of each sampling::tick
+            initial: 10
+            # Sets the sampling policy for subsequent messages after sampling::initial messages are logged. When sampling::thereafter is set to N, every Nth message is logged and all others are dropped. If N is zero, the logger drops all messages after sampling::initial messages are logged.
+            thereafter: 100
+          processors:
+            - batch:
+                exporter:
+                  otlp:
+                    protocol: http/protobuf
+                    endpoint: "${env:INTERNAL_TELEMETRY_OTLP_ENDPOINT:-https://otlp.nr-data.net}"
+                    headers:
+                      - name: api-key
+                        value: "${env:INTERNAL_TELEMETRY_NEW_RELIC_LICENSE_KEY}"
+        traces:
+          # traces are disabled by default due to experimental status and lack of default sampling rate that works across use cases
+          level: "${env:INTERNAL_TELEMETRY_TRACE_LEVEL:-none}"
+          sampler:
+            parent_based:
+              root:
+                trace_id_ratio_based:
+                  ratio: ${env:INTERNAL_TELEMETRY_TRACE_SAMPLE_RATIO:-0.01}
+          processors:
+            - batch:
+                exporter:
+                  otlp:
+                    protocol: http/protobuf
+                    endpoint: "${env:INTERNAL_TELEMETRY_OTLP_ENDPOINT:-https://otlp.nr-data.net}"
+                    headers:
+                      - name: api-key
+                        value: "${env:INTERNAL_TELEMETRY_NEW_RELIC_LICENSE_KEY}"
+        resource:
+          newrelic.collector_telemetry.version: 0.4.0
+          newrelic.service.type: otel_collector
+          service.name: "${env:INTERNAL_TELEMETRY_SERVICE_NAME:-otel-collector}"
+          k8s.cluster.name: "${env:CLUSTER_NAME}"
+          k8s.namespace.name: "${env:NAMESPACE}"
+          k8s.pod.name: "${env:POD_NAME}"
+          k8s.container.name: "${env:CONTAINER_NAME}"

other-examples/collector/internal-telemetry-nr-k8s-otel-collector/k8s/secrets.yaml.template

@@ -0,0 +1,14 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: nr-otel-internal-telemetry-secret
+  namespace: internal-telemetry-nr-k8s-otel-collector
+stringData:
+  # New Relic license key to authenticate the export requests.
+  # docs: https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/#license-key
+  NEW_RELIC_LICENSE_KEY: <INSERT_LICENSE_KEY>
+  # The default US endpoint is set here. If your account is based in the EU, use `https://otlp.eu01.nr-data.net` instead.
+  # docs: https://docs.newrelic.com/docs/more-integrations/open-source-telemetry-integrations/opentelemetry/best-practices/opentelemetry-otlp/#configure-endpoint-port-protocol
+  NEW_RELIC_OTLP_ENDPOINT: https://otlp.nr-data.net
+  # Cluster name to add as k8s.cluster.name resource attribute (should match the cluster value passed to helm)
+  CLUSTER_NAME: <YOUR_CLUSTER_NAME>

other-examples/collector/internal-telemetry-nr-k8s-otel-collector/values.yaml

@@ -0,0 +1,107 @@
+# Helm values for nr-k8s-otel-collector with internal telemetry enabled
+# This configuration adds comprehensive internal telemetry to both the deployment and daemonset collectors
+
+# Configuration for the deployment collector (cluster-level metrics)
+deployment:
+  # Mount the internal telemetry config as a volume
+  extraVolumes:
+    - name: internal-telemetry-config
+      configMap:
+        name: otel-internal-telemetry-config
+
+  extraVolumeMounts:
+    - name: internal-telemetry-config
+      mountPath: /etc/otel-internal-telemetry
+      readOnly: true
+
+  # Add the config file path as an additional argument
+  extraArgs:
+    - --config=file:/etc/otel-internal-telemetry/config.yaml
+
+  # Environment variables for internal telemetry configuration
+  envs:
+    - name: INTERNAL_TELEMETRY_SERVICE_NAME
+      value: "kbauer-feb18-nr-k8s-otel-collector-deployment"
+    - name: INTERNAL_TELEMETRY_OTLP_ENDPOINT
+      valueFrom:
+        secretKeyRef:
+          name: nr-otel-internal-telemetry-secret
+          key: NEW_RELIC_OTLP_ENDPOINT
+    - name: INTERNAL_TELEMETRY_NEW_RELIC_LICENSE_KEY
+      valueFrom:
+        secretKeyRef:
+          name: nr-otel-internal-telemetry-secret
+          key: NEW_RELIC_LICENSE_KEY
+    # K8s resource attributes for internal telemetry
+    - name: CLUSTER_NAME
+      valueFrom:
+        secretKeyRef:
+          name: nr-otel-internal-telemetry-secret
+          key: CLUSTER_NAME
+    - name: NAMESPACE
+      valueFrom:
+        fieldRef:
+          fieldPath: metadata.namespace
+    - name: POD_NAME
+      valueFrom:
+        fieldRef:
+          fieldPath: metadata.name
+    - name: CONTAINER_NAME
+      value: "otel-collector-deployment"
+
+# Configuration for the daemonset collector (node-level metrics)
+daemonset:
+  # Mount the internal telemetry config as a volume
+  extraVolumes:
+    - name: internal-telemetry-config
+      configMap:
+        name: otel-internal-telemetry-config
+
+  extraVolumeMounts:
+    - name: internal-telemetry-config
+      mountPath: /etc/otel-internal-telemetry
+      readOnly: true
+
+  # Add the config file path as an additional argument
+  extraArgs:
+    - --config=file:/etc/otel-internal-telemetry/config.yaml
+
+  # Environment variables for internal telemetry configuration
+  envs:
+    - name: INTERNAL_TELEMETRY_SERVICE_NAME
+      value: "kbauer-feb18-nr-k8s-otel-collector-daemonset"
+    - name: INTERNAL_TELEMETRY_OTLP_ENDPOINT
+      valueFrom:
+        secretKeyRef:
+          name: nr-otel-internal-telemetry-secret
+          key: NEW_RELIC_OTLP_ENDPOINT
+    - name: INTERNAL_TELEMETRY_NEW_RELIC_LICENSE_KEY
+      valueFrom:
+        secretKeyRef:
+          name: nr-otel-internal-telemetry-secret
+          key: NEW_RELIC_LICENSE_KEY
+    # K8s resource attributes for internal telemetry
+    - name: CLUSTER_NAME
+      valueFrom:
+        secretKeyRef:
+          name: nr-otel-internal-telemetry-secret
+          key: CLUSTER_NAME
+    - name: NAMESPACE
+      valueFrom:
+        fieldRef:
+          fieldPath: metadata.namespace
+    # POD_NAME is already defined by the chart for daemonset
+    - name: CONTAINER_NAME
+      value: "otel-collector-daemonset"
+
+# Optional: Customize internal telemetry levels
+# Uncomment and add these to extraEnvs in both deployment and daemonset sections above:
+#
+# - name: INTERNAL_TELEMETRY_METRICS_LEVEL
+#   value: "detailed"  # Options: detailed, normal, basic, none
+# - name: INTERNAL_TELEMETRY_LOG_LEVEL
+#   value: "INFO"  # Options: DEBUG, INFO, WARN, ERROR
+# - name: INTERNAL_TELEMETRY_TRACE_LEVEL
+#   value: "none"  # Options: none, basic (experimental)
+# - name: INTERNAL_TELEMETRY_TRACE_SAMPLE_RATIO
+#   value: "0.01"  # 1% sampling rate