Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[newrelic-logging] enable sendMetrics and fluent bit metrics config by default #1547

Closed
wants to merge 21 commits into from
Closed

[newrelic-logging] enable sendMetrics and fluent bit metrics config by default #1547

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
c0b69fa
fix: enable metrics and fb metrics config by default
rohit-bandlamudi-nr Jan 30, 2025
667213d
feat: minor release with configurable metrics and entity synthesis
rohit-bandlamudi-nr Feb 11, 2025
8528a70
fix: update readme file
rohit-bandlamudi-nr Feb 11, 2025
31df60f
fix: removed comment on sendMetrics
rohit-bandlamudi-nr Feb 11, 2025
c06e96a
[nri-bundle] Add eAPM to NRI Bundle (#1579)
kpattaswamy Jan 30, 2025
e10f75b
[agent-control] build: use specific system identity registration imag…
DavSanchez Jan 31, 2025
121917e
[nr-ebpf-agent] Release (#1593)
Philip-R-Beckwith Jan 31, 2025
fde5502
[git workflows] helm version bump for lint (#1594)
Philip-R-Beckwith Jan 31, 2025
81cf0ca
[NR eBPF Agent] Enable lookup for existing TLS certs. (#1596)
Philip-R-Beckwith Jan 31, 2025
886bbfe
[NRI-Bundle] Bumping eBPF Agent. (#1597)
Philip-R-Beckwith Feb 3, 2025
56cd1e0
bump release-0-31-0 (#1598)
vjripoll Feb 4, 2025
2fedcee
[newrelic-logging] chore(deps): Update common library to 1.3.1 (#1582)
dbudziwojskiNR Feb 4, 2025
0cba9f2
NR-361437-fix-jd-parse-error (#1586)
vjripoll Feb 5, 2025
c9853a2
NR-367081:Bump nr-ebpf-agent/client image versions. (#1601)
satya-nr Feb 6, 2025
2acbebf
[nr-ebpf-agent] Stop using host network for Otel Collector (#1603)
Philip-R-Beckwith Feb 6, 2025
c1053cc
fix concatenation error message during l2 identity (#1611)
vjripoll Feb 11, 2025
0b419ac
Update Bundle dependencies (#1595)
renovate[bot] Feb 11, 2025
46b191a
test: added tests to validate labels for fluentbit entity synthesis
rohit-bandlamudi-nr Jan 7, 2025
2727056
feat: enable metrics option and fb metrics config by default
rohit-bandlamudi-nr Jan 30, 2025
7cdfc3c
fix: fix values file
rohit-bandlamudi-nr Jan 30, 2025
fc83a2f
fix: unquote sendMetrics option in values file
rohit-bandlamudi-nr Jan 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion charts/newrelic-logging/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
apiVersion: v2
description: A Helm chart to deploy New Relic Kubernetes Logging as a DaemonSet, supporting both Linux and Windows nodes and containers
name: newrelic-logging
version: 1.23.5
version: 1.23.6-beta
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we using beta here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want to release this as a beta version and test the entity synthesis and forwarding of metrics (if possible, we want to test this with fleet control and agent control as well)

appVersion: 2.1.0
home: https://github.com/newrelic/kubernetes-logging
icon: https://newrelic.com/assets/newrelic/source/NewRelic-logo-square.svg
Expand Down
1 change: 0 additions & 1 deletion charts/newrelic-logging/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,6 @@ See [values.yaml](values.yaml) for the default values
| `fluentBit.config.extraOutputs` | Contains extra fluent-bit.conf Outputs config | |
| `fluentBit.config.parsers` | Contains parsers.conf Parsers config | |
| `fluentBit.retryLimit` | Amount of times to retry sending a given batch of logs to New Relic. This prevents data loss if there is a temporary network disruption, if a request to the Logs API is lost or when receiving a recoverable HTTP response. Set it to "False" for unlimited retries. | 5 |
| `fluentBit.sendMetrics` | Enable the collection of Fluent Bit internal metrics in Prometheus format as well as newrelic-fluent-bit-output internal plugin metrics. See [this documentation page](https://docs.newrelic.com/docs/logs/forward-logs/kubernetes-plugin-log-forwarding/#troubleshoot-installation) for more details. | `false` |
| `dnsConfig` | [DNS configuration](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-dns-config) that will be added to the pods. Can be configured also with `global.dnsConfig`. | `{}` |
| `fluentBit.criEnabled` | We assume that `kubelet`directly communicates with the container engine using the [CRI](https://kubernetes.io/docs/concepts/overview/components/#container-runtime) specification. Set this to `false` if your K8s installation uses [dockershim](https://kubernetes.io/docs/tasks/administer-cluster/migrating-from-dockershim/) instead, in order to get the logs properly parsed. | `true` |

Expand Down
33 changes: 33 additions & 0 deletions charts/newrelic-logging/templates/_helpers.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,39 @@ https://log-api.newrelic.com/log/v1
{{- end -}}
{{- end -}}

{{/*
Returns fluentbit config to collect and forward its metrics to New Relic
*/}}
{{- define "newrelic-logging.fluentBit.monitoring.config" -}}
[INPUT]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why don't we just keep this in values.yaml for better visibility.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are enabling sending of metrics by default. We don't want this to be configurable by the user using the values file. Basically, disabling metrics should not be an option

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think visibility is important. If a customer really wants to remove it, can't they do it anyway?

name prometheus_scrape
Alias fb-metrics-collector
host 127.0.0.1
port 2020
tag fb_metrics
metrics_path /api/v2/metrics/prometheus
scrape_interval 10s

[OUTPUT]
Name prometheus_remote_write
Match fb_metrics
Alias fb-metrics-forwarder
Host ${METRICS_HOST}
Port 443
Uri /prometheus/v1/write?prometheus_server=${CLUSTER_NAME}
Header Authorization Bearer ${LICENSE_KEY}
Tls On
Tls.verify Off
add_label app fluent-bit
add_label source kubernetes
add_label pod_name ${HOSTNAME}
add_label node_name ${NODE_NAME}
{{- printf "add_label cluster_name %s" (include "newrelic-logging.cluster" .) | nindent 4 -}}
{{- printf "add_label namespace %s" .Release.Namespace | nindent 4 -}}
{{- printf "add_label daemonset_name %s" (include "newrelic-logging.fullname" .) | nindent 4 -}}
{{- end -}}


{{/*
Returns metricsHost
*/}}
Expand Down
4 changes: 1 addition & 3 deletions charts/newrelic-logging/templates/configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,7 @@ data:
{{- if .Values.fluentBit.config.extraOutputs }}
{{- .Values.fluentBit.config.extraOutputs | nindent 4}}
{{- end }}
{{- if and (.Values.fluentBit.sendMetrics) (.Values.fluentBit.config.metricInstrumentation) }}
{{- .Values.fluentBit.config.metricInstrumentation | nindent 4}}
{{- end }}
{{- (include "newrelic-logging.fluentBit.monitoring.config" .) | nindent 4}}
parsers.conf: |
{{- if .Values.fluentBit.config.parsers }}
{{- .Values.fluentBit.config.parsers | nindent 4}}
Expand Down
2 changes: 0 additions & 2 deletions charts/newrelic-logging/templates/daemonset-windows.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,8 +111,6 @@ spec:
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: SEND_OUTPUT_PLUGIN_METRICS
value: {{ $.Values.fluentBit.sendMetrics | default "false" | quote }}
- name: METRICS_HOST
value: {{ include "newrelic-logging.metricsHost" $ | quote }}
{{- include "newrelic-logging.extraEnv" $ | nindent 12 }}
Expand Down
2 changes: 0 additions & 2 deletions charts/newrelic-logging/templates/daemonset.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,8 +125,6 @@ spec:
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: SEND_OUTPUT_PLUGIN_METRICS
value: {{ $.Values.fluentBit.sendMetrics | default "false" | quote }}
- name: METRICS_HOST
value: {{ include "newrelic-logging.metricsHost" . | quote }}
{{- include "newrelic-logging.extraEnv" . | nindent 12 }}
Expand Down
13 changes: 0 additions & 13 deletions charts/newrelic-logging/tests/fluentbit_sendmetrics_test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ tests:
set:
licenseKey: nr_license_key
enableWindows: true
fluentBit.sendMetrics: true
asserts:
# Linux
- contains:
Expand All @@ -31,12 +30,6 @@ tests:
fieldRef:
fieldPath: metadata.name
template: templates/daemonset.yaml
- contains:
path: spec.template.spec.containers[0].env
content:
name: SEND_OUTPUT_PLUGIN_METRICS
value: "true"
template: templates/daemonset.yaml
- contains:
path: spec.template.spec.containers[0].env
content:
Expand All @@ -60,12 +53,6 @@ tests:
fieldRef:
fieldPath: metadata.name
template: templates/daemonset-windows.yaml
- contains:
path: spec.template.spec.containers[0].env
content:
name: SEND_OUTPUT_PLUGIN_METRICS
value: "true"
template: templates/daemonset-windows.yaml
- contains:
path: spec.template.spec.containers[0].env
content:
Expand Down
32 changes: 3 additions & 29 deletions charts/newrelic-logging/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,6 @@ fluentBit:
k8sBufferSize: "32k"
k8sLoggingExclude: "false"
retryLimit: 5
sendMetrics: false
extraEnv: []
# extraEnv:
# - name: HTTPS_PROXY
Expand Down Expand Up @@ -166,6 +165,8 @@ fluentBit:
Allowlist_key message
Allowlist_key log

# sendMetrics is set to false by default, this has to be defaulted to true in the future, in the newrelic-fluent-bit-output plugin

outputs: |
[OUTPUT]
Name newrelic
Expand All @@ -174,7 +175,7 @@ fluentBit:
licenseKey ${LICENSE_KEY}
endpoint ${ENDPOINT}
lowDataMode ${LOW_DATA_MODE}
sendMetrics ${SEND_OUTPUT_PLUGIN_METRICS}
sendMetrics true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to enable this for fluent bit? It seems to me that its specific for sending output plugin related metrics. : https://github.com/newrelic/newrelic-fluent-bit-output/pull/142/files @jsubirat

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, sendMetrics are defaulted to false in the newrelic-fluentbit-output repo. We can change that to true. Until then, we have set this to true for testing

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to enable this for fluent bit? It seems to me that its specific for sending output plugin related metrics. : https://github.com/newrelic/newrelic-fluent-bit-output/pull/142/files @jsubirat

Yeah, note that , these metrics are not required for creation of fluentbit entity. But we will be missing on some key metrics forwarded by the newrelic fluentbit output plugin that are used to power few charts on the fluentbit entity view

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed offline with @nr-rkallempudi and Joshua and we aligned on not changing anything related fluent bit output plugin metrics. The metrics added by fluent bit output plugin are not being used for fluent bit dashboard AFAIK.

Retry_Limit ${RETRY_LIMIT}

# extraOutputs: |
Expand All @@ -189,33 +190,6 @@ fluentBit:
# Time_Key time
# Time_Format %Y-%m-%dT%H:%M:%S.%L
# Time_Keep On
metricInstrumentation: |
[INPUT]
name prometheus_scrape
Alias fb-metrics-collector
host 127.0.0.1
port 2020
tag fb_metrics
metrics_path /api/v2/metrics/prometheus
scrape_interval 10s

[OUTPUT]
Name prometheus_remote_write
Match fb_metrics
Alias fb-metrics-forwarder
Host ${METRICS_HOST}
Port 443
Uri /prometheus/v1/write?prometheus_server=${CLUSTER_NAME}
Header Authorization Bearer ${LICENSE_KEY}
Tls On
# Windows pods using prometheus_remote_write currently have issues if TLS verify is On
Tls.verify Off
# User-defined labels
add_label app fluent-bit
add_label cluster_name "${CLUSTER_NAME}"
add_label pod_name ${HOSTNAME}
add_label node_name ${NODE_NAME}
add_label source kubernetes

image:
repository: newrelic/newrelic-fluentbit-output
Expand Down
Loading