Enable Fluentd Prometheus plugin

[kaiohenricunha]

Is your feature request related to a problem? Please describe.

It would be a nice feature to have the power to enable Prometheus metrics scraping just like we do with Fluentbit, and how it is done with Fluentd's chart.

Describe the solution you'd like

Being able to easily enable Prometheus metrics scraping.

Additional context

Right now, it is not a straightforward task to integrate fluentd with prometheus when deployed with the fluent-operator. I'm not sure it's even possible.

https://docs.fluentd.org/monitoring-fluentd/monitoring-prometheus

https://github.com/fluent/fluent-plugin-prometheus

[erhudy]

This would be great to have, even if it's implemented in a way that basically just takes the configurations from https://docs.fluentd.org/monitoring-fluentd/monitoring-prometheus and puts them behind a single flag in the CRD (like spec.enablePrometheusMetrics).

[andrecastro]

This is almost a "must to have" for us to use this operator. Can you prioritize this?

[kaiohenricunha]

Has anyone found a workaround for this? Fluentd monitoring is critical.

[erhudy]

I am going to look at it when I have time.

[erhudy]

I have published a draft PR at #813 (intentionally broken for the time being). This is lacking tests and some refinements but does work (see below screenshot).

Questions to be answered about the PR:

• Just a general check that the approach I'm taking is okay.
• I refactored a few things in the operator while following the setup instructions at https://docs.fluentd.org/monitoring-fluentd/monitoring-prometheus, most notably moving tag to a separate field on the relevant struct and getting rid of the special treatment of the string "tag" (because the examples contained that as a label and the previous code was stripping it)
• Ideally this would be released alongside updated fluentd images that install fluent-plugin-prometheus by default. I could not find a repo for those Dockerfiles, but I imagine @benjaminhuo knows.

[huzhekun]

are there any plans to also enable such a prometheus metric endpoint for the fluentbit component? (preferably with a service/podmonitor) I found this issue from a while ago asking for the same thing but I believe it was mistakenly closed because the referred config line is related to the prometheus node exporter plugin rather than prometheus metrics of the fluentbit pods themselves. currently because of a lack of fluentbit/fluentd monitoring we are forced to use the fluentbit helm chart directly which does have the feature.

[kaiohenricunha]

are there any plans to also enable such a prometheus metric endpoint for the fluentbit component? (preferably with a service/podmonitor) I found this issue from a while ago asking for the same thing but I believe it was mistakenly closed because the referred config line is related to the prometheus node exporter plugin rather than prometheus metrics of the fluentbit pods themselves. currently because of a lack of fluentbit/fluentd monitoring we are forced to use the fluentbit helm chart directly which does have the feature.

I guess you just have to add an annotation to fluentbit:

https://github.com/fluent/fluent-operator/blob/master/charts/fluent-operator/values.yaml#L82

[joshuabaird]

@erhudy Would love to see your PR completed and merged! I went ahead and opened #966 to add the fluent-plugin-prometheus plugin to the fluentd image.

[erhudy]

@joshuabaird I am no longer using the fluentd operator. If you would like to take what I have done and reopen a new PR with my changes as a basis, please feel free to take it.