Skip to content

Too many informations related to opensearch are printed in the logs #162

@gibz2

Description

@gibz2

(check apply)

  • read the contribution guideline
  • (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.

Steps to replicate

Just use the latest plugin 1.1.5 with opensearch 3.2.0

Expected Behavior or What you need to ask

Hello,

I'm using the plugin since 1 year and since a few weeks it seems that it is displaying a lot of opensearch informations insides our fluentd logs.

Here is a part of my dockerfile, creating a new image from a fluentd basis for my deployment

FROM ${REGISTRY}fluent/fluentd:v1.19.0-1.0
....
gem install --verbose --version 1.1.5 fluent-plugin-opensearch --no-document

And here is the part of the configuration dealing with opensearch :

@type opensearch
hosts xxxxxxx.svc.cluster.local
port 9200
user "xxxxxx"
password "xxxxxxxx"
scheme https
ssl_verify false
reload_on_failure true
reload_connections true
reload_after 100
# Don't check opensearch version at startup
# It means fluentd can start even if opensearch is down
verify_os_version_at_startup false
logstash_format true
logstash_dateformat %Y.%m

template_name toto-prod
template_file /index_templates/generated/toto_default_template.json
template_overwrite true

utc_index true
include_timestamp true
include_tag_key true
use_legacy_template false
suppress_type_name true

request_timeout 2147483648
bulk_message_request_threshold 8m

<buffer>
  @type file
  total_limit_size 128MB
  flush_interval 60s
</buffer>

Finally when I run the container, a lot of log lines like this are constantly displayed :

{id: "zMy-zil4adDReLs35z", name: "toto-open-search-cluster-data-2", version: "3.2.0", host: "192.168.1.34", port: "9200", roles: ["data", "ingest"], attributes: {"shard_indexing_pressure_enabled" => "true"}}

If I change the fluentd loglevel, there is no impact, this messages are always spawning frequently on container logs.
It seems to be related to a 'print' on stdout related to the result of an opensearch request to '/_nodes'.
It is displayed very frequently (maybe each time I get 100 msg in fluentd ?)

Do you have any idea about this new behaviour and can I disable it ?
Somewhere else in the plugin, another 'opensearch api call' is done, printing the password we use for connection ! (like the opensearch client object used by fluentd config)
Which (dependency?) source code is raising this informations among the entire logs produced by fluentd ?

I expect to see nothing about my opensearch cluster informations in fluentd logs

Using Fluentd and OpenSearch plugin versions

  • OS version
    openshift 4

  • Bare Metal or within Docker or Kubernetes or others?
    docker

  • Fluentd v1.0 or later

    • paste result of fluentd --version or td-agent --version
      fluentd:v1.19.0-1.0 in container
  • OpenSearch plugin version

    • paste boot log of fluentd or td-agent
Generate index patterns
Generating and pushing index deletion policies
Launch Fluentd
2025-09-23 09:55:38 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2025-09-23 09:55:38 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2025-09-23 09:55:38 +0000 [info]: inaccessible include directory was specified path="/etc/fluent/conf.d"
2025-09-23 09:55:38 +0000 [info]: gem 'fluent-plugin-http-healthcheck' version '0.1.0'
2025-09-23 09:55:38 +0000 [info]: gem 'fluent-plugin-kafka' version '0.19.5'
2025-09-23 09:55:38 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.1.5'
2025-09-23 09:55:38 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.2.1'
2025-09-23 09:55:38 +0000 [info]: gem 'fluent-plugin-remote_syslog' version '1.1.0'
2025-09-23 09:55:38 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2025-09-23 09:55:38 +0000 [info]: gem 'fluentd' version '1.19.0'
  • paste result of fluent-gem list, td-agent-gem list or your Gemfile.lock
  • OpenSearch version (optional)
    3.2.0 container
  • OpenSearch template(s) (optional)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions