(check apply)
Problem
I have an IPv6-only network, and my Elasticsearch service is behind a reverse proxy, where I have no direct access. The hostname for Elasticsearch is configured as a CNAME pointing to a dual-stack setup with both A (IPv4) and AAAA (IPv6) records. Fluentd is running in a container (fluentd-1.18-debian), which prefers IPv4 by default, even though I’ve set gai.conf to prefer IPv6. Despite this, Fluentd fails to connect to Elasticsearch and throws a connection_write error. However, if I configure extra_hosts in my container with the hostname and its corresponding IPv6 address, Fluentd is able to connect and write to Elasticsearch.
Steps to replicate
Provide example config and message
gai.conf
# Prefer IPv6 over IPv4
precedence ::ffff:0:0/96 10
precedence ::/0
99-outputs.conf
cat config.d/99-outputs.conf
<match **>
@type copy
<store>
@type elasticsearch_data_stream
verify_es_version_at_startup false
default_elasticsearch_version 8
@log_level debug
log_es_400_reason true
data_stream_name my-data-stream
data_stream_ilm_name some-lifecycle-policy
include_tag_key true
reload_connections false
reload_on_failure false
request_timeout 20s
hosts "https://$USER:$REDACTED@$URL:$PORT"
suppress_type_name true
<buffer>
@type memory
flush_thread_count 8
flush_interval 10s
flush_mode interval
overflow_action drop_oldest_chunk
retry_timeout 15m
disable_chunk_backup true
</buffer>
# Set content_type to application/json for ES 8.x
content_type application/json
</store>
</match>
<label @FLUENT_LOG>
<match fluent.*>
@type stdout
</match>
</label>
Logs without extra_hosts for container
May 13 10:46:21 $HOSTNAME fluentd[1092482]: The client is unable to verify that the server is Elasticsearch. Some functionality may not be compatible if the server is running an unsupported product.
May 13 10:46:21 $HOSTNAME fluentd[1092482]: 2025-05-13 10:46:21 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Failed to create data stream: <my-data-stream> connect_write timeout reached"
May 13 10:46:21 $HOSTNAME podman[1092591]: 2025-05-13 10:46:21.367187306 +0000 UTC m=+0.033227174 container died 526b58d90c5acb1a287a8e4c53084094d2bf398bed8b38e3b099881e3fdd9972 (image=docker/fluentd:v1.18-debian, name=fluentd, maintainer=Fluentd developers <fluentd@googlegroups.com>, Description=Fluentd docker image, PODMAN_SYSTEMD_UNIT=fluentd.service, Vendor=Fluent Organization, Version=1.18.0)
May 13 10:46:21 $HOSTNAME podman[1092591]: 2025-05-13 10:46:21.403289505 +0000 UTC m=+0.069329373 container remove 526b58d90c5acb1a287a8e4c53084094d2bf398bed8b38e3b099881e3fdd9972 (image=docker/fluentd:v1.18-5, name=fluentd, PODMAN_SYSTEMD_UNIT=fluentd.service, Vendor=Fluent Organization, Version=1.18.0, maintainer=Fluentd developers <fluentd@googlegroups.com>, Description=Fluentd docker image)
<snip>
With extra_hosts for container it just works without connect_timeout reached and I can see my logs in Kibana.
(off-topic)
I also tested fluentbit where I saw in the debug logs that the output would connect to the ipv4 address from DNS. But in the Fluentbit ES output i could set net.dns.prefer_ipv6: true which worked to get logs into Kibana. Maybe it would also an idea for fluentd when you have ipv6 only and have to use hostnames with A and AAAA records.
Expected Behavior or What you need to ask
I expect Fluentd to connect to Elasticsearch using IPv6 automatically, but it fails due to the default preference for IPv4. I need help troubleshooting this issue with Fluentd's container network settings or ensuring that it respects the IPv6 preference.
Using Fluentd and ES plugin versions
- OS version: Debian-based (Fluentd container: fluentd-1.18-debian)
- Bare Metal or within Docker or Kubernetes or others? Docker container
- Fluentd v1.18
- paste result of
fluentd --version or td-agent --version
$ fluentd --version fluentd 1.18.0
- ES plugin 5.4.3
- paste boot log of fluentd or td-agent
- paste result of
fluent-gem list, td-agent-gem list or your Gemfile.lock
-
/$ fluent-gem list
*** LOCAL GEMS ***
abbrev (default: 0.1.1)
async (1.32.1)
async-http (0.64.2)
async-io (1.43.2)
async-pool (0.10.3)
base64 (0.2.0, default: 0.1.1)
benchmark (default: 0.2.1)
bigdecimal (default: 3.1.3)
bundler (default: 2.4.19)
cgi (default: 0.3.7)
concurrent-ruby (1.3.5)
console (1.30.2)
cool.io (1.9.0)
csv (3.3.4, default: 3.2.6)
date (default: 3.3.3)
debug (1.7.1)
delegate (default: 0.3.0)
did_you_mean (default: 1.6.3)
dig_rb (1.0.1)
digest (default: 3.1.1)
drb (2.2.1, default: 2.1.1)
elastic-transport (8.4.0)
elasticsearch (8.14.0)
elasticsearch-api (8.14.0)
english (default: 0.7.2)
erb (default: 4.0.2)
error_highlight (default: 0.5.1)
etc (default: 1.4.2)
excon (1.2.5)
faraday (2.13.1)
faraday-excon (2.3.0)
faraday-net_http (3.4.0)
fcntl (default: 1.0.2)
ffi (1.17.2 x86_64-linux-gnu)
fiber-annotation (0.2.0)
fiber-local (1.1.0)
fiber-storage (1.0.0)
fiddle (default: 1.1.1)
fileutils (default: 1.7.0)
find (default: 0.1.1)
fluent-config-regexp-type (1.0.0)
fluent-plugin-elasticsearch (5.4.3)
fluent-plugin-geoip (1.3.2)
fluent-plugin-http-healthcheck (0.1.0)
fluent-plugin-prometheus (2.0.3)
fluent-plugin-record-modifier (2.1.1)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-systemd (1.0.5)
fluentd (1.18.0)
forwardable (default: 1.3.3)
geoip-c (0.9.1)
geoip2_c (0.3.4)
getoptlong (default: 0.2.0)
http_parser.rb (0.8.0)
io-console (default: 0.6.0)
io-nonblock (default: 0.2.0)
io-wait (default: 0.3.0)
ipaddr (default: 1.2.5)
irb (default: 1.6.2)
json (2.11.3, 2.7.4, default: 2.6.3)
logger (1.7.0, default: 1.5.3)
matrix (0.4.2)
minitest (5.25.1)
msgpack (1.8.0)
multi_json (1.15.0)
mutex_m (default: 0.1.2)
net-ftp (0.2.1)
net-http (0.6.0, default: 0.4.1)
net-imap (0.3.8)
net-pop (0.1.2)
net-protocol (default: 0.2.1)
net-smtp (0.3.4)
nio4r (2.7.4)
nkf (default: 0.1.2)
observer (default: 0.1.1)
oj (3.16.5)
open-uri (default: 0.3.0)
open3 (default: 0.1.2)
openssl (default: 3.1.0)
optparse (default: 0.3.1)
ostruct (default: 0.5.5)
pathname (default: 0.2.1)
power_assert (2.0.3)
pp (default: 0.4.0)
prettyprint (default: 0.1.1)
prime (0.1.2)
prometheus-client (4.2.4)
protocol-hpack (1.5.1)
protocol-http (0.26.8)
protocol-http1 (0.19.1)
protocol-http2 (0.16.0)
pstore (default: 0.1.2)
psych (default: 5.0.1)
racc (default: 1.6.2)
rake (13.0.6)
rbs (2.8.2)
rdoc (default: 6.5.1.1)
readline (default: 0.0.3)
readline-ext (default: 0.1.5)
reline (default: 0.3.2)
resolv (default: 0.2.2)
resolv-replace (default: 0.1.1)
rexml (3.3.9)
rinda (default: 0.1.1)
rss (0.3.1)
ruby2_keywords (default: 0.0.5)
securerandom (default: 0.2.2)
serverengine (2.4.0)
set (default: 1.0.3)
shellwords (default: 0.1.0)
sigdump (0.2.5)
singleton (default: 0.1.1)
stringio (default: 3.0.4)
strptime (0.2.5)
strscan (default: 3.0.7)
syntax_suggest (default: 1.1.0)
syslog (default: 0.1.1)
systemd-journal (1.4.2)
tempfile (default: 0.1.3)
test-unit (3.5.7)
time (default: 0.2.2)
timeout (default: 0.3.1)
timers (4.4.0)
tmpdir (default: 0.1.3)
traces (0.15.2)
tsort (default: 0.1.1)
typeprof (0.21.3)
tzinfo (2.0.6)
tzinfo-data (1.2025.2)
un (default: 0.2.1)
uri (1.0.3, default: 0.12.4)
weakref (default: 0.1.2)
webrick (1.9.1)
yajl-ruby (1.4.3)
yaml (default: 0.2.1)
zlib (default: 3.0.0)
* ES version (optional)
* ES template(s) (optional)
(check apply)
Problem
I have an IPv6-only network, and my Elasticsearch service is behind a reverse proxy, where I have no direct access. The hostname for Elasticsearch is configured as a CNAME pointing to a dual-stack setup with both A (IPv4) and AAAA (IPv6) records. Fluentd is running in a container (fluentd-1.18-debian), which prefers IPv4 by default, even though I’ve set gai.conf to prefer IPv6. Despite this, Fluentd fails to connect to Elasticsearch and throws a connection_write error. However, if I configure extra_hosts in my container with the hostname and its corresponding IPv6 address, Fluentd is able to connect and write to Elasticsearch.
Steps to replicate
Provide example config and message
gai.conf
99-outputs.conf
Logs without extra_hosts for container
With extra_hosts for container it just works without connect_timeout reached and I can see my logs in Kibana.
(off-topic)
I also tested fluentbit where I saw in the debug logs that the output would connect to the ipv4 address from DNS. But in the Fluentbit ES output i could set
net.dns.prefer_ipv6: truewhich worked to get logs into Kibana. Maybe it would also an idea for fluentd when you have ipv6 only and have to use hostnames with A and AAAA records.Expected Behavior or What you need to ask
I expect Fluentd to connect to Elasticsearch using IPv6 automatically, but it fails due to the default preference for IPv4. I need help troubleshooting this issue with Fluentd's container network settings or ensuring that it respects the IPv6 preference.
Using Fluentd and ES plugin versions
fluentd --versionortd-agent --version$ fluentd --version fluentd 1.18.0fluent-gem list,td-agent-gem listor your Gemfile.lock