missing net_http_persistent faraday extension?

[sylvainOL]

Hello,
after failure to use 1.14 and 1.15, I decided to give a try to 1.16 but no luck :(

I've got an issue with fluent-plugin-enhance-k8s-metadataplugin:

2023-12-19 15:07:03 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-12-19 15:07:03 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2023-12-19 15:07:04 +0000 [info]: gem 'fluentd' version '1.16.3'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-aliyun-oss' version '0.0.1'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.14.3'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-datadog' version '0.14.2'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-datapoint' version '2.0.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.15'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.4.2'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-enhance-k8s-metadata' version '2.0.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-events' version '2.0.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-gcs' version '0.4.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-gelf-hs' version '1.0.8'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-geoip' version '1.3.2'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.20'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-kafka' version '0.19.2'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-kinesis' version '3.4.2'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-kube-events-timestamp' version '0.1.3'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-kubernetes-metadata-filter' version '2.5.3'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-kubernetes-sumologic' version '2.0.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-label-router' version '0.2.10'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-logzio' version '0.2.2'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-mattermost' version '0.2.2'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.1.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-mysqlslowquery' version '0.0.9'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-newrelic' version '1.2.2'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.1.4'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-parser-logfmt' version '0.0.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.1.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-prometheus-format' version '2.0.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-protobuf' version '2.0.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.2.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-redis' version '0.3.5'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-remote-syslog' version '1.1'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.3.3'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-sqs' version '3.0.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.8.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-syslog_rfc5424' version '0.9.0.rc.8'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-tag-normaliser' version '0.1.2'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-throttle' version '0.0.5'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-ua-parser' version '1.2.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-vmware-log-intelligence' version '2.0.8'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-vmware-loginsight' version '1.4.2'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.5.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluentd' version '1.16.2'
2023-12-19 15:07:04 +0000 [info]: [clusterflow:logging:fluent-cluster-flow:0] DeDot will recurse nested hashes and arrays
2023-12-19 15:07:05 +0000 [info]: [globalFilter:0] Kubernetes URL is not set - inspecting environment
2023-12-19 15:07:05 +0000 [info]: [globalFilter:0] cache_ttl: 7200, cache_size: 1000, cache_refresh: 3600, cache_refresh_variation: 900
2023-12-19 15:07:05 +0000 [info]: [globalFilter:0] Initializing Kubernetes API clients
2023-12-19 15:07:05 +0000 [info]: [globalFilter:0] create client with URL: https://10.43.0.1:443/api and apiVersion: v1
2023-12-19 15:07:05 +0000 [error]: [globalFilter:0] Error creating client (:net_http_persistent is not registered on Faraday::Adapter), retrying in 2 second(s)...
2023-12-19 15:07:07 +0000 [info]: [globalFilter:0] create client with URL: https://10.43.0.1:443/api and apiVersion: v1
2023-12-19 15:07:07 +0000 [error]: [globalFilter:0] Error creating client (:net_http_persistent is not registered on Faraday::Adapter), retrying in 4 second(s)...
2023-12-19 15:07:11 +0000 [info]: [globalFilter:0] create client with URL: https://10.43.0.1:443/api and apiVersion: v1
2023-12-19 15:07:11 +0000 [error]: [globalFilter:0] Error creating client (:net_http_persistent is not registered on Faraday::Adapter), retrying in 8 second(s)...
2023-12-19 15:07:19 +0000 [info]: [globalFilter:0] create client with URL: https://10.43.0.1:443/api and apiVersion: v1
2023-12-19 15:07:19 +0000 [error]: [globalFilter:0] Error creating client (:net_http_persistent is not registered on Faraday::Adapter), retrying in 16 second(s)...
2023-12-19 15:07:35 +0000 [info]: [globalFilter:0] create client with URL: https://10.43.0.1:443/api and apiVersion: v1
2023-12-19 15:07:35 +0000 [error]: [globalFilter:0] Error creating client (:net_http_persistent is not registered on Faraday::Adapter), retrying in 32 second(s)...
2023-12-19 15:08:07 +0000 [info]: [globalFilter:0] create client with URL: https://10.43.0.1:443/api and apiVersion: v1
2023-12-19 15:08:07 +0000 [error]: [globalFilter:0] Error creating client (:net_http_persistent is not registered on Faraday::Adapter), retrying in 64 second(s)...
2023-12-19 15:09:11 +0000 [info]: [globalFilter:0] create client with URL: https://10.43.0.1:443/api and apiVersion: v1
2023-12-19 15:09:11 +0000 [error]: [globalFilter:0] Error creating client (:net_http_persistent is not registered on Faraday::Adapter), retrying in 128 second(s)...
2023-12-19 15:11:19 +0000 [info]: [globalFilter:0] create client with URL: https://10.43.0.1:443/api and apiVersion: v1
/usr/local/bundle/gems/faraday-2.7.12/lib/faraday/middleware_registry.rb:57:in `lookup_middleware': :net_http_persistent is not registered on Faraday::Adapter (Faraday::Error)
	from /usr/local/bundle/gems/faraday-2.7.12/lib/faraday/rack_builder.rb:113:in `adapter'
	from /usr/local/lib/ruby/3.2.0/forwardable.rb:240:in `adapter'
	from /usr/local/bundle/gems/fluent-plugin-enhance-k8s-metadata-2.0.0/lib/sumologic/kubernetes/connector.rb:55:in `create_client'
	from /usr/local/bundle/gems/fluent-plugin-enhance-k8s-metadata-2.0.0/lib/sumologic/kubernetes/connector.rb:16:in `block in core_clients'
	from /usr/local/bundle/gems/fluent-plugin-enhance-k8s-metadata-2.0.0/lib/sumologic/kubernetes/connector.rb:15:in `map'
	from /usr/local/bundle/gems/fluent-plugin-enhance-k8s-metadata-2.0.0/lib/sumologic/kubernetes/connector.rb:15:in `core_clients'
	from /usr/local/bundle/gems/fluent-plugin-enhance-k8s-metadata-2.0.0/lib/sumologic/kubernetes/connector.rb:11:in `connect_kubernetes'
	from /usr/local/bundle/gems/fluent-plugin-enhance-k8s-metadata-2.0.0/lib/fluent/plugin/filter_enhance_k8s_metadata.rb:59:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/plugin.rb:187:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:154:in `add_filter'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:72:in `block in configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `each'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:149:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:105:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:80:in `run_configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/supervisor.rb:571:in `run_supervisor'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from /usr/local/bundle/gems/fluentd-1.16.3/bin/fluentd:15:in `<top (required)>'
	from /usr/local/bundle/bin/fluentd:25:in `load'
	from /usr/local/bundle/bin/fluentd:25:in `<main>'

I tried to see the issue and it seems the gem faraday-net_http_persistent is missing.

*** LOCAL GEMS ***

abbrev (default: 0.1.1)
activemodel (7.1.1)
activesupport (7.1.1)
addressable (2.8.5)
aes_key_wrap (1.1.0)
aliyun-sdk (0.7.0)
async (1.30.3)
async-http (0.60.2)
async-io (1.37.0)
async-pool (0.4.0)
attr_required (1.0.1)
aws-eventstream (1.2.0)
aws-partitions (1.851.0, 1.842.0)
aws-sdk-cloudwatchlogs (1.71.0)
aws-sdk-core (3.186.0, 3.185.1)
aws-sdk-firehose (1.58.0)
aws-sdk-kinesis (1.52.0)
aws-sdk-kms (1.72.0)
aws-sdk-s3 (1.136.0)
aws-sdk-sqs (1.67.0, 1.64.0)
aws-sigv4 (1.6.1)
base64 (0.2.0, default: 0.1.1)
benchmark (default: 0.2.1)
bigdecimal (3.1.4, default: 3.1.3, 1.4.4)
bindata (2.4.15)
bundler (2.4.22, default: 2.4.10)
cgi (default: 0.3.6)
concurrent-ruby (1.2.2, 1.1.10)
connection_pool (2.4.1)
console (1.23.2)
cool.io (1.8.0)
csv (default: 3.2.6)
date (default: 3.3.3)
debug (1.7.1)
declarative (0.0.20)
delegate (default: 0.3.0)
did_you_mean (default: 1.6.3)
dig_rb (1.0.1)
digest (default: 3.1.1)
digest-crc (0.6.5)
domain_name (0.6.20231109, 0.5.20190701)
drb (default: 2.1.1)
elastic-transport (8.3.0)
elasticsearch (8.11.0)
elasticsearch-api (8.11.0)
english (default: 0.7.2)
erb (default: 4.0.2)
error_highlight (default: 0.5.1)
etc (default: 1.4.2)
excon (0.104.0)
faraday (2.7.12, 1.10.3)
faraday-em_http (1.0.0)
faraday-em_synchrony (1.0.0)
faraday-excon (2.1.0, 1.1.0)
faraday-follow_redirects (0.3.0)
faraday-httpclient (1.0.1)
faraday-multipart (1.0.4)
faraday-net_http (3.0.2, 1.0.1)
faraday-net_http_persistent (1.2.0)
faraday-patron (1.0.0)
faraday-rack (1.0.0)
faraday-retry (1.0.3)
faraday_middleware (1.2.0)
faraday_middleware-aws-sigv4 (1.0.1)
fcntl (default: 1.0.2)
ffi (1.16.3)
ffi-compiler (1.0.1)
fiber-annotation (0.2.0)
fiber-local (1.0.0)
fiddle (default: 1.1.1)
fileutils (default: 1.7.0)
find (default: 0.1.1)
fluent-config-regexp-type (1.0.0)
fluent-mixin-config-placeholders (0.4.0)
fluent-plugin-aliyun-oss (0.0.1)
fluent-plugin-cloudwatch-logs (0.14.3)
fluent-plugin-concat (2.5.0)
fluent-plugin-datadog (0.14.2)
fluent-plugin-datapoint (2.0.0)
fluent-plugin-dedot_filter (1.0.0)
fluent-plugin-detect-exceptions (0.0.15)
fluent-plugin-elasticsearch (5.4.2)
fluent-plugin-enhance-k8s-metadata (2.0.0)
fluent-plugin-events (2.0.0)
fluent-plugin-gcs (0.4.0)
fluent-plugin-gelf-hs (1.0.8)
fluent-plugin-geoip (1.3.2)
fluent-plugin-grafana-loki (1.2.20)
fluent-plugin-grok-parser (2.6.2)
fluent-plugin-kafka (0.19.2)
fluent-plugin-kinesis (3.4.2)
fluent-plugin-kube-events-timestamp (0.1.3)
fluent-plugin-kubernetes-metadata-filter (2.5.3)
fluent-plugin-kubernetes-sumologic (2.0.0)
fluent-plugin-label-router (0.2.10)
fluent-plugin-logzio (0.2.2)
fluent-plugin-mattermost (0.2.2)
fluent-plugin-multi-format-parser (1.1.0)
fluent-plugin-mysqlslowquery (0.0.9)
fluent-plugin-newrelic (1.2.2)
fluent-plugin-opensearch (1.1.4)
fluent-plugin-parser-logfmt (0.0.0)
fluent-plugin-prometheus (2.1.0)
fluent-plugin-prometheus-format (2.0.0)
fluent-plugin-protobuf (2.0.0)
fluent-plugin-record-modifier (2.2.0, 2.1.1)
fluent-plugin-redis (0.3.5)
fluent-plugin-remote-syslog (1.1)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-s3 (1.7.2)
fluent-plugin-splunk-hec (1.3.3)
fluent-plugin-sqs (3.0.0)
fluent-plugin-sumologic_output (1.8.0)
fluent-plugin-syslog_rfc5424 (0.9.0.rc.8)
fluent-plugin-systemd (1.0.5)
fluent-plugin-tag-normaliser (0.1.2)
fluent-plugin-throttle (0.0.5)
fluent-plugin-ua-parser (1.2.0)
fluent-plugin-vmware-log-intelligence (2.0.8)
fluent-plugin-vmware-loginsight (1.4.2)
fluent-plugin-webhdfs (1.5.0)
fluentd (1.16.3, 1.16.2)
forwardable (default: 1.3.3)
gelf (3.1.0)
geoip-c (0.9.1)
geoip2_c (0.3.4)
getoptlong (default: 0.2.0)
google-apis-core (0.11.2)
google-apis-iamcredentials_v1 (0.17.0)
google-apis-storage_v1 (0.29.0)
google-cloud-core (1.6.1)
google-cloud-env (2.0.1)
google-cloud-errors (1.3.1)
google-cloud-storage (1.45.0)
google-protobuf (3.24.4, 3.21.12 ruby x86_64-linux)
googleauth (1.9.0)
http (5.1.1, 4.4.1)
http-accept (1.7.0)
http-cookie (1.0.5)
http-form_data (2.3.0)
http-parser (1.2.3)
http_parser.rb (0.8.0)
httpclient (2.8.3)
i18n (1.14.1)
io-console (default: 0.6.0)
io-nonblock (default: 0.2.0)
io-wait (default: 0.3.0)
ipaddr (default: 1.2.5)
irb (default: 1.6.2)
jmespath (1.6.2)
json (2.6.3)
json-jwt (1.15.3)
jwt (2.7.1)
kubeclient (4.10.1, 4.9.3)
llhttp-ffi (0.4.0)
logfmt (0.0.10)
logger (default: 1.5.3)
lru_redux (1.1.0)
ltsv (0.1.2)
mail (2.8.1)
matrix (0.4.2)
mime-types (3.5.1)
mime-types-data (3.2023.1003)
mini_mime (1.1.5)
mini_portile2 (2.8.5)
minitest (5.20.0, 5.16.3)
msgpack (1.7.2)
multi_json (1.15.0)
multipart-post (2.3.0)
mutex_m (default: 0.1.2)
myslog (0.1.1)
net-ftp (0.2.0)
net-http (default: 0.3.2)
net-http-persistent (4.0.2)
net-imap (0.4.2, 0.3.4)
net-pop (0.1.2)
net-protocol (default: 0.2.1)
net-smtp (0.4.0, 0.3.3)
netrc (0.11.0)
nio4r (2.6.0)
nkf (default: 0.1.2)
nokogiri (1.15.4)
observer (default: 0.1.1)
oj (3.15.0)
open-uri (default: 0.3.0)
open3 (default: 0.1.2)
openid_connect (1.1.8)
opensearch-ruby (3.0.1)
openssl (default: 3.1.0)
optparse (default: 0.3.1)
os (1.1.4)
ostruct (default: 0.5.5)
pathname (default: 0.2.1)
power_assert (2.0.3)
pp (default: 0.4.0)
prettyprint (default: 0.1.1)
prime (0.1.2)
prometheus-client (4.2.2, 4.2.1)
protocol-hpack (1.4.2)
protocol-http (0.24.7)
protocol-http1 (0.15.1)
protocol-http2 (0.15.1)
pstore (default: 0.1.2)
psych (default: 5.0.1)
public_suffix (5.0.3)
racc (1.7.1, default: 1.6.2)
rack (3.0.8)
rack-oauth2 (1.21.3)
rake (13.0.6)
rbs (2.8.2)
rdoc (default: 6.5.0)
readline (default: 0.0.3)
readline-ext (default: 0.1.5)
recursive-open-struct (1.1.3)
redis (3.3.5)
reline (default: 0.3.2)
representable (3.2.0)
resolv (default: 0.2.2)
resolv-replace (default: 0.1.1)
rest-client (2.1.0)
retriable (3.1.2)
rexml (3.2.5)
rinda (default: 0.1.1)
rss (0.2.9)
ruby-kafka (1.5.0)
ruby2_keywords (default: 0.0.5)
securerandom (default: 0.2.2)
serverengine (2.3.2)
set (default: 1.0.3)
shellwords (default: 0.1.0)
sigdump (0.2.5)
signet (0.18.0)
singleton (default: 0.1.1)
snappy (0.3.0)
specific_install (0.3.8)
stringio (default: 3.0.4)
strptime (0.2.5)
strscan (default: 3.0.5)
swd (2.0.2)
syntax_suggest (default: 1.0.2)
syslog (default: 0.1.1)
syslog_protocol (0.9.2)
systemd-journal (1.4.2)
tempfile (default: 0.1.3)
test-unit (3.5.7)
time (default: 0.2.2)
timeout (0.4.0, default: 0.3.1)
timers (4.3.5)
tmpdir (default: 0.1.3)
traces (0.11.1)
trailblazer-option (0.1.2)
tsort (default: 0.1.1)
typeprof (0.21.3)
tzinfo (2.0.6)
tzinfo-data (1.2023.3)
uber (0.1.0)
un (default: 0.2.1)
unf (0.1.4)
unf_ext (0.0.8.2)
uri (0.12.2, default: 0.12.1)
user_agent_parser (2.16.0)
uuidtools (2.2.0)
validate_email (0.1.6)
validate_url (1.0.15)
weakref (default: 0.1.2)
webfinger (2.1.2)
webhdfs (0.10.2)
webrick (1.8.1)
yajl-ruby (1.4.3)
yaml (default: 0.2.1)
zlib (default: 3.0.0)

My configuration is "basic":

---
apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  name: logging
  namespace: logging
spec:
  controlNamespace: logging
  enableRecreateWorkloadOnImmutableFieldChange: true
  flowConfigCheckDisabled: true
  fluentd:
    annotations:
      proxy.istio.io/config: '{ "holdApplicationUntilProxyStarts": true }'
      sidecar.istio.io/proxyMemory: '2Gi'
      sidecar.istio.io/proxyMemoryLimit: '5Gi'
    bufferStorageVolume:
      pvc:
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 3Gi
          storageClassName: fast
          volumeMode: Filesystem
    configCheckAnnotations:
      sidecar.istio.io/inject: 'false'
    extraVolumes:
      - containerName: fluentd
        path: /fluentd/log
        volume:
          emptyDir:
            sizeLimit: 1Gi
        volumeName: log-out
    image:
      pullPolicy: Always
      # renovate: datasource=docker
      repository: ghcr.io/kube-logging/fluentd
      tag: v1.16-full-build.122
    logLevel: info
      enabled: false
  globalFilters:
    - enhanceK8s: {}

[pepov]

I tried adding that gem, but it still fails for me with the same message