Envoy Segmentation Fault Due to "Too Many Open Files" Error

[rishabh78]

If you are reporting any crash or any potential security issue, do not
open an issue in this repo. Please report the issue via emailing
envoy-security@googlegroups.com where the issue will be triaged appropriately.

Title: Envoy Segmentation Fault Due to "Too Many Open Files" Error

Description:

Envoy process crashed with a segmentation fault due to an "Too many open files" error when attempting to create a new connection, leading to a critical failure in the system.

[2025-04-09 15:54:22.466][1954179][debug][pool] [source/common/conn_pool/conn_pool_base.cc:328] trying to create new connection
[2025-04-09 15:54:22.466][1954179][debug][pool] [source/common/conn_pool/conn_pool_base.cc:181] creating a new connection (connecting=0)
[2025-04-09 15:54:22.466][1954179][error][envoy_bug] [source/common/network/socket_interface_impl.cc:89] envoy bug failure: false. Details: socket(2) failed, got error: Too many open files
[2025-04-09 15:54:22.466][1954179][error][envoy_bug] [./source/common/common/assert.h:38] stacktrace for envoy bug
[symbolize_elf.inc : 1072] RAW: /proc/self/task/1954172/maps: errno=24
[2025-04-09 15:54:22.466][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #0 UNKNOWN [0x5637657cff03]
[2025-04-09 15:54:22.466][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #1 UNKNOWN [0x5637653fc4a5]
[2025-04-09 15:54:22.466][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #2 UNKNOWN [0x5637653f6cc5]
[2025-04-09 15:54:22.466][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #3 UNKNOWN [0x5637653eb1ef]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #4 UNKNOWN [0x563764e83836]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #5 UNKNOWN [0x5637652832aa]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #6 UNKNOWN [0x563765282c40]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #7 UNKNOWN [0x56376525edc7]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #8 UNKNOWN [0x563765260163]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #9 UNKNOWN [0x563765275cf7]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #10 UNKNOWN [0x5637652791b9]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #11 UNKNOWN [0x56376526002d]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #12 UNKNOWN [0x563763acb9b4]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #13 UNKNOWN [0x563763ab68d4]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #14 UNKNOWN [0x563763ab5f33]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #15 UNKNOWN [0x563763aba8e2]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [source/common/network/connection_impl.cc:89] envoy bug failure: false. Details: Client socket failure
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:38] stacktrace for envoy bug
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #0 UNKNOWN [0x5637653f6e35]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #1 UNKNOWN [0x5637653f6cec]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #2 UNKNOWN [0x5637653eb1ef]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #3 UNKNOWN [0x563764e83836]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #4 UNKNOWN [0x5637652832aa]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #5 UNKNOWN [0x563765282c40]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #6 UNKNOWN [0x56376525edc7]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #7 UNKNOWN [0x563765260163]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #8 UNKNOWN [0x563765275cf7]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #9 UNKNOWN [0x5637652791b9]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #10 UNKNOWN [0x56376526002d]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #11 UNKNOWN [0x563763acb9b4]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #12 UNKNOWN [0x563763ab68d4]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #13 UNKNOWN [0x563763ab5f33]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #14 UNKNOWN [0x563763aba8e2]
[2025-04-09 15:54:22.467][1954179][error][envoy_bug] [./source/common/common/assert.h:45] #15 UNKNOWN [0x5637657004c6]
[2025-04-09 15:54:22.467][1954179][debug][connection] [source/common/network/connection_impl.cc:1017] [Tags: "ConnectionId":"1201"] connecting to 10.177.216.53:636
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:129] Caught Segmentation fault, suspect faulting address 0x0
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:113] Backtrace (use tools/stack_decode.py to get line numbers):
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:114] Envoy version: 2135e1a42f002a939d60581096291acb6abce695/1.33.2/Clean/RELEASE/BoringSSL
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:116] Address mapping: 56376338a000-563765f2e000 /usr/local/bin/envoy
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #0: [0x7f61d2eb3520]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #1: [0x563763db4f63]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #2: [0x5637653f7f59]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #3: [0x56376525f026]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #4: [0x563765260163]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #5: [0x563765275cf7]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #6: [0x5637652791b9]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #7: [0x56376526002d]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #8: [0x563763acb9b4]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #9: [0x563763ab68d4]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #10: [0x563763ab5f33]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #11: [0x563763aba8e2]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #12: [0x5637657004c6]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #13: [0x5637653f6107]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #14: [0x5637653ba34e]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #15: [0x5637653e5cd1]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #16: [0x5637653e9158]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #17: [0x5637653e961b]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #18: [0x563764a14123]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #19: [0x5637653e3945]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #20: [0x5637655701b7]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #21: [0x563765571444]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #22: [0x563764e86696]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #23: [0x563764e87be5]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #24: [0x563765901af0]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #25: [0x563765900431]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #26: [0x563764e7dc8d]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #27: [0x563765988952]
[2025-04-09 15:54:22.467][1954179][critical][backtrace] [./source/server/backtrace.h:123] #28: [0x7f61d2f05ac3]
Segmentation fault

Envoy Configurations

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 636  # LDAPS port for Setup B
    access_log:
    - name: envoy.access_loggers.file
      filter:
        extension_filter:
          name: "envoy.filters.accesslog.cel"
          typed_config:
            "@type": "type.googleapis.com/envoy.extensions.access_loggers.filters.cel.v3.ExpressionFilter"
            expression: "(request.size > 0)"
      typed_config:
        "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog"
        path: "/var/log/envoy/access.log"
        log_format:
          json_format:
            timestamp: "%START_TIME%"
            start_time_epoch_micro: "%START_TIME(%s.%6f)%"
            duration: "%DURATION%"
            event: "TCP Connection"
            client_ip: "%DOWNSTREAM_REMOTE_ADDRESS_WITHOUT_PORT%"
            downstream_local_address: "%DOWNSTREAM_LOCAL_ADDRESS%"
            upstream_host: "%UPSTREAM_HOST%"
            original_dst_port: "%DOWNSTREAM_LOCAL_PORT%"
            cluster: "%UPSTREAM_CLUSTER%"
            bytes_sent: "%BYTES_SENT%"
            bytes_received: "%BYTES_RECEIVED%"
            connection_id: "%CONNECTION_ID%"
            domain: "%REQUESTED_SERVER_NAME%"
    filter_chains:
    - filters:
      - name: envoy.filters.network.tcp_proxy
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy"
          stat_prefix: "tcp_proxy"
          cluster: "ldap_backend"
      transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext"
          common_tls_context:
            tls_certificates:
            - certificate_chain:
                filename: "server.crt"
              private_key:
                filename: "server.key"

  clusters:
  - name: ldap_backend
    connect_timeout: 5s
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: ldap_backend
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: some.com
                port_value: ####
  

Envoy verison : 2135e1a/1.33.2/Clean/RELEASE/BoringSSL

[phlax]

@rishabh78

If you are reporting any crash or any potential security issue, do not
open an issue in this repo.

in this case i think the issue is related to your system ulimit and not a sec issue - but please read and follow the instructions in future

you can check with ulimit -n and increase with eg ulimit -n 65536

[yanavlasov]

I think this is a problem in Envoy. This condition should not result in a hard crash, just failed downstream requests.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.