EPIPE strikes back

[afharo]

🐛 Bug report

Follow-up to #2605: it looks like we found a new use case where EPIPE can be triggered and make the application crash.

While this seems fixed when running on Kibana on Self-Hosted (or in the local development), ECH deployments crash when the other end closes the connection if the payload is too large.

To reproduce

Reach out to the @elastic/kibana-core team to share the steps

Expected behavior

EPIPE does not emit.

Node.js version

22.21.1

@elastic/elasticsearch version

@elastic/elasticsearch@9.1.1, @elastic/transport@9.0.1

Operating system

Unknown

Any other relevant environment information

No response

[afharo]

Further investigation may indicate that it is caused by the response from the proxy due to the length of the URL/headers being too large.

[JoshMock]

URL length is currently not restricted explicitly in the JS client or transport, so it is going to support whatever Undici or node:http will allow, which may be higher than what the proxy supports.

I'm not sure what error handling to add here. Since URL length enforcement depends heavily on the environment ES is running in, it makes sense to err on the side of being permissive in the client. If Kibana is aware of what kind of environment it's currently running in (Cloud vs. Serverless or self-hosted), it may make more sense to handle it downstream there. But I'm very open to discussion on that.

[afharo]

I agree that we shouldn't limit it in the client, and let the server throw the error.

The reason I shared this piece of info is in case it helps with identifying where it could be that we're not handling the closed connection EPIPE. Since this error happens as part of the header transmission, it may be that we're not correctly handling when the server drops the connection (or replies with a 413/414).

For additional context: Kibana still uses the HTTP transport (not undici yet).

[JoshMock]

Yep, I assumed HttpConnection was still in use; I'll be throwing Kibana Core a party on the day the UndiciConnection switch happens. 🖤 😆

Do we have any stack trace info, what exception was raised, etc? Or does it crash hard enough that we never get that feedback? HttpConnection sets a catch-all error handler that should be run for anything that happens once the request is in flight, so I'm curious if it's happening before or while the request object is created.

[afharo]

I'll be throwing Kibana Core a party on the day the UndiciConnection switch happens. 🖤 😆

I'd love to do that switch as well! I can't wait to see the benefit of using HTTP/2 connections. I plan to research the impact in our next OnWeek.

Do we have any stack trace info, what exception was raised, etc? Or does it crash hard enough that we never get that feedback? HttpConnection sets a catch-all error handler that should be run for anything that happens once the request is in flight, so I'm curious if it's happening before or while the request object is created.

The stack trace that we get is minimal (you know... somehow our transpiler messes it up very badly):

Error: write EPIPE
    at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:87:19)
    at WriteWrap.callbackTrampoline (node:internal/async_hooks:130:17)

Googling around, I found this Node.js issue that might be relevant: nodejs/node#54366

[afharo]

Interestingly, in isolation, I managed to force the client to receive ResponseError: 431 Request Header Fields Too Large from our Cloud Proxy. But the client doesn't crash.

Testing code

import apm from 'elastic-apm-node'; import { Client, ClusterConnectionPool, HttpConnection } from '@elastic/elasticsearch'; import { LONG_ID } from './long_id.js'; // Imported from a separate file because it's 1MB.

apm.start({ contextPropagationOnly: true });

export async function indexDocument() {
const client = new Client({
nodes: [CLOUD_ES_ENDPOINT],
auth: { apiKey: 'REDACTED' },
Connection: HttpConnection,
ConnectionPool: ClusterConnectionPool,
maxRetries: 0,
});

const index = 'test-index';
const id = LONG_ID + LONG_ID + LONG_ID; // I tried with 1, 2, and 3 concatenations and get different errors, but all correctly handled.
const document = {
    title: 'Test Document',
    content: 'This is a test document',
};

try {
    await client.index({ index, id, document });
    console.log('Document indexed successfully');
} catch (err) {
    console.log('Error indexing document');
    console.error(err);
}

}

async function main() {
await indexDocument();
process.exit(0);
}

main();

const id = LONG_ID;

Error indexing document
ResponseError: too_long_http_line_exception
        Root causes:
                too_long_http_line_exception: An HTTP line is larger than 4096 bytes.
    at SniffingTransport._request (/Users/afharo/Desktop/epipe_error/node_modules/@elastic/transport/lib/Transport.js:533:27)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async /Users/afharo/Desktop/epipe_error/node_modules/@elastic/transport/lib/Transport.js:631:32
    at async SniffingTransport.request (/Users/afharo/Desktop/epipe_error/node_modules/@elastic/transport/lib/Transport.js:627:20)
    at async Client.IndexApi [as index] (/Users/afharo/Desktop/epipe_error/node_modules/@elastic/elasticsearch/lib/api/api/index.js:78:12)
    at async indexDocument (file:///Users/afharo/Desktop/epipe_error/index_document.js:34:9)
    at async main (file:///Users/afharo/Desktop/epipe_error/index.js:10:5) {
  options: { redaction: { type: 'replace', additionalKeys: [] } },
  meta: {
    body: { error: [Object], status: 400 },
    statusCode: 400,
    headers: {
      'content-length': '219',
      'content-type': 'application/json',
      'x-cloud-request-id': 'f0gHTv3lTn6VQ3MMKX-2XQ',
      'x-elastic-product': 'Elasticsearch',
      'x-found-handling-cluster': '1afb2844c141458e8396f13fafd11e42',
      'x-found-handling-instance': 'instance-0000000033',
      date: 'Tue, 27 Jan 2026 08:47:07 GMT'
    },
    meta: {
      context: null,
      request: [Object],
      name: 'elasticsearch-js',
      connection: [Object],
      attempts: 0,
      aborted: false
    },
    warnings: null
  }
}

const id = LONG_ID + LONG_ID;

Error indexing document
ResponseError: 431 Request Header Fields Too Large
    at SniffingTransport._request (/Users/afharo/Desktop/epipe_error/node_modules/@elastic/transport/lib/Transport.js:533:27)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async /Users/afharo/Desktop/epipe_error/node_modules/@elastic/transport/lib/Transport.js:631:32
    at async SniffingTransport.request (/Users/afharo/Desktop/epipe_error/node_modules/@elastic/transport/lib/Transport.js:627:20)
    at async Client.IndexApi [as index] (/Users/afharo/Desktop/epipe_error/node_modules/@elastic/elasticsearch/lib/api/api/index.js:78:12)
    at async indexDocument (file:///Users/afharo/Desktop/epipe_error/index_document.js:34:9)
    at async main (file:///Users/afharo/Desktop/epipe_error/index.js:10:5) {
  options: { redaction: { type: 'replace', additionalKeys: [] } },
  meta: {
    body: '431 Request Header Fields Too Large',
    statusCode: 431,
    headers: {
      'content-type': 'text/plain; charset=utf-8',
      connection: 'close'
    },
    meta: {
      context: null,
      request: [Object],
      name: 'elasticsearch-js',
      connection: [Object],
      attempts: 0,
      aborted: false
    },
    warnings: null
  }
}

const id = LONG_ID + LONG_ID + LONG_ID; (and subsequent increases)

Error indexing document
ConnectionError: Response aborted while reading the body
    at SniffingTransport._request (/Users/afharo/Desktop/epipe_error/node_modules/@elastic/transport/lib/Transport.js:599:31)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async /Users/afharo/Desktop/epipe_error/node_modules/@elastic/transport/lib/Transport.js:631:32
    at async SniffingTransport.request (/Users/afharo/Desktop/epipe_error/node_modules/@elastic/transport/lib/Transport.js:627:20)
    at async Client.IndexApi [as index] (/Users/afharo/Desktop/epipe_error/node_modules/@elastic/elasticsearch/lib/api/api/index.js:78:12)
    at async indexDocument (file:///Users/afharo/Desktop/epipe_error/index_document.js:34:9)
    at async main (file:///Users/afharo/Desktop/epipe_error/index.js:10:5) {
  options: { redaction: { type: 'replace', additionalKeys: [] } },
  meta: {
    body: undefined,
    statusCode: 0,
    headers: {},
    meta: {
      context: null,
      request: [Object],
      name: 'elasticsearch-js',
      connection: [Object],
      attempts: 0,
      aborted: false
    },
    warnings: null
  }
}

I'll try again from within the Kibana code to see if it breaks and we find a consistent way of making it crash for easier debugging.

[afharo]

I'll try again from within the Kibana code to see if it breaks and we find a consistent way of making it crash for easier debugging.

The same code in a local Kibana (writing to ECH ES endpoint) leads to the same behavior 🫠