[Placeholder] Codec adds new line when used with streaming input (TCP/UDP) plugins.

[mashhurs]

Description

A codec randomly adds new line (might be whitespace) in the event payload. The behavior happens with streaming input plugins such as TCP or UDP. As I have confirmed, when TCP buffer gets full (64K), the buffer gets flushed and moved to the codec. The buffer treated with new line in a reality is a part of event but codec doesn't have information about it, so continues with text.split("\n").each do |line| which eventually adds new line.

We have an effort in the past to fix it but it ended up with not landing since it was a broad (depending on use-case): #63

Related:

• Handle split reads #6
• [META][Discuss] add support for input_type in "dual modes" codecs elastic/logstash#11885
• Newline Characters Inserted #14
• stdin/multiline-code adds random newlines #37
• Correctly handle reads by the input module that are not aligned to a newline #38
• Multiline received as last input will be dropped #57

This issue is a placeholder to re-activate the discussion and bring a fix/solution/agreement to fix the issue for the long term.
Current quick thought to fix:

• Reviving the effort to add BufTok to the Multiline codec, with the cumbersome flag to tell the codec that it is running in "streaming mode".
• Bringing BufTok into the input plugins, to ensure that it only ever emits whole lines to the codec (opt-in, with a flag to tell the plugin that its expecting line-oriented input; will need to ensure a buffer flush on close)