Can http sink send single events

[poehlerflorian]

Question

We are trying to use (or rather misuse) vector as a simple http proxy that redacts some fields with sensitive data from the json payload. The batch mechanism of vector however sends the events wrapped in an array, even with batch.max_events=1.

Can this be configured somehow to only send a single event?

E.g. instead of sending

[
  {
    <payload>
  }
]

just send

{
  <payload>
}

Vector Config

sources:
  http_input:
    type: http_server
    address: 0.0.0.0:8686
    decoding:
      codec: json
transforms:
  payload_modifier:
    inputs:
      - http_input
    type: remap
    file: ${VECTOR_CONFIG_DIR}/sanitize.vrl # Simple redaction of various fields

sinks:
  http_output:
    inputs:
      - payload_modifier
    type: http
    uri: <grafana-url>
    encoding:
      codec: json

Vector Logs

No response

[thomasqueirozb]

Changing the framing method will change how the output in sent to the http server. Looking at the source code it looks like the array (AKA [ and ]) is only inserted into the payload if encoding is json, the framing is character_delimited and the delimiter is ,. Since you already set the batching to 1, you should be able to set the framing to newline_delimited like so (this may cause other issues but I think it should be fine. I'm pretty sure that if events were batched you'd get a jsonl style request):

sinks:
  http_output:
    inputs:
      - payload_modifier
    type: http
    uri: <grafana-url>
    encoding:
      codec: json
    framing:
      method: newline_delimited

[poehlerflorian]

Thanks! This works great, with one addition: I had to explicitly set the Content-Type header (otherwise I'd get a 415 Unsupported Media Type response, but I haven't actually looked up what Content-Type header is sent by default), so my config now looks like this:

sinks:
  http_output:
    inputs:
      - payload_modifier
    type: http
    uri: "SECRET[fromfile.oncall_url]/"
    batch:
      max_events: 1
    encoding:
      codec: json
    framing:
      method: newline_delimited
    request:
      headers:
        "Content-Type": application/json

Another option I found out is to set the encoding.codec to text, along with the Content-Type header and set the payload as a json encoded string to the .message property:

sinks:
  http_output:
    inputs:
      - payload_modifier
    type: http
    uri: "SECRET[fromfile.oncall_url]/"
    batch:
      max_events: 1
    encoding:
      codec: text
    request:
      headers:
        "Content-Type": application/json

And at the end of the vrl script:

[…]
alert = .
. = {}
.message = encode_json(alert)

This works great, but I think your solution is cleaner since it does not require any modifications of the vrl script.

Thank you for your help!