Back pressure-like flow control for in/out_forward protocol

[tagomoris]

Using opts (possibly v0.12)

[repeatedly]

It means in_forward returns own busy status to out_forward and out_forward coordinates buffer handling?

[tagomoris]

I don't have clear answer what we can do about this feature.
If buffer plugins has any APIs to explain its status, we can use it.

[lam-juice]

it means telling input plugins to not ACK / update pos file when the buffer is reaching its limits?

A possible way: create a mutex class and allow plugins to refer to it.

Usually, there are many plugins bug only a few places that need to implement backpressure. Under this approach, in the configuration for an input or output plugin you would explicitly refer to the mutex. Otherwise, plugins work as they do now.

[donato]

Is this a thing? Due to the us-east-1 outage from AWS today, we are experiencing some headaches with td-agent.

Td-agent reads as fast as it can, even though we can't write output to S3. The buffer fills until we run out of memory and td-agent gets reaped. If td-agent would use backpressure to not read past a certain level until the buffer is being flushed, then this wouldn't occur.

[fragatina]

+1

[seungyongshim]

+1