Stream Nginx Logs to Kafka via kcat | messages are not sent as logs are written

[moatazsalem2]

Hello,

I have started Nginx with access logs being directed to stdio
/usr/sbin/nginx -g 'daemon off;'

then updated the command to pipe the access logs to kcat
/usr/sbin/nginx -g 'daemon off;' | /usr/bin/kcat -b $KAFKA_HOST -t $KAFKA_TOPIC -P -X queue.buffering.max.ms=1 -l -X batch.num.messages=1 -X batch.size=1 -X queue.buffering.max.messages=1 -X queue.buffering.max.kbytes=1

As you can tell, i have played with many options, but still, out of each 5 page views, i get 4 entries in the topic which indicates that kcat is still buffering some access logs information.

Appreciate your support.

[OneCricketeer]

I'd recommend using something like Fluent-bit for this