vlagent: implement Kubernetes log collector

[vadimalekseev]

Describe Your Changes

Fixes: #538

Checklist

The following checks are mandatory:

• My change adheres to VictoriaMetrics contributing guidelines.
• My change adheres to VictoriaMetrics development goals.

[valyala]

Looks great overall!

@vadimalekseev , please document the added functionality in docs/victorialogs/vlagent.md, and mention it at docs/victorialogs/CHANGELOG.md.

[AndrewChubatiuk]

maybe makes sense to move k8s client configuration from this PR and https://github.com/VictoriaMetrics/VictoriaMetrics/blob/1a68d4ac8a032dfa96cea8b984e17a3cb23f825d/lib/promscrape/discovery/kubernetes/kubeconfig.go to a separate package

[vadimalekseev]

Added documentation.

Please, do not merge this PR until we resolve this issue: #816, because it significantly affects memory usage when collecting pod logs. In my tests, vlagent uses up to 5x more memory because of the double buffering issue.

After this, I need to run final tests to compare the FDs, RAM, CPU usage of vlagent with promtail, Vector and Fluent Bit.

upd:
Fixed by implementing new LogMessageProcessor, see:

VictoriaLogs/app/vlagent/kubernetescollector/processor.go

Line 34 in 01cb965

type logFileProcessor struct {

[valyala]

@vadimalekseev , thank you for implementing discovery and collecting of logs in Kubernetes by vlagent!