docs/victorialogs/vlagent.md: clarify docs about collecting Kubernetes logs via vlagent

This is a follow-up for the commit 2aa0ea1

Updates #815
Updates #538

Author: valyala

docs/victorialogs/vlagent.md

@@ -62,70 +62,82 @@ in a timely manner when some of the VictoriaLogs instances are unavailable.
 
 ### Collect Kubernetes Pod logs
 
-To collect Kubernetes Pod logs with `vlagent` use [`victoria-logs-collector`](https://docs.victoriametrics.com/helm/victoria-logs-collector/#quick-start) Helm chart.
-This Helm chart deploys `vlagent` as a DaemonSet, which collects logs from Pods on all nodes and sends them to VictoriaLogs instances.
-In case of network issues, `vlagent` automatically buffers the collected logs and sends them to VictoriaLogs instances as soon as they become available,
-as described in [Replication and high availability](https://docs.victoriametrics.com/victorialogs/vlagent/#replication-and-high-availability).
+The [`victoria-logs-collector`](https://docs.victoriametrics.com/helm/victoria-logs-collector/#quick-start) Helm chart deploys `vlagent`
+across all the Kubernetes nodes and enables automatic discovering and collecting all the logs generated by containers across all the pods.
+It sends the collected logs to the configured VictoriaLogs instances (either [single-node](https://docs.victoriametrics.com/helm/victoria-logs-single/)
+or [cluster](https://docs.victoriametrics.com/helm/victoria-logs-cluster/) instances).
 
-To quickly deploy `vlagent` and VictoriaLogs with Helm, see [VictoriaLogs Quick Start](https://docs.victoriametrics.com/helm/victoria-logs-single/#quick-start).
+If the configured VictoriaLogs are temporarily unavailable (for example, because of upgrades, configuration changes or other maintenance events),
+then `vlagent` buffers the collected logs at the configured persistent volume and sends them to VictoriaLogs instances as soon as they become available,
+as described in [Replication and high availability](https://docs.victoriametrics.com/victorialogs/vlagent/#replication-and-high-availability).
 
 ## Kubernetes Collector Configuration
 
-This section describes the existing configuration options for `vlagent`.
-In most cases, you don't need to configure `vlagent` manually, use
-[`victoria-logs-collector`](https://docs.victoriametrics.com/helm/victoria-logs-collector/#quick-start) Helm chart instead.
+In most cases, you don't need to configure `vlagent` manually.
+Use [`victoria-logs-collector`](https://docs.victoriametrics.com/helm/victoria-logs-collector/#quick-start) Helm chart instead.
+
+`vlagent` must run on Kubernetes nodes in [DaemonSet mode](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/).
+Pass `-kubernetesCollector` command-line flag to `vlagent` in order to start discovering and collecting logs from all the containers
+running inside all the pods on the given Kubernetes node. Pass VictoriaLogs address to store the collected logs
+via `-remoteWrite.url` command-line flag.
+
+Here is the minimal configuration for `vlagent` to collect logs on the current Kubernetes node and send them to `http://victoria-logs:9428/internal/insert`:
 
-To enable `vlagent` to collect Kubernetes Pod logs, pass `-kubernetesCollector` command-line flag to `vlagent` and specify the list 
-of `-remoteWrite.url` values to send the collected logs to.
-Minimal configuration for `vlagent` to collect Kubernetes Pod logs:
 ```sh
 ./vlagent -kubernetesCollector -remoteWrite.url=http://victoria-logs:9428/internal/insert
 ```
 
-`vlagent` needs to be able to access the Kubernetes API server to watch and get pods (the `watch` and `get` verbs on the `pods` resource).
-`vlagent` also needs to be able to access the `/var/log/containers` and `/var/log/pods` directories on the Kubernetes nodes.
+`vlagent` can send the copies of the collected logs into multiple destinations.
+See [replication and high availability docs](https://docs.victoriametrics.com/victorialogs/vlagent/#replication-and-high-availability) for details.
+
+`vlagent` requires access to Kubernetes API server to watch and get pods (the `watch` and `get` verbs on the `pods` resource).
+`vlagent` also requires access to the `/var/log/containers` and `/var/log/pods` directories on the Kubernetes node.
 For Kubernetes in Docker (in case you run `vlagent` using tools like minikube or kind), you may need to mount `/var/lib` directory.
 
 `vlagent` uses checkpoints to persist its state across restarts.
 Default location for checkpoints is `./vlagent-kubernetes-checkpoints.json`.
 You can specify a different location for checkpoints with `-kubernetesCollector.checkpointsPath` command-line flag.
 Make sure that this file is available for `vlagent` across restarts.
-Use [`hostPath`](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath) volume 
+Use [`hostPath`](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath) volume
 so the checkpoint file stays on the node disk and is reused across Pod restarts.
 
-`vlagent` automatically parses JSON logs from Kubernetes Pod logs and treats the following fields 
-as [`_msg`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) fields:
+`vlagent` automatically parses JSON logs from Kubernetes Pod logs and treats the following fields
+as [`_msg`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) field:
 - `message`
 - `msg`
 - `log`
 
-You can change the default list of `_msg` fields by passing `-kubernetesCollector.msgFields` command-line flag with a list of field names.  
+You can change the default list of `_msg` fields by passing `-kubernetesCollector.msgFields` command-line flag with comma-separated list of field names.
 
-`vlagent` extracts timestamp in RFC3339 format from the log line and uses it as
-[`_time`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field) field,
-using the following fields:
+`vlagent` extracts timestamp from the log line and uses it as
+[`_time`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field) field, using the following fields:
 - `time`
 - `timestamp`
 - `ts`
 
-You can change the default list of `_time` fields by passing `-kubernetesCollector.timeField` command-line flag with a list of field names.
+You can change the default list of `_time` fields by passing `-kubernetesCollector.timeField` command-line flag with comma-separated list of field names.
 If none of the specified fields are present in the log line, then `vlagent` uses timestamp when the log line was written by Container Runtime.
 Usually, this timestamp is close to the actual time when the log line was written, but it may differ slightly, normally less than a millisecond.
 
 To ignore fields from Kubernetes Pod logs, pass `-kubernetesCollector.ignoreFields` command-line flag with a list of field names to ignore.
-You can also configure `vlagent` to add additional fields to all collected logs,
+You can also configure `vlagent` to add additional fields to all the collected logs,
 using `-kubernetesCollector.extraFields` command-line flag, which accepts a JSON object with additional fields.
+For example, the following command instructs `vlagent` to add `env=dev` and `cluster=staging` [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model)
+for all the collected logs before sending them to `-remoteWrite.url`:
+
+```sh
+./vlagent -kubernetesCollector.extraFields='{"env":"dev","cluster":"staging"}' ...
+```
 
 To set the default [tenant](http://localhost:1313/victorialogs/#multitenancy) ID for logs collected from Kubernetes Pods, 
-pass `-kubernetesCollector.tenantID` command-line flag with a tenant ID in format `accountID:projectID`.
+pass `-kubernetesCollector.tenantID` command-line flag with a tenant ID in the format `accountID:projectID`.
 
 `vlagent` uses the following fields as [`_stream`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) fields for Kubernetes Pod logs:
 - `kubernetes.container_name`
 - `kubernetes.pod_name`
 - `kubernetes.pod_namespace`
 
-Use these fields for fast filtering and grouping of logs in VictoriaLogs. 
-See [stream filter](https://docs.victoriametrics.com/victorialogs/logsql/#stream-filter) for more details.
+Use these fields for fast filtering and grouping of logs in VictoriaLogs via [stream filters](https://docs.victoriametrics.com/victorialogs/logsql/#stream-filter).
 
 ## Monitoring