Improve the prometheus history provider docs

[ialidzhikov]

Which component are you using?:
/area vertical-pod-autoscaler

Is your feature request designed to solve a problem? If so describe the problem this feature should solve.:

No

Describe the solution you'd like.:
The only existing documentation for running vpa-recommender with a prometheus history provider is How can I use Prometheus as a history provider for the VPA recommender.
When the team evaluated this option they discovered several things that could be documented better or things which were not documented at all.

Findings:

• vpa-recommender falls back to NOT using history (and it does not write VPACheckpoints) for the metrics if it cannot reach to the prometheus instance.
• vpa-recommender uses the pod labels from kube-state-metrics to load metrics for old (no-longer-existing) pods. An internal map in vpa-recommender maps {pod's labels, container name, namespace} to {memory/cpu histograms}. This requires that the vpa-recommender be started with the arguments: --metric-for-pod-labels=kube_pod_labels{job=\"kube-state-metrics\"}[8d] and pod-label-prefix=label_ where label_ is the prefix that kube-state-metrics adds to pod labels when it writes them to prometheus.

Credits to @plkokanov and @vitanovs for the findings.

Describe any alternative solutions you've considered.:
N/A

Additional context.:
N/A

[adrianmoisey]

/remove-kind feature
/kind documentation

Something to note about the prometheus integration is that we don't have anyone to maintain it. I've been considering suggestion that we deprecate it.

[adrianmoisey]

/area vertical-pod-autoscaler/prometheus