e2e(vpa/recommender): Introduce e2e tests with prometheus history provider

[ialidzhikov]

Which component are you using?:
/area vertical-pod-autoscaler

Is your feature request designed to solve a problem? If so describe the problem this feature should solve.:

No.

Describe the solution you'd like.:

Outline

The currently present e2e tests of the vpa-recommender do not specifically target Prometheus as storage backend. There are test(s) that explicitly validate if VPA Checkpoint resources are created, which automatically fails if we run the test(s) against a Prometheus based configuration.

Strategy

Additional implementation details can be included once the card gets picked

• Deployment manifest of a separate prometheus instance used by the vpa-recommender
• Deployment of kube-state-metrics that scrape the spawned test resources
  • Scrape configuration in the dedicated prometheus
• Instrument existing recommender e2e tests with labels allowing quick distinctions b/w them.

Notes

• The current e2e tests expect a recommender to be deployed within the kube-system namespace.
  • We can instrument the recommender e2e tests to spawn a new per-suite namespace where to place the required recommender. ( In case we face issues deploying the recommender within the kube-system ns. Running the same test against an existing cluster would also require this specific [ recommender in kube-system ns ] to be present).

Credits to @vitanovs

Describe any alternative solutions you've considered.:

N/A

Additional context.:

N/A

[adrianmoisey]

/remove-kind feature
/kind cleanup

[adrianmoisey]

/area vertical-pod-autoscaler/prometheus