feat(softtainter): expose node classification metrics #977
Conversation
Add HTTPS metrics support to the softtainter pod using controller-runtime: - Configure metrics server with TLS on port 8443 - Create Service with automatic certificate provisioning - Create ServiceMonitor for Prometheus scraping Expose classification metrics to track node utilization and decisions: - descheduler_softtainter_node_utilization_threshold: threshold values (0-1) - descheduler_softtainter_node_utilization_value: actual utilization (0-1) - descheduler_softtainter_node_classification: classification result (0/1/2) - descheduler_softtainter_threshold_mode: static vs deviation-based (0/1) All utilization metrics use 0-1 ratio format for consistency with Prometheus best practices and easy comparison in queries. Signed-off-by: Simone Tiraboschi <[email protected]>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: tiraboschi The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
@tiraboschi: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
| image: ${SOFTTAINTER_IMAGE} | ||
| ports: | ||
| - name: metrics | ||
| containerPort: 8443 |
There was a problem hiding this comment.
I suggest to use a different port number since 443 suffix resembles an apiserver. We have 10258 for the descheduler as 10259 is used for the scheduler. There's 60000 for the operator. We could go with 60001 for the moment?
Another way is to expose the metrics through descheduler as a carry patch. In which case there's no need for the extra configuration.
There was a problem hiding this comment.
Question is how this will evolve if/when the soft-tainter logic is moved to the descheduler repository. Also, given the descheduler is the source of truth about the classification it's more natural to expose the metrics within.
There was a problem hiding this comment.
With the current approach the soft-tainter can run very often and thus provide almost an up-to-date classification of nodes. Which on the other hand does not need to correspond to what descheduler sees since the time of running the LowNodeUtilization plugin may vary depending on how quickly the plugin works.
There was a problem hiding this comment.
The softtainter is running with the same periodicity of the LowNodeUtilization plugin, but not in sync being two fully independent pods. If the interval is small enough, we can assume for simplicity that the observations about node classifications are consistent. This is the same assumption we are taking when applying the soft-taints to nodes.
The descheduler classification is the source of truth for evictions, while the softtainer ones is the source of truth for the soft taints.
I think that the choice is more about the process to push it to upstream:
- a simple carry patch now on the descheduler repo (I'm going to prepare it) with the idea of having a proper metrics framework for the various plugin upstream in the future (something like: Expose classification metrics from LowNodeUtilization plugin kubernetes-sigs/descheduler#1740 )
- a patch on the softainter now exposing independent metrics and tbd when softtainer will be moved to the descheduler repository
There was a problem hiding this comment.
please see kubernetes-sigs/descheduler#1777
There was a problem hiding this comment.
|
Closing for openshift/descheduler#293 which sounds like the most natural place for this |
2 participants
Add HTTPS metrics support to the softtainter pod using controller-runtime:
Expose classification metrics to track node utilization and decisions:
All utilization metrics use 0-1 ratio format for consistency with Prometheus best practices and easy comparison in queries.