feat: add promtheus podMonitor in helm-charts

[dongjiang1989]

What type of PR is this?

/kind feature

What this PR does / why we need it:

Add promtheus podMonitor in helm-charts

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

[hami-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: dongjiang1989
Once this PR has been reviewed and has the lgtm label, please assign wawa0210 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[gemini-code-assist]

Summary of Changes

Hello @dongjiang1989, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a PodMonitor resource to the hami-vgpu Helm chart, enabling Prometheus to monitor the scheduler component. A new configuration option is added to control the deployment of the PodMonitor.

Highlights

• PodMonitor Resource: Adds a PodMonitor resource definition to the Helm chart for monitoring the hami-vgpu scheduler using Prometheus. This allows Prometheus to automatically discover and collect metrics from the scheduler pods.
• Prometheus Configuration: Introduces a prometheus.enabled flag in the values.yaml file to control whether the PodMonitor resource is deployed. It is disabled by default.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a Prometheus PodMonitor for the hami-scheduler, which is a great addition for observability. The implementation is mostly correct, but I've identified a potential issue with the metrics port name that could prevent scraping. Additionally, I've suggested an improvement to make the PodMonitor more easily discoverable by Prometheus through a dedicated labels configuration in values.yaml. These changes will enhance the robustness and usability of the new monitoring feature.

[gemini-code-assist]

The podMetricsEndpoints configuration specifies port: metrics. However, the corresponding service definition in charts/hami/templates/scheduler/service.yaml defines the monitoring port with the name monitor. For the PodMonitor to correctly discover and scrape the metrics endpoint, the port name must match the name defined in the Pod's specification. Assuming the Pod spec follows the service's naming convention, this should be monitor to ensure metrics are collected.

      port: monitor

[gemini-code-assist]

To make the PodMonitor discoverable by the Prometheus Operator, it's good practice to allow users to specify custom labels. This change uses a new .Values.prometheus.labels value (suggested in a separate comment on values.yaml) to apply these labels. While global.labels can also be used, a dedicated prometheus.labels is more explicit and aligned with the new prometheus values section.

    {{- with .Values.prometheus.labels }}
    {{- toYaml . | nindent 4 }}
    {{- end }}
    {{- with .Values.global.labels }}
    {{- toYaml . | nindent 4 }}
    {{- end }}

[gemini-code-assist]

To allow users to easily add labels for Prometheus Operator discovery of the PodMonitor, it's a good practice to provide a dedicated labels field under the prometheus section. This makes the chart more configurable and user-friendly. This new value should then be used in charts/hami/templates/scheduler/monitor.yaml to apply the labels to the PodMonitor resource.

  enabled: false
  labels: {}

[codecov]

Codecov Report

❌ Patch coverage is 40.66852% with 1065 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...ce-plugin/nvidiadevice/nvinternal/plugin/server.go	16.01%	169 Missing and 4 partials ⚠️
...evice-plugin/nvidiadevice/nvinternal/rm/rm_mock.go	0.00%	130 Missing ⚠️
...vice-plugin/nvidiadevice/nvinternal/plugin/util.go	22.58%	93 Missing and 3 partials ⚠️
pkg/device/awsneuron/device.go	76.36%	61 Missing and 8 partials ⚠️
...g/device-plugin/nvidiadevice/nvinternal/cdi/cdi.go	0.00%	68 Missing ⚠️
...-plugin/nvidiadevice/nvinternal/plugin/register.go	0.00%	66 Missing ⚠️
...e-plugin/nvidiadevice/nvinternal/plugin/factory.go	0.00%	48 Missing ⚠️
pkg/device/ascend/device.go	74.19%	42 Missing and 6 partials ⚠️
...device-plugin/nvidiadevice/nvinternal/imex/imex.go	0.00%	35 Missing ⚠️
pkg/device/amd/device.go	75.86%	31 Missing and 4 partials ⚠️
... and 19 more

Flag	Coverage Δ
unittests	51.16% <40.66%> (-9.99%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...ice-plugin/nvidiadevice/nvinternal/info/version.go	0.00% <ø> (ø)
...device-plugin/nvidiadevice/nvinternal/rm/helper.go	100.00% <100.00%> (ø)
pkg/device/common/common.go	53.33% <ø> (ø)
pkg/device/devices.go	81.12% <ø> (+7.47%)	⬆️
pkg/device/enflame/config.go	100.00% <ø> (ø)
pkg/device/enflame/device.go	83.25% <ø> (+5.16%)	⬆️
pkg/device/enflame/gcu.go	90.35% <ø> (ø)
pkg/device/hygon/device.go	91.22% <ø> (-2.15%)	⬇️
pkg/device/iluvatar/device.go	59.29% <ø> (-26.94%)	⬇️
pkg/device/kunlun/config.go	0.00% <ø> (ø)
... and 62 more

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[FouoF]

I think add a serviceMonitor is better than podMonitor for the scheduler already create service to expose metrics.
#1255 and #1486 did something about it.

[dongjiang1989]

I think add a serviceMonitor is better than podMonitor for the scheduler already create service to expose metrics. #1255 and #1486 did something about it.

Thanks @FouoF
A ServiceMonitor must depend on a Service; however, for components like the scheduler, it is not necessary to create a Service resource solely for metrics. A PodMonitor serves the same purpose.

[archlitchi]

could you sync again with master branch, i've just rebase and force pushed several commits to minimize the repo size.