Skip to content

Add DCGM throttle alerts #307

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 12 commits into from
Closed

Add DCGM throttle alerts #307

wants to merge 12 commits into from

Conversation

@zxhdaze
Copy link
Contributor

@zxhdaze zxhdaze commented Sep 10, 2024 •
edited by Deezzir
Loading

Adds Throttle alerts for NVIDIA GPUs. I am using the DCGM_FI_DEV_CLOCK_THROTTLE_REASONS bitmask from the dcgm-exporter snap and values from this doc to determine a throttle reason.

Do not check for the following reasons:

  • DCGM_CLOCKS_THROTTLE_REASON_CLOCKS_SETTING - user settings to limit the clock speed
  • DCGM_CLOCKS_THROTTLE_REASON_DISPLAY_CLOCKS - display settings to limit the clock speed
  • DCGM_CLOCKS_THROTTLE_REASON_GPU_IDLE - system idling

PS, thanks @zxhdaze for the draft work. EDIT by @Deezzir

@Deezzir Deezzir force-pushed the SOLENG-534-dcgm-throttle-alerts branch from 8a4ee5f to 4cce89d Compare October 1, 2024 22:45
@Deezzir Deezzir force-pushed the SOLENG-534-dcgm-throttle-alerts branch from 4cce89d to 4eee72e Compare October 1, 2024 23:02
@Deezzir Deezzir marked this pull request as ready for review October 1, 2024 23:22
@Deezzir Deezzir requested a review from a team as a code owner October 1, 2024 23:22
@Deezzir Deezzir mentioned this pull request Oct 2, 2024
Merged
@Deezzir
Copy link
Contributor

Deezzir commented Oct 2, 2024

Closing, work moved to #331

@Deezzir Deezzir closed this Oct 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Pjack Pjack Awaiting requested review from Pjack Pjack is a code owner automatically assigned from canonical/soleng-reviewers

@aieri aieri Awaiting requested review from aieri aieri is a code owner automatically assigned from canonical/soleng-reviewers

@samuelallan72 samuelallan72 Awaiting requested review from samuelallan72 samuelallan72 is a code owner automatically assigned from canonical/soleng-reviewers

@jneo8 jneo8 Awaiting requested review from jneo8 jneo8 is a code owner automatically assigned from canonical/soleng-reviewers

@gabrielcocenza gabrielcocenza Awaiting requested review from gabrielcocenza gabrielcocenza is a code owner automatically assigned from canonical/soleng-reviewers

@rgildein rgildein Awaiting requested review from rgildein rgildein is a code owner automatically assigned from canonical/soleng-reviewers

@Deezzir Deezzir Awaiting requested review from Deezzir Deezzir is a code owner automatically assigned from canonical/soleng-reviewers

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@zxhdaze @Deezzir