Skip to content

feat: Add flags to optionally disable DCGM Exporter, Neuron Monitor resources - #190

Merged
sky333999 merged 9 commits into
aws-observability:mainfrom
KKamJi98:main
Jul 21, 2025
Merged

feat: Add flags to optionally disable DCGM Exporter, Neuron Monitor resources#190
sky333999 merged 9 commits into
aws-observability:mainfrom
KKamJi98:main

Conversation

@KKamJi98

Copy link
Copy Markdown
Contributor

Description of changes:
This change introduces new Helm chart flags:

  • .Values.dcgmExporter.enabled
  • .Values.neuronMonitor.enabled
  • .Values.windows.enabled
    By setting them to false, undesired resources (DCGM Exporter, Neuron Monitor, Windows) are not deployed, thereby preventing unnecessary warnings in logs.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@KKamJi98 KKamJi98 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR introduces optional flags to disable DCGM Exporter, Neuron Monitor, and Windows-specific CloudWatch Agent components in the Helm chart.

The goal is to make the chart more flexible across environments where those components are not needed.

Please let me know if any changes are required or if you'd like additional test coverage.

@dansimov04012022

Copy link
Copy Markdown

Can't get why admissionWebhooks is optional, but those components aren't. Having this will be very useful.

@KKamJi98

KKamJi98 commented Jun 4, 2025

Copy link
Copy Markdown
Contributor Author

@dansimov04012022
I completely agree. Even when no nodes require DCGM, Neuron or Windows support, the chart still spins up the related DaemonSets and Services. Because the agent keeps trying to scrape those endpoints, we see a constant stream of warnings like:

W! {"caller":"internal/transaction.go:125","msg":"Failed to scrape Prometheus endpoint","kind":"receiver", ... ,"target_labels":"{__name__=\"up\", instance=\"dcgm-exporter-service.amazon-cloudwatch.svc:9400\", job=\"containerInsightsDCGMExporterScraper\"}"}

Being able to disable these components with enabled: false flags will eliminate the unused resources and stop the noise in our logs.

@sky333999

Copy link
Copy Markdown
Contributor

Hi @KKamJi98 .. thanks for making these changes. I think its fair to have flags that control whether or not these resources should be created. (Do note that the resources such as the actual daemonset objects are created, but the actual pods are only spun if they find appropriate nodes - so you shouldnt be seeing wasted cpu or mem even in the current state)

Would you mind leaving out windows from this change? There's some ongoing discussions on how to restructure windows to better match the multi-agent pattern under agents, so this issue should likely get resolved as part of that. We can still track windows as a separate Github issue if you'd like to create one.

Comment thread RELEASE_NOTES Outdated

@KKamJi98 KKamJi98 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sky333999 Thanks for the review! I’ve made the requested changes and pushed the commit.

@KKamJi98
KKamJi98 requested a review from sky333999 July 10, 2025 13:37
@KKamJi98 KKamJi98 changed the title feat: Add flags to optionally disable DCGM Exporter, Neuron Monitor, and Windows resources feat: Add flags to optionally disable DCGM Exporter, Neuron Monitor resources Jul 10, 2025
@sky333999
sky333999 merged commit 8de9850 into aws-observability:main Jul 21, 2025
0 of 3 checks passed
@sky333999

Copy link
Copy Markdown
Contributor

Thanks for your contribution @KKamJi98 !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants