Skip to content

NeuronCore mapping bug fix - #341

Merged
sky333999 merged 17 commits into
amazon-contributing:aws-cwa-devfrom
petruanica:aws-cwa-dev
Aug 14, 2025
Merged

NeuronCore mapping bug fix#341
sky333999 merged 17 commits into
amazon-contributing:aws-cwa-devfrom
petruanica:aws-cwa-dev

Conversation

@petruanica

Copy link
Copy Markdown

Description

  • Fixing a bug for NeuronCore mapping. Initially, it was assumed that one NeuronDevice will have 2 NeuronCores. With the introduction of Trn2 instances, NeuronDevices can have up to 4 NeuronCores. So we are adding logic to extract the number of NeuronCores per NeuronDevice from the metrics emitted by NeuronMonitor.

Related PR

}

// method to add neuroncore_per_device_count attribute to all metrics
func (ed *EmptyMetricDecorator) addNeuronCorePerDeviceAttribute(metrics pmetric.MetricSlice, neuronCoresPerDevice int) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this have to be a datapoint attribute? Cant it be a resource attr?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We chose to do it like this because we don't have access to the resource later on. But this would be a good idea for future refactoring.
https://github.com/aws/amazon-cloudwatch-agent/blob/main/plugins/processors/gpuattributes/internal/awsneuron_metric_modifier.go#L152

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its the same pmetric getting passed to the processor right? I dont get why it wont be available later on.
Please track this as a to-do.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do, thank you for pointing this out

sky333999
sky333999 previously approved these changes Aug 14, 2025
neuronHardwareInfo, neuronHardwareInfoFound := findNeuronHardwareInfo(metrics)
if neuronHardwareInfoFound {
ed.addEmptyMetrics(neuronHardwareInfo, metrics)
neuronCoresPerDevice, foundCoresPerDevice := getNeuronCoresPerDevice(neuronHardwareInfo)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit for better readability

Suggested change
neuronCoresPerDevice, foundCoresPerDevice := getNeuronCoresPerDevice(neuronHardwareInfo)
neuronCoresPerDevice, foundCoresPerDevice := getNeuronCoresPerDevice(neuronHardwareInfo)
if !foundCoresPerDevice {
neuronCoresPerDevice = DefaultNeuronCorePerDevice
}
ed.addNeuronCorePerDeviceAttribute(metrics, neuronCoresPerDevice)

@sky333999
sky333999 merged commit 726f1bd into amazon-contributing:aws-cwa-dev Aug 14, 2025
134 of 141 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants