Skip to content

Commit ae8e33e

Browse files
NouemanKHALdkirov-ddsarah-witt
authored
Add Nutanix Integration (DataDog#22086)
* initial scaffolding * working nutanix.health.up metric * working nutanix.cluster.count metric * work in progress * work in progress: assert tags in unit tests * tests cleanup * cleanup * health check * collecting basic cluster metrics * collecting cluster stats and basic node metrics * remove upgrade status tag and lint * fix basic auth + add integration tests * refactor and cleanup * collecting node stats metrics * lint * add cluster namespace to cluster metrics * rename metrics, remove unit suffixes * use host instead of nodes as much as possible * collecting basic vm metrics * fix query param typo * collecting vm stats metric * add missing argument required for VmStats and passing integration tests * add metadata.csv * fix typo in test name * update integration tests to stop checking for values * add nutanix overview dashboard * update manifest description and classifier tags * update manifest metric to check for * little cleanup * remove unused dependency from pyproject * set default min_collection_interval to 120s * update dashboard with more units and improvements * update dashboard description * report host metrics and vm metrics with their correspondig hostname * report external host tags for hosts and vms * switch to list all vm stats endpoint for better rate limit - update metdata.csv with new metrics * add ntnx_type:host and ntnx_type:vm as tags * add cluster_name and host_name tags to all hosts and vm metrics + fix integration tests * improve metrics descriptions in metadata.csv * update dashboard * add compact legend to all cluster/host/vm widgets for better ux * fix stats sampling interval to match the min_collection_interval * add support for pagination * add page_limit parameter for pagination size limit * update fixtures and tests for the new paginated requests * rename paginated methods to start with list instead of get * add support for retry logic to handle PC rate limiting * add process signatures * update nutanix process signatures * fix error deleting page and limit params * fix manifest.json extra comma in process_signatures * collect events * add bash script to record fixtures * Fix log message for error collecting vm metrics * refactor pagination method and improve logging * ddev validate ci --sync * update dashboard and add new nutanix logos * add debug logs for HTTP requests and payloads * add support for port in pc_ip * swap nutanix.vm.hypervisor.memory_usage_ppm with nutanix.vm.memory.usage_ppm for more accurate VM memory usage * improve logging: reduce HTTP logging noise to only rate limits and error responses * fix validate dashboards * bump python version to 3.13 and min base check version * fix typo in min base check * fix Mock() has no len error in test_retry.py * wip * add collect_events property * change remaining references to nutanix.vm.hypervisor.memory_usage_ppm to nutanix.vm.memory.usage_ppm to fix VM memory usage widgets * add support for tasks collection, update fixtures * add ntnx_type tag to events and tasks * small cleanup * dashboard: change all bytes in binary to bytes in decimal * cleanup and small refactor * make events and tasks match implementation, fix handling of start_time, improve tests and small refactor * split check.py into modules, fix integration tests * improve error messages for non 2xx http error responses * add missing dd licence headers to some files * rename health_check_score metric * improve metric names batch 1 * improve cluster and host metric names * improve vm metric names * split unit tests into multiple files * add support for audits collection * cleanup and improving tests setup * improve duplication logic tests in events,audits and tasks * add support for alerts collection * use alerts v4.2 API that supports filtering by creationTime * sync all API calls to use the same time window (start, end) * add extra filtering to avoid events/tasks/audits/alerts duplicates * fallback to alerts v4.0 API if v4.2 is not available * fix self.last__x_collection_time fields to be the max timestamp: fixes duplicates * persist information about v4.2 API in the persistence cache * wip: host and vm stats not working? * improve vm stats collection by cluster, improve info logs and debug logs * improve type hints and method comments * add support for capacity metrics * add nutanix tag to all entities * report node status metric * ddev validate models and config * add collect_tasks and collect_audits properties for nutanix * add filter propreties for alerts * add filter by severity and type for alerts * add filter events by type * add filter tasks by status * add resource filters support for infra resources and activity resources * improve resource_filters * cleanup * fetch and cache categories * attach categories as tags with option to add ntnx_ suffix * improve categories collection/attachment, improve tests, update all fixtures * improve categories collection and testing * add owner to manifest.json * remove duplicate self.last_audit_collection_time assignment * fix alert messages parameter rendering * add more tests * reduce info logs, improve info log summary, and change rest of logs to debug * improve audits timpestamp tracking, improve logging, code cleanup * improve resource_filters logging, log error messages * fix integration tests + add support for fake docker server testing * fix nutanix wheel version * reset teleport change * ddev validate ci --sync * fix licence headers * fix more licence headers * fix one more licence header * ddev validate labeler --sync * Fix labeler config * reduce audits.json size * reduce audits.json to 50KB * reduce alerts.json to 20 items * replace bash script for recording fixtures with python implementation * update resource_filters description * add starting check info log and add comment about sampling interval * improve categories tests around default behavior, remove duplicate record_fixtures.py * udpate resource_filters to match default desired category behavior * add collect_subtasks properties, use persistent cache to track last collected items correctly before filtering * Apply suggestions from code review Co-authored-by: dkirov-dd <166512750+dkirov-dd@users.noreply.github.com> * cleanup rate limit retry implementation * add nutanix.api.rate_limited metric for visibility * fix port configuration log message test * fix rate limit tests and update with the new metric * improve retry_limit implementation * update README.md * add missing prefix_category_tags property * add tests for prefix_category_tags property * address david review: always add ntnx_is_agent_vm tag * document categories in the README * add example in the README for collecting categories * fix dashboard validation * fix readme validate * address review: raise ConfigurationError when its not set * fix ntnx_is_agent_vm tag in tests * add missing spec.yaml properties * update powerConsumptionInstantWatt name for consistency * set creates_events to true in manifest.json * fix retry_on_rate_limit behavior on non 429 responses * fix changelog file name * remove unnecessary file * fix retry_on_rate_limit behavior and add type hints * address sarah review: add log message when skipping a resource * address sarah review: add test all metrics + fix missing metric in metadata.csv * update fixtures with new prism_central url and new VM in OFF state * report nutanix.vm.status metric * address review: by default only collect VMs with powerState ON * collect only ON VM even if other VM resource_filters are set that are not powerState * fix imports and paths for record_fixtures.py * add note about duplicate hostname issue * sync config with the new VM collection comment * update README to explain that a single agent can monitor a prism central environment * fix license headers * refactor activity_monitor by reducing code duplication and extraching a sharing method * refactor infrastructure_monitor * refactor resource_filters * refactor check.py / simplyfing * add support for rendering audit messages * add support for rendering nutanix event messages * remove all X_id tags * add caches for activity entities * update test fixtures and adjust tests to work with the new data * add support for displaying affected alerts in tasks * refactor: reduce code duplication in activity_monitor * remove debugging code * improve readability of activity_monitor.py code related to filtering * code cleanup + reduce code duplication * split collect_cluster_metrics into isolated collection phases to enable partial data collection on failures * improve error isolation for host processing to allow non-blocking errors when single host fails * replace log.exception with log.error for more user friendly log message for known errors * make resource_filters proprety always required and remove its default values * switch super() call to python3 style * address some code smells * update page_limit default value from 50 to 100 to reduce API calls * Revert accidental version modification for dev Co-authored-by: Sarah Witt <sarah.witt@datadoghq.com> * add support for batch_vm_collection by default to avoid rate limits * fix batch collection mode to process all vms regardless of the batch mode * improve testing vms filtering in both vm collection modes --------- Co-authored-by: dkirov-dd <166512750+dkirov-dd@users.noreply.github.com> Co-authored-by: Sarah Witt <sarah.witt@datadoghq.com>
1 parent 2cb2760 commit ae8e33e

60 files changed

Lines changed: 29392 additions & 4 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.codecov.yml

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -502,6 +502,10 @@ coverage:
502502
target: 75
503503
flags:
504504
- network
505+
Nutanix:
506+
target: 75
507+
flags:
508+
- nutanix
505509
Nvidia_Triton:
506510
target: 75
507511
flags:
@@ -1483,6 +1487,11 @@ flags:
14831487
paths:
14841488
- nginx_ingress_controller/datadog_checks/nginx_ingress_controller
14851489
- nginx_ingress_controller/tests
1490+
nutanix:
1491+
carryforward: true
1492+
paths:
1493+
- nutanix/datadog_checks/nutanix
1494+
- nutanix/tests
14861495
nvidia_nim:
14871496
carryforward: true
14881497
paths:

.github/workflows/config/labeler.yml

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -998,6 +998,10 @@ integration/ntp:
998998
- changed-files:
999999
- any-glob-to-any-file:
10001000
- ntp/**/*
1001+
integration/nutanix:
1002+
- changed-files:
1003+
- any-glob-to-any-file:
1004+
- nutanix/**/*
10011005
integration/nvidia_jetson:
10021006
- changed-files:
10031007
- any-glob-to-any-file:
@@ -1334,10 +1338,6 @@ integration/sqlserver:
13341338
- changed-files:
13351339
- any-glob-to-any-file:
13361340
- sqlserver/**/*
1337-
integration/storage_management:
1338-
- changed-files:
1339-
- any-glob-to-any-file:
1340-
- storage_management/**/*
13411341
integration/squid:
13421342
- changed-files:
13431343
- any-glob-to-any-file:
@@ -1350,6 +1350,10 @@ integration/statsd:
13501350
- changed-files:
13511351
- any-glob-to-any-file:
13521352
- statsd/**/*
1353+
integration/storage_management:
1354+
- changed-files:
1355+
- any-glob-to-any-file:
1356+
- storage_management/**/*
13531357
integration/streamnative:
13541358
- changed-files:
13551359
- any-glob-to-any-file:

.github/workflows/test-all.yml

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2766,6 +2766,26 @@ jobs:
27662766
minimum-base-package: ${{ inputs.minimum-base-package }}
27672767
pytest-args: ${{ inputs.pytest-args }}
27682768
secrets: inherit
2769+
jcc7e1d2:
2770+
uses: ./.github/workflows/test-target.yml
2771+
with:
2772+
job-name: Nutanix
2773+
target: nutanix
2774+
platform: linux
2775+
runner: '["ubuntu-22.04"]'
2776+
repo: "${{ inputs.repo }}"
2777+
context: ${{ inputs.context }}
2778+
python-version: "${{ inputs.python-version }}"
2779+
latest: ${{ inputs.latest }}
2780+
agent-image: "${{ inputs.agent-image }}"
2781+
agent-image-py2: "${{ inputs.agent-image-py2 }}"
2782+
agent-image-windows: "${{ inputs.agent-image-windows }}"
2783+
agent-image-windows-py2: "${{ inputs.agent-image-windows-py2 }}"
2784+
test-py2: ${{ inputs.test-py2 }}
2785+
test-py3: ${{ inputs.test-py3 }}
2786+
minimum-base-package: ${{ inputs.minimum-base-package }}
2787+
pytest-args: ${{ inputs.pytest-args }}
2788+
secrets: inherit
27692789
jb705691:
27702790
uses: ./.github/workflows/test-target.yml
27712791
with:

nutanix/CHANGELOG.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# CHANGELOG - Nutanix
2+
3+
<!-- towncrier release notes start -->
4+

nutanix/README.md

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
# Agent Check: Nutanix
2+
3+
## Overview
4+
5+
This check collects resource usage metrics from your Nutanix cluster, CPU, memory, storage, and I/O performance for clusters, hosts, and VMs. It also collects operational activity data from Prism Central, including events, tasks, audits, and alerts.
6+
7+
## Setup
8+
9+
### Installation
10+
11+
The Nutanix check is included in the [Datadog Agent][1] package, so you don't need to install anything else on your server.
12+
13+
### Configuration
14+
15+
In Prism Central, create a user with the following roles:
16+
17+
- Cluster Viewer
18+
- Virtual Machine Viewer
19+
- Prism Viewer
20+
- Monitoring Admin
21+
22+
Then, edit the `nutanix.d/conf.yaml` file in the `conf.d/` folder at the root of your [Agent's configuration directory][2]. See the [sample nutanix.d/conf.yaml][3] for all available configuration options.
23+
24+
A single Agent instance connected to Prism Central is enough to monitor all clusters, hosts, and VMs managed by that Prism Central.
25+
26+
[Restart the Agent][4] to start sending Nutanix metrics and activity data to Datadog.
27+
28+
**Note**: The default collection interval is 120 seconds. In practice, setting the interval to 60 seconds or higher results in more reliable and consistent metric collection.
29+
30+
### Validation
31+
32+
Run the [Agent's status subcommand][5] and look for `nutanix` under the Checks section.
33+
34+
## Data Collected
35+
36+
### Metrics
37+
38+
See [metadata.csv][6] for a list of metrics provided by this check.
39+
40+
The integration collects metrics across three resource types, each prefixed with its resource name:
41+
42+
- **Cluster** (`nutanix.cluster.*`): storage capacity and usage, CPU and memory allocation, I/O performance, health score, VM counts.
43+
- **Host** (`nutanix.host.*`): per-host CPU, memory, storage, and controller I/O metrics.
44+
- **VM** (`nutanix.vm.*`): per-VM CPU, memory, disk, network, and storage tier metrics.
45+
46+
A `nutanix.health.up` metric reports Prism Central connectivity status (`1` for reachable, `0` otherwise).
47+
48+
#### Collecting activity data
49+
50+
The integration collects operational activity data from Prism Central by default. Each activity type can be toggled independently in the `nutanix.d/conf.yaml` file:
51+
52+
- `collect_events`: Prism Central events (default: `true`)
53+
- `collect_alerts`: alerts with severity information (default: `true`)
54+
- `collect_tasks`: infrastructure tasks, parent tasks only (default: `true`)
55+
- `collect_subtasks`: include subtasks alongside parent tasks (default: `false`)
56+
- `collect_audits`: user audit logs (default: `true`)
57+
58+
### Events
59+
60+
This check collects activity data from Prism Central and emits them as Datadog events. Each activity type is identified by the `ntnx_type` tag:
61+
62+
- `ntnx_type:event`: Prism Central Events
63+
- `ntnx_type:alert`: Prism Central Alerts
64+
- `ntnx_type:task`: Prism Central Tasks
65+
- `ntnx_type:audit`: Prism Central Audits
66+
67+
Use the `collect_events`, `collect_alerts`, `collect_tasks`, and `collect_audits` parameters in the [sample nutanix.d/conf.yaml][3] to toggle each activity type.
68+
69+
**Note**: By default, only parent tasks are collected. Set `collect_subtasks: true` to include subtasks.
70+
71+
### Service Checks
72+
73+
The integration does not emit any service checks.
74+
75+
## Troubleshooting
76+
77+
### VM collection
78+
79+
By default, only VMs with `powerState: ON` are collected. To collect VMs in other power states (OFF, PAUSED), add an explicit `powerState` VM filter in `resource_filters`:
80+
81+
```yaml
82+
resource_filters:
83+
- resource: vm
84+
property: powerState
85+
patterns:
86+
- '.*'
87+
```
88+
89+
Other VM filters (e.g., by name) do not override this default. Only a `powerState` filter does.
90+
91+
### Category tags
92+
93+
Nutanix categories are attached as tags to metrics. By default, only `USER` category tags are collected. To include `SYSTEM` or `INTERNAL` categories, add an explicit category filter in `resource_filters`.
94+
95+
To collect `SYSTEM` and `INTERNAL` categories alongside `USER`:
96+
97+
```yaml
98+
resource_filters:
99+
- resource: category
100+
property: type
101+
patterns:
102+
- '^SYSTEM$'
103+
- '^INTERNAL$'
104+
- '^USER$'
105+
```
106+
107+
Category tags use the Nutanix category key as the tag name (e.g., `Environment:Production`). Set `prefix_category_tags: true` to prefix them with `ntnx_` (e.g., `ntnx_Environment:Production`) to avoid collisions with existing Datadog tags.
108+
109+
### Duplicate hostnames
110+
111+
The Nutanix API does not expose the real hostname of VMs. VM metrics use the VM name from Prism Central as the hostname. If the Datadog Agent is installed on a Nutanix VM, its auto-detected hostname may differ from the VM name, causing duplicate hosts in Datadog. To fix this, set `hostname` in `datadog.yaml` (or the `DD_HOSTNAME` environment variable) to match the VM name in Prism Central.
112+
113+
### Filtering resources
114+
115+
Use the `resource_filters` option to control which resources are collected. Each filter requires a `resource` type, a `property` (the API field name to match against), and a list of regex `patterns`. Optionally, set `type` (`include` or `exclude`, default: `include`). Exclude filters take precedence over include filters.
116+
117+
Supported resource types: `cluster`, `host`, `vm`, `event`, `task`, `alert`, `audit`, `category`. Nested properties are supported using `/` as a separator (e.g., `userReference/name`). Note that hosts use `hostName` as the API field for the host name, not `name`.
118+
119+
```yaml
120+
resource_filters:
121+
- resource: cluster
122+
property: name
123+
patterns:
124+
- '^prod-'
125+
- resource: host
126+
property: hostName
127+
type: exclude
128+
patterns:
129+
- '^standby-'
130+
- resource: alert
131+
property: severity
132+
patterns:
133+
- '^WARNING$'
134+
- '^CRITICAL$'
135+
```
136+
137+
Need help? Contact [Datadog support][7].
138+
139+
[1]: https://app.datadoghq.com/account/settings/agent/latest
140+
[2]: https://docs.datadoghq.com/agent/guide/agent-configuration-files/#agent-configuration-directory
141+
[3]: https://github.com/DataDog/integrations-core/blob/master/nutanix/datadog_checks/nutanix/data/conf.yaml.example
142+
[4]: https://docs.datadoghq.com/agent/guide/agent-commands/#start-stop-and-restart-the-agent
143+
[5]: https://docs.datadoghq.com/agent/guide/agent-commands/#agent-status-and-information
144+
[6]: https://github.com/DataDog/integrations-core/blob/master/nutanix/metadata.csv
145+
[7]: https://docs.datadoghq.com/help/

0 commit comments

Comments
 (0)