[Feature]: Infiniband health monitor

[MaxFedotov]

Prerequisites

• I searched existing issues

Feature Summary

Hey folks, don't know if it is in scope\roadmap for NVSentiel, but would like to start the discussion about infiniband health monitoring.

Infiniband is also a very important part of HPC and it's health and performance directly influence the workloads running on the nodes.

Problem/Use Case

We are using gpud together with node-problem-detector and draino for health-checking, node condition setting and draining and are planning to investigate NVSentiel as a single replacement for these components.

One of the great features that gpud supports is the ability to run infiniband checks and report interfaces that are either down or flap between active<>down states. This allows us to cordon the problematic nodes and evict the workloads from them until the problem is investigated and fixed.

Proposed Solution

Introduce a new infiniband-health-monitor (integrate gpud healthchecks logic?) for NVSentiel, that will implement ib ports down and flapping cheks.

Component

New Component

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord or schedule a call!

[lalitadithya]

Hey Max, thank you for opening this issue! I think Infiniband monitoring is also in scope for NVSentinel.

We were planning to start wrking on this post the winter holidays. @KaivalyaMDabhadkar has some experience with IB monitoring and he said he had some thoughts around how this can work. I think we can start a shared google document to discuss our thoughts and then proceed to implementation. What do you think, Max, Kaivalya?

[KaivalyaMDabhadkar]

Hey Max, thank you for opening this issue! I think Infiniband monitoring is also in scope for NVSentinel.

We were planning to start wrking on this post the winter holidays. @KaivalyaMDabhadkar has some experience with IB monitoring and he said he had some thoughts around how this can work. I think we can start a shared google document to discuss our thoughts and then proceed to implementation. What do you think, Max, Kaivalya?

Absolutely. I have been brainstorming on how to design the NIC health monitor since the past few days, will create and share a doc here for the initial design proposal so that we can discuss it further.

[MaxFedotov]

Hey @lalitadithya and @KaivalyaMDabhadkar!

Thanks, will be glad to work on the proposal together.

Happy holidays!