feat: scout_heartbeat_timeout health alert

[krish-nvidia]

Description

Problem: Machines can become unresponsive (e.g., left in BIOS menu, OS issues, power faults) while they are in the Ready state, creating silent failures that go undetected by carbide until instance creation attempts fail. A metric was created for this issue but a host's health state never reflected the timeout, so allocations weren't blocked.

Fix: Add scout heartbeat timeout health alert for machines in the Ready state:

• Create a merge health override when last_scout_contact_time exceeds scout_reporting_timeout (default 5 minutes)
• Remove the override automatically when scout heartbeat recovers in Ready
• Clear the scout heartbeat timeout alert whenever the host transitions out of Ready, so stale alerts do not leak across state changes.
• Continue emitting hosts_with_scout_heartbeat_timeout metric

Type of Change

• Add - New feature or capability
• Change - Changes in existing functionality
• Fix - Bug fixes
• Remove - Removed features or deprecated functionality
• Internal - Internal changes (refactoring, tests, docs, etc.)

Related Issues (Optional)

Breaking Changes

• This PR contains breaking changes

Testing

• Unit tests added/updated
• Integration tests added/updated
• Manual testing performed
• No testing required (docs, internal refactor, etc.)

Additional Notes

[github-actions]

🔐 TruffleHog Secret Scan

✅ No secrets or credentials found!

Your code has been scanned for 700+ types of secrets and credentials. All clear! 🎉

🔗 View scan details

_{🕐 Last updated: 2026-03-11 14:11:52 UTC | Commit: 76f2032}

[github-actions]

🛡️ Vulnerability Scan

🚨 Found 74 vulnerability(ies)
📊 vs main: 74 (no change)

Severity Breakdown:

• 🔴 Critical/High: 74
• 🟡 Medium: 0
• 🔵 Low/Info: 0

🔗 View full details in Security tab

_{🕐 Last updated: 2026-03-11 14:12:04 UTC | Commit: 76f2032}