Skip to content

[system-health]: Honor 'psu' and 'pdb' device ignore independently in hardware checker#28415

Merged
yxieca merged 1 commit into
sonic-net:masterfrom
vvolam:vvolam/system-health-psu-pdb-ignore
Jul 21, 2026
Merged

[system-health]: Honor 'psu' and 'pdb' device ignore independently in hardware checker#28415
yxieca merged 1 commit into
sonic-net:masterfrom
vvolam:vvolam/system-health-psu-pdb-ignore

Conversation

@vvolam

@vvolam vvolam commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Why I did it

show system-health detail reports Hardware: Not OK with reason "Failed to get PSU information" on platforms that have no PSUs and whose system_health_monitoring_config.json lists psu in devices_to_ignore but not pdb — even though psu is shown as ignored.

This is a regression from #26829 ("Add PDB support for system-health"), which changed the _check_psu_status early-return guard to require both psu and pdb to be ignored. When only psu is ignored, the checker proceeds, finds an empty PSU_INFO, and reports the failure. The same change also silently re-enabled per-PSU checks on switches that ignore the psu category but not pdb.

fixes #28414

Work item tracking
  • Microsoft ADO (number only): 38803669

How I did it

In src/system-health/health_checker/hardware_checker.py, _check_psu_status now honors psu and pdb independently:

  • Skip the whole check only when both psu and pdb are ignored.
  • Report the empty-PSU_INFO "Failed to get PSU information" failure only when psu is not ignored (so a genuinely broken psud on a real switch is still detected).
  • Per row, skip PSU rows when psu is ignored and PDB rows (keyed PDB N) when pdb is ignored.

Added regression unit tests in src/system-health/tests/test_system_health.py.

How to verify it

  1. Unit tests:
    cd src/system-health && python3 -m pytest tests/test_system_health.py -k "hardware or psu or pdb or config"
    
  2. Manual: on a platform with no PSU_INFO in STATE_DB, set devices_to_ignore to include psu (and fan) but not pdb, restart the system-health service, and confirm show system-health detail reports Hardware: OK (previously Not OK — Failed to get PSU information).
  3. Confirm a platform that exposes PSUs still reports PSU failures when psu is not ignored, and that PDB rows are still evaluated when only psu is ignored.

Verified with both 202605 and master image. PSU is correctly ignored on a DPU, while it is correctly checked on the NPU.

Which release branch to backport (provide reason below if selected)

  • 202305
  • 202311
  • 202405
  • 202411
  • 202505
  • 202511
  • 202512
  • 202605
  • 202608

Reason: the regression (#26829) is present on master and 202605; the fix should follow it wherever #26829 is merged.

Tested branch (Please provide the tested image version)

  • Validated via the unit tests above on master; the regression was originally observed on 20260510.

Description for the changelog

[system-health] Fix false "Failed to get PSU information" when only 'psu' (not 'pdb') is in devices_to_ignore

Link to config_db schema for YANG module changes

N/A — no YANG/config_db schema changes.

A picture of a cute animal (not mandatory but encouraged)

🐾🐱

… hardware checker

The PDB-support change (sonic-net#26829) required BOTH 'psu' and 'pdb' to be in devices_to_ignore before _check_psu_status would skip. Platforms that ignore only 'psu' (e.g. DPUs with no PSU_INFO) therefore fell through to the empty-table branch and reported "Failed to get PSU information", flipping system-health Hardware to Not OK. It also silently re-enabled per-PSU checks on switches that ignore category 'psu' but not 'pdb'.

Honor 'psu' and 'pdb' independently: skip the whole check only when both are ignored; report the empty PSU_INFO failure only when 'psu' is not ignored; and per row, skip PSU rows when 'psu' is ignored and PDB rows when 'pdb' is ignored.

Signed-off-by: Vasundhara Volam <vvolam@microsoft.com>
Copilot AI review requested due to automatic review settings July 14, 2026 20:56
@vvolam
vvolam requested a review from lguohan as a code owner July 14, 2026 20:56
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run Azure.sonic-buildimage

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a regression in system-health where the hardware checker would incorrectly report “Failed to get PSU information” (and re-enable PSU checks) when devices_to_ignore contains psu but not pdb. The change refines how PSU vs PDB entries (both sourced from STATE_DB PSU_INFO) are filtered so that psu and pdb ignore behavior works independently and matches the PR/issue expectations.

Changes:

  • Update _check_psu_status to treat psu and pdb category ignores independently, while still skipping the entire PSU/PDB pass only when both categories are ignored.
  • Suppress the empty-PSU_INFO failure when psu is ignored (to support platforms with no PSUs), but keep it for platforms where PSU monitoring is expected.
  • Add regression unit tests covering the “ignore psu but not pdb” scenarios, including the empty-table case.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
src/system-health/health_checker/hardware_checker.py Refines PSU/PDB ignore logic so empty PSU_INFO and per-row processing honor psu/pdb independently.
src/system-health/tests/test_system_health.py Adds unit tests to prevent regressions for “ignore psu only” behaviors (empty PSU_INFO, and mixed PSU/PDB rows).

@vvolam
vvolam requested a review from saiarcot895 July 14, 2026 21:04
@saiarcot895
saiarcot895 requested a review from yuazhe July 14, 2026 22:27
@saiarcot895

Copy link
Copy Markdown
Contributor

@yuazhe please review as well.

@mssonicbld

Copy link
Copy Markdown
Collaborator

This PR has backport request for branch(es): 202605.
Added label(s) for branch(es) 202605.

---Powered by SONiC BuildBot

@vvolam
vvolam requested a review from judyjoseph July 21, 2026 17:14
@vvolam vvolam added the Tested for 202605 branch Tested for 202605 branch label Jul 21, 2026
@yxieca
yxieca merged commit 4e3f979 into sonic-net:master Jul 21, 2026
29 checks passed
@vvolam
vvolam deleted the vvolam/system-health-psu-pdb-ignore branch July 21, 2026 20:46
@mssonicbld

Copy link
Copy Markdown
Collaborator

Cherry-pick PR to 202605: #28616

@vaibhavhd

Copy link
Copy Markdown
Contributor

@vvolam approving cherry pick for 202605. Please:

  1. Add results of 202605 testing that you did. I understand the label is added, but please update the results here apart from the label.
  2. What are we doing to prevent such regressions in future?

@vvolam

vvolam commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

@vvolam approving cherry pick for 202605. Please:

  1. Add results of 202605 testing that you did. I understand the label is added, but please update the results here apart from the label.
  2. What are we doing to prevent such regressions in future?

@vaibhavhd Patch is already verified on a testbed with 202605 image and I have updated same in PR description as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: system-health reports Hardware "Not OK - Failed to get PSU information" when devices_to_ignore has 'psu' but not 'pdb'

6 participants