Treat kubelet NodeAffinity status.reason as retryable system error

[mhotan]

Why are the changes needed?

Identified a unique, fairly rare, edge case in certain cloud providers (e.g GKE) where the kubelet rejects the pod due to lacking NodeAffinity requirements prior the underlying controller manager applying the appropriate labels.

We would see workflows failing with

[...]: [...] currentAttempt done. Last Error: USER::Pod was rejected: Predicate NodeAffinity failed: node(s) didn't match Pod's node affinity/selector

What changes were proposed in this pull request?

Treat Pod failures with status.reason of NodeAffinity as a retryable system error.

How was this patch tested?

Unit tests.

This is important to improve the readability of release notes.

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

Summary by Bito

This pull request enhances the handling of pod failures in cloud environments by treating 'NodeAffinity' status as a retryable error, improving workflow reliability. It also includes new unit tests to verify this behavior.

Unit tests added: True

Estimated effort to review (1-5, lower is better): 2

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 58.48%. Comparing base (a75cea0) to head (840006a).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6461      +/-   ##
==========================================
- Coverage   58.48%   58.48%   -0.01%     
==========================================
  Files         940      940              
  Lines       71584    71584              
==========================================
- Hits        41867    41865       -2     
- Misses      26534    26536       +2     
  Partials     3183     3183              

Flag	Coverage Δ
unittests-datacatalog	59.03% <ø> (ø)
unittests-flyteadmin	56.23% <ø> (-0.03%)	⬇️
unittests-flytecopilot	30.99% <ø> (ø)
unittests-flytectl	64.78% <ø> (+0.05%)	⬆️
unittests-flyteidl	76.12% <ø> (ø)
unittests-flyteplugins	60.95% <100.00%> (ø)
unittests-flytepropeller	54.78% <ø> (ø)
unittests-flytestdlib	64.04% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[hamersaw]

Are we sure that the NodeAffinity reason is enough to determine if this should be retried? Worse case scenario is that we retry something that should not be (not all that bad), but was to gauge confidence here.

[mhotan]

Yeah, this was my concern as well. I have more confidence in that the node is marked as Failed with the following

status: {
message: "Pod was rejected: Predicate NodeAffinity failed: node(s) didn't match Pod's node affinity/selector"
phase: "Failed"
reason: "NodeAffinity"
startTime: "2025-05-16T04:53:42Z"
}
}

.

Given the specific nature of this status, I am assuming a small likelihood that we would have a prevalance of false positives causing us to retry needlessly.