Skip to content

WIP: Add KEP-4680 device health reporting with message field support - #127

Draft
harche wants to merge 1 commit into
kubernetes-sigs:mainfrom
harche:healt_status
Draft

WIP: Add KEP-4680 device health reporting with message field support#127
harche wants to merge 1 commit into
kubernetes-sigs:mainfrom
harche:healt_status

Conversation

@harche

@harche harche commented Nov 11, 2025

Copy link
Copy Markdown
Contributor

Summary

Adds KEP-4680 device health reporting to the example driver, serving as the reference implementation for the version-neutral WatchHealthStatus API from the kubeletplugin helper (kubernetes/kubernetes#139477). No versioned DRAResourceHealth gRPC types appear in driver code; the helper serves v1 and v1alpha1, so allocated device health surfaces in pod.status.containerStatuses[].allocatedResourcesStatus on both new and 1.36-era kubelets.

The driver reports all devices healthy and re-sends the report periodically as the heartbeat the kubelet's health lease requires. For demos and e2e tests, annotating the plugin pod with health.example.com/<device>: Unhealthy|Healthy|Unknown overrides a device's simulated health (see demo/examples/health-reporting/).

go.mod temporarily pins the k8s.io staging modules to master pseudo-versions until WatchHealthStatus ships in a k8s.io/dynamic-resource-allocation release.

Verification

The e2e test deploys a pod with a claim, asserts Healthy in pod status, flips the device to Unhealthy via annotation, and asserts recovery back to Healthy. Verified on kind against both supported kubelet generations:

  • v1alpha1 path: released kindest/node:v1.36.1 (pre-v1 kubelet) — e2e passes via the helper's v1alpha1 compatibility serving.
  • v1 path: node image built from kubernetes master (v1.37.0-beta.0.446+4b1379a4a44) — e2e passes, and driver logs confirm the kubelet opened /v1.DRAResourceHealth/NodeWatchResources.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Nov 11, 2025
@k8s-ci-robot

Copy link
Copy Markdown
Contributor

Welcome @harche!

It looks like this is your first PR to kubernetes-sigs/dra-example-driver 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/dra-example-driver has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: harche
Once this PR has been reviewed and has the lgtm label, please assign klueska for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot
k8s-ci-robot requested review from byako and pohly November 11, 2025 17:23
@k8s-ci-robot k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Nov 11, 2025
@harche

harche commented Nov 11, 2025

Copy link
Copy Markdown
Contributor Author

/hold for the implementation PRs to merge,

kubernetes/enhancements#4680 (comment)

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 11, 2025
@nojnhuh nojnhuh moved this from 🆕 New to 🏗 In progress in Dynamic Resource Allocation Nov 13, 2025
@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 10, 2026
@pohly pohly moved this from 🏗 In progress to 👀 In review in Dynamic Resource Allocation Feb 20, 2026
@pohly pohly moved this from 👀 In review to 🏗 In progress in Dynamic Resource Allocation Mar 2, 2026
@harche

harche commented Mar 13, 2026

Copy link
Copy Markdown
Contributor Author

I will update once 1.36 beta is cut

@pohly pohly moved this from 🏗 In progress to 👀 In review in Dynamic Resource Allocation Mar 16, 2026
@pohly pohly moved this from 👀 In review to 🏗 In progress in Dynamic Resource Allocation Mar 16, 2026
@pohly pohly moved this from 🏗 In progress to 👀 In review in Dynamic Resource Allocation Apr 7, 2026
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 13, 2026
@harche

harche commented May 13, 2026

Copy link
Copy Markdown
Contributor Author

/cc @SergeyKanzhelev @nojnhuh

@harche

harche commented May 13, 2026

Copy link
Copy Markdown
Contributor Author

I will update once 1.36 beta is cut

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 13, 2026
@harche

harche commented May 13, 2026

Copy link
Copy Markdown
Contributor Author

/cc @SergeyKanzhelev @nojnhuh

/hold I am adding a feature to trigger the health status update in this example driver.

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 13, 2026
@harche
harche marked this pull request as draft May 14, 2026 16:13
@k8s-ci-robot k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 14, 2026
@harche
harche force-pushed the healt_status branch 2 times, most recently from e8b7bce to 79e477e Compare May 14, 2026 17:32
@harche

harche commented May 14, 2026

Copy link
Copy Markdown
Contributor Author

Putting this on hold until #201 merges. Both PRs modify the e2e test infrastructure (e2e_setup_test.go, e2e_test.go) and will conflict. Will rebase after #201 lands.

/hold

@pohly

pohly commented May 15, 2026

Copy link
Copy Markdown
Contributor

As mentioned before, I don't like that DRA drivers have to interact with the gRPC interface (drahealthv1alpha1 currently, despite the beta graduation of the feature?!). Can we first improve the kubeletplugin support for this feature (abstract from underlying gRPC interface similar to how it's done for the main DRA kubelet interace, better documentation) and only then make the DRA example driver use that improved support?

@pohly pohly moved this from 👀 In review to 📋 Backlog in Dynamic Resource Allocation Jun 2, 2026
@kubernetes-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: harche
Once this PR has been reviewed and has the lgtm label, please assign klueska for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow kubernetes-prow Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 6, 2026
@harche harche changed the title Add KEP-4680 device health reporting with message field support WIP: Add KEP-4680 device health reporting with message field support Jul 6, 2026
@kubernetes-prow kubernetes-prow Bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 6, 2026
@harche
harche force-pushed the healt_status branch 2 times, most recently from 19c069d to 42a8863 Compare July 6, 2026 19:20
@harche
harche force-pushed the healt_status branch 2 times, most recently from 11a364c to 0413d2c Compare July 6, 2026 20:11
@kubernetes-prow kubernetes-prow Bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jul 6, 2026
@kubernetes-prow kubernetes-prow Bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Jul 24, 2026
Implement the kubeletplugin WatchHealthStatus API with a simulated health
source (driven by pod annotations) so that allocated device health surfaces
in pod status, covered by an e2e test.

Signed-off-by: Harshal Patil <12152047+harche@users.noreply.github.com>
@harche

harche commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

I tested both DRAResourceHealth API generations end-to-end on kind by temporarily redirecting the k8s.io staging modules in go.mod to kubernetes master pseudo-versions (which contain the merged WatchHealthStatus API from kubernetes/kubernetes#139477):

  • v1alpha1 path: against a released kindest/node:v1.36.1 kubelet (pre-v1), the health e2e passes via the helper's v1alpha1 compatibility serving — initial Healthy, annotation-forced Unhealthy, and recovery back to Healthy all surface in pod.status.containerStatuses[].allocatedResourcesStatus.
  • v1 path: against a kubelet built from kubernetes master (v1.37.0-beta.0.446+4b1379a4a44), the same e2e passes, and the driver logs confirm the kubelet opened the v1 stream (/v1.DRAResourceHealth/NodeWatchResources) rather than falling back to v1alpha1.

The replace block in go.mod is temporary: once WatchHealthStatus ships in a tagged k8s.io/dynamic-resource-allocation release, I'll drop the redirect and bump to the release version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants