WIP: simple discovery server #17818

justinsb · 2025-12-15T17:16:46Z

No description provided.

k8s-ci-robot · 2025-12-15T17:16:48Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

k8s-ci-robot · 2025-12-15T17:16:52Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign justinsb for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ameukam · 2025-12-15T19:19:50Z

discovery/cmd/discovery-server/main.go

+		MinVersion: tls.VersionTLS12,
+	}
+
+	server := &http.Server{


http3 ? I had white wine before

Sorry, I don't get it ... would you like to use http3? Have I opted in to http3 automatically? Do you want us to be the "guinea pig" for http3 in kube (if so, I'm game!)

I don't think there's anything special we need from http3. We do need the client certificate information. I was thinking we would probably end up deploying this directly behind an L4 load balancer, or (failing that) using ingress or gateway with SNI routing.

In terms of backends, right now I have this with a simple in-memory implementation. Honestly that's probably good enough to get started, as we will not be offering any guarantee as to retention of these objects.

But ... if we wanted to do better, I think we should put them into etcd because (1) we should be able to run etcd pretty cheaply and we don't have to worry about wracking up a huge GCS bill if someone figures out how to make us send queries to GCS etc and (2) it means that we can use etcd-operator, which would be good from the "all the wood behind one arrow" perspective

I was half-joking about using http3 for the discovery server but looks like the OIDC protocol is only compatible with HTTP 1.1.

I think we can support http3, but let's start with whatever go gives us out of the box (which I think is still http1 or http2)

I do think a controversial one would be to support DNS over HTTP, if you're feeling spicy :-)

justinsb · 2025-12-16T12:23:55Z

discovery/docs/kubeconfig.md

@@ -0,0 +1,82 @@
+# Using kubectl with Discovery Service
+
+Since the Discovery Service now emulates the Kubernetes API for `DiscoveryEndpoint` resources, you can use `kubectl` to interact with it.


I'm going to ask gemini to clean up these instructions / demo scripts.

Cleaned up!

justinsb · 2025-12-16T12:24:31Z

discovery/pkg/discovery/k8s_types.go

+}
+
+// DiscoveryEndpoint represents a registered client in the discovery service.
+type DiscoveryEndpoint struct {


I should move this to apis/discovery.kops.k8s.io/v1alpha1 for consistency (and probably change the version to v1alpha1)

justinsb · 2025-12-16T12:25:51Z

discovery/pkg/discovery/server.go

+			{
+				Name:         "discoveryendpoints",
+				SingularName: "discoveryendpoint",
+				Namespaced:   true,


I think this probably should be cluster scoped, although I guess we could use the namespace to indicate the cluster if we we wanted to allow multiple clusters to share the same CA (which isn't a terrible idea if someone is doing multicluster).

Do we consider RBAC in the equation ?

So this isn't technically kube-apiserver, and I haven't implemented RBAC.

Right now we have this: anyone that has any cert signed by a CA can read the objects for that CA's universe (defined by the hash of the CA public key). You can write an object that matches your own CN only.

I probably need to build out the client side here to better understand what we actually need, whether it's acceptable to have the same CA certificate etc. (e.g. maybe we should only let kubelet certificates register, or maybe we should only let control plane nodes register, or maybe we should create a dedicated CA only for discovery)

justinsb · 2025-12-16T12:27:16Z

discovery/go.mod

+
+require (
+	k8s.io/apimachinery v0.34.3
+	k8s.io/client-go v0.34.3


Technically client-go / apimachinery is only used by the clients / tests, so it might be nice to split them out. But to do that would require a separate go.mod, which is a bit of a pain.

ameukam · 2025-12-16T20:39:32Z

discovery/pkg/discovery/server.go

+	return s
+}
+
+func (s *Server) registerRoutes() {


so we never delete endpoints ?

Not currently, no. You're right - I should add a TTL. (Maybe 2 hours, and then we can have nodes register every hour?). It won't be a "hard" TTL - we reserve the rights to remove objects at any time (and I should add that to the README.md / GEMINI.md)

I should probably also add explicit deletion support.

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 15, 2025

k8s-ci-robot requested review from johngmyers and zetaab December 15, 2025 17:16

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Dec 15, 2025

justinsb force-pushed the discovery branch 2 times, most recently from 8396a06 to eead354 Compare December 15, 2025 18:52

ameukam reviewed Dec 15, 2025

View reviewed changes

justinsb force-pushed the discovery branch from eead354 to 4e75f10 Compare December 16, 2025 12:23

justinsb commented Dec 16, 2025

View reviewed changes

justinsb force-pushed the discovery branch 3 times, most recently from 62736c6 to 2136f14 Compare December 16, 2025 14:22

spike: discovery server

294dd35

justinsb force-pushed the discovery branch from 2136f14 to 294dd35 Compare December 16, 2025 18:00

ameukam reviewed Dec 16, 2025

View reviewed changes

		@@ -0,0 +1,82 @@
		# Using kubectl with Discovery Service

		Since the Discovery Service now emulates the Kubernetes API for `DiscoveryEndpoint` resources, you can use `kubectl` to interact with it.

WIP: simple discovery server #17818

Are you sure you want to change the base?

WIP: simple discovery server #17818

Uh oh!

Conversation

justinsb commented Dec 15, 2025

Uh oh!

k8s-ci-robot commented Dec 15, 2025

Uh oh!

k8s-ci-robot commented Dec 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants