KEP-6060: API Server Authentication to Webhooks

[pmengelbert]

• One-line PR description: Add API Server Authentication to Webhooks KEP
• Issue link: API Server Authentication to Webhooks #6060

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: pmengelbert
Once this PR has been reviewed and has the lgtm label, please assign deads2k for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• keps/prod-readiness/OWNERS
• keps/sig-auth/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[liggitt]

Historically, tokens only identified, and authorization checks were done by whatever received the token. This is proposing doing the authorization check at token issuance time so the webhook merely has to look for a particular claim in the token?

If we're doing an authorization check on the service account at the time of issuance, would making the bound object and authorization check be related to the Service / ValidatingWebhookConfiguration / MutatingWebhookConfiguration / CustomResourceDefinition the token is going to be used against make more sense than an APIService? That would let us verify the requested audience was coherent with it as well.

[pmengelbert]

This is proposing doing the authorization check at token issuance time so the webhook merely has to look for a particular claim in the token?

Yes. In order to avoid the webhook talking back to kube-apiserver for TokenReview or SubjectAccessReview. This is all done to prevent aggregated API servers from probing policy information by asking questions about resources it does not control.

The purpose of using the APIService as the bound object is to make it possible for the Kubernetes API Server to check whether the webhook token acquisition service account has permission to talk to the webhook, and whether it may ask do so in order to ask questions about this APIService.

With the {Validating,Mutating}WebhookConfiguration as the bound object, I can't think of a way to perform the same authorization check as with the APIService, without the webhook having to call back to kube-apiserver. At token issuance, kube-apiserver does not know what question is being asked; it knows the audience and the bound object.

Because it is proposed that the audience be derived from the webhook config, the audience from the TokenRequest is used to correlate the APIService (from the bound object) and the webhook config corresponding to the audience. From there, it can be determined whether or not the webhook is even relevant to that APIService. Furthermore, kube-apiserver can check whether or not the token acquisition service account has the requisite permissions to ask questions about resources in that APIService. Without the bound APIService, kube-apiserver cannot do that check at credential issuance (it doesn't know the question being asked). We can't assume that the webhook author will have knowledge of which API Services it must support, so the webhook must instead answer the question (of whether or not the principal may ask questions about resources in a particular APIService) by making a SAR request to kube-apiserver. We want to avoid the webhook calling back to the Kubernetes API server for SAR and TokenReview, for obvious reasons.

[pmengelbert]

After discussing further with @enj, we propose the following:

Support 3 types of bindings:

• MutatingWebhookConfiguration
• ValidatingWebhookConfiguration
• APIService

When either of the webhook configurations is used as the bound object, the permissions required for token issuance are "attest" on "*", because in essence that is what is being permitted by such a token.

Tokens bound to APIService won't require such broad permission, and will be narrowly scoped to that APIService.

How does this strike you?

[liggitt]

why the unusual prefix? Isn't audience typically the host / URL where the token will be presented?

Is there a reason not to use https://$url/with/path or https://$servicename.$servicenamespace.svc:port/with/path as the audience? Webhooks already are required to have valid serving certificates for those hosts, so no new knowledge should be required to make the webhooks able to validate a similar audience in the token.

[pmengelbert]

The reason for this is that the webhook author may not know the URL where the webhook is deployed, and there will be more plumbing needed on the webhook side to make that information available to the token verification routine.

That said, we don't want this to be a sticking point, so we'll accept your suggestion if you don't agree with the above reasoning.

[liggitt]

Audience as a URL is more typical ... and since webhooks already are required to have valid serving certificates for their host, it seems more likely to me they can know or be told the host/URL more easily

[liggitt]

Does this mean a server that serves lots of different API groups/versions (like kube-apiserver) combined with a webhook that intercepts lots of different groups/versions (as a generic label protection or something) would need a distinct token for every API group/version type it sent to the webhook? That's not ideal.

[pmengelbert]

That is an unfortunate consequence.

In practice, aggregated API Servers will typically have one, and almost always fewer than five, relevant APIService objects. kube-apiserver is a distinct case (could have thousands of APIServices), and I share your concern about the explosion of tokens there.

We could permit kube-apiserver to use a "wildcard" APIService as the bound object. During bootstrapping, the requisite permissions (i.e. "attest" on "*" or an equivalent rule) would be set up for kube-apiserver's token acquisition service account. Then kube-apiserver only has to maintain one token per webhook, while aggregated API servers should be able to manage with one per webhook/apiservice combination.

[pmengelbert]

Note, rather than using the wildcard APIService, we think it's better to allow bindings to either an APIService or one of the two WebhookConfiguration types (see comment linked below):

#6156 (comment)

[liggitt]

A request to $group/v2 of an API can be converted to $group/v1 and sent to a webhook that asked to intercept $group/v1. See https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-matchpolicy

The object sent to the webhook would be in $group/v1, the kind and resource in the AdmissionReview would be $group/v1 and the requestKind and requestResource in the AdmissionReview would be $group/v2.

What APIService / version would you expect to be in the token sent to a webhook for that?

(I commented elsewhere about the implications of tying the token to a particular presented $group/$version ... we may want to tie it to the target webhook or service)

[pmengelbert]

In this case:

• Aggregated API Servers (likely) don't need to worry about this, since they don't serve custom resources. Any conversion (or request to convert) will have to be done by the aggregated API server itself, so it can just request the token post-conversion.
• kube-apiserver won't need to worry about this either, if it can request (as proposed in my comment above) tokens bound to one of the WebhookConfiguration types.

Alternatively, we can ignore the APIVersion and decide that we only care about the APIGroup. I don't think it's ever particularly relevant.

If we do decide we care about the version, then this is covered by the webhook: it will check whether or not the (APIGroup, APIVersion) match between those stated in the token and those from the AdmissionReview request body.

[liggitt]

Maybe it's just me, but I found defining new terms / acronyms for things we already have a bit confusing.

Also, be careful with language like "constrained" unless you really mean these tokens would be invalid to use as service account tokens in other ways because of these claims (which would make them sort of not normal service account tokens)

[pmengelbert]

Maybe it's just me, but I found defining new terms / acronyms for things we already have a bit confusing.

Not my intention to confuse. It helped with the writing to have those terms laid out, so that I have a consistent way to refer to a specific use of something in a particular context. It also helped with conciseness (ex. "token acquisition service account", vs "the service account token named in the TokenRequest for a token to authenticate to webhooks"). The latter is hard to work into more complex sentences.

Also, be careful with language like "constrained" [. . .]

Noted. I'll give this another pass.

[liggitt]

We don't want to have to mint a token for every webhook invocation, a token should be able to be retained and renewed/rotated after some percentage of its lifetime is used. I was expecting each client to only have to maintain one token per webhook, not one token per resource $group/$version per webhook.

[pmengelbert]

We don't want to have to mint a token for every webhook invocation, a token should be able to be retained and renewed/rotated after some percentage of its lifetime is used.

Yes, the tokens will be cached until their expiration, at which time they will be refreshed. We're in agreement on that. I'll make sure it gets mentioned here.

[liggitt]

We don't randomize built-in service account names for other things ... this seems more confusing than helpful

Rather than randomizing the kube-apiserver serviceaccount, let's just make sure we have workable user stories / examples / documentation / defaults for how:

1. aggregated servers configure their own distinct service account
2. permissions are granted to the server's service account (what does the server admin do, what does the webhook configuration author do, what knowledge does each of those actors have, how does that combine to route permissions to the right spots?)

[pmengelbert]

Agreed. I don't think we can really do anything to prevent people from abusing it, so there's no need to complicate things. I'll update the user stories to include these.

[liggitt]

if it's presenting a pod (Group: "", Version: "v1"), why is it getting a token bound to APIService v1.networking.k8s.io?

[pmengelbert]

Mistake, thanks for catching. I meant to use a different resource (to avoid the weirdness of having an empty string APIGroup in the example).

[liggitt]

similar comment about token reuse and wanting to maintain one reusable token per client per webhook, not per resource $group/$version

[pmengelbert]

This will be resolved if we are in agreement on this:

#6156 (comment)

[liggitt]

let's make sure it's possible to set up permissions correctly given the knowledge of these two personas:

• webhook author / webhook config manifest author who knows the things they are intercepting, but not server identities
• aggregated server admin who knows the identity they are giving the server, but not the webhooks that will need to be called by the server

[pmengelbert]

This is covered by:
#6156 (comment)

Basically, webhook config manifest authors can set up a service account with "attest" on the "*" APIService (or equivalent authz check); that allows TokenRequests for that service account to be bound to a MutatingWebhookConfiguration or a ValidatingWebhookConfiguration.

Aggregated API Server admins will set up a service account with "attest" on the relevant APIService (or equivalent).

[benjaminapetersen]

/hold

@pmengelbert lost me in the commit history 😄

[ahmedtd]

Nit --- this is doubled up "is in the format is in the format"

[pmengelbert]

Thanks. This and other things have since been updated. The version I just pushed is much closer to final.

[ahmedtd]

While this is a good default, I think there are probably setups where the webhook backend expects a different audience? It might need to be configurable in the Mutating/ValidatingAdmissionWebhook object.

In general, it's the relying party that's in control of which audience value(s) are expected.

[ahmedtd]

I would prefer if we give kube-apiserver one identity that it can use for all purposes (contacting webhooks, contacting peer replicas, uploading metrics, etc).

If that needs to be a service account, then that's fine, but it should probably be something like system:serviceaccount:kube-system:kube-apiserver.

[k8s-ci-robot]

@pmengelbert: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-enhancements-verify	a82528c	link	true	/test pull-enhancements-verify

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.