Is it possible to deploy the image with tag as well ?

[MageshSrinivasulu]

@hectorj2f I have one question. Generally, the policy controller looks for the digest to validate the image right?

In the documentation, there is an option to add additional details like a tag while signing

https://docs.sigstore.dev/cosign/other_types/#tag-signing.
https://github.com/sigstore/helm-charts/tree/main/charts/policy-controller#testing-the-webhook

So if we sign the image with additional details like a tag then we can deploy the image with tag details, right?

Why I am asking is I used to get the below error when I tried to deploy the image with tag where the policy controller only expects digest and throws an error like below

admission webhook "policy.sigstore.dev" denied the request: validation failed: invalid value: <IMAGE> must be an image digest: spec.template.spec.containers[0].image

Please let me know If my below three understandings are right and if there are only three possible scenarios. Let me know if any other scenario exists

1. When we deploy an image with digest details validation is done as usual
2. If we signed an image with tag details as well then we can deploy the image with a tag ( !!! I AM TRYING TO ACHIEVE THIS )
3. We cannot deploy an unsigned image with tag details because we will get the above error right?

Today I tried deploying the changes to achieve point 2 but got the same error requesting the digest. Do I need to do any special configuration for that?

[hectorj2f]

@MageshSrinivasulu As discussed over slack, this scenario works similarly to how cosign v2 does https://github.com/sigstore/cosign/pull/2650/files#diff-72911b6572099462e5f2e8ff920f4f9228587ab100820a26d3e90377c36bd73fR118. You can use a tag but if it doesn't resolve to a digest, you get a warning in cosign.

For the policy-controller, we enforce the usage of digests and reject any image tag that cannot get resolved.

[hectorj2f]

The other main reason to get the mentioned error could be due to problems pulling the secrets to access to that image. However if you are getting a similar error when using the cosign cli that is due to the image tag not getting resolved to a digest.

[MageshSrinivasulu]

@MageshSrinivasulu As discussed over slack, this scenario works similarly to how cosign v2 does https://github.com/sigstore/cosign/pull/2650/files#diff-72911b6572099462e5f2e8ff920f4f9228587ab100820a26d3e90377c36bd73fR118. You can use a tag but if it doesn't resolve to a digest, you get a warning in cosign.

For the policy-controller, we enforce the usage of digests and reject any image tag that cannot get resolved.

@hectorj2f What might be a possible reason for the resolution of tag --> digest to fail? Will the reason for failure will be logged? I suspect this might be the reason why I am facing that error.

[hectorj2f]

We're investigating the issue in two channels slack (https://sigstore.slack.com/archives/C01PZKDL4DP/p1675689427673029?thread_ts=1675658446.116119&cid=C01PZKDL4DP) and here. Perhaps we should move our conversation here.

[MageshSrinivasulu]

@hectorj2f. Sure let me consolidate the investigation done so far so that I might be useful to the people who might read this.

For enabling debug mode in the policy controller for further investigation update the below config map from info --> debug

https://github.com/sigstore/helm-charts/blob/main/charts/policy-controller/templates/webhook/configmap.yaml#L12

Respecting the possible levels, as listed here https://knative.dev/docs/serving/observability/logging/config-logging/

Once the debug is enabled. Able to see the logs stating tag to digest resolution is failing because of authentication

{"level":"debug","ts":"2023-02-06T12:38:35.046Z","logger":"policy-controller","caller":"webhook/validator.go:1009","msg":"Unable to resolve digest \"<IMAGE>\": GET https://<REPO>/oauth2/token?scope=repository%3Apull&service=<REPO>: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.","commit":"21c7eb0-dirty","[knative.dev/kind](http://knative.dev/kind)":"apps/v1, Kind=Deployment","[knative.dev/namespace](http://knative.dev/namespace)":"cosign-test","[knative.dev/name](http://knative.dev/name)":"webapp1","[knative.dev/operation](http://knative.dev/operation)":"CREATE","[knative.dev/resource](http://knative.dev/resource)":"apps/v1, Resource=deployments","[knative.dev/subresource](http://knative.dev/subresource)":"","[knative.dev/userinfo](http://knative.dev/userinfo)":"<USER>"}

The docker secrets are mentioned in below cluster image policies but still, policy controller is failing in the tag to digest the resolution step

apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
  creationTimestamp: "2022-12-23T05:15:00Z"
  finalizers:
  - clusterimagepolicies.policy.sigstore.dev
  generation: 1
  name: cluster-image-policy-acr
  resourceVersion: "1161425501"
  uid: 2160ab6e-dfe9-4895-95aa-e83ee8ba23c8
spec:
  authorities:
  - key:
      secretRef:
        name: cosigned-public-key
    name: authority-0
    source:
    - signaturePullSecrets:
      - name: acr-dev
  images:
  - glob: <REPO>.azurecr.io/**
  mode: warn

Understanding is policy-controller to use the same secret for both the below two steps

1. Tag to Digest conversion ( !!! But this is not working )
2. Using digest to validate the signature ( This part is working as expected )

[shani-e]

I am encountering similar issue with a private registry when deploying something to labeled namespace.
I receive the following error message:

admission webhook "policy.sigstore.dev" denied the request: validation failed: invalid value: <registry-domain>/shani/shani-playground/hello:v2 must be an image digest: spec.containers[0].image

When I check the logs of the webhook pod in debug mode, I see the following message:

{"level":"debug","ts":"2023-02-06T15:33:50.634Z","logger":"policy-controller","caller":"webhook/validator.go:1009","msg":"Unable to resolve digest \"<registry-domain>/shani/shani-playground/hello:v2\": GET https://registry-domain>/jwt/auth?scope=repository%3Ashani%2Fshani-playground%2Fhello%3Apull&service=container_registry: DENIED: access forbidden","commit":"21c7eb0-dirty","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"shani-temp","knative.dev/name":"shani1","knative.dev/operation":"CREATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"","knative.dev/userinfo":"kubernetes-admin"}

The log does not provide any further details or specify if it is an authentication issue.

I have a secret called regcred that stores the token for my private registry and is stored in the same namespace as the policy-controller.

I tried to provide the token in the CIP as described above, and I also tried to add the regcred secret to the ImagePullSecret section of the webhook SA and webhhok deployment YAML files, but none of these efforts have resolved the error.

Here is the code for the service account:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    meta.helm.sh/release-name: policy-controller
    meta.helm.sh/release-namespace: cosign-system
  creationTimestamp: "2023-02-06T13:32:48Z"
  labels:
    app.kubernetes.io/instance: policy-controller
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: policy-controller
    app.kubernetes.io/version: 0.5.2
    control-plane: policy-controller-webhook
    helm.sh/chart: policy-controller-0.3.9
  name: policy-controller-webhook
  namespace: cosign-system
  resourceVersion: "28985310"
  uid: b45ec1fb-e148-4aa6-8c92-74d5022f65fb
secrets:
- name: policy-controller-webhook-token-8f5lb
imagePullSecrets:
- name: regcred

[MageshSrinivasulu]

@hectorj2f Any update on how to proceed further?

[hectorj2f]

@MageshSrinivasulu have you tried to deploy the pod in namespace where label is not enabled and to use acr-dev secret in imagePullSecrets section of the pod?

[MageshSrinivasulu]

@hectorj2f Yes I was able to deploy the pod in the namespace in which the label is not enabled and using acr-dev on imagePullSecrets

[hectorj2f]

After attempting to trust immutable tags, we realized it is impossible to detect this type of tags. I am concerned of the security risks that can cause the usage of tags in the policy-controller. So I don't see a solution for this request at the moment.

[barthek]

@hectorj2f I understand the admission controller won't accept images without explicit mention of the digest, however I would think that the mutating webhook's job would be to resolve the tags and re-submit the request internally (now as image@digest). Is my understanding wrong?

[hectorj2f]

@barthek Sure, that is what it happens. However there are certain tags that cannot get resolved to a digest.
The mutating webhook attempts to resolve those tags consuming the imagePullSecrets from the pod spec.

[JRolfe-Gen]

@hectorj2f - Can you expand on which tags do not get resolved to a digest? It wouldn't seem right that you could pull the image by the tag, but not figure out the digest. Can you also expand on the security issues that you see using tags? Going tag > digest > signature would still always verify that image is signed... I could see that it would not verify which specific image is being used, But that is always the issue with using tags. In this case we just want to know that the image is signed. Even the cosign tool will follow the tag and verify the image is signed.

[hectorj2f]

@JRolfe-Gen There are multiple reasons why to reject any usage of tags even if immutable (which are difficult to detect).
When signing a tag, nothing tells you if a tag has changed once it has been signed. Moreover, registries can sometimes change the image themselves. As a consequence, we recommend to sign/verify digests which are immutable. To avoid exposing the security of the users, we decided to be strict and reject tags not resolvable into digests.

Note that our mutation webhook attempts resolving an image tag to a digest analogously how cosign library does it.