images-are-certified flags non-certified images coming from a chart's tests

[komish]

Chart Verifier should consider whether or not a chart's test images can be not certified, vs. the application's images itself.

An example chart would be the dotnet imagestream chart, merged here openshift-helm-charts/charts#1653

This chart only installs imagestreams in to the cluster. In collaborating with this maintainer, we identified a path that would allow this maintainer to leverage those imagestreams in pods to test that the imagestream's installation works as expected. This does work (e.g. helm test of a helm install here works), but the images-are-certified check flags all of the test pods as having uncertified images because the image manifests are not fully qualified, and therefore cannot be certified "as they are written in manifest".

Here's a smallerized copy of the report against that chart

apiversion: v1
kind: verify-report
metadata:
    tool:
        verifier-version: 1.13.8
        # ... truncated ...
    chart:
        name: redhat-dotnet-imagestreams
        # ... truncated ...
results:
    - check: v1.0/is-helm-v3
      type: Mandatory
      outcome: PASS
      reason: API version is V2, used in Helm 3
    - check: v1.1/images-are-certified
      type: Mandatory
      outcome: FAIL
      reason: |-
        Image is not Red Hat certified : dotnet-runtime:latest : repository not found: dotnet-runtime
        Image is not Red Hat certified : dotnet-runtime:latest
        Image is not Red Hat certified : dotnet:6.0 : repository not found: dotnet
        Image is not Red Hat certified : dotnet:6.0
        Image is not Red Hat certified : dotnet:8.0 : repository not found: dotnet
        Image is not Red Hat certified : dotnet:8.0
        Image is not Red Hat certified : dotnet-runtime:9.0 : repository not found: dotnet-runtime
        Image is not Red Hat certified : dotnet-runtime:9.0
        Image is not Red Hat certified : dotnet-runtime:9.0-ubi8 : repository not found: dotnet-runtime
        Image is not Red Hat certified : dotnet-runtime:9.0-ubi8
        Image is not Red Hat certified : dotnet:6.0-ubi8 : repository not found: dotnet
        Image is not Red Hat certified : dotnet:6.0-ubi8
        Image is not Red Hat certified : dotnet:8.0-ubi8 : repository not found: dotnet
        Image is not Red Hat certified : dotnet:8.0-ubi8
        Image is not Red Hat certified : dotnet:9.0-ubi8 : repository not found: dotnet
        Image is not Red Hat certified : dotnet:9.0-ubi8
        Image is not Red Hat certified : dotnet-runtime:6.0-ubi8 : repository not found: dotnet-runtime
        Image is not Red Hat certified : dotnet-runtime:6.0-ubi8
        Image is not Red Hat certified : dotnet:9.0 : repository not found: dotnet
        Image is not Red Hat certified : dotnet:9.0
        Image is not Red Hat certified : dotnet:latest : repository not found: dotnet
        Image is not Red Hat certified : dotnet:latest
        Image is not Red Hat certified : dotnet-runtime:6.0 : repository not found: dotnet-runtime
        Image is not Red Hat certified : dotnet-runtime:6.0
        Image is not Red Hat certified : dotnet-runtime:8.0 : repository not found: dotnet-runtime
        Image is not Red Hat certified : dotnet-runtime:8.0
        Image is not Red Hat certified : dotnet-runtime:8.0-ubi8 : repository not found: dotnet-runtime
        Image is not Red Hat certified : dotnet-runtime:8.0-ubi8
    - check: v1.0/has-readme
      type: Mandatory
      outcome: PASS
      reason: Chart has a README
    - check: v1.0/not-contains-crds
      type: Mandatory
      outcome: PASS
      reason: Chart does not contain CRDs
    - check: v1.1/has-kubeversion
      type: Mandatory
      outcome: PASS
      reason: Kubernetes version specified
    - check: v1.0/required-annotations-present
      type: Mandatory
      outcome: PASS
      reason: All required annotations present
    - check: v1.0/helm-lint
      # ... truncated ...

Having non-fully qualified names here is expected, as the expectation is that these image references will be swapped out to use the imagestreams when applied, but you won't see that when manifests are rendered.

I think it may be reasonable to exclude the image references in helm chart tests from the images-are-certified check.

[komish]

cc @tmds

[mgoerens]

@komish

I'm trying to make sense of the work that needs to be done here. I'm not extremely familiar with ImageStreams so:

• I understand that an ImageStream is used to create a reference to another image
• Pods can use that stable reference instead of a potentially moving image / tag.
• Issue is that we check if the reference is certified, which doesn't make sense and fail

So we want to detect when a Pod uses an image/reference that is provided by an ImageStream and we should ignore it when it comes to checking if it's certified.

Is my understanding somehow correct ? Is there more to this ?

[komish]

It's simpler. We just need to consider if images referenced in a chart's tests should be checked for certification status.

The ImageStream example here is just a manifestation of the problem. The developer is providing an ImageStream chart, and to test that it's applied correctly, their test chart tries to use the ImageStream. To use the ImageStream, the developer is using the stubbed image references, along with a special annotation on the workload to allow its image to be substituted at application time with that of the ImageStream

    "alpha.image.policy.openshift.io/resolve-names": '*'

The resolution doesn't happen until application of the chart onto the cluster, but chart-verifier is looking at the image references in manifests pre-application, so it sees unqualified image references and throw the images-are-certified failure.

Here's the chart: https://github.com/openshift-helm-charts/charts/releases/tag/redhat-redhat-dotnet-imagestreams-0.0.2

To summarize, the ImageStream example is just a manifestation, but the issue is simply that it would seem images in tests are counting against the images-are-certified check, and we might want to exclude those (or determine valid reasons why it should stay the way it is implemented. I haven't thought through that much yet).