Support private container registries for Pods which do not specifying image pull secrets

[avdhoot]

What steps did you take and what happened:

Followed install instructions. Tried to run scan using below on command.

$ kubectl starboard  find vulnerabilities deployment/XXX-qa-web -n XXXX-qa -v 3
I0814 12:52:38.530373   28112 scanner.go:56] Getting Pod template for workload: {Deployment XXXX-qa-web XXXX-qa}
I0814 12:52:40.842053   28112 scanner.go:71] Scanning with options: {ScanJobTimeout:0s DeleteScanJob:true}
I0814 12:52:41.183767   28112 runner.go:79] Running task and waiting forever
I0814 12:52:41.183840   28112 runnable_job.go:47] Creating runnable job: starboard/b75ba5e8-82c9-4915-ad35-4b35c37987ab
I0814 12:52:41.535929   28112 reflector.go:207] Starting reflector *v1.Job (30m0s) from pkg/mod/k8s.io/client-go@v0.19.0-alpha.3/tools/cache/reflector.go:156
I0814 12:52:41.535978   28112 reflector.go:243] Listing and watching *v1.Job from pkg/mod/k8s.io/client-go@v0.19.0-alpha.3/tools/cache/reflector.go:156
I0814 12:52:50.479003   28112 runnable_job.go:73] Stopping runnable job on task failure with status: Failed
I0814 12:52:50.479115   28112 runner.go:83] Stopping runner on task completion with error: job failed: BackoffLimitExceeded: Job has reached the specified backoff limit
E0814 12:52:52.784556   28112 manager.go:177] Container default terminated with Error: 2020-08-14T07:22:49.629Z FATAL   unable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:
        * unable to inspect the image (us.gcr.io/XXXX-1/XXXX:116579-23d73da-release-2019-10): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
        * GET https://us.gcr.io/v2/token?scope=repository%3xxxl-1%2FXXXX%3Apull&service=us.gcr.io: UNKNOWN: Unable to parse json key.


error: running scan job: job failed: BackoffLimitExceeded: Job has reached the specified backoff limit

What did you expect to happen:
Scan should completed without error

Anything else you would like to add:
trivy support gcr. But I am not to able find way to pass custom ENV to trivy using starboard.

Environment:

• Starboard version (use starboard version): Starboard Version: {Version:0.2.6 Commit:d43faefc56021ae55d4574054ce7de13175ca206 Date:2020-07-09T20:30:45Z}
• Kubernetes version (use kubectl version): client:v1.17.10, server: v1.17.2
• OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc): Ubuntu 18.04

[avdhoot]

@danielpacak In case you missed it ^^

[danielpacak]

Hi @avdhoot ! Thank you for reporting this issue. I'm going to follow up on that and link it to a parent story or epic about configuring scanners. In your case the root cause is that we currently cannot pass TRIVY_USERNAME or TRIVY_PASSWORD envs, but there're more config options that users might want to pass to a scanner.

[danielpacak]

@avdhoot Actually starboard can use image pull secret attached to the Pod template or the service account. In your case did you set the secret on the Pod template level or at the service account level?

[avdhoot]

Our cluster created using kops. We seeded it with registry secrets using below command. We do not have image_pull secret attached to pod. @danielpacak
kops create secret dockerconfig -f ~/.docker/config.json

[danielpacak]

Thank you for providing all the details @avdhoot So just to rephrase and confirm my understanding:

You neither use image pull Secret set at Pod template not Service Account. You're using a feature¹ of kops to store the config.json in /root/.docker/config.json on all nodes (including masters) so that both Kubernetes and system containers may use registries defined in it.

[avdhoot]

Thanks @danielpacak you are understanding is correct.