Description
We are using Trivy Operator with trivy.command=filesystem because we want to scan workload images from the node-local image cache and avoid pulling images from the registry during scan jobs.
Based on the Starboard filesystem scan design, our expectation is that the operator should schedule the scan job onto the same node where the target workload pod is running, and use imagePullPolicy: Never for the target image container.
However, in our cluster the generated scan-vulnerabilityreport-* pod is not pinned to the same node as the target workload pod. As a result, the scan pod may land on a different node where the target image is not present locally, and the main scan container fails with:
text ErrImageNeverPull Container image "" is not present with pull policy of Never
This makes filesystem mode unreliable unless all workload images are pre-pulled onto all nodes, which seems to defeat the intended purpose of this mode.
Configuration
Helm chart: trivy-operator-0.32.0
Relevant config:
yaml trivy.command: filesystem trivy.imagePullPolicy: IfNotPresent
Observed behavior
Generated scan pod:
yaml containers: - name: nginx image: nginx:1.27 imagePullPolicy: Never
Pod status:
yaml state: waiting: reason: ErrImageNeverPull message: Container image "nginx:1.27" is not present with pull policy of Never
The init containers using the Trivy image complete successfully. The failure is only on the main container created from the target workload image.
Why this looks wrong
In filesystem mode, the operator appears to rely on the target image already being present on the node. That is reasonable, but then the scan job must be scheduled onto the same node as the scanned workload pod. In our case this does not seem to happen.
Expected behavior
When trivy.command=filesystem is used, the operator should schedule the scan job onto the same node as the target workload pod, or otherwise guarantee access to the node-local image cache of the scanned workload image.
Actual behavior
The scan job is scheduled independently, can land on a different node, and fails with ErrImageNeverPull.
Question
Is this a regression / known limitation in current Trivy Operator versions?
Is there any supported workaround to make filesystem mode schedule scan jobs onto the same node as the target workload without requiring pre-pulling all workload images onto all nodes?
Environment
- Trivy Operator chart version: 0.32.0
- Trivy image tag: 0.69.1
- Kubernetes: EKS
- Scan mode: filesystem
Related issue
This looks similar to #1978, but the core problem here seems to be that the scan pod is not being constrained to the workload node.
Description
We are using Trivy Operator with trivy.command=filesystem because we want to scan workload images from the node-local image cache and avoid pulling images from the registry during scan jobs.
Based on the Starboard filesystem scan design, our expectation is that the operator should schedule the scan job onto the same node where the target workload pod is running, and use imagePullPolicy: Never for the target image container.
However, in our cluster the generated scan-vulnerabilityreport-* pod is not pinned to the same node as the target workload pod. As a result, the scan pod may land on a different node where the target image is not present locally, and the main scan container fails with:
text ErrImageNeverPull Container image "" is not present with pull policy of Never
This makes filesystem mode unreliable unless all workload images are pre-pulled onto all nodes, which seems to defeat the intended purpose of this mode.
Configuration
Helm chart: trivy-operator-0.32.0
Relevant config:
yaml trivy.command: filesystem trivy.imagePullPolicy: IfNotPresent
Observed behavior
Generated scan pod:
yaml containers: - name: nginx image: nginx:1.27 imagePullPolicy: Never
Pod status:
yaml state: waiting: reason: ErrImageNeverPull message: Container image "nginx:1.27" is not present with pull policy of Never
The init containers using the Trivy image complete successfully. The failure is only on the main container created from the target workload image.
Why this looks wrong
In filesystem mode, the operator appears to rely on the target image already being present on the node. That is reasonable, but then the scan job must be scheduled onto the same node as the scanned workload pod. In our case this does not seem to happen.
Expected behavior
When trivy.command=filesystem is used, the operator should schedule the scan job onto the same node as the target workload pod, or otherwise guarantee access to the node-local image cache of the scanned workload image.
Actual behavior
The scan job is scheduled independently, can land on a different node, and fails with ErrImageNeverPull.
Question
Is this a regression / known limitation in current Trivy Operator versions?
Is there any supported workaround to make filesystem mode schedule scan jobs onto the same node as the target workload without requiring pre-pulling all workload images onto all nodes?
Environment
Related issue
This looks similar to #1978, but the core problem here seems to be that the scan pod is not being constrained to the workload node.