ScanJobs fail in Trivy Operator Server Mode — all ScanJob pods in Error state, no vulnerabilities detected

[hichem-belhocine]

What steps did you take and what happened:

When running Trivy Operator in server mode, all ScanJob pods end up in an Error state and no vulnerability reports are generated. The operator keeps attempting to create ScanJobs, but each one fails immediately. However, when switching to standalone mode, vulnerability scanning works correctly and reports are created as expected.

What did you expect to happen:

• ScanJobs should run successfully.
• Trivy Operator should retrieve vulnerability results from the Trivy server.
• VulnerabilityReports should be created in the scanned namespaces.

Environment:

• Trivy-Operator version (use trivy-operator version): 0.31.0
• Kubernetes version (use kubectl version): v1.33.5

Values.yaml

global:
  image:
    registry: "private-registry"

operator:
  replicas: 2
  scanJobsConcurrentLimit: 5
  scanJobTTL: "30s"
  # We use Server mode  
  builtInTrivyServer: true

trivyOperator:
  scanJobTolerations:
  scanJobCustomVolumesMount:
    - name: trusted-ca
      mountPath: /etc/ssl/certs/trusted-ca-bundle.crt
      subPath: trusted-ca-bundle.crt
      readOnly: true
    - name: tmp
      mountPath: /tmp
  scanJobCustomVolumes:
    - name: trusted-ca
      secret:
        secretName: trusted-ca
    - emptyDir: {}
      name: tmp
  scanJobPodTemplatePodSecurityContext:
    runAsUser: 0
    runAsNonRoot: false
    seccompProfile:
      type: RuntimeDefault
  scanJobPodTemplateContainerSecurityContext:
    allowPrivilegeEscalation: false
    capabilities:
      drop:
        - ALL
    privileged: false
    readOnlyRootFilesystem: true
    # -- For filesystem scanning, Trivy needs to run as the root user
    # runAsUser: 0

trivy:
  mode: ClientServer # Standalone by default.
  storageSize: "30Gi"
  resources:
    requests:
      cpu: 10m
      memory: 100M
      # ephemeralStorage: "2Gi"
    limits:
      cpu: 500m
      memory: 1000M
      # ephemeralStorage: "2Gi"
  httpProxy: xxxxxxxxxxxxx
  httpsProxy: xxxxxxxxxxxxx
  noProxy: xxxxxxxxxxxxxx
  slow: true
  ignoreUnfixed: true
  dbRegistry: "private-registry"
  javaDbRegistry: "private-registry"
  command: image
  server:
    # -- resources set trivy-server resource
    resources:
      requests:
        cpu: 10m
        memory: 512Mi
        # ephemeral-storage: "2Gi"
      limits:
        cpu: 1
        memory: 1Gi
        # ephemeral-storage: "2Gi" 
    replicas: 2
    podSecurityContext:
      runAsUser: 65534
      runAsNonRoot: true
      fsGroup: 65534
      seccompProfile:
        type: RuntimeDefault
    securityContext:
      runAsNonRoot: true
      privileged: false
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      seccompProfile:
        type: RuntimeDefault
      capabilities:
        drop:
        - ALL 
    extraServerVolumes:
      volumeMounts:
        - name: trusted-ca
          mountPath: /etc/ssl/certs/trusted-ca-bundle.crt
          subPath: trusted-ca-bundle.crt
          readOnly: true
      volumes:
        - name: trusted-ca
          secret:
            secretName: trusted-ca

resources:
  limits:
    cpu: 500m
    memory: 2048Mi
  requests:
    cpu: 100m
    memory: 512Mi

podAnnotations:
  prometheus.io/port: '8080'
  prometheus.io/scrape: 'true'
  prometheus.io/honor_labels: "true"

podSecurityContext:
  runAsNonRoot: true
  seccompProfile:
    type: RuntimeDefault

# -- securityContext security context
securityContext:
  runAsNonRoot: true
  privileged: false
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop:
      - ALL
  seccompProfile:
    type: RuntimeDefault
volumeMounts:
  - name: trusted-ca
    mountPath: /etc/ssl/certs/trusted-ca-bundle.crt
    subPath: trusted-ca-bundle.crt
    readOnly: true
  - name: tmp
    mountPath: /tmp
volumes:
  - name: trusted-ca
    secret:
      secretName: trusted-ca
  - emptyDir: {}
    name: tmp
tolerations:
affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app.kubernetes.io/name: trivy-operator
        topologyKey: topology.kubernetes.io/zone
      - labelSelector:
          matchLabels:
            app.kubernetes.io/name: trivy-operator
        topologyKey: kubernetes.io/hostname
policiesBundle:
  registry: private-registry
nodeCollector:
  registry: private-registry

> k get po
NAME                                        READY   STATUS    RESTARTS   AGE
scan-vulnerabilityreport-66d6c58f66-8vtlm   1/2     Error     0          3s
scan-vulnerabilityreport-67dd5bcdbb-fpl7r   0/1     Error     0          33s
scan-vulnerabilityreport-698c55c4f9-hvbsr   0/1     Error     0          3s
scan-vulnerabilityreport-84656d9585-s2dqj   0/1     Error     0          33s
scan-vulnerabilityreport-86fbbf6f5-s6g2w    0/1     Error     0          3s
trivy-operator-55b44b8579-8g8kk             1/1     Running   0          15m
trivy-server-0                              1/1     Running   0          22m

trivy-operator pod logs has hundrends of this issue

{"level":"error","ts":"2025-11-17T14:33:50Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-59599f5f8d","container":"xxxxx","status.reason":"Error","status.message":"","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).completedContainers\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:441\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).SetupWithManager.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:103\nsigs.k8s.io/controller-runtime/pkg/reconcile.TypedFunc[...].Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/reconcile/reconcile.go:134\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:296"}

Removing the whole ns as suggested here #1325 (comment) does not help

[hichem-belhocine]

Now i can get vuln after adding trivy-service.trivy-system to no-proxy or removing the proxy

But some scan job pods still in Error state

> k get po -n trivy-system
NAME                                        READY   STATUS     RESTARTS   AGE
scan-vulnerabilityreport-5558d6f8fd-mb4zb   1/1     Running    0          3m22s
scan-vulnerabilityreport-5c589c967f-mdhxw   1/1     Running    0          3m17s
scan-vulnerabilityreport-64d9fc576b-ftcwc   1/4     Error      0          15s
scan-vulnerabilityreport-7bddbd8449-hhzhz   1/2     NotReady   0          76s
trivy-operator-54fd547589-h49lm             1/1     Running    0          4m15s
trivy-operator-54fd547589-xmwl9             1/1     Running    0          4m15s
trivy-server-0                              1/1     Running    0          4m18s
trivy-server-1                              1/1     Running    0          4m17s

trivy-operator pod logs still has of this error

{"level":"error","ts":"2025-11-17T14:33:50Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-59599f5f8d","container":"xxxxx","status.reason":"Error","status.message":"","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).completedContainers\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:441\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).SetupWithManager.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:103\nsigs.k8s.io/controller-runtime/pkg/reconcile.TypedFunc[...].Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/reconcile/reconcile.go:134\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:296"}

[daanschipper]

Running into the same issue. The empty status.message is due to not flushing the stdout before the container is exited, see #2825. Running a patched version shows the reason why it is failing:

{"level":"error","ts":"2025-12-01T13:31:27Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-8446dc7b5d","container":"dashboard","status.reason":"Error","status.message":"2025-12-01T13:31:23Z\tINFO\tLoaded\tfile_path=\"/etc/trivy/trivy-config.yaml\"\n2025-12-01T13:31:23Z\tINFO\tAdding schema version to the DB repository for backward compatibility\trepository=\"aquasec/trivy-java-db:1\"\n2025-12-01T13:31:23Z\tWARN\t\"--slow\" is deprecated. Use \"--parallel 1\" instead.\n2025-12-01T13:31:23Z\tINFO\t[vuln] Vulnerability scanning is enabled\n2025-12-01T13:31:24Z\tFATAL\tFatal error\trun error: image scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:e97c2cb5d4d99161ebc7ba9204decc914ffad16056e
6658182d220faf398e8a7): unable to get post analysis filesystem: unable to create temporary directory: stat /tmp/trivy-7: no such file or directory\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).completedContainers\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:441\ngithub.com/aquasecurity/trivy-operator/pkg/v
ulnerabilityreport/controller.(*ScanJobController).SetupWithManager.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:103\nsigs.k8s.io/controller-runtime/pkg/reconcile.TypedFunc[...].Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/reconcile/reconcile.go:134\nsigs.k8s.io/controller-runtime/pkg/internal/co
ntroller.(*Controller[...]).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).proc
essNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:296"}                                                                                                       

2025-12-01T13:35:25Z    FATAL    Fatal error    run error: image scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:e97c2cb5d4d99161ebc7ba9204decc914ffad16056e6658182d220faf398e8a7): walk error: failed to process the file: failed to analyze file: failed to copy file to temp: create temp error: open /tmp/trivy-7/analyzer-composite-1748982865/analyzer-file-78
7276398: no such file or directory                                                                                                                                                                                                                                                                                                                                                                                                       

[daanschipper]

I could replicate this with trivy-operator running in client server mode and a pod with two containers. One container was super quick to scan, the other more than a minute. When the first container of the scan job is completed the second scan job container errors. I patched it by using per container subpath within the /tmp directory daanschipper@1749113, I suspect the two parallel trivy runs do not behave nicely with the creation of the tmp directory from https://github.com/aquasecurity/trivy/blob/main/pkg/x/os/os.go#L23

[henkka]

We're having same issue, thank you @daanschipper for the analysis!

I believe the issue was introduced in aquasecurity/trivy#9241 which implemented process-safe temp file cleanup, but it doesn't work with the way Trivy Operator wants to run scans.

When scanning pod with multiple containers, Trivy Operator spawns simultaneous scan jobs sharing a single volume. The change in aquasecurity/trivy#9241 relies on os.GetPid() to generate unique temporary directory names. However, inside Kubernetes, containers have isolated PID namespaces. This means every scan process runs as PID 1.

Instead of unique directories, all parallel jobs unintentionally share the same path (e.g., /tmp/trivy-1). This creates a race condition where the first job to complete triggers Cleanup() and removes the directory, causing all other active jobs to fail.

ping @afdesk just so you're aware of the issue

[afdesk]

@henkka thanks!
I'll take a look today

[afdesk]

yes, it seems there is an issue here. trying to fix it