feat: allow transform of reports before upload

r-oldenburg · r-oldenburg · commit 835a3b414b2b · 2026-03-09T16:26:48.000+01:00
diff --git a/Dockerfile b/Dockerfile
@@ -16,6 +16,10 @@ RUN groupadd --gid 1000 app && \
 
 COPY --from=build /app /app
 
+RUN apt-get update -y -qq && \
+    apt-get install -y -qq --no-install-recommends jq=1.7.1-6+deb13u1 kubectl=1.32.3+ds-2 curl=8.14.1-2+deb13u2 && \
+    apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
+
 COPY src/* /app/
 
 RUN chown -R app:app /app
diff --git a/README.md b/README.md
@@ -10,6 +10,7 @@ analysis and tracking.
 
 * Monitor Kubernetes for new Trivy vulnerability reports.
 * Push vulnerability reports to a configured Defect Dojo instance.
+* **Transformation Hook**: Intercept and modify reports (e.g., for custom deduplication or enrichment) before upload.
 * Seamless integration with your existing Kubernetes cluster and security workflow.
 * Developed using the Pythonic Kopf framework for easy maintenance and extensibility.
 
@@ -127,6 +128,11 @@ For a local development setup, please take a look at
 | `http_proxy`                             | *(empty)*                   | HTTP proxy setting (optional).                                                                                                                                                                 |
 | `https_proxy`                            | *(empty)*                   | HTTPS proxy setting (optional).                                                                                                                                                                |
 | `excludedNamespaces`                     | *(empty)*                   | List of namespace globs patterns to exclude from processing. Each pattern is converted into a --namespace=!<pattern> CLI argument passed to the operator Deployment. Reports from these namespaces will be ignored (optional). |
+| `transformation.enabled`                 | `false`                     | Enable the transformation hook to modify reports before upload.                                                                                                                                |
+| `transformation.scriptConfigMap`         | *(empty)*                   | Name of a ConfigMap containing the transformation script.                                                                                                                                     |
+| `transformation.scriptFilename`          | `transform.py`              | Filename of the script within the ConfigMap.                                                                                                                                                  |
+| `transformation.interpreter`             | `python3`                   | Command used to execute the script (e.g., `python3`, `bash`, `jq`).                                                                                                                            |
+| `transformation.scanType`                | `Generic Findings Import`   | The DefectDojo scanner type (parser) to use if the transformation is successful.                                                                                                            |
 
 ### A note on eval
 
@@ -141,6 +147,45 @@ evaluated and used as the engagement name.
 If you set defectDojoEngagementName to `body["report"]["artifact"]["tag"]`,
 then the engagement will get the name of the specified image-tag.
 
+## Transformation Hook
+
+The transformation hook allows you to manipulate the report data exactly as you need it before it reaches DefectDojo. Common use cases include:
+*   Custom deduplication logic.
+*   Enriching findings with specific labels from the scanned image.
+*   Filtering out specific findings.
+
+### How it works
+The operator serializes the raw Trivy report to a JSON string and pipes it into your script's `stdin`. Your script must write the final modified JSON to `stdout`.
+
+If the script exits with code `0`, the operator uses the modified content and switches the DefectDojo `scan_type` to the one configured in `transformation.scanType`. If the script fails, the operator automatically falls back to sending the original report with the default "Trivy Operator Scan" type.
+
+### Example: Using JQ
+To simply remove a specific field from the report, you can use `jq` as your interpreter:
+
+```yaml
+transformation:
+  enabled: true
+  interpreter: "jq"
+  scriptConfigMap: "jq-transform-cm"
+  scriptFilename: "filter.jq"
+  scanType: "Generic Findings Import"
+```
+
+### Example: Custom Shell Script
+If you need more complex logic, use a shell script:
+
+```bash
+#!/bin/bash
+# Read stdin into a variable or file
+REPORT=$(cat -)
+
+# Perform transformation (e.g., using jq)
+MODIFIED_REPORT=$(echo "$REPORT" | jq '.metrics += {"source": "trivy-operator"}')
+
+# Output back to stdout
+echo "$MODIFIED_REPORT"
+```
+
 ## Metrics
 
 The operator provides a Prometheus metrics endpoint(:9090/metrics), where
diff --git a/agents.md b/agents.md
@@ -0,0 +1,47 @@
+# Codebase Overview and Agents
+
+## Project Summary
+`trivy-dojo-report-operator` is a Kubernetes operator that automates the export of security reports from the Trivy Operator to DefectDojo. It is built using Python and the Kopf (Kubernetes Operator Pythonic Framework) framework.
+
+## Architecture
+
+The operator acts as a bridge between the Kubernetes cluster's security state (managed by Trivy) and the vulnerability management system (DefectDojo).
+
+### "Agents" (Handlers)
+
+The system consists of the following event handlers (agents) defined in `src/handlers.py`:
+
+1.  **Startup Agent (`configure`)**
+    -   **Type:** `@kopf.on.startup()`
+    -   **Responsibility:** Initializes the operator configuration, setting timeouts for watching resources to prevent connection drops, and configuring the persistence storage mechanism (`StatusDiffBaseStorage`) to handle large resource objects efficiently without overloading Kubernetes annotations.
+
+2.  **Report Forwarding Agent (`send_to_dojo`)**
+    -   **Type:** `@kopf.on.create(...)` (Dynamic Registration)
+    -   **Trigger:** Creation of Trivy-generated Custom Resources (CRs). The specific resources watched are configurable, but typically include:
+        -   `vulnerabilityreports.aquasecurity.github.io`
+        -   `configauditreports.aquasecurity.github.io`
+        -   `exposedsecretreports.aquasecurity.github.io`
+        -   `infraassessmentreports.aquasecurity.github.io`
+        -   `rbacassessmentreports.aquasecurity.github.io`
+    -   **Responsibility:**
+        -   **Extraction:** Extracts the full manifest of the created report.
+        -   **Transformation:** Converts the Kubernetes object into a JSON-compatible dictionary.
+        -   **Context Resolution:** dynamic evaluation of DefectDojo engagement parameters (Product Type, Product, Environment, Engagement Name) using the logic in `settings.py`.
+        -   **Transmission:** Uploads the report to the DefectDojo API (`/api/v2/import-scan/`) using the `requests` library.
+        -   **Observability:** Records Prometheus metrics, including processing time (`request_processing_seconds`) and request counters (`requests_total`).
+
+## Configuration
+
+The agents are configured via environment variables (loaded in `src/settings.py` and `src/env_vars.py`), controlling:
+
+-   **Connectivity:** DefectDojo URL (`DEFECT_DOJO_URL`) and API Key (`DEFECT_DOJO_API_KEY`).
+-   **Scope:** optional filtering by Kubernetes Label (`LABEL`, `LABEL_VALUE`).
+-   **Import Logic:**
+    -   `DEFECT_DOJO_ACTIVE`: Whether findings are marked as active.
+    -   `DEFECT_DOJO_VERIFIED`: Whether findings are marked as verified.
+    -   `DEFECT_DOJO_CLOSE_OLD_FINDINGS`: Whether to close old findings in DefectDojo.
+    -   `DEFECT_DOJO_PUSH_TO_JIRA`: Whether to push findings to Jira.
+
+## Deployment
+
+The operator is packaged as a Docker container and deployed via a Helm chart located in the `charts/` directory.
diff --git a/charts/templates/deployment.yaml b/charts/templates/deployment.yaml
@@ -101,7 +101,15 @@ spec:
           value: {{ quote .Values.operator.trivyDojoReportOperator.env.http_proxy }}
         - name: HTTPS_PROXY
           value: {{ quote .Values.operator.trivyDojoReportOperator.env.https_proxy }}
-        image: {{ .Values.operator.trivyDojoReportOperator.image.repository }}:{{ .Values.operator.trivyDojoReportOperator.image.tag | default .Chart.AppVersion }}
+        - name: TRANSFORMATION_ENABLED
+          value: {{ quote .Values.operator.trivyDojoReportOperator.transformation.enabled }}
+        - name: TRANSFORMATION_SCRIPT_PATH
+          value: "/scripts/{{ .Values.operator.trivyDojoReportOperator.transformation.scriptFilename }}"
+        - name: TRANSFORMATION_INTERPRETER
+          value: {{ quote .Values.operator.trivyDojoReportOperator.transformation.interpreter }}
+        - name: DEFECT_DOJO_SCAN_TYPE_OVERRIDE
+          value: {{ quote .Values.operator.trivyDojoReportOperator.transformation.scanType }}
+        image: {{ printf "%s:%s" .Values.operator.trivyDojoReportOperator.image.repository (.Values.operator.trivyDojoReportOperator.image.tag | default .Chart.AppVersion) | quote }}
         livenessProbe:
           httpGet:
             path: /healthz
@@ -121,20 +129,33 @@ spec:
         {{- end }}
         securityContext: {{- toYaml .Values.operator.trivyDojoReportOperator.containerSecurityContext
           | nindent 10 }}
-        {{- with .Values.operator.trivyDojoReportOperator.extraVolumeMounts }}
         volumeMounts:
+          - mountPath: /tmp
+            name: tmp
+          {{- if .Values.operator.trivyDojoReportOperator.transformation.scriptConfigMap }}
+          - mountPath: /scripts
+            name: transformation-scripts
+          {{- end }}
+          {{- with .Values.operator.trivyDojoReportOperator.extraVolumeMounts }}
           {{- toYaml . | nindent 10 }}
-        {{- end }}
+          {{- end }}
       {{- with .Values.operator.trivyDojoReportOperator.imagePullSecrets }}
       imagePullSecrets:
         {{- toYaml . | nindent 8 }}
       {{- end }}
       securityContext: {{- toYaml .Values.operator.podSecurityContext | nindent 8 }}
       # Additional volumes on the output Deployment definition.
-      {{- with .Values.operator.trivyDojoReportOperator.extraVolumes }}
       volumes:
+        - name: tmp
+          emptyDir: {}
+        {{- if .Values.operator.trivyDojoReportOperator.transformation.scriptConfigMap }}
+        - name: transformation-scripts
+          configMap:
+            name: {{ .Values.operator.trivyDojoReportOperator.transformation.scriptConfigMap }}
+        {{- end }}
+        {{- with .Values.operator.trivyDojoReportOperator.extraVolumes }}
         {{- toYaml . | nindent 8 }}
-      {{- end }}
+        {{- end }}
       serviceAccountName: {{ include "charts.fullname" . }}-account
       {{- with .Values.operator.trivyDojoReportOperator.nodeSelector }}
       nodeSelector:
diff --git a/charts/templates/rbac.yaml b/charts/templates/rbac.yaml
@@ -38,9 +38,29 @@ rules:
   - ""
   resources:
   - namespaces
+  - pods
+  - secrets
+  - replicationcontrollers
   verbs:
   - list
   - watch
+  - get
+- apiGroups:
+  - apps
+  resources:
+  - deployments
+  - statefulsets
+  - daemonsets
+  - replicasets
+  verbs:
+  - get
+- apiGroups:
+  - batch
+  resources:
+  - jobs
+  - cronjobs
+  verbs:
+  - get
 - apiGroups:
   - ""
   resources:
diff --git a/charts/values.yaml b/charts/values.yaml
@@ -39,16 +39,20 @@ operator:
       defectDojoCloseOldFindingsProductScope: "false"
       defectDojoDeduplicationOnEngagement: "true"
       defectDojoDoNotReactivate: "true"
-      defectDojoEngagementName: engagement
+      defectDojoEngagementName: "full_object.get('meta_engagement_name', 'Default Engagement')"
+      defectDojoTags: "[f\"base_image:{full_object.get('meta_base_image', 'unknown')}\", f\"component:{full_object.get('meta_engagement_name', 'unknown')}\"]"
       defectDojoEnvName: Development
-      defectDojoEvalEngagementName: "false"
+      defectDojoProductTypeName: "Research and Development"
+      defectDojoProductName: "BARMER CoreMedia"
+      defectDojoEvalEngagementName: "true"
       defectDojoEvalEnvName: "false"
       defectDojoEvalProductName: "false"
+      defectDojoServiceName: ""
+      defectDojoEvalServiceName: "false"
       defectDojoEvalProductTypeName: "false"
       defectDojoEvalTestTitle: "false"
+      defectDojoEvalTags: "true"
       defectDojoMinimumSeverity: Info
-      defectDojoProductName: product
-      defectDojoProductTypeName: Research and Development
       defectDojoPushToJira: "false"
       defectDojoTestTitle: Kubernetes
       defectDojoVerified: "false"
@@ -59,6 +63,12 @@ operator:
       repository: ghcr.io/telekom-mms/docker-trivy-dojo-operator
       tag: 0.9.2
     imagePullSecrets: []
+    transformation:
+      enabled: true
+      scriptConfigMap: "trivy-dojo-transform-script"
+      scriptFilename: "wrapper.sh"
+      interpreter: "bash"
+      scanType: "Generic Findings Import"
   type: ClusterIP
   podSecurityContext:
     runAsNonRoot: true
diff --git a/src/handlers.py b/src/handlers.py
@@ -3,6 +3,8 @@
 import prometheus_client
 import requests
 import settings
+import subprocess
+import os
 
 from requests.exceptions import HTTPError
 from io import BytesIO
@@ -36,6 +38,63 @@ def check_allowed_reports(report: str):
         exit(1)
 
 
+def run_transformation(raw_report: dict, logger) -> dict | None:
+    """
+    Runs the transformation script on the raw report using piping (stdin/stdout).
+    Returns the transformed report as a dict or None if transformation failed.
+    """
+    try:
+        # 1. Serialize Input
+        input_data = json.dumps(raw_report)
+
+        # 2. Execute Script via pipe
+        cmd = [
+            settings.TRANSFORMATION_INTERPRETER,
+            settings.TRANSFORMATION_SCRIPT_PATH,
+        ]
+        logger.info(f"Executing transformation: {' '.join(cmd)}")
+
+        result = subprocess.run(
+            cmd,
+            input=input_data,
+            capture_output=True,
+            text=True,
+            env=os.environ,
+            check=True,
+        )
+
+        # 3. Parse Output
+        if not result.stdout.strip():
+            logger.error("Transformation script returned empty stdout")
+            if result.stderr:
+                logger.error(f"Transformation stderr: {result.stderr}")
+            return None
+
+        transformed_data = json.loads(result.stdout)
+        if result.stderr:
+            for line in result.stderr.splitlines():
+                logger.info(f"Transformation: {line}")
+        return transformed_data
+
+    except subprocess.CalledProcessError as e:
+        logger.error(f"Transformation script failed with exit code {e.returncode}")
+        if e.stderr:
+            logger.error(f"Transformation stderr: {e.stderr}")
+        if e.stdout:
+            logger.error(f"Transformation stdout (partial): {e.stdout[:500]}")
+        return None
+    except json.JSONDecodeError as e:
+        logger.error(f"Failed to parse transformation output as JSON: {e}")
+        if result.stdout:
+            logger.error(f"Raw output (partial): {result.stdout[:500]}...")
+        if result.stderr:
+            logger.error(f"Transformation stderr: {result.stderr}")
+        return None
+    except Exception as e:
+        logger.error(f"Error during transformation: {e}")
+        return None
+
+
 @kopf.on.startup()
 def configure(settings: kopf.OperatorSettings, **_):
     """
@@ -95,6 +154,17 @@ def send_to_dojo(body, meta, logger, **_):
 
         logger.debug(full_object)
 
+        scan_type = "Trivy Operator Scan"
+        if settings.TRANSFORMATION_ENABLED:
+            logger.info("Transformation Hook is enabled")
+            transformed_object = run_transformation(full_object, logger)
+            if transformed_object:
+                logger.info("Transformation successful")
+                full_object = transformed_object
+                scan_type = settings.DEFECT_DOJO_SCAN_TYPE_OVERRIDE
+            else:
+                logger.warning("Transformation failed, falling back to raw report")
+
         _DEFECT_DOJO_ENGAGEMENT_NAME = (
             eval(settings.DEFECT_DOJO_ENGAGEMENT_NAME)
             if settings.DEFECT_DOJO_EVAL_ENGAGEMENT_NAME
@@ -130,6 +200,16 @@ def send_to_dojo(body, meta, logger, **_):
             else settings.DEFECT_DOJO_TEST_TITLE
         )
 
+        _DEFECT_DOJO_TAGS = (
+            eval(settings.DEFECT_DOJO_TAGS)
+            if settings.DEFECT_DOJO_EVAL_TAGS
+            else (settings.DEFECT_DOJO_TAGS.split(",") if settings.DEFECT_DOJO_TAGS else [])
+        )
+
+        logger.info(f"DefectDojo Config - Engagement: {_DEFECT_DOJO_ENGAGEMENT_NAME}, Test: {_DEFECT_DOJO_TEST_TITLE}, Service: {_DEFECT_DOJO_SERVICE_NAME}")
+        logger.info(f"Transformation Metadata - base_image: {full_object.get('meta_base_image')}, tag: {full_object.get('meta_tag')}")
+
+
         # define the vulnerabilityreport as a json-file so DD accepts it
         json_string: str = json.dumps(full_object)
         json_file: BytesIO = BytesIO(json_string.encode("utf-8"))
@@ -149,14 +229,15 @@ def send_to_dojo(body, meta, logger, **_):
             "minimum_severity": settings.DEFECT_DOJO_MINIMUM_SEVERITY,
             "auto_create_context": settings.DEFECT_DOJO_AUTO_CREATE_CONTEXT,
             "deduplication_on_engagement": settings.DEFECT_DOJO_DEDUPLICATION_ON_ENGAGEMENT,
-            "scan_type": "Trivy Operator Scan",
+            "scan_type": scan_type,
             "engagement_name": _DEFECT_DOJO_ENGAGEMENT_NAME,
             "product_name": _DEFECT_DOJO_PRODUCT_NAME,
             "product_type_name": _DEFECT_DOJO_PRODUCT_TYPE_NAME,
             "service": _DEFECT_DOJO_SERVICE_NAME,
             "environment": _DEFECT_DOJO_ENV_NAME,
             "test_title": _DEFECT_DOJO_TEST_TITLE,
             "do_not_reactivate": settings.DEFECT_DOJO_DO_NOT_REACTIVATE,
+            "tags": _DEFECT_DOJO_TAGS,
         }
 
         logger.debug(data)
diff --git a/src/settings.py b/src/settings.py
@@ -50,6 +50,12 @@
 DEFECT_DOJO_TEST_TITLE: str = os.getenv("DEFECT_DOJO_TEST_TITLE", "Kubernetes")
 DEFECT_DOJO_EVAL_TEST_TITLE: bool = get_env_var_bool("DEFECT_DOJO_EVAL_TEST_TITLE")
 
+DEFECT_DOJO_VERSION: str = os.getenv("DEFECT_DOJO_VERSION", "")
+DEFECT_DOJO_EVAL_VERSION: bool = get_env_var_bool("DEFECT_DOJO_EVAL_VERSION")
+
+DEFECT_DOJO_TAGS: str = os.getenv("DEFECT_DOJO_TAGS", "")
+DEFECT_DOJO_EVAL_TAGS: bool = get_env_var_bool("DEFECT_DOJO_EVAL_TAGS")
+
 DEFECT_DOJO_ENGAGEMENT_NAME: str | None = os.getenv("DEFECT_DOJO_ENGAGEMENT_NAME")
 DEFECT_DOJO_EVAL_ENGAGEMENT_NAME: bool = get_env_var_bool(
     "DEFECT_DOJO_EVAL_ENGAGEMENT_NAME"
@@ -66,3 +72,12 @@
 
 HTTP_PROXY: str = os.getenv("HTTP_PROXY") or os.getenv("http_proxy")
 HTTPS_PROXY: str = os.getenv("HTTPS_PROXY") or os.getenv("https_proxy")
+
+TRANSFORMATION_ENABLED: bool = get_env_var_bool("TRANSFORMATION_ENABLED")
+TRANSFORMATION_SCRIPT_PATH: str = os.getenv(
+    "TRANSFORMATION_SCRIPT_PATH", "/scripts/transform.py"
+)
+TRANSFORMATION_INTERPRETER: str = os.getenv("TRANSFORMATION_INTERPRETER", "python3")
+DEFECT_DOJO_SCAN_TYPE_OVERRIDE: str = os.getenv(
+    "DEFECT_DOJO_SCAN_TYPE_OVERRIDE", "Generic Findings Import"
+)
