Release v0.5.5

Author: adonm

ISSUES.md

@@ -6,19 +6,19 @@
 
 | # | Severity | Title | Status | Reference |
 |---|---|---|---|---|
-| 1 | High | SQL injection via Helm value `mysql.table` in SQL template | **Accepted** | `chart/harvest.sql:243`, `chart/templates/configmap.yaml:10` |
-| 2 | High | Missing Secret template `mariadb-credentials` — chart broken | **Regressed** | `chart/templates/statefulset.yaml:39,47,52`; `cronjob.yaml:44,49` |
-| 3 | Medium | Default weak MariaDB credentials (`harvest`/`harvest`) | **New** | `chart/values.yaml:9-11` |
+| 1 | High | SQL injection via Helm value `mysql.table` in SQL template | **Accepted** | `chart/harvest.sql`, `chart/templates/configmap.yaml` |
+| 2 | High | Missing Secret template `mariadb-credentials` — chart broken | **Resolved** — `chart/templates/secret.yaml` present | `chart/templates/secret.yaml` |
+| 3 | Medium | Default weak MariaDB credentials (`harvest`/`harvest`) | **Documented** — override for production | `README.md`, `chart/values.yaml:9-11` |
 
-> All previously reported issues #2–9 from ISFSMS.md remain resolved; only the Secret template (#1 originally) has regressed.
+> Previously reported hardening issues remain resolved. Current accepted/documented risks are trusted Helm values and development-only default credentials.
 
 ## Detailed findings
 
 ### 1. SQL injection via Helm template `mysql.table` value (High)
 
 **Category:** V5 Validation — Injection (CWE-89)  
 **Trust boundary:** Helm values supplied at `helm install` / `--set` or via `values.yaml` (operator-controlled, potentially from CI or external configs).  
-**Sink:** `chart/harvest.sql` line 243, rendered through `tpl` in `chart/templates/configmap.yaml:10`.  
+**Sink:** `chart/harvest.sql`, rendered through `tpl` in `chart/templates/configmap.yaml`.  
 
 **Evidence:**  
 The SQL file contains:
@@ -48,37 +48,20 @@ The Helm template risk is accepted. Chart installers (`helm install`, `--set`) a
 
 ### 2. Missing Secret template for MariaDB credentials (High)
 
-**Category:** V14 Configuration — Missing security resource
+**Category:** V14 Configuration — Missing security resource  
+**Status:** Resolved. `chart/templates/secret.yaml` exists and renders the `mariadb-credentials` Secret used by the StatefulSet and CronJob.
 
-**Evidence:**  
-The chart defines credential values in `chart/values.yaml` (`mariadb.rootPassword`, `mariadb.user`, `mariadb.password`), and both the StatefulSet (`chart/templates/statefulset.yaml` lines 39, 47, 52) and the CronJob (`chart/templates/cronjob.yaml` lines 44, 49) reference a Kubernetes Secret named `mariadb-credentials`. No Secret template exists in the chart (`chart/templates/secret.yaml` is absent from the repository contents).  
-
-**Impact:**  
-Any attempt to install the chart will result in pod errors (`CreateContainerConfigError`) because the required Secret is missing. The chart is unusable without manual intervention, defeating its purpose as a self-contained deployment.
-
-**Preconditions:**  
-None — the chart fails to deploy immediately with default values (as shown in the README commands).
-
-**Fix:**  
-Add a `chart/templates/secret.yaml` with content similar to:
-```yaml
-apiVersion: v1
-kind: Secret
-metadata:
-  name: mariadb-credentials
-type: Opaque
-stringData:
-  MARIADB_ROOT_PASSWORD: {{ .Values.mariadb.rootPassword | quote }}
-  MARIADB_USER: {{ .Values.mariadb.user | quote }}
-  MARIADB_PASSWORD: {{ .Values.mariadb.password | quote }}
+**Verification:**
+```bash
+helm template harvest chart | grep -A8 'name: mariadb-credentials'
 ```
 
 ### 3. Default weak MariaDB credentials (Medium)
 
 **Category:** V2 Authentication — Weak credentials; V14 Configuration — Insecure defaults (CWE-521, CWE-1392)
 
 **Evidence:**  
-In `chart/values.yaml`, the entries `mariadb.rootPassword`, `mariadb.user`, and `mariadb.password` are all set to the literal string `harvest`. The NetworkPolicy is disabled by default (`networkPolicy.enabled: false`), meaning the database is accessible from any pod in the cluster with these well-known credentials.
+In `chart/values.yaml`, the bundled MariaDB defaults (`mariadb.rootPassword`, `mariadb.user`, and `mariadb.password`) are all set to the literal string `harvest`. When the bundled database is enabled, the NetworkPolicy is disabled by default (`networkPolicy.enabled: false`), meaning MariaDB is accessible from any pod in the cluster with these well-known credentials.
 
 **Impact:**  
 If a deployment uses the defaults (e.g., an automated pipeline that neglects to override them), an attacker who gains a foothold in the cluster (any pod) can connect to MariaDB as `harvest`/`harvest` and exfiltrate or destroy the consultation data.
@@ -87,7 +70,6 @@ If a deployment uses the defaults (e.g., an automated pipeline that neglects to
 The chart is installed with default values, a likely accidental scenario for users who skip reading the “override for production” note.
 
 **Fix:**  
-- Enforce that credentials must be provided by using `required` in the Secret template:  
-  `{{ required "mariadb.password is required" .Values.mariadb.password }}`  
-- Or generate strong random passwords at install time (e.g., with `randAlphaNum`) and store them in the Secret.  
-- At minimum, set the defaults to empty strings so the deployment fails clearly rather than running with weak credentials.
+- For production, set `mariadb.enabled=false` and point `mysql.host` at an externally managed database.  
+- Always override `mariadb.user` and `mariadb.password` for production/external databases.  
+- A future hardening change could enforce non-empty credentials with `required` or generate strong random install-time passwords.

README.md

@@ -1,21 +1,27 @@
 # DuckDB SQL Pipeline for Harvesting Consultations
 
 A single SQL file pulls consultation data from WA government Citizen Space and
-EngagementHQ APIs, normalises it, and mirrors it into MariaDB for downstream consumption.
+EngagementHQ APIs, normalises it, and mirrors it into a MariaDB table for downstream consumption.
 The runtime harvest path is DuckDB SQL only; Python is not required.
 
 ## Quick start
 
-Set the standard [DuckDB MySQL environment variables](https://duckdb.org/docs/current/core_extensions/mysql#configuration) before running locally:
+The recommended path is Helm, because `chart/harvest.sql` is templated with the
+configured output table name.
+
+For local DuckDB runs, set the standard [DuckDB MySQL environment variables](https://duckdb.org/docs/current/core_extensions/mysql#configuration), render the table name, then run the rendered SQL:
 
 ```bash
 export MYSQL_HOST=localhost MYSQL_USER=harvest MYSQL_PWD=harvest MYSQL_DATABASE=harvest
-duckdb -c ".read chart/harvest.sql"
+sed 's/{{ .Values.mysql.table }}/consultations/g' chart/harvest.sql > /tmp/harvest.sql
+duckdb -c ".read /tmp/harvest.sql"
 ```
 
 > **Note:** The pipeline needs a dedicated MySQL user with write access to the
-> `harvest` database. The local defaults use user/password `harvest` for developer
-> convenience. Override credentials for production.
+> target database/table. Helm installs write to `mysql.database`.`mysql.table`,
+> defaulting to `harvest`.`consultations`. Use a simple table identifier such as
+> `consultations`; do not pass untrusted input to `mysql.table`. The local defaults
+> use user/password `harvest` for developer convenience. Override credentials for production.
 
 ## Kubernetes
 
@@ -28,23 +34,31 @@ just test          # Trigger a one-off harvest job
 just clean         # Tear down cluster
 ```
 
+Defaults deploy an in-cluster MariaDB for local/dev use and write to table
+`harvest.consultations`. For production, disable the bundled database with
+`--set mariadb.enabled=false` and point `mysql.host` at an externally managed
+MySQL/MariaDB service. Change the table with `--set mysql.table=...` or the
+`table` variable in `justfile`.
+
 ### Helm install
 
 ```bash
 helm upgrade --install harvest chart \
   --namespace harvest-consultations --create-namespace \
-  --set mysql.host=mariadb
+  --set mysql.host=mariadb \
+  --set mysql.table=consultations
 ```
 
-Override for external databases or production credentials:
+For production/external databases, disable the bundled MariaDB StatefulSet and provide the external host plus credentials:
 
 ```bash
 helm upgrade --install harvest chart \
   --namespace harvest-consultations --create-namespace \
-  --set mysql.host=external-db \
+  --set mariadb.enabled=false \
+  --set mysql.host=external-db.example.internal \
   --set mysql.database=harvest \
+  --set mysql.table=consultations \
   --set harvest.schedule="@daily" \
-  --set mariadb.rootPassword='change-me' \
   --set mariadb.user=harvest \
   --set mariadb.password='change-me'
 ```
@@ -66,19 +80,21 @@ just ci-test         # kind → helm install → harvest job → dump → valida
 | Key | Default | Description |
 |-----|---------|-------------|
 | `mysql.host` | `mariadb` | MySQL hostname for the harvest job |
-| `mysql.database` | `harvest` | Database name |
-| `mariadb.rootPassword` | `harvest` | MariaDB root password (init only; not used by app/healthcheck) |
-| `mariadb.user` | `harvest` | Application database user |
-| `mariadb.password` | `harvest` | Application database password |
-| `mariadb.image.repository` | `mariadb` | MariaDB image |
-| `mariadb.image.tag` | `11` | MariaDB image tag |
-| `mariadb.storage.size` | `1Gi` | PVC size for MariaDB data |
-| `mariadb.storage.storageClassName` | (unset) | StorageClass for PVC (set to `"encrypted"` for at-rest encryption) |
-| `mariadb.resources.requests.memory` | `256Mi` | MariaDB memory request |
-| `mariadb.resources.requests.cpu` | `100m` | MariaDB CPU request |
-| `mariadb.resources.limits.memory` | `1Gi` | MariaDB memory limit |
-| `mariadb.resources.limits.cpu` | `1` | MariaDB CPU limit |
-| `networkPolicy.enabled` | `false` | Enable NetworkPolicy to restrict ingress to MariaDB |
+| `mysql.database` | `harvest` | Target MySQL database name |
+| `mysql.table` | `consultations` | Target MySQL table replaced by each harvest run |
+| `mariadb.enabled` | `true` | Deploy bundled MariaDB StatefulSet/Service for local/dev; set `false` for production/external databases |
+| `mariadb.rootPassword` | `harvest` | Bundled MariaDB root password (init only; not rendered when `mariadb.enabled=false`) |
+| `mariadb.user` | `harvest` | Application database user for bundled or external DB |
+| `mariadb.password` | `harvest` | Application database password for bundled or external DB |
+| `mariadb.image.repository` | `mariadb` | Bundled MariaDB image |
+| `mariadb.image.tag` | `11` | Bundled MariaDB image tag |
+| `mariadb.storage.size` | `1Gi` | PVC size for bundled MariaDB data |
+| `mariadb.storage.storageClassName` | (unset) | StorageClass for bundled MariaDB PVC (set to `"encrypted"` for at-rest encryption) |
+| `mariadb.resources.requests.memory` | `256Mi` | Bundled MariaDB memory request |
+| `mariadb.resources.requests.cpu` | `100m` | Bundled MariaDB CPU request |
+| `mariadb.resources.limits.memory` | `1Gi` | Bundled MariaDB memory limit |
+| `mariadb.resources.limits.cpu` | `1` | Bundled MariaDB CPU limit |
+| `networkPolicy.enabled` | `false` | Enable NetworkPolicy to restrict ingress to bundled MariaDB; ignored when `mariadb.enabled=false` |
 | `harvest.schedule` | `@hourly` | CronJob schedule |
 | `harvest.image.repository` | `ghcr.io/wagov-dtt/harvest-duckdb` | DuckDB image with extensions pre-installed |
 | `harvest.image.tag` | `""` | Image tag override; empty computes `{chart-version}-duckdb{appVersion without dots}` |
@@ -91,7 +107,7 @@ just ci-test         # kind → helm install → harvest job → dump → valida
 
 | Path | Purpose |
 |------|---------|
-| `chart/harvest.sql` | SQL-only DuckDB harvest pipeline |
+| `chart/harvest.sql` | SQL-only DuckDB harvest pipeline; Helm templates `mysql.table` into the final write statement |
 | `chart/templates/secret.yaml` | Mariadb-credentials Secret (auto-generated from values) |
 | `chart/templates/networkpolicy.yaml` | Optional NetworkPolicy for MariaDB ingress isolation |
 | `chart/` | Helm chart (hand-written, source of truth) |

chart/Chart.yaml

@@ -2,5 +2,5 @@ apiVersion: v2
 name: harvest-consultations
 description: DuckDB harvest pipeline for WA government consultation data
 type: application
-version: 0.5.4
+version: 0.5.5
 appVersion: "1.5.2"

chart/harvest.sql

@@ -1,5 +1,5 @@
 -- harvest.sql — DuckDB SQL pipeline for harvesting consultations
--- Run:  duckdb -c ".read harvest.sql"
+-- Rendered by Helm so mysql.table can set the target table name.
 -- MySQL target via DuckDB MySQL env vars: MYSQL_HOST, MYSQL_USER, MYSQL_PWD, MYSQL_DATABASE
 -- https://duckdb.org/docs/current/core_extensions/mysql#configuration
 --

chart/templates/networkpolicy.yaml

@@ -1,4 +1,4 @@
-{{- if .Values.networkPolicy.enabled }}
+{{- if and .Values.networkPolicy.enabled .Values.mariadb.enabled }}
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
 metadata:

chart/templates/secret.yaml

@@ -7,6 +7,8 @@ metadata:
     {{- include "harvest-consultations.labels" . | nindent 4 }}
 type: Opaque
 data:
+  {{- if .Values.mariadb.enabled }}
   MARIADB_ROOT_PASSWORD: {{ .Values.mariadb.rootPassword | b64enc | quote }}
+  {{- end }}
   MARIADB_USER: {{ .Values.mariadb.user | b64enc | quote }}
   MARIADB_PASSWORD: {{ .Values.mariadb.password | b64enc | quote }}

chart/templates/service.yaml

@@ -1,3 +1,4 @@
+{{- if .Values.mariadb.enabled }}
 apiVersion: v1
 kind: Service
 metadata:
@@ -12,3 +13,4 @@ spec:
       targetPort: 3306
   selector:
     app: mariadb
+{{- end }}

chart/templates/statefulset.yaml

@@ -1,3 +1,4 @@
+{{- if .Values.mariadb.enabled }}
 apiVersion: apps/v1
 kind: StatefulSet
 metadata:
@@ -75,3 +76,4 @@ spec:
         resources:
           requests:
             storage: {{ .Values.mariadb.storage.size | quote }}
+{{- end }}

chart/values.yaml

@@ -4,8 +4,11 @@ mysql:
   table: consultations
 
 mariadb:
-  # rootPassword is only used for MariaDB initialization; the application and
-  # readiness probe connect via the dedicated user/password below.
+  # Enable only for local/dev installs. For production, use an external database
+  # and set mysql.host plus mariadb.user/password.
+  enabled: true
+  # rootPassword is only used when the bundled MariaDB StatefulSet is enabled;
+  # the application and readiness probe connect via the dedicated user/password below.
   rootPassword: harvest
   user: harvest
   password: harvest