Tekton Results broken on dogfooding cluster (postgres image gone) — temporarily disabled

[vdemeester]

Summary

Tekton Results has been fully broken on the oci-ci-cd dogfooding cluster for 45+ days and is being temporarily disabled until it can be fixed properly.

Symptoms (as of 2026-06-08)

tekton-results-postgres-0                      0/1  ImageInspectError   (32d)
tekton-results-api-...                         0/1  CrashLoopBackOff    (45d, 15728 restarts)
tekton-results-watcher-...                     0/1  CrashLoopBackOff    (47d)
tekton-results-retention-policy-agent-...      0/1  CrashLoopBackOff    (45d)

Root cause

The postgres image pinned in the overlay is no longer pullable:

docker.io/bitnami/postgresql@sha256:ac8dd0d6512c4c5fb146c16b1c5f05862bd5f600d73348506ab4252587e7fcc6

Bitnami deprecated their free Docker Hub catalog in 2025, so the digest can no longer be inspected/pulled (ImageInspectError). With no DB, the api/watcher/retention-agent crashloop.

Impact: PipelineRun / PVC pileup

The results-watcher adds a results.tekton.dev/pipelinerun finalizer to PipelineRuns so it can store the record before deletion. While Results is broken, the finalizer is added but never removed, so completed PipelineRuns get stuck in Terminating forever. This previously trapped ~33 PipelineRuns (some since February) and blocked cascade-deletion of their PVCs, contributing to block-storage sprawl (~1.5 TB of orphaned 50Gi PVCs were manually reclaimed on 2026-06-08).

Temporary mitigation (this issue's PR)

Scale all Results workloads (api, watcher, retention-policy-agent, postgres) to replicas: 0 via a kustomize patch in tekton/cd/results/overlays/oci-ci-cd. This stops the crashloops and prevents the finalizer pileup. Manifests and the postgres PVC are kept intact so re-enabling is a trivial revert.

Proper fix (follow-up)

Replace the unavailable Bitnami image with a maintained postgres option, e.g.:

• bitnamilegacy/postgresql (stopgap), or
• official postgres image (needs to match Results' expected env/initdb), or
• CloudNativePG operator-managed Postgres.

Then re-enable Results (revert the disable patch) and verify the watcher removes finalizers cleanly.

Related

• Bitnami Docker Hub deprecation
• PVC reduction work (Reduce pruner history limits on dogfooding cluster #3376, Collapse simple 2-task pipelines into single Tasks to eliminate PVC usage #3379, Deploy in-cluster OCI registry (zot) for artifact transport between tasks #3380, Replace Fluentd with Vector for Tekton log collection and Results API integration #3381)

[vdemeester]

cc @tektoncd/results-maintainers

[enarha]

The image is actually pull-able. Running ./test/e2e/00-setup.sh && ./test/e2e/01-install.sh from the repo root deploys Tekton results locally on Kind, including the Postgres image from Bitnami. skopeo inspect docker://bitnami/postgresql@sha256:ac8dd0d6512c4c5fb146c16b1c5f05862bd5f600d73348506ab4252587e7fcc6 also returns the image data. So not sure if the issue was intermittent or what, but it's working now. With that said, I do not mind switching to more robust and future proof solution.

[vdemeester]

@enarha ah that might be because of cri-o setup on oracle cloud enforcing full image reference (docker.io/library/...).