upgrade kubearchive to 1.23.0 on staging

[olegbet]

Summary

• Upgrade kubearchive staging from v1.22.2 to v1.23.0
• Updates development/kubearchive.yaml release manifest (images + version labels)
• Bumps vacuum image tags in development/kustomization.yaml
• No schema migration version change needed

Test plan

• Verify ArgoCD syncs successfully on staging clusters
• Confirm kubearchive pods start with v1.22.1 images

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: olegbet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• components/kubearchive/OWNERS [olegbet]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[qodo-for-redhat-appstudio]

PR Summary by Qodo

Upgrade staging KubeArchive to v1.23.0

⚙️ Configuration changes ✨ Enhancement 🕐 10-20 Minutes

AI Description

• Promote staging KubeArchive workloads and metadata from v1.22.2 to v1.23.0.
• Configure API database pooling, query timeout, and request rate limits.
• Keep schema v13 while aligning migration and vacuum images with the release.

Diagram

graph TD
  K["Development overlay"] --> M["Release manifest"] --> A["API server"] --> D[("PostgreSQL")]
  M --> O["Operator"] --> S["Event sink"]
  M --> J["Maintenance jobs"] --> D

Loading

High-Level Assessment

The current approach is appropriate because it matches prior KubeArchive staging upgrades: update the rendered release manifest, preserve digest pinning, and keep overlay-specific vacuum and migration references synchronized. A centralized image transformer would reduce repeated tags but would not cover version labels, per-image digests, API defaults, or the versioned immutable migration Job.

Files changed (2) +63 / -45

Other (2) +63 / -45

kubearchive.yamlDeploy KubeArchive v1.23.0 with API safeguards +59/-41

Deploy KubeArchive v1.23.0 with API safeguards

• Updates all release labels and pinned API, operator, sink, vacuum, and PostgreSQL images to v1.23.0. Adds API database connection-pool limits, connection lifetimes, a query timeout, and overall/log-specific rate limits.

components/kubearchive/development/kubearchive.yaml

kustomization.yamlAlign staging vacuum and migration jobs with v1.23.0 +4/-4

Align staging vacuum and migration jobs with v1.23.0

• Bumps all staging vacuum CronJob image overrides to v1.23.0 and gives the immutable migration Job a v1.23.0-specific name. The target database schema remains version 13.

components/kubearchive/development/kustomization.yaml

[github-actions]

Kustomize Render Diff

Comparing d6fbabef8 → 78fa49545

Component	Environment	Changes
components/kubearchive/development	development	+54 -36
components/kubearchive/staging/lightwell-dev	staging	+52 -34
components/kubearchive/staging/stone-stage-p01	staging	+52 -34
components/kubearchive/staging/stone-stg-rh01	staging	+52 -34

Total: 4 components, +210 -138 lines

📋 Full diff available in the workflow summary and as a downloadable artifact.

[qodo-for-redhat-appstudio]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (1) 📜 Skill insights (0)

1. Cluster templates retain v1.22.2 📘 Rule violation ⚙ Maintainability

Description

The PR upgrades five KubeArchive component images to v1.23.0, but the cluster-creation templates
still pin the corresponding v1.22.2 images and digests. New clusters would therefore deploy the
previous release instead of the component versions introduced by this PR.

Code

components/kubearchive/development/kubearchive.yaml[1151]

+          image: quay.io/kubearchive/api:v1.23.0@sha256:e2b69c81967cd9fb0b7e6b75de19a5c1271554e4991c945f718e3f6af09d82af

Relevance

●●● Strong

Accepted template-alignment findings in PR #13490 show reviewers require generated cluster templates
to match component upgrades.

PR-#13490

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

PR Compliance ID 490 requires every matching image under hack/new-cluster/templates/ to follow
component image changes. The PR branch has v1.23.0 component images, while the cluster template
still contains the previous v1.22.2 API, operator, sink, vacuum, and PostgreSQL references, and
its vacuum kustomize patch is likewise stale.

Rule 490: Update template image references with component image changes
components/kubearchive/development/kubearchive.yaml[1151-1151]
components/kubearchive/development/kubearchive.yaml[1253-1253]
components/kubearchive/development/kubearchive.yaml[1367-1367]
components/kubearchive/development/kubearchive.yaml[1429-1429]
components/kubearchive/development/kubearchive.yaml[1459-1459]
hack/new-cluster/templates/kubearchive/kubearchive.yaml[1135-1135]
hack/new-cluster/templates/kubearchive/kubearchive.yaml[1237-1237]
hack/new-cluster/templates/kubearchive/kubearchive.yaml[1351-1351]
hack/new-cluster/templates/kubearchive/kubearchive.yaml[1413-1413]
hack/new-cluster/templates/kubearchive/kubearchive.yaml[1443-1443]
hack/new-cluster/templates/kubearchive/kustomization.yaml[117-117]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Update the KubeArchive cluster-creation templates so every changed component image uses the same `v1.23.0` tag and digest as the development manifests.

## Issue Context
The PR updates the API, operator, sink, vacuum, and PostgreSQL images, while matching files under `hack/new-cluster/templates/` still reference `v1.22.2`. Also update all vacuum patch occurrences in the template kustomization.

## Fix Focus Areas
- hack/new-cluster/templates/kubearchive/kubearchive.yaml[1135-1443]
- hack/new-cluster/templates/kubearchive/kustomization.yaml[109-117]
- components/kubearchive/development/kubearchive.yaml[1151-1459]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context sources

✅ Compliance rules (platform): 3 rules

Tip of the day

💡 Did you know, you can group findings by type and pick your Finding display, from Minimal to Full

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[qodo-for-redhat-appstudio]

1. Cluster templates retain v1.22.2 📘 Rule violation ⚙ Maintainability

The PR upgrades five KubeArchive component images to v1.23.0, but the cluster-creation templates
still pin the corresponding v1.22.2 images and digests. New clusters would therefore deploy the
previous release instead of the component versions introduced by this PR.

Agent Prompt

## Issue description
Update the KubeArchive cluster-creation templates so every changed component image uses the same `v1.23.0` tag and digest as the development manifests.

## Issue Context
The PR updates the API, operator, sink, vacuum, and PostgreSQL images, while matching files under `hack/new-cluster/templates/` still reference `v1.22.2`. Also update all vacuum patch occurrences in the template kustomization.

## Fix Focus Areas
- hack/new-cluster/templates/kubearchive/kubearchive.yaml[1135-1443]
- hack/new-cluster/templates/kubearchive/kustomization.yaml[109-117]
- components/kubearchive/development/kubearchive.yaml[1151-1459]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

[konflux-ci-qe-bot]

🤖 Pipeline Failure Analysis

Category: Infrastructure

The pipeline failed to install Konflux because the client lost connection to the Kubernetes API server, indicating a critical underlying infrastructure issue.

📋 Technical Details

Immediate Cause

The appstudio-e2e-tests/konflux-ci-install-konflux step failed with an "Unable to connect to the server: EOF" error, directly preventing the installation of Konflux due to a dropped or unestablished connection to the Kubernetes API server.

Contributing Factors

Multiple subsequent diagnostic gather steps, including gather-audit-logs, gather-extra, gather-must-gather, and redhat-appstudio-gather, also failed. These failures consistently showed "context deadline exceeded" errors for must-gather pods and widespread "remote error: tls: internal error" when oc commands attempted to access logs or resources from various cluster components, confirming systemic instability in the OpenShift cluster's control plane, network, or TLS configuration.

Impact

The inability to establish and maintain a stable connection to the Kubernetes API server rendered the cluster non-functional for the pipeline's purposes, completely blocking the Konflux installation and preventing the execution of any subsequent e2e tests.

🔍 Evidence

appstudio-e2e-tests/gather-audit-logs

Category: infrastructure
Root Cause: The must-gather operation failed due to a "context deadline exceeded" error, indicating that the must-gather pod could not start collecting data within the timeout period, likely due to underlying cluster performance or resource constraints.

Logs:

artifacts/appstudio-e2e-tests/gather-audit-logs/build-log.txt

[must-gather      ] OUT 2026-08-29T15:20:09.268601714Z the server is currently unable to handle the request (get imagestreams.image.openshift.io must-gather)

artifacts/appstudio-e2e-tests/gather-audit-logs/build-log.txt

[must-gather-5stlz] OUT 2026-08-29T15:30:09.862740977Z gather did not start: context deadline exceeded

artifacts/appstudio-e2e-tests/gather-audit-logs/build-log.txt

error: gather did not start for pod must-gather-5stlz: context deadline exceeded

appstudio-e2e-tests/gather-extra

Category: infrastructure
Root Cause: The OpenShift cluster is experiencing a critical infrastructure issue, evidenced by widespread "remote error: tls: internal error" messages when oc commands attempt to communicate with various cluster components (e.g., Kubelet on nodes for logs) or the API server. This indicates a breakdown in secure communication or health of core Kubernetes services, leading to the failure of the gather process.

Logs:

artifacts/appstudio-e2e-tests/gather-extra/build-log.txt line 67

error: inspection completed with the errors occurred while gathering data:
    [skipping gathering routes.route.openshift.io/oauth-openshift due to error: the server doesn't have a resource type "routes", skipping gathering namespaces/openshift-authentication due to error: one or more errors occurred while gathering pod-specific data for namespace: openshift-authentication

    [one or more errors occurred while gathering container data for pod oauth-openshift-795fc57547-h2bcs:

    [Get "https://10.0.64.42:10250/containerLogs/openshift-authentication/oauth-openshift-795fc57547-h2bcs/oauth-openshift?timestamps=true": remote error: tls: internal error, Get "https://10.0.64.42:10250/containerLogs/openshift-authentication/oauth-openshift-795fc57547-h2bcs/oauth-openshift?previous=true&timestamps=true": remote error: tls: internal error]

artifacts/appstudio-e2e-tests/gather-extra/build-log.txt line 1618

[must-gather-dfpmr] OUT 2026-08-29T15:19:28.08619222Z gather did not start: context deadline exceeded

artifacts/appstudio-e2e-tests/gather-extra/build-log.txt line 1629

error: gather did not start for pod must-gather-dfpmr: context deadline exceeded

artifacts/appstudio-e2e-tests/gather-extra/build-log.txt line 1642

Error from server (ServiceUnavailable): error trying to reach service: remote error: tls: internal error

artifacts/appstudio-e2e-tests/gather-extra/build-log.txt line 1667

Error from server: error dialing backend: remote error: tls: internal error

appstudio-e2e-tests/gather-must-gather

Category: infrastructure
Root Cause: The OpenShift cluster is in an unhealthy state, preventing the must-gather diagnostic tool from functioning. This is evidenced by the must-gather pod failing to start due to a timeout and widespread "remote error: tls: internal error" messages during subsequent log collection attempts from numerous cluster components, indicating underlying issues with node kubelets or internal cluster networking/TLS.

Logs:

artifacts/appstudio-e2e-tests/gather-must-gather/build-log.txt

Error running must-gather collection:
    gather did not start for pod must-gather-nrmnb: context deadline exceeded

artifacts/appstudio-e2e-tests/gather-must-gather/build-log.txt

Get "https://10.0.64.42:10250/containerLogs/openshift-authentication/oauth-openshift-795fc57547-h2bcs/oauth-openshift?previous=true&timestamps=true": remote error: tls: internal error

artifacts/appstudio-e2e-tests/gather-must-gather/build-log.txt

error: gather did not start for pod must-gather-nrmnb: context deadline exceeded

appstudio-e2e-tests/konflux-ci-install-konflux

Category: infrastructure
Root Cause: The Konflux installation failed because the client could not connect to the Kubernetes API server due to an unexpected connection closure (EOF error), likely caused by an unresponsive cluster or network connectivity issues.

Logs:

artifacts/appstudio-e2e-tests/konflux-ci-install-konflux/build-log.txt

Unable to connect to the server: EOF

appstudio-e2e-tests/redhat-appstudio-gather

Category: infrastructure
Root Cause: The OpenShift cluster is in an unhealthy state, preventing the must-gather tool from starting and oc adm inspect from collecting logs due to widespread "remote error: tls: internal error." This indicates fundamental issues with network communication, node health, or TLS configuration affecting core cluster components.

Logs:

artifacts/appstudio-e2e-tests/redhat-appstudio-gather/build-log.txt

error: the server doesn't have a resource type "applications"

artifacts/appstudio-e2e-tests/redhat-appstudio-gather/build-log.txt

Error running must-gather collection:
    gather did not start for pod must-gather-kf8wx: context deadline exceeded

artifacts/appstudio-e2e-tests/redhat-appstudio-gather/build-log.txt

Get "https://10.0.64.42:10250/containerLogs/openshift-authentication/oauth-openshift-795fc57547-h2bcs/oauth-openshift?previous=true&timestamps=true": remote error: tls: internal error

artifacts/appstudio-e2e-tests/redhat-appstudio-gather/build-log.txt

skipping gathering routes.route.openshift.io/oauth-openshift due to error: the server doesn't have a resource type "routes"

---

Analysis powered by prow-failure-analysis | Build: 2093695136671207424

[openshift-ci]

@olegbet: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/appstudio-e2e-tests	aadef24	link	true	/test appstudio-e2e-tests

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.