Skip to content

NO-ISSUE: Fix pre-OS Image Stream OS validation for OKD clusters - #31617

Open
sarthakpurohit wants to merge 1 commit into
openshift:mainfrom
sarthakpurohit:fix-okd-pre-os-image-stream-test
Open

NO-ISSUE: Fix pre-OS Image Stream OS validation for OKD clusters#31617
sarthakpurohit wants to merge 1 commit into
openshift:mainfrom
sarthakpurohit:fix-okd-pre-os-image-stream-test

Conversation

@sarthakpurohit

@sarthakpurohit sarthakpurohit commented Sep 7, 2026

Copy link
Copy Markdown

Problem

The validatePreOSImageStreamsNodeOS function in test/extended/ci/job_names.go hardcodes an assertion that pre-OS Image Stream clusters should use RHEL CoreOS 9 nodes ("CoreOS 9."). However, OKD SCOS clusters use CentOS Stream CoreOS 10 even in pre-OS Image Stream releases (4.22 and earlier).

This causes the [sig-ci] [Early] prow job name should match os version test to fail deterministically on every OKD SCOS aws-upgrade-minor job (4.22 → 5.0 upgrade):

fail [job_names.go:206]: Pre OS Image Stream cluster should use RHEL 9 nodes
Expected
    <string>: CentOS Stream CoreOS 10.0.20260806-0 (Coughlan)
to contain substring
    <string>: CoreOS 9.

The [Early] tests run BEFORE the upgrade against the 4.22 OKD SCOS cluster, which uses CentOS Stream CoreOS 10. The test then hits the pre-OSImageStream code path (since 4.22 < 4.23 and OSImageStream CR does not exist), which unconditionally expects CoreOS 9.

The actual cluster upgrade itself succeeds — this is purely a test assertion bug.

Root Cause

PR #31487 (ff28c753e9) correctly added OKD detection and set targetStream = "centos-10" for the OSImageStream-enabled code path. However, it did not update the validatePreOSImageStreamsNodeOS fallback, which is reached when:

  1. The OSImageStream CR does not exist (all pre-4.23 clusters), AND
  2. The cluster desired version is < 4.23.0

This fallback unconditionally asserts "CoreOS 9." regardless of whether the cluster is OKD.

Fix

Pass the isOKD flag into validatePreOSImageStreamsNodeOS so it expects:

  • "CentOS Stream CoreOS 10" for OKD SCOS clusters
  • "CoreOS 9." for OCP clusters (unchanged behavior)

Evidence

Confirmed on multiple failing nightly runs:

Tests binary 5.0.0-202609050031.p2.g6bffdda (Sep 5 build, includes PR #31487) still fails, confirming the fix is incomplete.

Cherry-pick Plan

After merge to main, cherry-pick to release-5.0 is needed (the 5.0 OKD nightly aws-upgrade-minor job is the one failing).

/cc @pablintino

Summary by CodeRabbit

  • Bug Fixes
    • Updated pre-OS image stream validation to recognize the appropriate operating system for both OpenShift and OKD clusters.
    • Improved validation messaging to clearly identify the detected cluster type.

The validatePreOSImageStreamsNodeOS function hardcodes an assertion that
pre-OS Image Stream clusters should use RHEL CoreOS 9 nodes. However,
OKD SCOS clusters use CentOS Stream CoreOS 10 even in pre-OS Image
Stream releases (4.22 and earlier).

This causes the [sig-ci] [Early] prow job name should match os version
test to fail deterministically on every OKD SCOS 4.22 -> 5.0 upgrade
job (aws-upgrade-minor), because the [Early] tests run against the 4.22
cluster before the upgrade begins. The 4.22 OKD nodes report
"CentOS Stream CoreOS 10" but the test asserts "CoreOS 9.".

The actual cluster upgrade succeeds — this is purely a test assertion
bug in the pre-OSImageStream fallback path.

PR openshift#31487 (ff28c75) correctly added OKD detection and set
targetStream = "centos-10", but did not update the
validatePreOSImageStreamsNodeOS fallback which is reached when the
OSImageStream CR does not exist (i.e., all pre-4.23 clusters).

This change passes the isOKD flag into validatePreOSImageStreamsNodeOS
so it expects CentOS Stream CoreOS 10 for OKD clusters.

Signed-off-by: Sarthak Purohit <sapurohi@redhat.com>
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci
openshift-ci Bot requested a review from pablintino September 7, 2026 19:15
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Sep 7, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@sarthakpurohit: This pull request explicitly references no jira issue.

Details

In response to this:

Problem

The validatePreOSImageStreamsNodeOS function in test/extended/ci/job_names.go hardcodes an assertion that pre-OS Image Stream clusters should use RHEL CoreOS 9 nodes ("CoreOS 9."). However, OKD SCOS clusters use CentOS Stream CoreOS 10 even in pre-OS Image Stream releases (4.22 and earlier).

This causes the [sig-ci] [Early] prow job name should match os version test to fail deterministically on every OKD SCOS aws-upgrade-minor job (4.22 → 5.0 upgrade):

fail [job_names.go:206]: Pre OS Image Stream cluster should use RHEL 9 nodes
Expected
   <string>: CentOS Stream CoreOS 10.0.20260806-0 (Coughlan)
to contain substring
   <string>: CoreOS 9.

The [Early] tests run BEFORE the upgrade against the 4.22 OKD SCOS cluster, which uses CentOS Stream CoreOS 10. The test then hits the pre-OSImageStream code path (since 4.22 < 4.23 and OSImageStream CR does not exist), which unconditionally expects CoreOS 9.

The actual cluster upgrade itself succeeds — this is purely a test assertion bug.

Root Cause

PR #31487 (ff28c753e9) correctly added OKD detection and set targetStream = "centos-10" for the OSImageStream-enabled code path. However, it did not update the validatePreOSImageStreamsNodeOS fallback, which is reached when:

  1. The OSImageStream CR does not exist (all pre-4.23 clusters), AND
  2. The cluster desired version is < 4.23.0

This fallback unconditionally asserts "CoreOS 9." regardless of whether the cluster is OKD.

Fix

Pass the isOKD flag into validatePreOSImageStreamsNodeOS so it expects:

  • "CentOS Stream CoreOS 10" for OKD SCOS clusters
  • "CoreOS 9." for OCP clusters (unchanged behavior)

Evidence

Confirmed on multiple failing nightly runs:

Tests binary 5.0.0-202609050031.p2.g6bffdda (Sep 5 build, includes PR #31487) still fails, confirming the fix is incomplete.

Cherry-pick Plan

After merge to main, cherry-pick to release-5.0 is needed (the 5.0 OKD nightly aws-upgrade-minor job is the one failing).

/cc @pablintino

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Sep 7, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Team

Run ID: 99580f60-b630-49fb-8c61-b4dbed973817

📥 Commits

Reviewing files that changed from the base of the PR and between c36d7ed and 599e19d.

📒 Files selected for processing (1)
  • test/extended/ci/job_names.go

Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review.


Walkthrough

The pre-OS Image Stream node validation now receives the detected OKD status. It expects RHEL CoreOS 9 for OCP and CentOS Stream CoreOS 10 for OKD.

Changes

Pre-OS Image Stream validation

Layer / File(s) Summary
Select OS expectations
test/extended/ci/job_names.go
The validator accepts isOKD, selects the expected node OS and diagnostic text, and receives the detected OKD status at the call site.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 599e1

Pre-OS Image Stream validation now recognizes CentOS Stream CoreOS 10 on OKD while retaining the CoreOS 9 expectation for OCP, preventing the incorrect OKD early-job failure without affecting cluster upgrades.

Suggested reviewers: redhat-chai-bot

🚥 Pre-merge checks | ✅ 14 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 1 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (14 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: fixing pre-OS Image Stream OS validation for OKD clusters.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The check passes. The PR changes only the pre-OS Image Stream validation helper and its call site. The available base-to-head diff adds no Ginkgo title and does not modify any Describe, It, `Conte…
Test Structure And Quality ✅ Passed PASS: The patch changes only the existing node OS validation helper and its single call site. It does not add or split an It block, create resources, or add wait operations. The node list is read-on…
Microshift Test Compatibility ✅ Passed PASS — The commit adds no new Ginkgo test. It changes the existing should match os version test's validation helper and call site only. The added code lists core Kubernetes Nodes and selects an OS-i…
Single Node Openshift (Sno) Test Compatibility ✅ Passed PASS: The pull request modifies the existing [sig-ci] [Early] prow job name test and its helper. It adds no Ginkgo test. The changed helper lists nodes and validates each node OS image; it does not …
Topology-Aware Scheduling Compatibility ✅ Passed PASS — The pull request changes only test/extended/ci/job_names.go, a test helper that validates node OS strings and passes isOKD. The diff adds no deployment manifests, operators, controllers, re…
Ote Binary Stdout Contract ✅ Passed The patch changes only OS validation logic in test/extended/ci/job_names.go: it adds an isOKD parameter, selects an expected OS substring, and passes the value from the existing test path. The cha…
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The pull request changes an existing Ginkgo test helper and its call site. It adds no new It, Describe, Context, or When declarations. The changed code only lists cluster Nodes and selects an …
No-Weak-Crypto ✅ Passed The pull request changes only OS-version validation in test/extended/ci/job_names.go. The diff adds an isOKD branch and changes expected OS strings; it adds no MD5, SHA1, DES, RC4, 3DES, Blowfish,…
Container-Privileges ✅ Passed PASS. The pull request changes only test/extended/ci/job_names.go. The diff updates node OS validation and passes isOKD; it adds no container or Kubernetes manifest settings. Searches of the chang…
No-Sensitive-Data-In-Logs ✅ Passed The pull request changes only OS-version validation in test/extended/ci/job_names.go. It adds isOKD branching and changes the assertion description to RHEL 9 or CentOS Stream CoreOS 10. It doe…
  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci

openshift-ci Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sarthakpurohit
Once this PR has been reviewed and has the lgtm label, please assign mkowalski for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label Sep 7, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling required tests:
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@redhat-chai-bot

Copy link
Copy Markdown
Contributor

/override-sticky ci/prow/e2e-vsphere-ovn

Automated triage: This failure appears unrelated to the PR changes.

Job classification: Eligible long-running presubmit vSphere OVN end-to-end job. The definition uses the openshift-e2e-vsphere workflow with IPI installation on vsphere-elastic/vsphere02 and runs openshift-e2e-test.

Revision check: Run SHA 599e19d8144fdc1c8abed5a3578444bcee3d3fae; current PR HEAD 599e19d8144fdc1c8abed5a3578444bcee3d3fae; match.

Execution status: Tests executed. The run completed the e2e suite for 1h58m16s: 2344 passed, 1 blocking failure, 4 informing failures, 2211 skipped, and 0 flaky. The blocking failure was RouteExternalCertificate: after secret deletion, the route condition remained ExternalCertificateSARCompleted,True instead of becoming ExternalCertificateValidationFailed,False; polling ended with Interrupted by User at test/extended/router/external_certificate.go:246.

Completed supporting jobs: ci/prow/e2e-vsphere-ovn-upi passed on this revision. The unit, lint, verify, dependency, image, and agentic-image checks also passed. Pending checks: ci/prow/e2e-metal-ipi-ovn-ipv6 and tide.

Overlap assessment: The PR changes only test/extended/ci/job_names.go, adjusting pre-OS image-stream node OS validation for OKD versus OCP. It does not change router code, external-certificate reconciliation, route conditions, or shared dependencies used by the failing test. No plausible direct or indirect overlap was found.

Missing-coverage risk: Low for this PR. The job ran the full long-running e2e suite and exercised the relevant coverage; the sole blocking failure is the recurring router external-certificate condition race, while 2344 tests passed. The sibling vSphere OVN UPI e2e job also passed.

Rationale: The exact condition mismatch and timeout signature recur in unrelated CI runs and are consistent with a known router reconciliation race after external-certificate secret deletion, not with the PR's OS-image validation change. Override is appropriate for this unrelated failure.

If you disagree with this assessment, rerun the current job with /test e2e-vsphere-ovn.


AI-generated. Review for accuracy.

@openshift-ci

openshift-ci Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

@redhat-chai-bot: Overrode contexts on behalf of redhat-chai-bot: ci/prow/e2e-vsphere-ovn

These overrides will persist across retests on the current HEAD SHA. Pushing a new commit will clear them. Use /override-cancel to remove them.

Details

In response to this:

/override-sticky ci/prow/e2e-vsphere-ovn

Automated triage: This failure appears unrelated to the PR changes.

Job classification: Eligible long-running presubmit vSphere OVN end-to-end job. The definition uses the openshift-e2e-vsphere workflow with IPI installation on vsphere-elastic/vsphere02 and runs openshift-e2e-test.

Revision check: Run SHA 599e19d8144fdc1c8abed5a3578444bcee3d3fae; current PR HEAD 599e19d8144fdc1c8abed5a3578444bcee3d3fae; match.

Execution status: Tests executed. The run completed the e2e suite for 1h58m16s: 2344 passed, 1 blocking failure, 4 informing failures, 2211 skipped, and 0 flaky. The blocking failure was RouteExternalCertificate: after secret deletion, the route condition remained ExternalCertificateSARCompleted,True instead of becoming ExternalCertificateValidationFailed,False; polling ended with Interrupted by User at test/extended/router/external_certificate.go:246.

Completed supporting jobs: ci/prow/e2e-vsphere-ovn-upi passed on this revision. The unit, lint, verify, dependency, image, and agentic-image checks also passed. Pending checks: ci/prow/e2e-metal-ipi-ovn-ipv6 and tide.

Overlap assessment: The PR changes only test/extended/ci/job_names.go, adjusting pre-OS image-stream node OS validation for OKD versus OCP. It does not change router code, external-certificate reconciliation, route conditions, or shared dependencies used by the failing test. No plausible direct or indirect overlap was found.

Missing-coverage risk: Low for this PR. The job ran the full long-running e2e suite and exercised the relevant coverage; the sole blocking failure is the recurring router external-certificate condition race, while 2344 tests passed. The sibling vSphere OVN UPI e2e job also passed.

Rationale: The exact condition mismatch and timeout signature recur in unrelated CI runs and are consistent with a known router reconciliation race after external-certificate secret deletion, not with the PR's OS-image validation change. Override is appropriate for this unrelated failure.

If you disagree with this assessment, rerun the current job with /test e2e-vsphere-ovn.


AI-generated. Review for accuracy.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci

openshift-ci Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

@sarthakpurohit: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants