CI: Updates Makefile and GH Workflow to Support Multiple Gateway API Versions #10390

danehans · 2024-11-21T22:27:55Z

Description

Updates the Makefile and conformance GH workflow to support multiple Gateway API versions.

Fixes #10359

API changes

None

Code changes

Makefile- Refactors conformance testing targets to support multiple Gateway API versions.

CI changes

.github/workflows/composite-actions/kube-gateway-api-conformance-tests/action.yaml: Adds the k8sgateway-api-version as input to set the CONFORMANCE_VERSION environment variable used by the conformance target.
.github/workflows/conformance-tests.yaml: Creates a Gateway API version matrix for conformance testing in CI.
.github/workflows/nightly-tests.yaml: Creates a Gateway API version matrix for nightly testing in CI.
.github/workflows/regression-tests.yaml: Creates a Gateway API version matrix for regression testing in CI.
-.github/workflows/.env/pr-tests/{min,max}_versions.env: Adds Dotenv files to define the source min/max GW API versions used for conformance testing.

Docs changes

N/A
BOT NOTES:
resolves kgateway-dev#10359

solo-changelog-bot · 2024-11-21T22:28:06Z

Issues linked to changelog:
kgateway-dev#10359

danehans · 2024-11-21T22:29:13Z

.github/workflows/composite-actions/kube-gateway-api-conformance-tests/action.yaml

@@ -53,6 +57,8 @@ runs:
      fi
  - name: Run the kubernetes gateway API conformance tests
    shell: bash
+    env:
+      GATEWAY_API_VERSION: ${{ inputs.gateway-api-version }}


@sam-heilbron @nfuden this env var is not getting set causing the CI failure. PTAL and let me know if you see the issue or if you suggest anyone else with more familiarity with CI that can review.

One thing that stands out to me is that we mixing "matrix" and "input" in this composite-action. I prefer your approach, since it makes the action standalone, and unaware that it's called within a matrix. I'm not sure if it causes issues combingin both, but from a mainteance perspective, I think it would be better if we adjusted the syntax to be entirely input based, as you have it

github-actions · 2024-11-26T19:05:58Z

Visit the preview URL for this PR (updated for commit 240b7ad):

https://gloo-edge--pr10390-issue-10359-9hzllf0a.web.app

_{(expires Thu, 23 Jan 2025 16:27:13 GMT)}

_{🔥 via Firebase Hosting GitHub Action 🌎}

_{Sign: 77c2b86e287749579b7ff9cadb81e099042ef677}

sam-heilbron · 2024-11-26T23:14:34Z

.github/workflows/conformance-tests.yaml

      - name: Run Conformance Tests
        uses: ./.github/workflows/composite-actions/kube-gateway-api-conformance-tests
+        with:
+          gateway-api-version: ${{ matrix.gateway-api-version }}


https://github.com/solo-io/gloo/blob/main/.github/workflows/nightly-tests.yaml#L86 we rely on an env file to define inputs for some of our other tests. I think it could be great to do something similar for this job. it can be a separate file, but it would standardize our way of defining these variables and make it easier to keep in sync

sam-heilbron · 2024-12-10T15:59:19Z

@danehans this PR is out of date, but I can approve once you update it

danehans · 2024-12-10T17:18:58Z

@sam-heilbron I am still working through the implementation. I have tried implementing your feedback but no luck thus far. I should have time later today to continue troubleshooting.

danehans · 2024-12-11T02:55:08Z

.github/workflows/.env/pr-tests/min_versions.env

@@ -0,0 +1 @@
+k8sgateway_api_version='v1.1.0'


I initially set this value to v1.0.0 to match .github/workflows/.env/nightly-tests/min_versions.env but the following conformance test consistently failed:

=== RUN TestConformance/GatewayWithAttachedRoutes/Gateway_listener_should_have_AttachedRoutes_set_even_when_Gateway_has_unresolved_refs helpers.go:650: Gateway expected observedGeneration to be updated to 1 for all conditions, only 0/2 were updated. stale conditions are: Accepted (generation 0), Programmed (generation 0) helpers.go:655: Conditions matched expectations helpers.go:655: Gateway status listeners matched expectations helpers.go:681: Accepted condition set to Status True with Reason Accepted, expected Status False helpers.go:681: Accepted was not in conditions list [[{ResolvedRefs False 1 2024-12-11 00:21:38 +0000 UTC BackendNotFound Service "does-not-exist" not found} {Accepted True 1 2024-12-11 00:21:38 +0000 UTC Accepted }]]

^ is a test bug that was fixed in kubernetes-sigs/gateway-api@8bd5491 (included in v1.1.0). @sam-heilbron should .github/workflows/.env/nightly-tests/min_versions.env be updated accordingly?

weren't we previously passing conformance on 1.0.0? (before we upgraded to 1.1 / 1.2)

…PI versions Signed-off-by: Daneyon Hansen <[email protected]>

Signed-off-by: Daneyon Hansen <[email protected]>

danehans · 2025-01-06T23:05:49Z

@sam-heilbron @jenshu PTAL

cc: @josh-pritchard

sam-heilbron

A question about what tests run on PRs, but generally the changes look good to me

sam-heilbron · 2025-01-07T16:32:14Z

.github/workflows/conformance-tests.yaml

@@ -33,12 +33,27 @@ jobs:
          - ${{ inputs.image-variant }}
        version:
          - ${{ inputs.version }}
+        version-files:


This is a shift in behavior. Previously, we ran min/max versions on an async schedule (ie not on PRs) and it seems we are shifting that to run this on PRs. This isn't bad per se, I just wanted to call it out. What are we trying to achieve by running min/max on PRs, that running these nightly did not achieve? Or, if we didn't run the min/max for conformance tests nightly, perhaps we should do that instead?

sheidkamp

Overall makes sense, a few comments.

Can you also kick off a nightly run for this branch and link the results in the PR description?

sheidkamp · 2025-01-07T17:55:02Z

.github/workflows/nightly-tests.yaml

+      uses: falti/[email protected]
+      id: dotenv
+      with:
+        path: ${{ matrix.version-files.file }}


Looks like the matrix (line 484) does not have version-files defined and is getting its versions from the inline kube-version array. I think k8sgateway_api_version can get added to the nightly env files and then those files can be used here.

sheidkamp · 2025-01-07T17:56:58Z

.github/workflows/nightly-tests.yaml

+        uses: falti/[email protected]
+        id: dotenv
+        with:
+          path: ${{ matrix.version-files.file }}


Same comment as above (version-files not defined in matrix).

Also, this won't work until the env files are backported to 1.18.x

sheidkamp · 2025-01-07T18:07:33Z

.github/workflows/regression-tests.yaml

+          - label: 'min'
+            file: './.github/workflows/.env/pr-tests/min_versions.env'
+          - label: 'max'
+            file: './.github/workflows/.env/pr-tests/max_versions.env'


Can we move the kube-version versions into the env file as well so we don't mix ways of tracking versions? I wouldn't be opposed to re-using the nightly env files, though renaming the files to be more accurate would be tricky.

github-actions bot added keep pr updated work in progress labels Nov 21, 2024

danehans commented Nov 21, 2024

View reviewed changes

danehans mentioned this pull request Nov 21, 2024

Set k8sgateway api version when setting up e2e test cluster #10386

Merged

4 tasks

sam-heilbron reviewed Nov 26, 2024

View reviewed changes

danehans marked this pull request as draft December 3, 2024 19:35

danehans force-pushed the issue_10359 branch 3 times, most recently from ab977eb to 7bb39ce Compare December 5, 2024 18:00

danehans marked this pull request as ready for review December 5, 2024 18:00

danehans force-pushed the issue_10359 branch 2 times, most recently from 1982e91 to 1131566 Compare December 5, 2024 20:10

danehans mentioned this pull request Dec 9, 2024

Bumps Gateway API Dependencies #10457

Merged

danehans force-pushed the issue_10359 branch from 05207ef to 9fc26f7 Compare December 9, 2024 22:45

danehans force-pushed the issue_10359 branch 5 times, most recently from 4d1200f to aac5da1 Compare December 11, 2024 01:09

danehans requested a review from sam-heilbron December 11, 2024 02:43

danehans commented Dec 11, 2024

View reviewed changes

danehans force-pushed the issue_10359 branch from 1123630 to eb5b0e4 Compare December 13, 2024 18:45

danehans removed the work in progress label Dec 16, 2024

danehans added 2 commits December 16, 2024 08:06

CI: Updates Makefile and conformance GH action for multiple Gateway A…

1652737

…PI versions Signed-off-by: Daneyon Hansen <[email protected]>

Refactors to use dotenv-action envs for min/max GW API versions

9dae4d1

Signed-off-by: Daneyon Hansen <[email protected]>

danehans force-pushed the issue_10359 branch from eb5b0e4 to 3b413a0 Compare December 16, 2024 16:06

Moves gw api env var to Setup test env step

411c3d8

Signed-off-by: Daneyon Hansen <[email protected]>

danehans force-pushed the issue_10359 branch from 3b413a0 to 411c3d8 Compare December 16, 2024 16:15

jenshu requested a review from sheidkamp January 7, 2025 16:22

sam-heilbron reviewed Jan 7, 2025

View reviewed changes

sheidkamp reviewed Jan 7, 2025

View reviewed changes

changelog-bot added 2 commits January 16, 2025 16:12

Adding changelog file to new location

f3ded4a

Deleting changelog file from old location

240b7ad

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: Updates Makefile and GH Workflow to Support Multiple Gateway API Versions #10390

CI: Updates Makefile and GH Workflow to Support Multiple Gateway API Versions #10390

danehans commented Nov 21, 2024 •

edited by solo-changelog-bot bot

Loading

solo-changelog-bot bot commented Nov 21, 2024

danehans Nov 21, 2024

sam-heilbron Nov 26, 2024

github-actions bot commented Nov 26, 2024 •

edited

Loading

sam-heilbron Nov 26, 2024

sam-heilbron commented Dec 10, 2024

danehans commented Dec 10, 2024

danehans Dec 11, 2024

jenshu Jan 7, 2025

danehans commented Jan 6, 2025

sam-heilbron left a comment

sam-heilbron Jan 7, 2025

sheidkamp left a comment

sheidkamp Jan 7, 2025

sheidkamp Jan 7, 2025

sheidkamp Jan 7, 2025

		@@ -0,0 +1 @@
		k8sgateway_api_version='v1.1.0'

CI: Updates Makefile and GH Workflow to Support Multiple Gateway API Versions #10390

Are you sure you want to change the base?

CI: Updates Makefile and GH Workflow to Support Multiple Gateway API Versions #10390

Conversation

danehans commented Nov 21, 2024 • edited by solo-changelog-bot bot Loading

Description

API changes

Code changes

CI changes

Docs changes

solo-changelog-bot bot commented Nov 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Nov 26, 2024 • edited Loading

Choose a reason for hiding this comment

sam-heilbron commented Dec 10, 2024

danehans commented Dec 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danehans commented Jan 6, 2025

sam-heilbron left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sheidkamp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danehans commented Nov 21, 2024 •

edited by solo-changelog-bot bot

Loading

github-actions bot commented Nov 26, 2024 •

edited

Loading