Improve longevity tests (nginx#4769)

- Disable access logging to reduce noise and cost
- Scale the controller to multiple replicas to verify leader election works
- Reduce runtime from 4 days to 3 days
- Fix workflow results collection path

Author: sjberman

.github/workflows/longevity-start.yml

@@ -132,6 +132,5 @@ jobs:
           echo "Longevity Test Run ID: ${{ github.run_id }}" >> $GITHUB_STEP_SUMMARY
 
       - name: Start Longevity Tests
-        continue-on-error: true
         working-directory: ./tests
         run: make start-longevity-test

.github/workflows/longevity-stop.yml

@@ -117,15 +117,14 @@ jobs:
         run: make update-firewall-with-local-ip
 
       - name: Stop Longevity Tests
-        continue-on-error: true
         working-directory: ./tests
         run: make stop-longevity-test
 
       - name: Upload Artifacts
         uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
         with:
           name: results-${{ matrix.type }}
-          path: tests/results/longevity/*-${{ matrix.type }}.*
+          path: tests/results/longevity/**/*-${{ matrix.type }}.*
 
       - name: Cleanup
         working-directory: ./tests

.yamllint.yaml

@@ -3,7 +3,6 @@ ignore:
   - charts/nginx-gateway-fabric/templates
   - config/crd/bases/
   - deploy
-  - site/static
 
 rules:
   braces: enable
@@ -32,6 +31,7 @@ rules:
       .github/workflows/redhat-certification.yml
       examples/proxy-settings-policy/app.yaml
       tests/suite/manifests/proxy-settings-policy/app.yaml
+      tests/suite/manifests/longevity
   key-duplicates: enable
   key-ordering: disable
   line-length:

docs/developer/release-process.md

@@ -42,7 +42,7 @@ To create a new release, follow these steps:
 3. Create a release branch following the `release-X.Y` naming convention.
     - Once the release branch is created, reach out to the infra team to get it added to the runner permissions.
 4. Once the release branch pipeline completes, run tests using the `release-X.X-rc` images that are pushed to Github (for example, `release-1.3-rc`).
-   1. Follow the [longevity testing](https://github.com/nginx/nginx-gateway-fabric/blob/main/tests/README.md#longevity-testing) steps to start the tests in the pipeline. For `image_tag`, use `release-X.X-rc`, and for `version`, use the upcoming `X.Y.Z` NGF version. Run the workflow on the new release branch. This process takes 4 days, where you'll collect results and tear down at the end.
+   1. Follow the [longevity testing](https://github.com/nginx/nginx-gateway-fabric/blob/main/tests/README.md#longevity-testing) steps to start the tests in the pipeline. For `image_tag`, use `release-X.X-rc`, and for `version`, use the upcoming `X.Y.Z` NGF version. Run the workflow on the new release branch. This process takes 3 days, where you'll collect results and tear down at the end.
    2. Kick off the [NFR workflow](https://github.com/nginx/nginx-gateway-fabric/actions/workflows/nfr.yml) in the browser. For `image_tag`, use `release-X.X-rc`, and for `version`, use the upcoming `X.Y.Z` NGF version. Run the workflow on the new release branch. This will run all of the NFR tests which are automated and open a PR with the results files when it is complete. Review this PR and make any necessary changes before merging. Once merged, be sure to cherry-pick the commit to the release branch (set the `needs cherry pick` label on the main PR).
    3. Run the IPv6 tests using the `make ipv6-tests` target. This must be run from within the `tests` directory. An example of running this script for release 2.1.0 would look like this: `make ipv6-test TAG=release-2.1-rc`
 5. Run the [Release PR](https://github.com/nginx/nginx-gateway-fabric/actions/workflows/release-pr.yml) workflow to update the repo files for the release. Then there are a few manual steps to complete:

tests/Makefile

@@ -160,7 +160,7 @@ nfr-test: check-for-plus-usage-endpoint build-crossplane-image  ## Run the NFR t
 
 .PHONY: start-longevity-test
 start-longevity-test: export START_LONGEVITY=true
-start-longevity-test: nfr-test ## Start the longevity test to run for 4 days in GKE
+start-longevity-test: nfr-test ## Start the longevity test to run for 3 days in GKE
 
 .PHONY: stop-longevity-test
 stop-longevity-test: export STOP_LONGEVITY=true

tests/README.md

@@ -345,12 +345,12 @@ make nfr-test
 
 ##### Longevity testing
 
-This test is run on its own due to its long-running nature. It will run for 4 days (as defined in `suite/scripts/longevity-wrk.sh`) before
+This test is run on its own due to its long-running nature. It will run for 3 days (as defined in `suite/scripts/longevity-wrk.sh`) before
 the tester must collect the results and complete the test.
 
 To run in the pipeline, [run the workflow](https://github.com/nginx/nginx-gateway-fabric/actions/workflows/longevity-start.yml) to start the tests. Once the workflow completes, the job ID will be included in the summary. This must be used as input when stopping the longevity tests.
 
-After 4 days (96h), visit the [GCP Monitoring Dashboards](https://console.cloud.google.com/monitoring/dashboards) page and select the `NGF Longevity Test` dashboard. Update the `cluster_name` filter to the names of the longevity clusters. Take PNG screenshots of each chart for the time period in which your test ran, and save those to be added to the results file. Then you can [stop the longevity tests](https://github.com/nginx/nginx-gateway-fabric/actions/workflows/longevity-stop.yml). If done too early, the traffic will still be flowing and results may not be collected properly, so be sure to wait the full time period.
+After 3 days (72h) from the time that the startup workflow **finished**, visit the [GCP Monitoring Dashboards](https://console.cloud.google.com/monitoring/dashboards) page and select the `NGF Longevity Test` dashboard. Update the `cluster_name` filter to the names of the longevity clusters. Take PNG screenshots of each chart for the time period in which your test ran, and save those to be added to the results file. Then you can [stop the longevity tests](https://github.com/nginx/nginx-gateway-fabric/actions/workflows/longevity-stop.yml). If done too early, the traffic will still be flowing and results may not be collected properly, so be sure to wait the full time period.
 
 The final workflow will tear down the test and open a PR with the results. The PNGs you took should be added, and any summaries as well. Combine any results files if necessary. If you don't want to open a PR, you can toggle it off in the input when running the workflow.
 

tests/suite/longevity_test.go

@@ -53,6 +53,11 @@ var _ = Describe("Longevity", Label("longevity-setup", "longevity-teardown"), fu
 			Skip("'longevity-setup' label not specified; skipping...")
 		}
 
+		// scale controller to test leader election
+		ngfDeployment, err := resourceManager.GetNGFDeployment(ngfNamespace, "ngf-longevity")
+		Expect(err).ToNot(HaveOccurred())
+		Expect(resourceManager.ScaleDeployment(ngfNamespace, ngfDeployment.GetName(), 2)).To(Succeed())
+
 		Expect(resourceManager.Apply([]client.Object{&ns})).To(Succeed())
 		Expect(resourceManager.ApplyFromFiles(files, ns.Name)).To(Succeed())
 		Expect(resourceManager.ApplyFromFiles(promFile, ngfNamespace)).To(Succeed())

tests/suite/manifests/longevity/cafe-routes.yaml

@@ -13,6 +13,12 @@ spec:
     - path:
         type: PathPrefix
         value: /coffee
+    filters:
+    - type: ExtensionRef
+      extensionRef:
+        group: gateway.nginx.org
+        kind: SnippetsFilter
+        name: loggable-snippet
     backendRefs:
     - name: coffee
       port: 80
@@ -32,6 +38,27 @@ spec:
     - path:
         type: PathPrefix
         value: /tea
+    filters:
+    - type: ExtensionRef
+      extensionRef:
+        group: gateway.nginx.org
+        kind: SnippetsFilter
+        name: loggable-snippet
     backendRefs:
     - name: tea
       port: 80
+---
+# Only log non-200 responses
+apiVersion: gateway.nginx.org/v1alpha1
+kind: SnippetsFilter
+metadata:
+  name: loggable-snippet
+spec:
+  snippets:
+  - context: http
+    value: |
+      map $status $loggable {
+        ~^[2] 0;
+        default 1;
+      }
+      access_log /dev/stdout combined if=$loggable;

tests/suite/manifests/longevity/cafe.yaml

@@ -25,6 +25,14 @@ spec:
           preStop:
             exec:
               command: ["/bin/sleep", "15"]
+        volumeMounts:
+        - name: nginx-config
+          mountPath: /etc/nginx/nginx.conf
+          subPath: nginx.conf
+      volumes:
+      - name: nginx-config
+        configMap:
+          name: nginx-no-access-log
 ---
 apiVersion: v1
 kind: Service
@@ -66,6 +74,14 @@ spec:
           preStop:
             exec:
               command: ["/bin/sleep", "15"]
+        volumeMounts:
+        - name: nginx-config
+          mountPath: /etc/nginx/nginx.conf
+          subPath: nginx.conf
+      volumes:
+      - name: nginx-config
+        configMap:
+          name: nginx-no-access-log
 ---
 apiVersion: v1
 kind: Service
@@ -79,3 +95,20 @@ spec:
     name: http
   selector:
     app: tea
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: nginx-no-access-log
+data:
+  nginx.conf: |
+    events {}
+    http {
+      access_log off;
+      server {
+        listen 8080;
+        location / {
+          return 200;
+        }
+      }
+    }

tests/suite/scripts/longevity-wrk.sh

@@ -44,5 +44,5 @@ if ((ELAPSED >= MAX_WAIT)); then
     exit 1
 fi
 
-nohup wrk -t2 -c100 -d96h http://cafe.example.com/coffee &>~/coffee.txt &
-nohup wrk -t2 -c100 -d96h https://cafe.example.com/tea &>~/tea.txt &
+nohup wrk -t2 -c100 -d72h http://cafe.example.com/coffee &>~/coffee.txt &
+nohup wrk -t2 -c100 -d72h https://cafe.example.com/tea &>~/tea.txt &