Add end-to-end testing for application accessibility and authentication

[Copilot]

End-to-End Testing Implementation

This PR adds E2E tests that verify applications deployed via ArgoCD are functionally accessible, not just healthy at the Kubernetes resource level.

Changes

Test Scripts (.github/scripts/)

• test-http-endpoints.sh: HTTP health checks via kubectl port-forward to envoy-gateway proxy, using curl --resolve for hostname routing
• test-certificates.sh: TLS certificate validation via port-forward with OpenSSL SNI
• test-argocd-auth.sh: ArgoCD CLI authentication and API testing via --port-forward
• All scripts are modular, executable, and can be run independently

Chainsaw Test (chainsaw-test.yaml)

• Three E2E test steps calling external scripts
• Catch blocks reference Envoy Gateway resources for debugging on failure

GitHub Workflow (.github/workflows/pipeline.yaml)

• ArgoCD CLI installation with checksum verification (cli_checksums.txt)
• Binary installed with restrictive permissions (555)

Documentation (README.md)

• Test coverage section referencing script locations and prerequisites

Connectivity Approach

The envoy-gateway proxy service gets a MetalLB-assigned IP (e.g. 10.89.0.200) on the KinD Docker network, which is not routable from the GitHub Actions runner host. Tests use kubectl port-forward to tunnel traffic to the gateway proxy service:

• HTTP test discovers the proxy service by label (gateway.envoyproxy.io/owning-gateway-name=envoy-private)
• Uses curl --resolve for hostname routing without modifying /etc/hosts
• Certificate test uses openssl s_client with SNI via localhost
• Retry loop checks port-forward readiness instead of hardcoded sleep
• Proper cleanup via trap EXIT

Test Coverage

1. HTTP Health Checks: ArgoCD, Keycloak, OpenCloud UI accessibility via HTTPS
2. Certificate Validation: TLS certificates are present and not expired for all applications
3. ArgoCD CLI Authentication: Login with admin credentials and application listing via API

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Test applications end to end</issue_title>
<issue_description>Currently, the pipeline just checks for application health by inspecting the pods, deployments and other kubernetes resources.
This can miss things like misconfigurations and doesn't test that traffic can even reach the application. The tests should be extended with functionality to test the applications in an end to end fashion. For example:

• Log in to argocd using the CLI
• Check that the ingress/gateway controller is reachable
• Log in to keycloak to check admin access
• Log in to opencloud and check that it is possible to upload/download a file
• Verify that certificates are as expected when accessing these resources</issue_description>

<agent_instructions>This issue may be too large for a single PR. Consider implementing only one part of it.</agent_instructions>

Comments on the Issue (you are @copilot in this section)

• Fixes Test applications end to end #794

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[lentzi90]

I would prefer to define the scripts outside the chainsaw test and then just call them.
Please also avoid installing tools in the test itself. The test should assume any needed tools are available. These need to be installed before starting the test, e.g. in a previous step in the github workflow.
Note that the test fails to find the argo CLI when trying to download.
Please update with these fixes @copilot

[Copilot]

I would prefer to define the scripts outside the chainsaw test and then just call them.
Please also avoid installing tools in the test itself. The test should assume any needed tools are available. Th...

I've refactored the E2E tests as requested:

✅ Scripts extracted: Test logic moved to separate shell scripts in .github/scripts/:

• test-http-endpoints.sh - HTTP health checks
• test-certificates.sh - TLS certificate validation
• test-argocd-auth.sh - ArgoCD CLI authentication

✅ Tools installed in workflow: Added ArgoCD CLI installation step in .github/workflows/pipeline.yaml with proper checksum verification (validates exactly one match before verification)

✅ Tests assume tools available: Scripts no longer install tools - they expect argocd, kubectl, curl, openssl, and jq to be available

✅ Modular design: Scripts can be called from chainsaw or run independently

Changes in commits: 30ddbf8, d829bd3, 22e3516

[lentzi90]

Please check how the release artifacts look like here: https://github.com/argoproj/argo-cd/releases. The workflow fails to find the checksum file! It is named cli_checksums.txt. @copilot

[Copilot]

Please check how the release artifacts look like here: https://github.com/argoproj/argo-cd/releases. The workflow fails to find the checksum file! It is named cli_checksums.txt. @copilot

Fixed! Updated both the workflow and README to use the correct checksum filename cli_checksums.txt instead of argocd-checksums.txt.

Commit: d124578

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• https://api.github.com/repos/argoproj/argo-cd/releases/latest
  • Triggering command: /usr/bin/curl curl -s REDACTED (http block)
  • Triggering command: /usr/bin/curl curl -sL REDACTED (http block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[lentzi90]

This needs a rebase @copilot . The ingress controller has been removed and we are using only gateway API now.
The test failed with this:

        --- Testing Ingress Controller ---
        Testing Nginx Ingress Controller at http://localhost/healthz...
        ✗ Nginx Ingress Controller returned HTTP 200 (expected 404)
    sink.go:56: ������������| 17:43:03 | kind-deploy | step-22   | SCRIPT    | ERROR |
        === ERROR
        exit status 1

That should no longer be relevant after a rebase.

[Copilot]

This needs a rebase @copilot . The ingress controller has been removed and we are using only gateway API now.
The test failed with this:

...

Merged main and updated all E2E tests for Gateway API:

• Replaced nginx-ingress wait/check with envoy-gateway (gateway.envoyproxy.io/owning-gateway-name=envoy-private)
• Removed the nginx ingress healthz test that was failing
• Updated chainsaw catch blocks to reference Gateway resources in envoy-gateway-system
• Updated README test coverage docs

Commits: 5045c80 (merge), d52f8cf, 28c9e0d

[lentzi90]

@copilot the test failed with the below error. Please check and fix!

        === Testing HTTP Endpoints ===
        Waiting for envoy gateway to be ready...
        pod/envoy-envoy-gateway-system-envoy-private-2f8a770a-97548ff4fc2qv condition met
        127.0.0.1 argocd.local
        127.0.0.1 keycloak.local
        127.0.0.1 opencloud.local
        
        --- Testing ArgoCD ---
        Testing ArgoCD UI at https://argocd.local/...
        ✗ ArgoCD UI returned HTTP 000000 (expected 200)
    sink.go:56: ������������| 20:17:22 | kind-deploy | step-20   | SCRIPT    | ERROR |
        === ERROR
        exit status 1

[Copilot]

@copilot the test failed with the below error. Please check and fix!

...

The root cause was that the test was hardcoding 127.0.0.1 in /etc/hosts, but the envoy-gateway proxy gets its IP from MetalLB (e.g. 10.89.0.200), so curl couldn't connect.

Two fixes applied:

1. Dynamically discover the gateway IP via kubectl get gateway envoy-private -n envoy-gateway-system -o jsonpath='{.status.addresses[0].value}' and use it in /etc/hosts
2. Fixed curl error handling — || echo "000" was concatenating with curl's -w "%{http_code}" output (producing 000000), changed to || true

Also updated the certificate test to connect directly to the gateway IP with SNI.

Commit: eb22156

[lentzi90]

Still no luck.

[59](https://github.com/lentzi90/personal-cloud/actions/runs/22528899440/job/65359912691?pr=795#step:6:360)
        === COMMAND
        /usr/bin/sh -c .github/scripts/test-http-endpoints.sh
    sink.go:56: ������������| 07:14:21 | kind-deploy | step-20   | SCRIPT    | LOG   |
        === STDOUT
        === Testing HTTP Endpoints ===
        Waiting for envoy gateway to be ready...
        pod/envoy-envoy-gateway-system-envoy-private-2f8a770a-97548ff4p87w5 condition met
        Getting gateway IP...
        Gateway IP: 10.89.0.200
        10.89.0.200 argocd.local
        10.89.0.200 keycloak.local
        10.89.0.200 opencloud.local
        
        --- Testing ArgoCD ---
        Testing ArgoCD UI at https://argocd.local/...
        ✗ ArgoCD UI returned HTTP 000 (expected 200)
    sink.go:56: ������������| 07:14:21 | kind-deploy | step-20   | SCRIPT    | ERROR |
        === ERROR
        exit status 1
    sink.go:56: ������������| 07:14:21 | kind-deploy | step-20   | TRY       | END   |
    sink.go:56: ������������| 07:14:21 | kind-deploy | step-20   | CATCH     | BEGIN |
    sink.go:56: ������������| 07:14:21 | kind-deploy | step-20   | CMD       | RUN   |
        === COMMAND
        /opt/hostedtoolcache/kind/v0.31.0/amd64/kubectl/bin/kubectl describe gateways.v1.gateway.networking.k8s.io envoy-private -n envoy-gateway-system
    sink.go:56: ������������| 07:14:21 | kind-deploy | step-20   | CMD       | LOG   |
        === STDOUT
        Name:         envoy-private
        Namespace:    envoy-gateway-system
        Labels:       environment=kind
        Annotations:  argocd.argoproj.io/tracking-id: envoy-gateways:gateway.networking.k8s.io/Gateway:envoy-gateway-system/envoy-private
        API Version:  gateway.networking.k8s.io/v1
        Kind:         Gateway
        Metadata:
          Creation Timestamp:  2026-03-02T07:10:41Z
          Generation:          1
          Resource Version:    3420
          UID:                 866f3b57-1ab8-43d3-89a3-e95d1207b45a
        Spec:
          Gateway Class Name:  envoy
          Listeners:
            Allowed Routes:
              Namespaces:
                From:  All
            Name:      http
            Port:      80
            Protocol:  HTTP
            Allowed Routes:
              Namespaces:
                From:  All
            Hostname:  *.local
            Name:      https
            Port:      443
            Protocol:  HTTPS
            Tls:
              Certificate Refs:
                Group:  
                Kind:   Secret
                Name:   wildcard-local-cert
              Mode:     Terminate
        Status:
          Addresses:
            Type:   IPAddress
            Value:  10.89.0.200
          Conditions:
            Last Transition Time:  2026-03-02T07:14:01Z
            Message:               The Gateway has been scheduled by Envoy Gateway
            Observed Generation:   1
            Reason:                Accepted
            Status:                True
            Type:                  Accepted
            Last Transition Time:  2026-03-02T07:14:01Z
            Message:               Address assigned to the Gateway, 1/1 envoy replicas available
            Observed Generation:   1
            Reason:                Programmed
            Status:                True
            Type:                  Programmed
          Listeners:
            Attached Routes:  0
            Conditions:
              Last Transition Time:  2026-03-02T07:14:01Z
              Message:               Sending translated listener configuration to the data plane
              Observed Generation:   1
              Reason:                Programmed
              Status:                True
              Type:                  Programmed
              Last Transition Time:  2026-03-02T07:14:01Z
              Message:               Listener has been successfully translated
              Observed Generation:   1
              Reason:                Accepted
              Status:                True
              Type:                  Accepted
              Last Transition Time:  2026-03-02T07:14:01Z
              Message:               Listener references have been resolved
              Observed Generation:   1
              Reason:                ResolvedRefs
              Status:                True
              Type:                  ResolvedRefs
            Name:                    http
            Supported Kinds:
              Group:          gateway.networking.k8s.io
              Kind:           HTTPRoute
              Group:          gateway.networking.k8s.io
              Kind:           GRPCRoute
            Attached Routes:  4
            Conditions:
              Last Transition Time:  2026-03-02T07:14:01Z
              Message:               Sending translated listener configuration to the data plane
              Observed Generation:   1
              Reason:                Programmed
              Status:                True
              Type:                  Programmed
              Last Transition Time:  2026-03-02T07:14:01Z
              Message:               Listener has been successfully translated
              Observed Generation:   1
              Reason:                Accepted
              Status:                True
              Type:                  Accepted
              Last Transition Time:  2026-03-02T07:14:01Z
              Message:               Listener references have been resolved
              Observed Generation:   1
              Reason:                ResolvedRefs
              Status:                True
              Type:                  ResolvedRefs
            Name:                    https
            Supported Kinds:
              Group:  gateway.networking.k8s.io
              Kind:   HTTPRoute
              Group:  gateway.networking.k8s.io
              Kind:   GRPCRoute
        Events:       <none>
    sink.go:56: ������������| 07:14:21 | kind-deploy | step-20   | CMD       | DONE  |
    sink.go:56: ������������| 07:14:21 | kind-deploy | step-20   | CMD       | RUN   |
        === COMMAND
        /opt/hostedtoolcache/kind/v0.31.0/amd64/kubectl/bin/kubectl logs --prefix -l gateway.envoyproxy.io/owning-gateway-name=envoy-private -n envoy-gateway-system --all-containers
    sink.go:56: ������������| 07:14:21 | kind-deploy | step-20   | CMD       | LOG   |
        === STDOUT
        [pod/envoy-envoy-gateway-system-envoy-private-2f8a770a-97548ff4p87w5/shutdown-manager] 2026-03-02T07:10:45.008Z	INFO	shutdown-manager	envoy/shutdown_manager.go:74	starting shutdown manager
    sink.go:56: ������������| 07:14:21 | kind-deploy | step-20   | CMD       | DONE  |
    sink.go:56: ������������| 07:14:21 | kind-deploy | step-20   | CATCH     | END   |
    sink.go:56: ������������| 07:14:21 | kind-deploy | @chainsaw | CLEANUP   | BEGIN |
    sink.go:56: ������������| 07:14:21 | kind-deploy | @chainsaw | DELETE    | OK    | v1/Namespace @ chainsaw-calm-mammal
Error: some tests failed
    sink.go:56: ������������| 07:14:27 | kind-deploy | @chainsaw | CLEANUP   | END   |
--- FAIL: chainsaw (0.00s)
    --- FAIL: chainsaw/kind-deploy (346.54s)
FAIL
Tests Summary...
- Passed  tests 0
- Failed  tests 1
- Skipped tests 0

@copilot

[Copilot]

Still no luck.

...

The MetalLB IP (10.89.0.200) is on the KinD Docker network, which is not routable from the GitHub Actions runner host. Switched to kubectl port-forward to tunnel to the envoy-gateway proxy service instead:

• Discovers proxy service by label gateway.envoyproxy.io/owning-gateway-name=envoy-private
• Uses curl --resolve for hostname routing (no /etc/hosts modification needed)
• Certificate test uses openssl s_client with SNI via localhost
• Retry loop for port-forward readiness instead of hardcoded sleep
• Proper cleanup via trap EXIT

Commit: c376305