Skip to content

[ci] run e2e tests from the runner using a remote-access kubeconfig - #1948

Open
abrarshivani wants to merge 2 commits into
NVIDIA:mainfrom
abrarshivani:ci-e2e-remote-access
Open

[ci] run e2e tests from the runner using a remote-access kubeconfig#1948
abrarshivani wants to merge 2 commits into
NVIDIA:mainfrom
abrarshivani:ci-e2e-remote-access

Conversation

@abrarshivani

@abrarshivani abrarshivani commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

This reverts the remote test execution workaround from #1763 now that Holodeck can hand us a kubeconfig the runner can actually use.

Problem

#1763 was needed because Holodeck tightened the permissions on the generated kubeconfig. The file was no longer readable from the GitHub Actions runner, so the e2e tests could not reach the cluster and we moved the whole test run into the EC2 instance instead.

That workaround costs us a few things:

  • the workflow writes the SSH private key to key.pem on the runner, so the secret exists as a file on disk
  • Go and make have to be installed and maintained inside the instance on every run
  • the source tree is rsynced into the instance, environment variables are forwarded over SSH, and logs have to be copied back afterwards

Solution

Holodeck added a kubernetes.remoteAccess option. When it is set, Holodeck rewrites the kubeconfig server URL to https://<PublicDnsName>:6443 and chowns the file to the owner of the bind-mounted workspace, so the runner user can read it. The security group still opens 6443 only to the auto-detected caller egress IP, and the file mode stays 0600.

So this PR sets remoteAccess: true in tests/e2e/infra/aws.yaml and drops the rsync/SSH scaffolding. The workflow goes back to running

make -f tests/e2e/Makefile test

on the runner, which is what it did before #1763.

The Holodeck action is already on v0.3.7 and the feature landed in v0.3.4, so no version bump is needed.

Notes

AWS_SSH_KEY is still passed to the Holodeck action, which needs it to provision the instance. Only the manual step that materialized it as key.pem is gone.

KUBECONFIG goes back to being a required environment variable in the test suite. The fallback to ~/.kube/config only made sense while the tests ran inside the instance, and in CI a silent fallback would point at the wrong cluster rather than fail.

Both LOG_ARTIFACTS and LOG_ARTIFACTS_DIR are set now. The Makefile uses the first for the ginkgo reports and the suite uses the second for its own logs, and they need to agree for the upload step to pick everything up.

The second commit removes the Archive Ginkgo logs step. It uploads ginkgo.json, which the Makefile has never written, so every run has been logging No files were found with the provided path: ginkgo.json. The reports it was meant to capture now sit under e2e_logs/, which the step above it already uploads on failure.

Validation

A full CI run passed on the first commit. Ginkgo ran from the runner workspace rather than inside the instance, and the suite reported Ran 4 of 4 Specs, 4 Passed | 0 Failed, matching the spec count on main.

Left as a follow-up: gpu-operator uses the same remote execution pattern and could be simplified the same way, but its logging and test environment differ enough to be worth a separate change.

Holodeck gained a kubernetes.remoteAccess option, which rewrites the
generated kubeconfig's server URL to the node's public DNS name on port
6443 and chowns the file to the owner of the bind-mounted workspace. The
runner user can therefore read the kubeconfig and talk to the cluster
directly, which is what we needed all along.

Enable that option and revert the remote-execution workaround added in
NVIDIA#1763. The workflow no longer writes the SSH private key to key.pem,
rsyncs the source tree into the instance, installs Go and make there, or
copies logs back afterwards. It just runs

    make -f tests/e2e/Makefile test

on the runner, as it did before the workaround.

Both LOG_ARTIFACTS and LOG_ARTIFACTS_DIR are set: the Makefile uses the
former for the ginkgo reports and the test suite uses the latter for its
own logs, and they need to agree for the upload-artifact step to pick
everything up.

KUBECONFIG goes back to being a required environment variable. Falling
back to ~/.kube/config only made sense while the tests ran inside the
instance, and in CI a silent fallback would point at the wrong cluster
rather than fail.

The security group still exposes 6443 only to the auto-detected caller
egress IP, and AWS_SSH_KEY is still passed to the Holodeck action, which
needs it to provision the instance.

Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
@abrarshivani abrarshivani self-assigned this Aug 4, 2026
The step uploads ginkgo.json, but the Makefile writes its ginkgo reports
to $(LOG_ARTIFACTS)/report.json and $(LOG_ARTIFACTS)/junit.xml, so that
path has never existed. Every run logs

    No files were found with the provided path: ginkgo.json.
    No artifacts will be uploaded.

The reports now sit under e2e_logs/, which the archive step above already
picks up on failure, so the step is redundant as well as broken.

Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
@abrarshivani
abrarshivani marked this pull request as ready for review August 4, 2026 20:23
@rahulait

rahulait commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Thanks @abrarshivani , LGTM. Lets wait for the release and then we should be good to merge this. We can do the similar changes to gpu-operator's CI as well.

@abrarshivani

abrarshivani commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

@rahulait Sounds good. Will merge this after the release. Sure, will take a look at gpu-operator's CI as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants