Enable OpenShift cluster deployment via Pulumi on GCP and local environments by shaina-patel-23 · Pull Request #1591 · DataDog/test-infra-definitions

shaina-patel-23 · 2025-06-20T18:19:32Z

What does this PR do?

This PR introduces support for provisioning an OpenShift cluster via Pulumi. More specifically, it adds:

A new component NewOpenShiftCluster in components/kubernetes/openshift.go to provision an Openshift cluster on a remote VM.
A local version NewLocalCRCCLuster for testing CRC clusters locally
A new scenario gcp/openshiftvm that provisions a GCP VM with nested virtualization and deploys an OpenShift cluster on it
Enhancements to GCP VM provisioning in resources/gcp/compute/vm.go to support:
- Nested virtualization
- Larger disk size needed by CRC
Helper options in scenations/gcp/compute/vmargs.go for setting custom OS descriptors

Which scenarios this will impact?

Introduces a new GCP scenario: gcp\openshiftvm, available in the ScenarioRegistry

Motivation

Lays the groundwork for e2e testing of the Agent on an OpenShift cluster
Aligns OpenShift testing with the patterns already in place for EKS, GKE, etc.

Usage

Download and export you pull secret:
- Create a free "Personal" RedHat account using your Datadog email at: https://console.redhat.com/openshift/install
- Once logged in, go to Clusters > Create cluster > Local: https://console.redhat.com/openshift/create/local
- Download:
  - The MacOS CRC package appropriate to your architecture (x86_64 for Intel, aarch64 for M1/M2).
  - The Pull Secret file: pull-secret.txt
Use invoke commands
- Run invoke setup and set the pull secret path using it. Then run invoke gcp.create-openshift and invoke gcp.destroy-openshift
- Or just run invoke gcp.create-openshift -p <pull_secret_path> and invoke gcp.destroy-openshift

Additional Notes

Manual verification steps for the cluster are:
- SSH into the VM on GCP after pulumi up command completes successfully
- Enable OC: eval $(crc oc-env)
- Login as kubeadmin: oc login -u kubeadmin https://api.crc.testing:6443 (password retrieved from cat ~/.crc/machines/crc/kubeadmin-password)
- Verify cluster: oc get nodes
The instance type n2-standard-8 was chosen because CRC requires at least 4 vCPUs and 9 GiB of RAM to function, and n2-standard-8 provides 8 vCPUs and 32 GiB of memory; additionally, it supports nested virtualization, which is needed for running CRC inside a VM on GCP

…lly start a CRC cluster

…rocess for CRC Cluster

Co-authored-by: Lénaïc Huard <L3n41c@users.noreply.github.com>

…t open: Permission denied errors I recieved when testing locally

KevinFairise2 · 2025-07-09T12:59:10Z

+			OSCommand: command.NewUnixOSCommand(),
+		})
+
+		pullSecretPath := os.Getenv("PULL_SECRET_PATH")


Why do we need it to be passed as an environment variable? Can we make it a params of the function instead?

Updated the setup too use function params instead of env variables by reading the pull secret path from Pulumi config in openshift.go, adding support for invoke gcp.create-openshift.go in tasks/gcp/openshift.py, and updating setup.py to optionally prompt for the path in invoke setup

KevinFairise2 · 2025-07-09T13:00:22Z

+	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
+)
+
+func NewLocalOpenShiftCluster(env config.Env, name string, opts ...pulumi.ResourceOption) (*Cluster, error) {


What are the condition to be able to run that locally? Is it working on MacOS laptops?

This was a commit where I ran the function and confirmed it worked on macOS. I modeled this after the unused NewLocalKindCluster setup. In both situations the local setups aren't referenced anywhere, so I'm fine with removing it if you think that's appropriate.

Actually the NewLocalKindCluster setup is used in a provisioner on datadog-agent side: https://github.com/DataDog/datadog-agent/blob/main/test/new-e2e/pkg/provisioners/local/kubernetes/kind.go#L132
The goal is provide a local setup that allows you to iterate faster when creating a test for the first time

Ah, I see. I’ve implemented functionality for passing the pull secret as a parameter for both the local cluster and cluster on GCP. However, I didn’t add an invoke command for the local cluster creation to keep consistency with other cluster types. For the datadog-agent integration, I anticipate needing to adapt the pull secret config approach. Currently, the GCP OpenShift cluster receives the pull secret through the invoke setup process, but the datadog-agent environment may require a different configuration pattern. I’d appreciate any guidance for how to handle that…

KevinFairise2 · 2025-07-09T13:03:46Z

+		AllowStoppingForUpdate: pulumi.Bool(true),
+		AdvancedMachineFeatures: &compute.InstanceAdvancedMachineFeaturesArgs{


Are these ones supported for all the machine types? I know that on AWS you can only enable nested virtualization on specific machines

On GCP, nested virtualization is only supported on specific machine types like N1, N2, etc., and not on E2 or shared-core instances. I chose n2-standard-8 to meet vCPU and memory needs for CRC. Also, it also supports nested virtualization, which is needed to run CRC inside a VM on GCP.

In that case what happens if we set instanceType to a type that does not support nested virtualization? Your change enforce the NestedVirtualization to be enabled all the time, so I am afraid it will break when some instance type are used. Maybe we should make it configurable?

I’ve addressed the nested virt concern by making the feature configurable. EnableNestedVirtualization now defaults to false for all GCP VM instances, with the OpenShift scenario setting it to true. I also removed the AllowStoppingForUpdate setting as I had added it only for testing. Also dropped the new WithInstanceType function I introduced in vmargs.go after realizing I could leverage the existing ddinfra:gcp/defaultInstanceType configuration.

…as a parameter

KevinFairise2 · 2025-07-11T07:57:57Z

+package openshiftvm
+
+import (
+	localKubernetes "github.com/DataDog/test-infra-definitions/components/kubernetes"


Let's not alias it localKubernetes here, because it actually create the openshift cluster in a remote VM on GCP

KevinFairise2 · 2025-07-11T07:58:59Z

+        "scenario": scenario_name,
+        "ddinfra:env": f"gcp/{cfg.get_gcp().account}",
+        "ddinfra:gcp/defaultPublicKeyPath": cfg.get_gcp().publicKeyPath,
+        "ddinfra:openShiftPullSecretPath": pull_secret_path,


Let's call the parameter ddinfra:gcp/openshift/pullSecretPath since it is related to GCP and openshift only.
with that you will need to add that parameter here: https://github.com/DataDog/test-infra-definitions/blob/main/resources/gcp/environment.go#L28
And create the corresponding helper funcition to retrieve that parameter from the Pulumi config map, for example: https://github.com/DataDog/test-infra-definitions/blob/main/resources/gcp/environment.go#L159C1-L162C2.
You should then be able to use that helper in openshiftvm/run.go to retrieve the image pull secret parameters, and pass it to NewOpenShiftCluster function that creates the openshift cluster component

…r openshift, removed withinstancetype func, change ddinfra name, etc

CelianR

LGTM, verified that all comments are addressed. This one is remaining but is not a blocker and could be part of a future PR if necessary

shaina-patel-23 added 7 commits May 28, 2025 13:22

Starting stages of improving Agent on Openshift testing, code to loca…

432b2d4

…lly start a CRC cluster

Working instance of CRC cluster on a GCP VM instance.

5d1f3a0

Making Openshift an available infra type including the verification p…

98912a3

…rocess for CRC Cluster

Cleaned version of making Openshift an available infra type

c42c094

Cluster is now able to persist when pulumi program ends using nohup

fe5d3e4

Reverting nohup, not a viable fix.

3aec4cd

Using rhel rather than ubuntu

9e4fea6

shaina-patel-23 changed the title ~~Contint 4727/shaina/openshift infra type~~ Enable OpenShift cluster deployment via Pulumi on GCP and local environments Jun 26, 2025

shaina-patel-23 added 2 commits June 30, 2025 08:30

Trying to debug by setting network mode system

2b7865a

Reachable and running cluster using enable-linger command

d9f75b4

shaina-patel-23 marked this pull request as ready for review July 2, 2025 19:40

shaina-patel-23 requested a review from a team as a code owner July 2, 2025 19:40

L3n41c reviewed Jul 3, 2025

View reviewed changes

Comment thread components/kubernetes/openshift.go Outdated

Comment thread components/kubernetes/openshift.go Outdated

Comment thread components/kubernetes/openshift.go Outdated

Comment thread components/kubernetes/openshift.go Outdated

Comment thread components/kubernetes/openshift.go Outdated

shaina-patel-23 and others added 2 commits July 3, 2025 12:47

Apply suggestions from code review

eb8b3e0

Co-authored-by: Lénaïc Huard <L3n41c@users.noreply.github.com>

Simplied crc binary curl, but inserted a sudo due to: tar: crc: Canno…

019da76

…t open: Permission denied errors I recieved when testing locally

KevinFairise2 reviewed Jul 9, 2025

View reviewed changes

Added in openshift.py task as well as support to pass in pull secret …

77b4fc1

…as a parameter

KevinFairise2 reviewed Jul 11, 2025

View reviewed changes

shaina-patel-23 added 4 commits July 11, 2025 15:07

Addressed changes from PR - pullsecret parameter, nested virt only fo…

b387c19

…r openshift, removed withinstancetype func, change ddinfra name, etc

Merge branch 'main' into CONTINT-4727/Shaina/Openshift-infra-type

600b575

Reformatted python files

aea1e12

More reformatting linting python files

c749289

CelianR approved these changes Jul 17, 2025

View reviewed changes

shaina-patel-23 merged commit 9b3b4e3 into main Jul 17, 2025
8 checks passed

shaina-patel-23 deleted the CONTINT-4727/Shaina/Openshift-infra-type branch July 17, 2025 12:34

		AllowStoppingForUpdate: pulumi.Bool(true),
		AdvancedMachineFeatures: &compute.InstanceAdvancedMachineFeaturesArgs{

Uh oh!

Conversation

shaina-patel-23 commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Which scenarios this will impact?

Motivation

Usage

Additional Notes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shaina-patel-23 Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CelianR left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

shaina-patel-23 commented Jun 20, 2025 •

edited

Loading

shaina-patel-23 Jul 9, 2025 •

edited

Loading