Add terraform for provisioning s390x build cluster on ibmcloud

infra/ibmcloud/terraform/k8s-s390x-build-cluster/README.md

@@ -0,0 +1,127 @@
+# _TF: IBM K8s s390x Build Cluster_
+These terraform resources define a IBM Cloud project containing a s390xVS cluster intended to serve as a "build cluster" for prow.k8s.io.
+
+---
+## Initial Setup
+
+### Supporting infrastructure
+
+#### Deploy k8s-infra-setup resources
+
+- this covers things like Resource Group, s390x Virtual Server Instances, Virtual Private Cloud, IBM Cloud Secret Manager Secrets, etc.
+- Once the deployment successfully completes, the `secrets_manager_id` will be generated and should be used in the subsequent steps.
+
+---
+#### Deploy k8s-s390x-build-cluster resources
+
+**1. Navigate to the correct directory**
+<br> You need to be in the `k8s-s390x-build-cluster` directory to run the automation.
+
+**2. Export COS Secrets**
+<br> Export `access_key` and `secret_key` as environment variables.
+```
+export AWS_ACCESS_KEY_ID="<HMAC_ACCESS_KEY_ID>"
+export AWS_SECRET_ACCESS_KEY="<HMAC_SECRET_ACCESS_KEY>"
+```
+
+**3. Initialize Terraform**
+<br> Execute the following command to initialize Terraform in your project directory. This command will download the necessary provider plugins and prepare the working environment.
+```
+terraform init -reconfigure
+```
+
+**4. Check the `variables.tf` file**
+<br> Open the `variables.tf` file to review all the available variables. This file lists all customizable inputs for your Terraform configuration.
+
+`ibmcloud_api_key`, `secrets_manager_id` are the only required variables that you must set in order to proceed. You can set this key either by adding it to your `var.tfvars` file or by exporting it as an environment variable.
+
+**Option 1:** Set in `var.tfvars` file
+Create `var.tfvars` file and set the following variables in `var.tfvars` file:
+```
+ibmcloud_api_key    = "<YOUR_API_KEY>"
+secrets_manager_id  = "<SECRETS_MANAGER_ID>"
+```
+Tip: To get the secrets_manager_id (GUID) for IBM Cloud Secrets Manager instance:
+```
+ibmcloud resource service-instances --service-name secrets-manager --output JSON | \
+jq -r '.[] | select(.name | contains("k8s-s390x")) | .guid'
+```
+**Option 2:** Export as an environment variable
+Alternatively, you can export above as an environment variable before running Terraform:
+```
+export TF_VAR_ibmcloud_api_key="<YOUR_API_KEY>"
+export TF_VAR_secrets_manager_id=$(ibmcloud resource service-instances --service-name secrets-manager --output JSON | \
+jq -r '.[] | select(.name | contains("k8s-s390x")) | .guid')
+```
+
+**5. Run Terraform Apply**
+<br> After setting the necessary variables (particularly the API_KEY), execute the following command to apply the Terraform configuration and provision the infrastructure:
+```
+terraform apply -var-file var.tfvars
+```
+Terraform will display a plan of the actions it will take, and you'll be prompted to confirm the execution. Type `yes` to proceed.
+
+**6. Get Output Information**
+<br> Once the infrastructure has been provisioned, use the terraform output command to list details about the provisioned resources.
+```
+terraform output
+```
+
+**7. Set up the Kubernetes cluster using ansible**
+Clone the repository `https://github.com/kubernetes-sigs/provider-ibmcloud-test-infra` and change the directory to `kubetest2-tf/data/k8s-ansible`:
+```
+cd kubetest2-tf/data/k8s-ansible
+```
+
+**8. Install ansible on the deployer VM**
+```
+dnf install ansible -y
+```
+
+**9. Update the fields under `group_vars/all` to include the Kubernetes version to install**
+<br> The following lines will update the version to the latest stable release of Kubernetes. You can modify it accordingly to set up the CI (alpha) version.
+```
+K8S_VERSION=$(curl -Ls https://dl.k8s.io/release/stable.txt)
+LOADBALANCER_EP=<mention the loadbalancer endpoint obtained from terraform output>
+sed -i \
+-e "s/^directory: .*/directory: release/" \
+-e "s/build_version: .*/build_version: $K8S_VERSION/" \
+-e "s/release_marker: .*/release_marker: $K8S_VERSION/" \
+-e "s/loadbalancer: .*/loadbalancer: $LOADBALANCER_EP/" group_vars/all
+```
+
+**10. Update the fields under `examples/k8s-build-cluster/hosts.yml` to contain IP addresses of the VMs to set up Kubernetes**
+```
+For example:
+
+[bastion]
+56.77.34.6
+
+[masters]
+192.168.100.3
+192.168.100.4
+
+[workers]
+192.168.100.5
+192.168.100.6
+192.168.100.7
+
+[workers:vars]
+ansible_ssh_common_args='-o ProxyCommand="ssh -W %h:%p -i <path/to/private-key> -q [email protected]" -i <path/to/private-key>'
+
+[masters:vars]
+ansible_ssh_common_args='-o ProxyCommand="ssh -W %h:%p -i <path/to/private-key> -q [email protected]" -i <path/to/private-key>'
+```
+
+**11. Update the fields under `group_vars/bastion_configuration` to contain the information of the private network.**
+```
+For example:
+
+bastion_private_gateway: 192.168.100.1
+bastion_private_ip: 192.168.100.2
+```
+
+**12. Trigger the installation using ansible**
+```
+ansible-playbook -v -i examples/k8s-build-cluster/hosts.yml install-k8s-ha.yaml -e @group_vars/bastion_configuration --extra-vars @group_vars/all
+```

infra/ibmcloud/terraform/k8s-s390x-build-cluster/bastion.tf

@@ -0,0 +1,98 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+locals {
+  bastion_nodes = {
+    "primary" = {
+      profile = var.bastion_profile
+      boot_volume = {
+        size = var.bastion_boot_volume_size
+      }
+    }
+  }
+}
+
+resource "ibm_is_instance" "bastion" {
+  for_each       = local.bastion_nodes
+  name           = "bastion-s390x-${each.key}"
+  vpc            = data.ibm_is_vpc.vpc.id
+  zone           = var.zone
+  profile        = each.value.profile
+  image          = var.image_id
+  keys           = [ibm_is_ssh_key.k8s_ssh_key.id]
+  resource_group = data.ibm_resource_group.resource_group.id
+
+  primary_network_interface {
+    name            = "public-nic-${each.key}"
+    subnet          = data.ibm_is_subnet.subnet.id
+    security_groups = [data.ibm_is_security_group.bastion.id]
+  }
+
+  boot_volume {
+    name = "boot-vol-bastion-${each.key}"
+    size = each.value.boot_volume.size
+  }
+
+  user_data = <<-EOF
+              #cloud-config
+              package_update: true
+              package_upgrade: true
+              packages:
+                - tcpdump
+                - net-tools
+                - iptables-persistent
+              write_files:
+                - path: /etc/ssh/sshd_config.d/99-bastion.conf
+                  content: |
+                    AllowTcpForwarding yes
+                    GatewayPorts yes
+                    PermitTunnel yes
+                    PermitRootLogin prohibit-password
+                    PasswordAuthentication no
+                    ClientAliveInterval 120
+                    ClientAliveCountMax 3
+                    MaxSessions 50
+                    MaxStartups 50:30:100
+                - path: /etc/systemd/network/10-eth1.network
+                  content: |
+                    [Match]
+                    Name=eth1
+                    [Network]
+                    Address=${data.ibm_is_subnet.subnet.ipv4_cidr_block}
+                    DNS=8.8.8.8
+                    DNS=8.8.4.4
+              runcmd:
+                - [sysctl, -w, net.ipv4.ip_forward=1]
+                - [echo, "net.ipv4.ip_forward = 1", ">>", /etc/sysctl.conf]
+                - [iptables, -t, nat, -A, POSTROUTING, -o, eth0, -j, MASQUERADE]
+                - [iptables, -A, FORWARD, -i, eth1, -o, eth0, -j, ACCEPT]
+                - [iptables, -A, FORWARD, -i, eth0, -o, eth1, -m, state, --state, RELATED,ESTABLISHED, -j, ACCEPT]
+                - [netfilter-persistent, save]
+                - [systemctl, restart, systemd-networkd]
+                - [systemctl, restart, sshd]
+                - [hostnamectl, set-hostname, "bastion-s390x-${each.key}.s390x-vpc.cloud.ibm.com"]
+                - [echo, "bastion-s390x-${each.key}.s390x-vpc.cloud.ibm.com", ">", /etc/hostname]
+                - [sed, -i, "s/^127.0.1.1.*/127.0.1.1\tbastion-s390x-${each.key}.s390x-vpc.cloud.ibm.com/", /etc/hosts]
+                - [touch, /var/lib/cloud/instance/bastion-setup-success]
+              EOF
+}
+
+resource "ibm_is_floating_ip" "bastion_fip" {
+  for_each       = ibm_is_instance.bastion
+  name           = "bastion-fip-${each.key}"
+  target         = each.value.primary_network_interface[0].id
+  resource_group = data.ibm_resource_group.resource_group.id
+}

infra/ibmcloud/terraform/k8s-s390x-build-cluster/data.tf

@@ -0,0 +1,55 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+data "ibm_resource_group" "resource_group" {
+  name = "rg-build-cluster"
+}
+
+data "ibm_is_vpc" "vpc" {
+  name = "k8s-s390x-vpc"
+}
+
+data "ibm_is_subnet" "subnet" {
+  name = "k8s-s390x-subnet"
+}
+
+data "ibm_is_security_group" "bastion" {
+  name = "k8s-vpc-s390x-bastion-sg"
+  vpc  = data.ibm_is_vpc.vpc.id
+}
+
+data "ibm_is_security_group" "control_plane_sg" {
+  name = "k8s-vpc-s390x-control-plane-sg"
+  vpc  = data.ibm_is_vpc.vpc.id
+}
+
+data "ibm_is_security_group" "worker_sg" {
+  name = "k8s-vpc-s390x-worker-sg"
+  vpc  = data.ibm_is_vpc.vpc.id
+}
+
+data "ibm_sm_arbitrary_secret" "ssh_private_key" {
+  instance_id       = var.secrets_manager_id
+  region            = var.region
+  name              = "zvsi-ssh-private-key"
+  secret_group_name = "default"
+}
+
+data "ibm_sm_arbitrary_secret" "ssh_public_key" {
+  instance_id       = var.secrets_manager_id
+  region            = var.region
+  name              = "zvsi-ssh-public-key"
+  secret_group_name = "default"
+}

infra/ibmcloud/terraform/k8s-s390x-build-cluster/load_balancer.tf

@@ -0,0 +1,51 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+resource "ibm_is_lb" "public" {
+  name            = "k8s-s390x-ci"
+  type            = "public"
+  subnets         = [data.ibm_is_subnet.subnet.id]
+  resource_group  = data.ibm_resource_group.resource_group.id
+  security_groups = [data.ibm_is_security_group.control_plane_sg.id]
+}
+
+resource "ibm_is_lb_pool" "k8s_api_pool" {
+  name                = "k8s-api-server-pool"
+  lb                  = ibm_is_lb.public.id
+  protocol            = "tcp"
+  algorithm           = "round_robin"
+  health_delay        = 5
+  health_retries      = 2
+  health_timeout      = 2
+  health_type         = "tcp"
+  health_monitor_url  = "/"
+  health_monitor_port = var.api_server_port
+}
+
+resource "ibm_is_lb_listener" "k8s_api_listener" {
+  lb           = ibm_is_lb.public.id
+  protocol     = "tcp"
+  port         = var.api_server_port
+  default_pool = ibm_is_lb_pool.k8s_api_pool.pool_id
+}
+
+resource "ibm_is_lb_pool_member" "k8s_api_members" {
+  for_each = ibm_is_instance.control_plane
+
+  lb             = ibm_is_lb.public.id
+  pool           = ibm_is_lb_pool.k8s_api_pool.pool_id
+  port           = var.api_server_port
+  target_address = each.value.primary_network_interface[0].primary_ipv4_address
+}