Hackweek- Refactor BV main.tf with SLC and module

[maximenoel8]

Goal

The goal of this PR is to refactor the BV (Build Validation) pipelines.
It removes all environment-specific main.tf files to create one generic main.tf that is configured using tfvars.

Changes

Pipeline Parameters

• Regroup parameters by pipeline type:
  • Build validation is now grouped in the jenkins_pipelines/environments/build-validation folder.
  • SLE maintenance updates are now grouped in the jenkins_pipelines/environments/sle-maintenance-update folder.
• Update tf_file to point to the new Terraform wrapper:
  • NUE BV and SLE maintenance update: susemanager-ci/terracumber_config/tf_files/templates/template-build-validation-single-provider.tf
  • SLC BV: susemanager-ci/terracumber_config/tf_files/templates/template-build-validation-multi-providers.tf
• Add new static parameters:
  • deployment_tfvars: Points to the related tfvars pipeline description.
  • bin_path / bin_plugins_path / sumaform_backend: Removed these values from parameter declarations

Groovy Logic

File: jenkins_pipelines/environments/common/pipeline-build-validation.groovy

• Declares and uses the new tfvars preparation Python script.
• Declares and uses the new Terraform variables file and environment file.
• Updates the common_params variable to correctly parse the new Terraform variables file for Terracumber.
• Refactors the deploy stage:
  • Parameters parsed in Jenkins are now added to terraform.tfvars rather than via TF ENV.
    • This simplifies setup reading and deployment manipulation.
    • Note: Secrets still use TF ENV parameters to avoid exposing values in the sumaform folder.
  • Uses the new prepare_tfvars.py script to update terraform.tfvars:
  • Concatenates specific platform tfvars descriptions (e.g., mlm51_micro_build_validation_nue.tfvars) and the locations.tfvars file.
  • Removes undeclared minions from terraform.tfvars.
  • Adds parameters parsed via Jenkins to terraform.tfvars.
  • Removes use-tf-resource-cleaner and tf-resources-to-keep parameters (no longer needed with terraform.tfvars).
• Updates getNodesHandler to correctly parse parameters passed to the function (fixes previous issue).

New Python Script: prepare_tfvars.py

Location: jenkins_pipelines/scripts/tf_vars_generator/prepare_tfvars.py

This script handles all tfvars manipulation. It was moved here from Terracumber to avoid issues and improve clarity as a standalone script. It performs the following:

• Concatenates files: -merge-files
• Cleans minions: --clean, --keep-resources, and --delete-all
• Injects values: --inject, --product-version, and --base-os
• Generates from environment: --env-file, --user, and --deploy-retail

New Variables File

Location: terracumber_config/tf_files/variables/build-validation-variables.tf

• Consolidates variables previously declared at the beginning of the old main.tf.
• Declares all variables needed for the main.tf template.
• Most values have no default and must be set at the tfvars level.

New tfvars Files

Locations:

• terracumber_config/tf_files/tfvars/build-validation-tfvars/
• terracumber_config/tf_files/tfvars/sle-update-tfvars/
• terracumber_config/tf_files/tfvars/location.tfvars

location.tfvars

A common file for all environments declaring location-specific variables (NUE and SLC), such as domain, mirror, and extensions.

Environment Specific tfvars

Stores all environment-specific values related to a deployment. Main values include:

• ENVIRONMENT_CONFIGURATION:
  • Minion list (name, mac, image, s390 userid).
  • Product version.
  • Name and URL prefixes.
• BASE_CONFIGURATIONS: List of base configurations (one for NUE; multiple by provider for SLC).
• MAIL_SUBJECT: Environment mail subject.
• MAIL_SUBJECT_ENV_FAIL: Mail subject on failure.
• LOCATION: Either nue or slc (used for location.tfvars).

New Templates

1. Single Provider Template
terracumber_config/tf_files/templates/template-build-validation-single-provider.tf
Used for deployments requiring only one Libvirt and base configuration for x86_64 architecture minions. It links to one default Libvirt and base configuration.

2. Multi-Provider Template
terracumber_config/tf_files/templates/template-build-validation-multi-providers.tf
Used for deploying BV SLC with multiple Libvirt providers and base configurations.

Both templates call the new build_validation module in Sumaform.

Old main.tf descriptions

They were all removed.

Depends on:

• sumaform: Hackweek - Add module BV  uyuni-project/sumaform#1961
• terracumber: Hackweek - Add module BV  uyuni-project/sumaform#1961

Related issue https://github.com/SUSE/spacewalk/issues/29146

[nodeg]

The GitHub Actions fail with:

(...)
╷
  │ Warning: Reference to undefined provider
  │ 
  │   on modules/build_validation/main.tf line 50, in module "base_arm":
  │   50:     libvirt = libvirt.suma-arm
  │ 
  │ There is no explicit declaration for local provider name "libvirt" in
  │ module.build_validation_module.module.base_arm, so OpenTofu is assuming you
  │ mean to pass a configuration for "hashicorp/libvirt".
  │ 
  │ If you also control the child module, add a required_providers entry named
  │ "libvirt" with the source address "hashicorp/libvirt".
  │ 
  │ (and 58 more similar warnings elsewhere)
  ╵
  
  Error: OpenTofu exited with code 1.
  Error: Process completed with exit code 1.

I wonder if we should exclude suma-arm for them?

[maximenoel8]

Green github actions !

[Copilot]

Pull request overview

This PR refactors the Build Validation (BV) and SLE maintenance update pipelines by consolidating environment-specific main.tf files into generic templates configured via tfvars files. The refactoring aims to reduce code duplication and improve maintainability.

Key changes:

• Introduces generic Terraform templates for single-provider and multi-provider deployments
• Creates centralized variable definitions in build-validation-variables.tf
• Consolidates environment-specific configurations into tfvars files organized by pipeline type
• Adds location-specific configuration file (location.tfvars) for common settings
• Removes old environment-specific main.tf files (e.g., Uyuni-Master-build-validation-NUE.tf)
• Adds Python dependency python-hcl2 for tfvars manipulation script

Reviewed changes

Copilot reviewed 64 out of 72 changed files in this pull request and generated 19 comments.

Show a summary per file

File	Description
terracumber_config/tf_files/variables/build-validation-variables.tf	New centralized variable definitions for build validation
terracumber_config/tf_files/tfvars/sle-update-tfvars/*.tfvars	SLE maintenance update configurations with critical syntax errors
terracumber_config/tf_files/tfvars/build-validation-tfvars/*.tfvars	Build validation configurations with critical syntax errors and incorrect location values
terracumber_config/tf_files/tfvars/location.tfvars	Common location-specific settings for NUE and SLC
terracumber_config/tf_files/templates/template-build-validation-*.tf	Generic templates for single and multi-provider deployments with variable reference error
terracumber_config/tf_files/tfvars/PR-tfvars/*.tfvars	PR testing configurations
jenkins_pipelines/scripts/tf_vars_generator/requirements.txt	Python dependency for tfvars manipulation
Various removed .tf files	Old environment-specific main.tf files being replaced

Critical Issues Found: Multiple syntax errors including missing equals signs in variable assignments, incorrect variable names, and wrong location values that must be fixed before merge.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[srbarrios]

The logic is becoming complex in both pipelines and terraform files, I suggest that we describe it well on the Readme file. At least, starting by the changes on this PR.