TSSC Multi-CI Development for RHTAP

This repository contains the innerloop development environment for the Multi-CI translations from the RHTAP Pipelines. This repository can also be directly tested in the CI system as it includes a copy of a sample source app for RHTAP (node.js) as well as a gitops repo for that deployment. This may be moved to a separate repo in future dev mode builds.

This repo includes Jenkins, Gitlab and Github Actions tests of the scripts

The tasks appear in the rhtap directory and are updated manually. Once updated they can be tested locally in the shell or pushed to the appropriate CI system and tested in that CI.

This includes:

1. A Jenkins shared library to be tested in Jenkins as well as Developer Hub as part of RHTAP.
2. A gitlab CI file which can be pushed to gitlab in a fork of this repo to test it
3. A github actions workflow which can be run manually in this repo or pushed to a test repo to validate. When github actions are created for some of the tasks, this will require publishing to github individual to test.

Development Mode

In development mode, the pipeline scripts can be tested using local shell scripts.

bash build-pipeline.sh to run a build which will create the Image, SBOM and other artifacts from your local shell. bash promote-pipeline to run a promotion script which will run the promotion flow, upload SBOM (some systems) and validate Enterprise Contract.

The local execution requires binaries to be installed in your cluster on your path. The shell scripts will print error message if any binaries are missing.

RHTAP-integrated mode

To test the pipelines in a real RHTAP environment:

1. Release the Templates to your forks
2. Deploy RHTAP after changing the DEVELOPER_HUB__CATALOG__URL in your .env (see .env.template) to point to your fork
3. Go to the Developer Hub url in your RHTAP instance, sign in and create a new component
   • Select the CI type that you want to test (Tekton, Jenkins, GitHub actions, ...)
4. Trigger the pipeline that you want to test (typically by creating or merging a PR)
   • Jenkins pipelines need some manual setup first, see Configuring Jenkins and Setting up Jenkins jobs

Deploying RHTAP for development

You'll need an OpenShift cluster for deploying RHTAP. Find the supported OpenShift versions in the Release notes for the latest version of RHTAP (in the Compatibility and support matrix section). If you have acess to the Red Hat Internal Slack workspace, a good way to provision a cluster can be to message Cluster Bot: rosa create <version> 6h.

Once you have a cluster ready:

1. Set up your .env file (or .envrc or whatever you prefer). Copy the .env.template file and fill in the values according to the inline instructions.
2. cd to your local clone of the https://github.com/redhat-appstudio/rhtap-cli repo
   • Compile the CLI with make if you haven't done so yet
3. Source your .env file
4. Run integration-tests/scripts/install.sh
5. When finished, the script will print a Homepage URL, Webhook URL and Callback URL. Go to the GitHub app that you used for RHTAP integration and set these urls in the settings.

Note: once you've set up your .env for the first time, most of the variables will be re-usable for future deployments.

Creating a cosign signing secret

Take a look in hack/create-sign-secret. If you run the build pipeline in development mode this script would have already been used to create the git ignored file tssc/signing-secret-env.sh, but you can run it again manually to generate a fresh signing secret.

When running in Jenkins, each of the three values generated by that script need to be created as "Secret text" credentials in Jenkins. Use the environment variable name as the credential ID and the long encoded string as the secret.

Quick-start for testing the build pipeline and the promote pipeline in development mode

Assuming that QUAY_IO_CREDS_USR, QUAY_IO_CREDS_PSW and MY_QUAY_USER environment variables are set and exported already.

export REKOR_HOST='' TUF_MIRROR=''

# if you have an RHTAP instance deployed including TPA
eval "$(hack/get-trustification-env.sh)"
# otherwise
# export FAIL_IF_TRUSTIFICATION_NOT_CONFIGURED=false

(bash build-pipeline.sh && export MY_IMAGES_TO_VERIFY=$(hack/show-image-ref) && bash promote-pipeline.sh) | tee log.out

Note that the MY_IMAGES_TO_VERIFY is a workaround so that the image being verified is the image that was just built. This is only for development purposes. If MY_IMAGES_TO_VERIFY is not set then the image list will be produced by the gather-deploy-images script as per usual.

Configuring Jenkins

Binaries The agent machines running jenkins (or if on master, that machine will need to have binaries configured for the jenkins user running the pipelines)

These will be checked prior to allowing execution to proceed. If any binaries are missing, there will be an error message printed. Install the required binary and re-run the shell mode or the jenkins agent.

ENV vars:
OK: IMAGE_URL
OK: IMAGE
OK: QUAY_IO_CREDS_USR
OK: QUAY_IO_CREDS_PSW
OK: DISABLE_ACS
OK: GITOPS_AUTH_PASSWORD
OK: POLICY_CONFIGURATION
OK: REKOR_HOST
OK: IGNORE_REKOR
OK: INFO
OK: STRICT
OK: EFFECTIVE_TIME
OK: HOMEDIR
Binaries:
OK: git in /usr/bin/git
OK: curl in /usr/bin/curl
OK: jq in /usr/bin/jq
OK: yq in /usr/local/bin/yq
OK: buildah in /usr/bin/buildah
OK: syft in /mnt/g/wslbin/syft
OK: cosign in /usr/local/bin/cosign
OK: python3 in /usr/bin/python3
Env vars and binaries ok

The library requires some secrets to be defined in your Jenkins cluster

There are utility scripts in the hack directory which allow automated setting of the Jenkins Secrets. You will need local env vars, and your Jenkins access token configured. Run the script and it will tell you what you are missing. It will set the credentials required for the Jenkins pipeline to work.

If you have an RHTAP instance deployed including TPA, run this before running the jenkins-set-secrets script: eval "$(hack/get-trustification-env.sh)". This will set TRUSTIFICATION_* environment variables for you, which jenkins-set-secrets will then deploy to your Jenkins instance.

To set the secrets, run hack/jenkins-set-secrets To validate secrets are set run hack/jenkins-get-secrets

You can also check the credentials view in your Jenkins instance.

Setting up Jenkins jobs

When you create a Jenkins-based component through Developer Hub (see RHTAP-integrated mode), you get a repository with a Jenkinsfile. Import the repository into your Jenkins instance as follows:

1. In the Jenkins homepage, click + New Item
2. Create a project of type Pipeline. Give it the same name as the Developer Hub component name.
3. In the next view, in the Pipeline section, select Definition: Pipeline script from SCM, then select SCM: Git
4. Fill in your Repository URL, fix the Branch Specifier, save the pipeline
5. You can now run the pipeline manually

CI Testing

For ease if testing Jenkins, Gitlab CI and Github Actions, there is a script in the root called ci-test.sh. This script will setup and run the build and promotion pipelines in an automated way. The script will perform the following

1. Reset the build and gitops repositories for each of the CI systems to a base source and application sample, these repos are typically called tssc-dev-source-{ci-name} and tssc-dev-gitops-{ci-name} . Note github Actions may have a different name due to history. The ci-test.sh script will print the names of the repos used as well as any manual steps on creating repos (gitlab)
2. Copy the CI pipelines and env setup for the ci tests into the build and gitops repos. The script can be run with RHTAP connected or not installed. If running without an RHTAP (usefull for testing build, sbom, ec) the scripts will detect that there is no cluster and will disable ACS, REKOR and TRUSTIFICATION. If you have a running cluster, this can also be accomplished with oc logout for simpler testing.'
3. Kick off a build on each CI if manually required.
4. To get automated pull requests, run bash hack/wait-for-gitops-update.sh in a separate window before running the ci-test.sh. It will watch the three gitops repos for updates from build and automatically send a PR to update stage from dev. This can be used to validate a full build/promotion cycle.
5. You can also run hack/rhtap-promote --repo repo-url to trigger a PR for an individual repo.

Gitlab CI, Github Action, Jenkins Configuration

ci-test will configure all three CI systems, specifically secrets for your configuration using the following scripts. When RHTAP auto-configures the CI systems secrets this will no longer be needed. These scripts could also be useful for customers. See the following scripts

hack/ghub-set-vars 
hack/glab-set-vars  
hack/jenkins-set-secrets

Release Process

This section outlines the simplified process for the v1.9+ release cycle, designed for transparency and ease of automation.

Before You Start (Initial Environment Setup)

If you haven't already set up your development environment, follow these steps to ensure your remotes are configured correctly for the release script:

1. Fork the Repository: Navigate to the redhat-appstudio/tssc-dev-multi-ci repository on GitHub and click the Fork button to create a copy in your own namespace.

2. Clone Your Fork:

   git clone git@github.com:<your-github-username>/tssc-dev-multi-ci.git
   cd tssc-dev-multi-ci

3. Add Upstream Remote:

   git remote add upstream git@github.com:redhat-appstudio/tssc-dev-multi-ci.git

4. Verify Remotes: Ensure your remotes contain both origin (your fork) and upstream (the main repo):

   git remote -v

Process Templates

Decide on the next minor version and initialize the environment:

export NEW_VERSION='1.9'

Phase A: Github Release

• Merge Pending PRs: Ensure all approved pull requests are merged into main.

• Run Automation Script:

  • Make sure the following are installed on your client before running script:
    • yq
    • npm
    • For mac clients gnu-sed is needed:
      • brew install gnu-sed
      • PATH="/opt/homebrew/opt/gnu-sed/libexec/gnubin:$PATH"

  ./hack/release.sh

• Merge PR generated if everything looks correct

  • CI will fail, this is expected cause Task Runner Image is not built yet

• Create GitHub Release: Target the new release branch (release-v${NEW_VERSION}.x).

  • Tag & Name Format: v${NEW_VERSION}.0

Phase B: Task Runner Image Build

Must be performed after the GitHub Release is finalized.

• Trigger Image Build: Follow the pattern in Konflux Release Data MR #12268.

Phase C: Miscellaneous

• Announce Release: Notify stakeholders of the new version availability.