Skip to content

Commit 3368893

Browse files
ralphbeanralphbean
committed
Application-layer new-cluster automation
See hack/new-cluster/README.md https://issues.redhat.com/browse/KONFLUX-7471
1 parent b19ea24 commit 3368893

File tree

58 files changed

+5080
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

58 files changed

+5080
-0
lines changed

‎hack/new-cluster/README.md

+67
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
# Application-layer new-cluster automation
2+
3+
This is step 2 of 3 in automating the rollout of a new cluster.
4+
5+
This automation creates yaml files in the infra-deployments repo locally.
6+
7+
## Prerequisite
8+
9+
1. The cluster-layer terraform automation has been run, and the new cluster is up. This entails that the AWS, IBM, and database secrets have been provisioned and injected into vault in the correct paths. This automation will verify that before proceeding.
10+
11+
2. You need the following installed on your system:
12+
13+
* ansible CLI
14+
* vault CLI
15+
* python3-hvac
16+
* python-requests
17+
18+
3. You have variables that correctly describe the new cluster,
19+
20+
* `longname` - Example: `kflux-prd-rh09.abe9`
21+
* `shortname` - Example: `kflux-prd-rh09`
22+
* `cutename` - Example: `rh09`
23+
* `env` - One of `production` or `staging`.
24+
* `network` - One of `public` or `private`.
25+
26+
4. You are connected to the VPN.
27+
28+
## Procedure
29+
30+
**Run the playbook**, which will prompt you for the five variables above:
31+
32+
```
33+
❯ ansible-playbook hack/new-cluster/playbook.yaml
34+
```
35+
36+
When the playbook completes, consult the output by inspecting `git diff`.
37+
38+
If satisifed, commit the results, push, and post a pull-request for review by your peers.
39+
40+
Include a description of steps you took to run and verify the automation in the description of your pull request to expedite review.
41+
42+
## Tips
43+
44+
If you do not want to run all steps, but only a subset **you can use tags** to run only tasks tagged with certain tags. For example, if you do not want to verify the vault settings or generate the applicationset changes, but you only want to generate the component overlays, use the `components` tag, like this:
45+
46+
```
47+
❯ ansible-playbook hack/new-cluster/playbook.yaml --tags components
48+
```
49+
50+
If you don't want to specify the variables at prompts, you can **specify variables when invoking the CLI**, like this:
51+
52+
```
53+
❯ ansible-playbook hack/new-cluster/playbook.yaml -e 'cutename=rh09 shortname=kflux-prd-rh09 longname=kflux-prd-rh09.abe9 env=production network=public'
54+
```
55+
56+
If you are **nervous about drift** between the current application manifests and those produced by this automation, you can inspect the different by running this automation and requesting it to produce the config **for an existing cluster**, and then investigate what changes it may have made by looking at `git diff`, like this.
57+
58+
```
59+
❯ ansible-playbook hack/new-cluster/playbook.yaml --skip-tags vault,chains,github -e 'cutename=rh03 shortname=kflux-prd-rh03 longname=kflux-prd-rh03.nnv1 env=production network=public'
60+
❯ git diff
61+
```
62+
63+
The playbook attempts to determine the correct version of some services by inspecting the `main` branch of their git repos. You can override this by setting commit ids specifically, like this:
64+
65+
```
66+
❯ ansible-playbook hack/new-cluster/playbook.yaml -e 'commit_id_multi_platform_controller=ec950d0cfb87bcfd6e3a79fc2b5ee40989126123 commit_id_build_definitions=ab6b0b8e40e440158e7288c73aff1cf83a2cc8a9 commit_id_tektoncd_results_for_konflux=425fcd0988b50965139238038e0d3bd3cb4f8bbc commit_id_pipeline_service_exporter=9d2439c8a77d2ce0527cc5aea3fc6561b7671b48'
67+
```

‎hack/new-cluster/playbook.yaml

+136
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
---
2+
# This ansible playbook will prepare the application-layer for a new cluster.
3+
#
4+
# Major steps:
5+
# * Provision new github application and insert credentials in vault
6+
# * Provision new chains signing secret and insert credentials in vault
7+
# * Locally write out configuration overlays for a new cluster based on templates.
8+
#
9+
10+
- name: Create and patch YAML files
11+
hosts: localhost
12+
gather_facts: no
13+
14+
vars_prompt:
15+
- name: cutename
16+
prompt: "Cute name of the cluster (like rh09)"
17+
private: false
18+
- name: shortname
19+
prompt: "Short name of the cluster (like kflux-prd-rh09)"
20+
private: false
21+
- name: longname
22+
prompt: "Long name of the cluster (like kflux-prd-rh09.1bz7)"
23+
private: false
24+
- name: env
25+
prompt: "What environment (one of {{allowed_envs}})"
26+
private: false
27+
- name: network
28+
prompt: "What network (one of {{allowed_networks}})"
29+
private: false
30+
31+
vars:
32+
vault_addr: "https://vault.devshift.net"
33+
34+
dst: "{{ lookup('env', 'PWD') }}"
35+
github_organization: "redhat-appstudio"
36+
github_application: "red-hat-konflux-{{ shortname }}"
37+
required_vars:
38+
- shortname
39+
- longname
40+
- env
41+
controlplane_cluster: appsrep09ue1
42+
allowed_envs:
43+
- production
44+
- staging
45+
allowed_networks:
46+
- public
47+
- private
48+
templated_components:
49+
- backup
50+
- build-service
51+
- integration
52+
- konflux-info
53+
- konflux-rbac
54+
- konflux-ui
55+
- kyverno
56+
- mintmaker
57+
- monitoring/logging
58+
- monitoring/prometheus
59+
- multi-platform-controller
60+
- namespace-lister
61+
- pipeline-service
62+
63+
argo_cd_app_files:
64+
- "{{ dst }}/argo-cd-apps/base/all-clusters/infra-deployments/monitoring-workload-logging/monitoring-workload-logging.yaml"
65+
- "{{ dst }}/argo-cd-apps/base/all-clusters/infra-deployments/monitoring-workload-prometheus/monitoring-workload-prometheus.yaml"
66+
- "{{ dst }}/argo-cd-apps/base/member/infra-deployments/build-service/build-service.yaml"
67+
- "{{ dst }}/argo-cd-apps/base/member/infra-deployments/integration/integration.yaml"
68+
- "{{ dst }}/argo-cd-apps/base/member/infra-deployments/konflux-info/konflux-info.yaml"
69+
- "{{ dst }}/argo-cd-apps/base/member/infra-deployments/konflux-rbac/konflux-rbac.yaml"
70+
- "{{ dst }}/argo-cd-apps/base/member/infra-deployments/konflux-ui/konflux-ui.yaml"
71+
- "{{ dst }}/argo-cd-apps/base/member/infra-deployments/kyverno/kyverno.yaml"
72+
- "{{ dst }}/argo-cd-apps/base/member/infra-deployments/mintmaker/mintmaker.yaml"
73+
- "{{ dst }}/argo-cd-apps/base/member/infra-deployments/multi-platform-controller/multi-platform-controller.yaml"
74+
- "{{ dst }}/argo-cd-apps/base/member/infra-deployments/namespace-lister/namespace-lister.yaml"
75+
- "{{ dst }}/argo-cd-apps/base/member/infra-deployments/pipeline-service/pipeline-service.yaml"
76+
77+
expected_vault_secrets:
78+
- 'stonesoup/{{ env }}/platform/ansible/generated/{{ shortname }}/github-app'
79+
- 'stonesoup/{{ env }}/platform/ansible/generated/{{ shortname }}/chains-signing-secret'
80+
- 'stonesoup/{{ env }}/platform/terraform/generated/{{ shortname }}/backup-bucket'
81+
- 'stonesoup/{{ env }}/platform/terraform/generated/{{ shortname }}/plnsvc-database'
82+
- 'stonesoup/{{ env }}/platform/terraform/generated/{{ shortname }}/tekton-bucket'
83+
- 'stonesoup/{{ env }}/platform/terraform/generated/{{ shortname }}/aws-ssh-key'
84+
- 'stonesoup/{{ env }}/platform/terraform/generated/{{ shortname }}/aws-account'
85+
- 'stonesoup/{{ env }}/platform/terraform/generated/{{ shortname }}/ibm-api-key'
86+
- 'stonesoup/{{ env }}/platform/terraform/generated/{{ shortname }}/ibm-s390x-ssh-key'
87+
- 'stonesoup/{{ env }}/platform/terraform/generated/{{ shortname }}/ibm-ppc64le-ssh-key'
88+
89+
github_commit_vars:
90+
- slug: openshift-pipelines/tektoncd-results-for-konflux
91+
varname: commit_id_tektoncd_results_for_konflux
92+
- slug: openshift-pipelines/pipeline-service-exporter
93+
varname: commit_id_pipeline_service_exporter
94+
- slug: konflux-ci/build-definitions
95+
varname: commit_id_build_definitions
96+
- slug: konflux-ci/multi-platform-controller
97+
varname: commit_id_multi_platform_controller
98+
99+
pre_tasks:
100+
- name: Validate and initialize variables
101+
tags: [github]
102+
include_tasks: tasks/variables.yaml
103+
104+
tasks:
105+
- name: Login to vault
106+
tags: [vault]
107+
include_tasks: tasks/vault/login.yaml
108+
109+
- name: Ensure the chains signing secret exists
110+
tags: [chains]
111+
include_tasks: tasks/vault/chains.yaml
112+
113+
- name: Ensure the github application exists
114+
tags: [github]
115+
include_tasks: tasks/github/app.yaml
116+
117+
- name: Verify all other vault secrets exist as expected
118+
tags: [vault]
119+
include_tasks: tasks/vault/verify.yaml
120+
loop: "{{ expected_vault_secrets }}"
121+
loop_control:
122+
loop_var: path
123+
124+
- name: Patch ApplicationSet files to include new cluster reference
125+
tags: [applicationsets]
126+
include_tasks: tasks/applicationsets.yaml
127+
loop: "{{ argo_cd_app_files }}"
128+
loop_control:
129+
loop_var: filename
130+
131+
- name: Copy cluster-specific component manifests
132+
tags: [components]
133+
include_tasks: tasks/components.yaml
134+
loop: "{{ templated_components }}"
135+
loop_control:
136+
loop_var: component

‎hack/new-cluster/tasks/applicationsets.yaml

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
- name: Look for {{ shortname }} in ApplicationSet {{ filename.split("/")[-1] }}
2+
tags: [applicationsets]
3+
command: |
4+
yq '.spec.generators[].merge.generators[] | select(has("list")) | .list.elements[] | select(.nameNormalized == "{{ shortname }}" )' {{ filename }}
5+
register: result
6+
changed_when: result.stdout == ""
7+
8+
- name: Insert {{ shortname }} into ApplicationSet {{ filename.split("/")[-1] }}
9+
tags: [applicationsets]
10+
command: |
11+
yq -i eval '(.spec.generators[].merge.generators[] | select(has("list")) | .list.elements) += {"nameNormalized": "{{ shortname }}", "values.clusterDir": "{{ shortname }}"}' {{ filename }}
12+
when: result.stdout == ""

‎hack/new-cluster/tasks/components.yaml

+36
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
- name: "Create the components/{{ component }}/{{ env }}/{{ shortname }}/ directory if it does not exist"
2+
tags: [components]
3+
file:
4+
path: "{{ dst }}/components/{{ component }}/{{ env }}/{{ shortname }}/"
5+
state: directory
6+
mode: '0755'
7+
8+
- name: Set subdir default value
9+
tags: [components]
10+
set_fact:
11+
subdir: "{{ env }}"
12+
13+
- name: Adjust subdir for special MPC case
14+
tags: [components]
15+
set_fact:
16+
subdir: "{{ env }}-downstream"
17+
when: network == 'private' and component == 'multi-platform-controller'
18+
19+
- name: "Create any subdirectories if we need them"
20+
tags: [components]
21+
file:
22+
path: "{{ dst }}/components/{{ component }}/{{ subdir }}/{{ shortname }}/{{ item['path'] }}"
23+
state: directory
24+
mode: '0755'
25+
with_filetree:
26+
- "templates/{{ component }}/"
27+
when: "item['state'] == 'directory'"
28+
29+
- name: "Create files from template for components/{{ component }}/{{ subdir }}/{{ shortname }}"
30+
tags: [components]
31+
template:
32+
src: "{{ item['src'] }}"
33+
dest: "{{ dst }}/components/{{ component }}/{{ subdir }}/{{ shortname }}/{{item['path'] }}"
34+
with_filetree:
35+
- "templates/{{ component }}/"
36+
when: "item['state'] == 'file'"

‎hack/new-cluster/tasks/github/app.yaml

+86
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
- name: Check to see if the {{ github_application }} github application exists
2+
tags: [github]
3+
uri:
4+
url: "https://github.com/apps/{{ github_application }}"
5+
method: GET
6+
return_content: no
7+
headers:
8+
Accept: "application/vnd.github.v3+json"
9+
register: exists
10+
failed_when: exists.status not in [200, 404]
11+
12+
# If the app doesn't exist in github, but a key does exist in vault - then
13+
# we're in a situation that doesn't make any sense. Fail hard.
14+
- name: "Verify that it is not already in vault. If it is, that's crazy."
15+
tags: [github]
16+
when: exists.status != 200
17+
command: >
18+
vault kv metadata get stonesoup/{{ env }}/platform/ansible/generated/{{ shortname }}/github-app
19+
register: sanity_check
20+
environment:
21+
VAULT_ADDR: "{{ vault_addr }}"
22+
VAULT_TOKEN: "{{ vault_token }}"
23+
changed_when: false
24+
failed_when: "sanity_check.rc != 2"
25+
26+
# Docs https://docs.github.com/en/apps/sharing-github-apps/registering-a-github-app-from-a-manifest
27+
- name: Create App Manifest
28+
tags: [github]
29+
when: exists.status != 200
30+
set_fact:
31+
app_manifest:
32+
name: "{{ github_application }}"
33+
url: "https://konflux-ui.apps.{{ longname }}.p1.openshiftapps.com"
34+
hook_attributes:
35+
url: "https://pipelines-as-code-controller-openshift-pipelines.apps.{{ longname }}.p1.openshiftapps.com"
36+
active: true
37+
redirect_url: http://localhost:8089
38+
description: >
39+
The app facilitates the integration between Github events and the OpenShift Pipelines Pipelines As Code component running on your production environment for Red Hat Konflux.
40+
41+
This will facilitate the running of any Tekton PipelineRuns you have defined in the `.tekton` folder of your repository based on the GitHub events you have associated with each of those PipelineRuns.
42+
43+
See the Konflux onboarding documentation for more details.
44+
public: true
45+
default_permissions:
46+
checks: write
47+
contents: write
48+
issues: write
49+
pull_requests: write
50+
organization_administration: read
51+
members: read
52+
default_events:
53+
- check_run
54+
- check_suite
55+
- commit_comment
56+
- issue_comment
57+
- pull_request
58+
- push
59+
60+
- name: Go see your browser and click to create the app. The playbook is now awaiting a redirect from github.com.
61+
tags: [github]
62+
when: exists.status != 200
63+
script: "github-app-flow.py {{ github_organization }} '{{ app_manifest | to_json | b64encode }}'"
64+
register: details
65+
66+
- name: Insert that into vault
67+
tags: [github]
68+
when: exists.status != 200
69+
command: >
70+
vault kv put stonesoup/{{ env }}/platform/ansible/generated/{{ shortname }}/github-app
71+
github-application-id={{ details.stdout | from_json | json_query('application_id') }}
72+
github-private-key='{{ details.stdout | from_json | json_query('pem') }}'
73+
webhook.secret={{ details.stdout | from_json | json_query('webhook_secret') }}
74+
environment:
75+
VAULT_ADDR: "{{ vault_addr }}"
76+
VAULT_TOKEN: "{{ vault_token }}"
77+
78+
- name: Verify that we put it in place correctly
79+
tags: [github]
80+
when: exists.status != 200
81+
command: >
82+
vault kv metadata get stonesoup/{{ env }}/platform/ansible/generated/{{ shortname }}/github-app
83+
environment:
84+
VAULT_ADDR: "{{ vault_addr }}"
85+
VAULT_TOKEN: "{{ vault_token }}"
86+
changed_when: false

‎hack/new-cluster/tasks/github/find-latest-commits.yaml

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
- name: Get latest commit SHA from GitHub API for {{ slug }} to be stored as {{ varname }}
2+
tags: [always]
3+
uri:
4+
url: "https://api.github.com/repos/{{ slug }}/commits/main"
5+
method: GET
6+
return_content: yes
7+
headers:
8+
Accept: "application/vnd.github.v3+json"
9+
register: github_commit
10+
when: "vars[varname] is undefined or vars[varname] == ''"
11+
12+
- set_fact: { "{{ varname }}": "{{ github_commit.json.sha }}" }
13+
tags: [always]
14+
when: "vars[varname] is undefined or vars[varname] == ''"

0 commit comments

Comments
 (0)