Skip to content

Commit cf81594

Browse files
ccardenosaclaude
ccardenosa
and
claude
committed
feat(telco-kpis): Add hub operator lockdown mechanism for reproducible installations
Implement comprehensive operator version pinning and lockdown mechanism to enable exact reproduction of hub operator deployments. This allows capturing the precise state of installed operators and reinstalling them with identical images, channels, and configurations. Key Features: 1. Hub Lockdown Capture System: - New role: hub_lockdown_capture - Captures installed operator state from hub clusters - Captures CSV versions, channels, catalog sources, and FBC upstream digests - Exports lockdown configuration as JSON file for version control - Playbook: capture-hub-lockdown.yml - Orchestrates capture process 2. Hub Lockdown Application: - Enhanced lockdown_hub_config role to consume lockdown JSON - Automatic operator enrichment with pinned versions and channels - GitLab symlink resolution for lockdown URIs (handles file:// protocol) - Channel override mechanism from lockdown configuration 3. FBC Upstream Digest Capture: - Captures file-based catalog (FBC) image digests for full reproducibility - Ensures exact catalog versions can be restored - Supports both connected and disconnected deployments 4. Lockdown Comparison Tool: - Playbook: compare-hub-lockdown.yml - Compares lockdown configurations - Identifies operator version drift between environments - Generates comparison reports for validation 5. Workarounds Framework: - New role: workarounds - Generic framework for operator deployment workarounds - TALM ManifestWork CRD workaround (fixes TALM installation when MCE without CR) - Executes before operator installation to prevent deployment failures 6. Supporting Infrastructure: - artifact_management role: Standardized artifact collection and JUnit generation - telco_kpis_defaults role: Centralized default variables for telco-kpis tests - Enhanced ocp_version_facts: Fixed multi-arch image version parsing Use Cases: - Capture production hub operator state for disaster recovery - Ensure identical operator versions across dev/staging/production - Enable regression testing with exact operator configurations - Support disconnected deployments with pinned catalog digests - Facilitate compliance requirements for version tracking Integration: - deploy-ocp-operators.yml: Automatically applies lockdown when hub_lockdown_uri provided - Works seamlessly with existing operator deployment workflows - Compatible with both mirror_only and full installation modes Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: Carlos Cardenosa <ccardeno@redhat.com>
1 parent 02fc5f3 commit cf81594

17 files changed

Lines changed: 1056 additions & 36 deletions

File tree

‎ansible.cfg‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ task_output_limit = 10
55
collections_path = ./collections
66
host_key_checking = False
77
force_color = True
8-
roles_path = ./playbooks/roles:./playbooks/compute/roles:./playbooks/infra/roles
8+
roles_path = ./playbooks/roles:./playbooks/compute/roles:./playbooks/infra/roles:./playbooks/telco-kpis/roles
99

1010
[ssh_connection]
1111
retries = 3

‎playbooks/deploy-ocp-operators.yml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -112,6 +112,31 @@
112112
environment:
113113
K8S_AUTH_KUBECONFIG: "{{ kubeconfig }}"
114114
tasks:
115+
# Hub Lockdown Integration (Telco-KPIs)
116+
# Parse hub lockdown JSON and use operators from lockdown
117+
- name: Parse hub lockdown JSON for operator deployment (Telco-KPIs)
118+
ansible.builtin.include_role:
119+
name: lockdown_hub_config
120+
when: hub_lockdown_uri is defined and hub_lockdown_uri | length > 0
121+
122+
# Use operators from hub lockdown if provided, otherwise use operators parameter
123+
- name: Select operator source (lockdown vs parameter)
124+
when: use_hub_lockdown | default(false)
125+
block:
126+
- name: Use operators from hub lockdown JSON (already enriched)
127+
ansible.builtin.set_fact:
128+
operators: "{{ hub_lockdown_operators }}"
129+
130+
- name: Display hub lockdown operators
131+
ansible.builtin.debug:
132+
msg:
133+
- "=========================================="
134+
- "Hub Lockdown Operators: ENABLED"
135+
- "=========================================="
136+
- "Using operators from lockdown JSON:"
137+
- "{{ operators | to_nice_json }}"
138+
- "=========================================="
139+
115140
- name: Install requirements
116141
ansible.builtin.pip:
117142
name:
@@ -225,6 +250,13 @@
225250
msg: "Mirror-only mode enabled. Skipping operator installation."
226251
when: mirror_only | bool
227252

253+
- name: Apply operator dependency and CRD workarounds (TALM/MCE/ACM)
254+
when: not (mirror_only | bool)
255+
ansible.builtin.include_role:
256+
name: workarounds
257+
vars:
258+
workarounds_talm_manifestwork_crd_operators: "{{ operator_input }}"
259+
228260
- name: Install Operators
229261
when: not (mirror_only | bool)
230262
ansible.builtin.include_role:

‎playbooks/roles/ocp_version_facts/tasks/pull-spec-provided.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
(ocp_version_facts_pull_spec | default(''))
1111
| regex_search('(?<=release:)([0-9]+\.[0-9]+\.[0-9]+(?:-[0-9a-zA-Z.-]+)?)')
1212
| default('')
13-
| regex_replace('-(x86|x64|x86_64|aarch64|ppc64le|s390x)$', '')
13+
| regex_replace('-(x86|x64|x86_64|aarch64|ppc64le|s390x|multi)$', '')
1414
}}
1515
1616
- name: Display parsed release version for debugging

‎playbooks/telco-kpis/capture-hub-lockdown.yml‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
---
2+
# Capture hub lockdown JSON for reproducible operator deployments
3+
#
4+
# This playbook generates a complete snapshot of installed hub operators
5+
# including catalog digests, operator CSVs, and deployment metadata.
6+
#
7+
# Usage:
8+
# ansible-playbook playbooks/telco-kpis/capture-hub-lockdown.yml \
9+
# -i inventories/ocp-deployment/build-inventory.py \
10+
# --extra-vars "hub_cluster=dev-kpi-01 ocp_version=4.17 kubeconfig=/path/to/kubeconfig"
11+
12+
- name: Capture Hub Lockdown State
13+
hosts: bastion
14+
gather_facts: true
15+
16+
vars:
17+
lockdown_output_file: "/tmp/hub-lockdown-{{ hub_cluster }}-{{ ansible_date_time.epoch }}.json"
18+
jenkins_build_number: ""
19+
jenkins_job_name: ""
20+
21+
tasks:
22+
- name: Validate required variables
23+
ansible.builtin.assert:
24+
that:
25+
- hub_cluster is defined
26+
- ocp_version is defined
27+
- kubeconfig is defined
28+
fail_msg: "Required variables: hub_cluster, ocp_version, kubeconfig"
29+
30+
- name: Include hub lockdown capture role
31+
ansible.builtin.include_role:
32+
name: hub_lockdown_capture
33+
vars:
34+
output_file: "{{ lockdown_output_file }}"
35+
cluster_name: "{{ hub_cluster }}"
36+
cluster_kubeconfig: "{{ kubeconfig }}"
37+
cluster_ocp_version: "{{ ocp_version }}"
38+
build_number: "{{ jenkins_build_number }}"
39+
job_name: "{{ jenkins_job_name }}"
40+
41+
- name: Ensure artifacts directory exists
42+
ansible.builtin.file:
43+
path: "{{ lookup('env', 'ARTIFACT_DIR') | default('/artifacts', true) }}"
44+
state: directory
45+
mode: '0755'
46+
delegate_to: localhost
47+
48+
- name: Fetch lockdown to artifacts directory
49+
ansible.builtin.fetch:
50+
src: "{{ lockdown_output_file }}"
51+
dest: "{{ lookup('env', 'ARTIFACT_DIR') | default('/artifacts', true) }}/"
52+
flat: true
53+
54+
- name: Display lockdown file location
55+
ansible.builtin.debug:
56+
msg:
57+
- "=========================================="
58+
- "Hub Lockdown Captured Successfully"
59+
- "=========================================="
60+
- "Hub Cluster: {{ hub_cluster }}"
61+
- "OCP Version: {{ ocp_version }}"
62+
- "Output File: {{ lockdown_output_file | basename }}"
63+
- "Artifact Location: {{ lookup('env', 'ARTIFACT_DIR') | default('/artifacts', true) }}/{{ lockdown_output_file | basename }}"
64+
- "=========================================="

‎playbooks/telco-kpis/compare-hub-lockdown.yml‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
---
2+
# Compare generated hub lockdown with input lockdown
3+
#
4+
# This playbook compares two hub lockdown JSON files and reports differences.
5+
# Used for validation when GENERATE_HUB_LOCKDOWN is enabled along with HUB_LOCKDOWN_URI.
6+
#
7+
# Usage:
8+
# ansible-playbook playbooks/telco-kpis/compare-hub-lockdown.yml \
9+
# -i inventories/ocp-deployment/build-inventory.py \
10+
# --extra-vars "generated_lockdown=/path/to/generated.json input_lockdown_uri=https://..."
11+
12+
- name: Compare Hub Lockdown Files
13+
hosts: bastion
14+
gather_facts: false
15+
16+
vars:
17+
comparison_output: "/tmp/lockdown-comparison.txt"
18+
19+
tasks:
20+
- name: Validate required variables
21+
ansible.builtin.assert:
22+
that:
23+
- generated_lockdown is defined
24+
- input_lockdown_uri is defined
25+
fail_msg: "Required variables: generated_lockdown, input_lockdown_uri"
26+
27+
- name: Resolve GitLab symlinks and download input lockdown JSON
28+
ansible.builtin.include_role:
29+
name: lockdown_hub_config
30+
tasks_from: resolve_gitlab_symlinks.yml
31+
vars:
32+
lockdown_url: "{{ input_lockdown_uri }}"
33+
lockdown_dest: /tmp/input-lockdown.json
34+
35+
- name: Check if generated lockdown exists
36+
ansible.builtin.stat:
37+
path: "{{ generated_lockdown }}"
38+
register: generated_stat
39+
40+
- name: Fail if generated lockdown not found
41+
ansible.builtin.fail:
42+
msg: "Generated lockdown file not found: {{ generated_lockdown }}"
43+
when: not generated_stat.stat.exists
44+
45+
- name: Compare lockdown files using jd (JSON diff)
46+
ansible.builtin.shell: |
47+
set -o pipefail
48+
49+
# Try jd first (better JSON diff tool)
50+
if command -v jd &>/dev/null; then
51+
jd -set /tmp/input-lockdown.json {{ generated_lockdown }}
52+
else
53+
# Fallback to jq diff
54+
echo "=== Input Lockdown (from {{ input_lockdown_uri }}) ==="
55+
jq -S . /tmp/input-lockdown.json
56+
echo ""
57+
echo "=== Generated Lockdown ==="
58+
jq -S . {{ generated_lockdown }}
59+
echo ""
60+
echo "=== Differences (- input, + generated) ==="
61+
diff -u <(jq -S . /tmp/input-lockdown.json) <(jq -S . {{ generated_lockdown }}) || true
62+
fi
63+
args:
64+
executable: /bin/bash
65+
register: lockdown_diff
66+
changed_when: false
67+
failed_when: false
68+
69+
- name: Save comparison output
70+
ansible.builtin.copy:
71+
content: "{{ lockdown_diff.stdout }}"
72+
dest: "{{ comparison_output }}"
73+
mode: '0644'
74+
75+
- name: Normalize and checksum input lockdown (excluding metadata)
76+
ansible.builtin.command:
77+
cmd: jq -S 'del(.hub.metadata)' /tmp/input-lockdown.json
78+
register: input_normalized
79+
changed_when: false
80+
81+
- name: Normalize and checksum generated lockdown (excluding metadata)
82+
ansible.builtin.command:
83+
cmd: jq -S 'del(.hub.metadata)' {{ generated_lockdown }}
84+
register: generated_normalized
85+
changed_when: false
86+
87+
- name: Set lockdown match result based on normalized content
88+
ansible.builtin.set_fact:
89+
lockdown_match:
90+
rc: "{{ 0 if input_normalized.stdout == generated_normalized.stdout else 1 }}"
91+
92+
- name: Display comparison results
93+
ansible.builtin.debug:
94+
msg:
95+
- "=========================================="
96+
- "Hub Lockdown Comparison"
97+
- "=========================================="
98+
- "Input Lockdown: {{ input_lockdown_uri }}"
99+
- "Generated Lockdown: {{ generated_lockdown }}"
100+
- "Match Status: {{ 'IDENTICAL ✓' if lockdown_match.rc == 0 else 'DIFFERENT ⚠' }}"
101+
- "=========================================="
102+
103+
- name: Display diff output
104+
ansible.builtin.debug:
105+
msg: "{{ lockdown_diff.stdout_lines }}"
106+
when: lockdown_match.rc != 0
107+
108+
- name: Warn if lockdowns don't match
109+
ansible.builtin.debug:
110+
msg:
111+
- "⚠⚠⚠ WARNING ⚠⚠⚠"
112+
- "Generated lockdown differs from input lockdown!"
113+
- "This may indicate:"
114+
- " - Operator versions were upgraded automatically"
115+
- " - CatalogSource digests changed"
116+
- " - Different operator configuration was deployed"
117+
- ""
118+
- "Review the diff above to understand the differences."
119+
- "Consider updating the input lockdown JSON if the generated version is correct."
120+
when: lockdown_match.rc != 0
121+
122+
- name: Copy comparison output to artifacts (only when ARTIFACT_DIR is explicitly set)
123+
when:
124+
- lockdown_match.rc != 0
125+
- lookup('env', 'ARTIFACT_DIR') | length > 0
126+
block:
127+
- name: Ensure artifacts directory exists
128+
ansible.builtin.file:
129+
path: "{{ lookup('env', 'ARTIFACT_DIR') }}"
130+
state: directory
131+
mode: '0755'
132+
133+
- name: Copy comparison output to artifacts
134+
ansible.builtin.copy:
135+
src: "{{ comparison_output }}"
136+
dest: "{{ lookup('env', 'ARTIFACT_DIR') }}/lockdown-comparison.txt"
137+
mode: '0644'
138+
remote_src: true

0 commit comments

Comments
 (0)