diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..1af497f8 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,149 @@ +# AGENTS.md + +This file provides guidance when working with code in this repository. + +## Project Overview + +Red Hat Sovereign Enclave (RHSE) is an infrastructure automation framework for deploying OpenShift Container Platform (OCP) on bare metal, supporting both connected and disconnected (air-gapped) environments. It uses Ansible as the primary automation engine, shell scripts for infrastructure management, and a 7-phase deployment pipeline. + +## Local Development Environment + +Ansible and all required tools are available in a local venv: + +```bash +source .venv/bin/activate +``` + +Activate before running any validation or deployment commands locally. + +## Validation Commands + +Run these before submitting changes: + +```bash +make -f Makefile.ci validate # Run all validators +make -f Makefile.ci validate-shell # Shellcheck on all shell scripts +make -f Makefile.ci validate-yaml # Yamllint on all YAML files +make -f Makefile.ci validate-ansible # Ansible-lint on playbooks +make -f Makefile.ci validate-json-schema # JSON schema validation +make -f Makefile.ci validate-tags # Validate Ansible playbook tags +make -f Makefile.ci validate-templates # Validate Jinja2 template rendering +make -f Makefile.ci validate-makefile # Makefile syntax check +``` + +## Deployment Commands + +```bash +# Full deployment (disconnected mode, default) +make deploy-cluster + +# Full deployment (connected mode, skips mirror registry) +ENCLAVE_DEPLOYMENT_MODE=connected make deploy-cluster + +# Custom configuration file +VARS_FILE=config/custom-global.yaml make deploy-cluster + +# Individual deployment phases +make deploy-cluster-prepare # Phase 1: Download binaries & content +make deploy-cluster-mirror # Phase 2: Setup local mirror registry +make deploy-cluster-install # Phase 3: Deploy OCP cluster +make deploy-cluster-post-install # Phase 4: Cluster configuration +make deploy-cluster-operators # Phase 5: Install operators +make deploy-cluster-day2 # Phase 6: Advanced features +make deploy-cluster-discovery # Phase 7: Hardware discovery +``` + +## Local Testing Infrastructure + +```bash +make environment # Create test VMs, networks, BMC emulation (libvirt/KVM) +make provision-landing-zone # Provision Landing Zone VM +make verify-landing-zone # Verify Landing Zone config +make install-enclave # Install Enclave on Landing Zone +make verify-enclave-installation # Verify installation +make verify # Verify infrastructure setup +make clean # Teardown infrastructure + +# Full CI flow simulation +make ci-flow-connected # Connected mode E2E +make ci-flow-disconnected # Disconnected mode E2E +``` + +Requires `DEV_SCRIPTS_PATH` environment variable pointing to a dev-scripts installation. + +## Architecture + +### 7-Phase Deployment Pipeline + +Orchestrated by `playbooks/main.yaml`: + +1. **Prepare** (`01-prepare.yaml`): Downloads OpenShift binaries and content images +2. **Mirror** (`02-mirror.yaml`): Sets up local Quay registry and mirrors images (disconnected only) +3. **Deploy** (`03-deploy.yaml`): Agent-Based Installer (ABI) deployment of OCP +4. **Post-Install** (`04-post-install.yaml`): SSL certificates, DNS, registry configuration +5. **Operators** (`05-operators.yaml`): Installs 18+ Red Hat operators +6. **Day-2** (`06-day2.yaml`): Advanced features, Clair, ACM policies, ML model config +7. **Discovery** (`07-configure-discovery.yaml`): Ironic and hardware discovery setup + +### Key Directories + +- `playbooks/tasks/` - 27 reusable Ansible task files shared across phases +- `playbooks/templates/` - Jinja2 templates for generating cluster configs +- `operators/` - Per-operator directories with subscription and config manifests +- `defaults/` - YAML defaults for operators, catalogs, binaries, images +- `config/` - User-provided configuration (copy from `*.example.yaml` files) +- `scripts/lib/` - Shared Bash libraries: `output.sh`, `config.sh`, `network.sh`, `ssh.sh`, `validation.sh`, `common.sh` + +### Configuration System + +User configuration lives in `config/` (gitignored). Start from examples: + +```bash +cp config/global.example.yaml config/global.yaml +cp config/certificates.example.yaml config/certificates.yaml +cp config/cloud_infra.example.yaml config/cloud_infra.yaml +``` + +Defaults in `defaults/` are merged with user config. The `defaults/operators.yaml` defines which operators are installed and their configurations. + +### Operator Configuration Pattern + +Each operator in `operators//` contains: +- `subscription.yaml` - OLM Subscription manifest +- `operator_group.yaml` (if needed) - OperatorGroup manifest +- Additional config manifests specific to the operator + +Operators are enabled/disabled and configured via `defaults/operators.yaml`. + +### Shell Script Organization + +`scripts/` is organized by function: +- `setup/` - Environment prerequisites and validation +- `infrastructure/` - VM provisioning, BMC emulation (sushy-tools), network setup +- `deployment/` - Cluster deployment orchestration +- `verification/` - Health checks and artifact collection +- `cleanup/` - Infrastructure teardown +- `utils/` - Concurrency control (`with_libvirt_lock.sh`), IP detection +- `diagnostics/` - Troubleshooting and log collection + +### Deployment Modes + +- **Disconnected** (default): Full air-gapped with local Quay mirror registry; mirrors all required images locally before deployment +- **Connected**: Skips mirror phase; pulls directly from Red Hat registries; used for faster development iteration + +## Code Quality Rules + +- Shell scripts: Must pass `shellcheck` (no exceptions) +- YAML: 200-character line limit; `yamllint` with `.yamllint.yml` config +- Ansible: `ansible-lint` with `basic` profile; config in `.ansible-lint` +- Avoid trailing whitespace and lines with only whitespace/tabs + +## Ansible Collections + +Defined in `requirements.yml`. Install with: + +```bash +ansible-galaxy collection install -r requirements.yml +``` + +Required: `ansible.utils`, `kubernetes.core`, `containers.podman`, `community.crypto`, `community.general` diff --git a/defaults/operators.yaml b/defaults/operators.yaml index 8b6300a9..bf0ac14b 100644 --- a/defaults/operators.yaml +++ b/defaults/operators.yaml @@ -7,6 +7,11 @@ operators: init_version: 3.15.0 namespace: quay-enterprise source: cs-redhat-operator-index-v4-20 + allowIngressFrom: + - name: ingress + namespaceSelector: + matchLabels: + network.openshift.io/policy-group: ingress - name: multicluster-engine version: 2.10.1 @@ -14,6 +19,15 @@ operators: init_version: 2.10.0 namespace: multicluster-engine source: cs-redhat-operator-index-v4-20 + allowIngressFrom: + - name: acm + namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: open-cluster-management + - name: console + namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: openshift-console - name: advanced-cluster-management version: 2.15.1 @@ -21,6 +35,15 @@ operators: init_version: 2.15.0 namespace: open-cluster-management source: cs-redhat-operator-index-v4-20 + allowIngressFrom: + - name: ingress + namespaceSelector: + matchLabels: + network.openshift.io/policy-group: ingress + - name: mce + namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: multicluster-engine - name: cincinnati-operator version: 5.0.3 @@ -30,6 +53,11 @@ operators: source: cs-redhat-operator-index-v4-20 csvNames: - update-service-operator + allowIngressFrom: + - name: ingress + namespaceSelector: + matchLabels: + network.openshift.io/policy-group: ingress - name: openshift-gitops-operator version: 1.19.2 @@ -46,6 +74,11 @@ operators: namespace: openshift-pipelines source: cs-redhat-operator-index-v4-20 global: true + allowIngressFrom: + - name: ingress + namespaceSelector: + matchLabels: + network.openshift.io/policy-group: ingress - name: netobserv-operator version: 1.11.0 @@ -56,6 +89,11 @@ operators: global: true csvNames: - network-observability-operator + allowIngressFrom: + - name: console + namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: openshift-console - name: cluster-logging version: 6.4.2 diff --git a/operators/openshift-cert-manager-operator/tasks.yaml b/operators/openshift-cert-manager-operator/tasks.yaml index 01246d59..c9d60b65 100644 --- a/operators/openshift-cert-manager-operator/tasks.yaml +++ b/operators/openshift-cert-manager-operator/tasks.yaml @@ -1,4 +1,61 @@ --- +- name: "Ensure same-namespace NetworkPolicy exists in cert-manager" + when: not (operator.disableDefaultNetworkPolicies | default(false)) + kubernetes.core.k8s: + state: present + definition: + apiVersion: networking.k8s.io/v1 + kind: NetworkPolicy + metadata: + name: allow-from-same-namespace + namespace: cert-manager + spec: + podSelector: {} + ingress: + - from: + - podSelector: {} + policyTypes: + - Ingress + +- name: "Ensure monitoring can reach cert-manager" + when: not (operator.disableDefaultNetworkPolicies | default(false)) + kubernetes.core.k8s: + state: present + definition: + apiVersion: networking.k8s.io/v1 + kind: NetworkPolicy + metadata: + name: allow-ingress-from-monitoring + namespace: cert-manager + spec: + podSelector: {} + ingress: + - from: + - namespaceSelector: + matchLabels: + network.openshift.io/policy-group: monitoring + policyTypes: + - Ingress + +- name: "Ensure kube-apiserver can reach cert-manager webhook" + when: not (operator.disableDefaultNetworkPolicies | default(false)) + kubernetes.core.k8s: + state: present + definition: + apiVersion: networking.k8s.io/v1 + kind: NetworkPolicy + metadata: + name: allow-ingress-from-kube-apiserver + namespace: cert-manager + spec: + podSelector: {} + ingress: + - from: + - ipBlock: + cidr: "{{ machineNetwork }}" + policyTypes: + - Ingress + - name: Patch CertManager to set resources kubernetes.core.k8s: state: patched diff --git a/operators/openshift-gitops-operator/tasks.yaml b/operators/openshift-gitops-operator/tasks.yaml index f713feef..ffacd4be 100644 --- a/operators/openshift-gitops-operator/tasks.yaml +++ b/operators/openshift-gitops-operator/tasks.yaml @@ -1,3 +1,81 @@ --- -- ansible.builtin.debug: - msg: "Operator configured properly, no further actions" +# The gitops operator creates the openshift-gitops namespace where ArgoCD +# components run. NetworkPolicies must be applied here separately since +# configure_operator.yaml only manages the operator namespace +# (openshift-gitops-operator). +- name: "Ensure same-namespace NetworkPolicy exists in openshift-gitops" + when: not (operator.disableDefaultNetworkPolicies | default(false)) + kubernetes.core.k8s: + state: present + definition: + apiVersion: networking.k8s.io/v1 + kind: NetworkPolicy + metadata: + name: allow-from-same-namespace + namespace: openshift-gitops + spec: + podSelector: {} + ingress: + - from: + - podSelector: {} + policyTypes: + - Ingress + +- name: "Ensure kube-apiserver can reach openshift-gitops webhooks" + when: not (operator.disableDefaultNetworkPolicies | default(false)) + kubernetes.core.k8s: + state: present + definition: + apiVersion: networking.k8s.io/v1 + kind: NetworkPolicy + metadata: + name: allow-ingress-from-kube-apiserver + namespace: openshift-gitops + spec: + podSelector: {} + ingress: + - from: + - ipBlock: + cidr: "{{ machineNetwork }}" + policyTypes: + - Ingress + +- name: "Ensure monitoring can reach openshift-gitops" + when: not (operator.disableDefaultNetworkPolicies | default(false)) + kubernetes.core.k8s: + state: present + definition: + apiVersion: networking.k8s.io/v1 + kind: NetworkPolicy + metadata: + name: allow-ingress-from-monitoring + namespace: openshift-gitops + spec: + podSelector: {} + ingress: + - from: + - namespaceSelector: + matchLabels: + network.openshift.io/policy-group: monitoring + policyTypes: + - Ingress + +- name: "Ensure ingress can reach openshift-gitops" + when: not (operator.disableDefaultNetworkPolicies | default(false)) + kubernetes.core.k8s: + state: present + definition: + apiVersion: networking.k8s.io/v1 + kind: NetworkPolicy + metadata: + name: allow-ingress-from-ingress + namespace: openshift-gitops + spec: + podSelector: {} + ingress: + - from: + - namespaceSelector: + matchLabels: + network.openshift.io/policy-group: ingress + policyTypes: + - Ingress diff --git a/playbooks/tasks/configure_operator.yaml b/playbooks/tasks/configure_operator.yaml index a715bdcb..35551015 100644 --- a/playbooks/tasks/configure_operator.yaml +++ b/playbooks/tasks/configure_operator.yaml @@ -10,6 +10,109 @@ delay: "{{ k8s_delay }}" until: r_namespace_exists is success +- name: "Ensure same-namespace NetworkPolicy exists" + when: + - operator.namespace != "openshift-operators" + - not (operator.disableDefaultNetworkPolicies | default(false)) + kubernetes.core.k8s: + state: present + definition: + apiVersion: networking.k8s.io/v1 + kind: NetworkPolicy + metadata: + name: allow-from-same-namespace + namespace: "{{ operator.namespace }}" + spec: + podSelector: {} + ingress: + - from: + - podSelector: {} + policyTypes: + - Ingress + register: r_same_namespace_netpol + retries: "{{ k8s_retries }}" + delay: "{{ k8s_delay }}" + until: r_same_namespace_netpol is success + +- name: "Ensure kube-apiserver NetworkPolicy exists" + when: + - operator.namespace != "openshift-operators" + - not (operator.disableDefaultNetworkPolicies | default(false)) + kubernetes.core.k8s: + state: present + definition: + apiVersion: networking.k8s.io/v1 + kind: NetworkPolicy + metadata: + name: allow-ingress-from-kube-apiserver + namespace: "{{ operator.namespace }}" + spec: + podSelector: {} + ingress: + - from: + - ipBlock: + cidr: "{{ machineNetwork }}" + policyTypes: + - Ingress + register: r_kube_apiserver_netpol + retries: "{{ k8s_retries }}" + delay: "{{ k8s_delay }}" + until: r_kube_apiserver_netpol is success + +- name: "Ensure monitoring NetworkPolicy exists" + when: + - operator.namespace != "openshift-operators" + - not (operator.disableDefaultNetworkPolicies | default(false)) + kubernetes.core.k8s: + state: present + definition: + apiVersion: networking.k8s.io/v1 + kind: NetworkPolicy + metadata: + name: allow-ingress-from-monitoring + namespace: "{{ operator.namespace }}" + spec: + podSelector: {} + ingress: + - from: + - namespaceSelector: + matchLabels: + network.openshift.io/policy-group: monitoring + policyTypes: + - Ingress + register: r_monitoring_netpol + retries: "{{ k8s_retries }}" + delay: "{{ k8s_delay }}" + until: r_monitoring_netpol is success + +- name: "Ensure ingress NetworkPolicies exist" + when: + - operator.namespace != "openshift-operators" + - not (operator.disableDefaultNetworkPolicies | default(false)) + kubernetes.core.k8s: + state: present + definition: + apiVersion: networking.k8s.io/v1 + kind: NetworkPolicy + metadata: + name: "allow-ingress-from-{{ item.name }}" + namespace: "{{ operator.namespace }}" + spec: + podSelector: {} + ingress: + - from: + - namespaceSelector: + matchLabels: "{{ item.namespaceSelector.matchLabels }}" + policyTypes: + - Ingress + loop: "{{ operator.allowIngressFrom | default([]) }}" + loop_control: + label: "{{ item.name }}" + register: r_ingress_netpols + retries: "{{ k8s_retries }}" + delay: "{{ k8s_delay }}" + until: r_ingress_netpols is success + - name: "Check if OperatorGroup already exists in namespace" when: operator.namespace != "openshift-operators" kubernetes.core.k8s_info: diff --git a/plugins/odf/plugin.yaml b/plugins/odf/plugin.yaml index 61f11b36..c6c5cf9d 100644 --- a/plugins/odf/plugin.yaml +++ b/plugins/odf/plugin.yaml @@ -12,6 +12,11 @@ operators: namespace: openshift-storage source: cs-redhat-operator-index-v4-20 csvMirror: true + allowIngressFrom: + - name: ingress + namespaceSelector: + matchLabels: + network.openshift.io/policy-group: ingress csvNames: - odf-operator - odf-dependencies diff --git a/schemas/operators.yaml b/schemas/operators.yaml index 2bf61830..cca02464 100644 --- a/schemas/operators.yaml +++ b/schemas/operators.yaml @@ -41,6 +41,48 @@ definitions: global: type: boolean description: Configure operator to watch the entire cluster. + disableDefaultNetworkPolicies: + type: boolean + description: >- + Opt out of all three default NetworkPolicies: allow-from-same-namespace, + allow-ingress-from-kube-apiserver, and allow-ingress-from-monitoring. + When true, no default ingress rules are created and allowIngressFrom is forbidden. + allowIngressFrom: + type: array + description: Additional ingress NetworkPolicies based on namespace label selectors. + items: + type: object + additionalProperties: false + required: + - name + - namespaceSelector + properties: + name: + type: string + description: Identifier used to name the NetworkPolicy (allow-ingress-from-). + pattern: "^[a-z0-9]([a-z0-9-]*[a-z0-9])?$" + maxLength: 253 + namespaceSelector: + type: object + additionalProperties: false + required: + - matchLabels + properties: + matchLabels: + type: object + description: Label selector to match source namespaces. + minProperties: 1 + additionalProperties: + type: string + if: + required: + - disableDefaultNetworkPolicies + properties: + disableDefaultNetworkPolicies: + const: true + then: + properties: + allowIngressFrom: false dependencies: csvMirror: - csvNames diff --git a/schemas/plugin.yaml b/schemas/plugin.yaml index 0b3961ea..45118ad0 100644 --- a/schemas/plugin.yaml +++ b/schemas/plugin.yaml @@ -42,6 +42,48 @@ definitions: global: type: boolean description: Configure operator to watch the entire cluster. + disableDefaultNetworkPolicies: + type: boolean + description: >- + Opt out of all three default NetworkPolicies: allow-from-same-namespace, + allow-ingress-from-kube-apiserver, and allow-ingress-from-monitoring. + When true, no default ingress rules are created and allowIngressFrom is forbidden. + allowIngressFrom: + type: array + description: Additional ingress NetworkPolicies based on namespace label selectors. + items: + type: object + additionalProperties: false + required: + - name + - namespaceSelector + properties: + name: + type: string + description: Identifier used to name the NetworkPolicy (allow-ingress-from-). + pattern: "^[a-z0-9]([a-z0-9-]*[a-z0-9])?$" + maxLength: 253 + namespaceSelector: + type: object + additionalProperties: false + required: + - matchLabels + properties: + matchLabels: + type: object + description: Label selector to match source namespaces. + minProperties: 1 + additionalProperties: + type: string + if: + required: + - disableDefaultNetworkPolicies + properties: + disableDefaultNetworkPolicies: + const: true + then: + properties: + allowIngressFrom: false dependencies: csvMirror: - csvNames