Skip to content

refactor(RELEASE-2471): convert check-data-keys to python #2308

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
johnbieren wants to merge 1 commit into
base: development
Choose a base branch
from
+28 −2,358
Open

refactor(RELEASE-2471): convert check-data-keys to python #2308

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions pipelines/managed/fbc-release/fbc-release.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,8 +165,6 @@ spec:
params:
- name: dataPath
value: "$(tasks.collect-data.results.data)"
- name: schema
value: $(params.taskGitUrl)/raw/$(params.taskGitRevision)/schema/dataKeys.json
- name: systems
value: |
[
Expand Down
2 changes: 0 additions & 2 deletions pipelines/managed/push-artifacts-to-cdn/push-artifacts-to-cdn.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -300,8 +300,6 @@ spec:
params:
- name: dataPath
value: "$(tasks.collect-data.results.data)"
- name: schema
value: $(params.taskGitUrl)/raw/$(params.taskGitRevision)/schema/dataKeys.json
- name: systems
value: |
[
Expand Down
2 changes: 0 additions & 2 deletions pipelines/managed/push-disk-images-to-cdn/push-disk-images-to-cdn.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -259,8 +259,6 @@ spec:
params:
- name: dataPath
value: "$(tasks.collect-data.results.data)"
- name: schema
value: $(params.taskGitUrl)/raw/$(params.taskGitRevision)/schema/dataKeys.json
- name: systems
value: |
[
Expand Down
2 changes: 0 additions & 2 deletions pipelines/managed/push-disk-images-to-marketplaces/push-disk-images-to-marketplaces.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -293,8 +293,6 @@ spec:
params:
- name: dataPath
value: "$(tasks.collect-data.results.data)"
- name: schema
value: $(params.taskGitUrl)/raw/$(params.taskGitRevision)/schema/dataKeys.json
- name: systems
value: |
[
Expand Down
2 changes: 0 additions & 2 deletions pipelines/managed/push-oot-kmods/push-oot-kmods.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,8 +186,6 @@ spec:
params:
- name: dataPath
value: "$(tasks.collect-data.results.data)"
- name: schema
value: $(params.taskGitUrl)/raw/$(params.taskGitRevision)/schema/dataKeys.json
- name: systems
value: |
[
Expand Down
2 changes: 0 additions & 2 deletions pipelines/managed/push-rpms-to-pulp/push-rpms-to-pulp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -686,8 +686,6 @@ spec:
params:
- name: dataPath
value: "$(tasks.collect-data.results.data)"
- name: schema
value: $(params.taskGitUrl)/raw/$(params.taskGitRevision)/schema/dataKeys.json
- name: systems
value: |
[
Expand Down
2 changes: 0 additions & 2 deletions ...kton-task-bundles-to-external-registry/push-tekton-task-bundles-to-external-registry.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,8 +161,6 @@ spec:
params:
- name: dataPath
value: "$(tasks.collect-data.results.data)"
- name: schema
value: $(params.taskGitUrl)/raw/$(params.taskGitRevision)/schema/dataKeys.json
- name: systems
value: |
[
Expand Down
2 changes: 0 additions & 2 deletions pipelines/managed/push-to-addons-registry/push-to-addons-registry.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,8 +162,6 @@ spec:
params:
- name: dataPath
value: "$(tasks.collect-data.results.data)"
- name: schema
value: $(params.taskGitUrl)/raw/$(params.taskGitRevision)/schema/dataKeys.json
- name: systems
value: |
[
Expand Down
2 changes: 0 additions & 2 deletions pipelines/managed/push-to-external-registry/push-to-external-registry.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,8 +168,6 @@ spec:
params:
- name: dataPath
value: "$(tasks.collect-data.results.data)"
- name: schema
value: $(params.taskGitUrl)/raw/$(params.taskGitRevision)/schema/dataKeys.json
- name: systems
value: |
[
Expand Down
2 changes: 0 additions & 2 deletions pipelines/managed/release-to-github/release-to-github.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -353,8 +353,6 @@ spec:
params:
- name: dataPath
value: "$(tasks.collect-data.results.data)"
- name: schema
value: $(params.taskGitUrl)/raw/$(params.taskGitRevision)/schema/dataKeys.json
- name: systems
value: |
[
Expand Down
2 changes: 0 additions & 2 deletions pipelines/managed/release-to-mrrc/release-to-mrrc.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,8 +168,6 @@ spec:
params:
- name: dataPath
value: "$(tasks.collect-data.results.data)"
- name: schema
value: $(params.taskGitUrl)/raw/$(params.taskGitRevision)/schema/dataKeys.json
- name: systems
value: |
[
Expand Down
2 changes: 0 additions & 2 deletions pipelines/managed/release-to-nrrc/release-to-nrrc.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,8 +168,6 @@ spec:
params:
- name: dataPath
value: "$(tasks.collect-data.results.data)"
- name: schema
value: $(params.taskGitUrl)/raw/$(params.taskGitRevision)/schema/dataKeys.json
- name: systems
value: |
[
Expand Down
2 changes: 0 additions & 2 deletions pipelines/managed/rh-advisories/rh-advisories.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -975,8 +975,6 @@ spec:
params:
- name: dataPath
value: "$(tasks.collect-data.results.data)"
- name: schema
value: $(params.taskGitUrl)/raw/$(params.taskGitRevision)/schema/dataKeys.json
- name: systems
value: |
[
Expand Down
2 changes: 0 additions & 2 deletions ...ed/rh-push-helm-chart-to-registry-redhat-io/rh-push-helm-chart-to-registry-redhat-io.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,8 +167,6 @@ spec:
params:
- name: dataPath
value: "$(tasks.collect-data.results.data)"
- name: schema
value: $(params.taskGitUrl)/raw/$(params.taskGitRevision)/schema/dataKeys.json
- name: systems
value: |
[
Expand Down
2 changes: 0 additions & 2 deletions pipelines/managed/rh-push-to-external-registry/rh-push-to-external-registry.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,8 +196,6 @@ spec:
params:
- name: dataPath
value: "$(tasks.collect-data.results.data)"
- name: schema
value: $(params.taskGitUrl)/raw/$(params.taskGitRevision)/schema/dataKeys.json
- name: systems
value: |
[
Expand Down
2 changes: 0 additions & 2 deletions pipelines/managed/rh-push-to-registry-redhat-io/rh-push-to-registry-redhat-io.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,8 +165,6 @@ spec:
params:
- name: dataPath
value: "$(tasks.collect-data.results.data)"
- name: schema
value: $(params.taskGitUrl)/raw/$(params.taskGitRevision)/schema/dataKeys.json
- name: systems
value: |
[
Expand Down
2 changes: 0 additions & 2 deletions pipelines/managed/rhtap-service-push/rhtap-service-push.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,8 +169,6 @@ spec:
params:
- name: dataPath
value: "$(tasks.collect-data.results.data)"
- name: schema
value: $(params.taskGitUrl)/raw/$(params.taskGitRevision)/schema/dataKeys.json
- name: systems
value: |
[
Expand Down
34 changes: 17 additions & 17 deletions tasks/managed/check-data-keys/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,23 +8,23 @@ meaning all the data keys must be allowed and formatted correctly.
For example, if `releaseNotes` is passed as a system and the data file does not have all the required
releaseNotes keys, the schema will give validation errors, and the task will fail.

The validation schema is defined in `schema/dataKeys.json` in this repository.
The validation schema is bundled in the release-service-utils image at
`/home/schemas/dataKeys.json`.

## Parameters

| Name | Description | Optional | Default value |
|-------------------------|----------------------------------------------------------------------------------------------------------------------------|----------|------------------------------------------------------------------------------------------------------------------|
| dataPath | Path to the JSON string of the merged data to use | No | - |
| schema | URL to the JSON schema file to validate the data against | Yes | https://raw.githubusercontent.com/konflux-ci/release-service-catalog/refs/heads/development/schema/dataKeys.json |
| systems | The systems to check that all data keys are present for | Yes | "" |
| ociStorage | The OCI repository where the Trusted Artifacts are stored | Yes | empty |
| ociArtifactExpiresAfter | Expiration date for the trusted artifacts created in the OCI repository. An empty string means the artifacts do not expire | Yes | 1d |
| trustedArtifactsDebug | Flag to enable debug logging in trusted artifacts. Set to a non-empty string to enable | Yes | "" |
| orasOptions | oras options to pass to Trusted Artifacts calls | Yes | "" |
| sourceDataArtifact | Location of trusted artifacts to be used to populate data directory | Yes | "" |
| dataDir | The location where data will be stored | Yes | /var/workdir/release |
| taskGitUrl | The url to the git repo where the release-service-catalog tasks and stepactions to be used are stored | No | - |
| taskGitRevision | The revision in the taskGitUrl repo to be used | No | - |
| caTrustConfigMapName | The name of the ConfigMap to read CA bundle data from | Yes | trusted-ca |
| caTrustConfigMapKey | The name of the key in the ConfigMap that contains the CA bundle data | Yes | ca-bundle.crt |
| caCertPath | Path to CA certificate bundle for TLS verification with self-signed certificates | Yes | /mnt/trusted-ca/ca-bundle.crt |
| Name | Description | Optional | Default value |
|-------------------------|----------------------------------------------------------------------------------------------------------------------------|----------|-------------------------------|
| dataPath | Path to the JSON string of the merged data to use | No | - |
| systems | The systems to check that all data keys are present for | Yes | "" |
| ociStorage | The OCI repository where the Trusted Artifacts are stored | Yes | empty |
| ociArtifactExpiresAfter | Expiration date for the trusted artifacts created in the OCI repository. An empty string means the artifacts do not expire | Yes | 1d |
| trustedArtifactsDebug | Flag to enable debug logging in trusted artifacts. Set to a non-empty string to enable | Yes | "" |
| orasOptions | oras options to pass to Trusted Artifacts calls | Yes | "" |
| sourceDataArtifact | Location of trusted artifacts to be used to populate data directory | Yes | "" |
| dataDir | The location where data will be stored | Yes | /var/workdir/release |
| taskGitUrl | The url to the git repo where the release-service-catalog tasks and stepactions to be used are stored | No | - |
| taskGitRevision | The revision in the taskGitUrl repo to be used | No | - |
| caTrustConfigMapName | The name of the ConfigMap to read CA bundle data from | Yes | trusted-ca |
| caTrustConfigMapKey | The name of the key in the ConfigMap that contains the CA bundle data | Yes | ca-bundle.crt |
| caCertPath | Path to CA certificate bundle for TLS verification with self-signed certificates | Yes | /mnt/trusted-ca/ca-bundle.crt |
Comment thread
johnbieren marked this conversation as resolved.
44 changes: 10 additions & 34 deletions tasks/managed/check-data-keys/check-data-keys.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,15 +16,12 @@ spec:
For example, if `releaseNotes` is passed as a system and the data file does not have all the required
releaseNotes keys, the schema will give validation errors, and the task will fail.

The validation schema is defined in `schema/dataKeys.json` in this repository.
The validation schema is bundled in the release-service-utils image at
`/home/schemas/dataKeys.json`.
params:
- name: dataPath
description: Path to the JSON string of the merged data to use
type: string
- name: schema
description: URL to the JSON schema file to validate the data against
type: string
default: https://raw.githubusercontent.com/konflux-ci/release-service-catalog/refs/heads/development/schema/dataKeys.json
- name: systems
description: The systems to check that all data keys are present for
type: string
Expand Down Expand Up @@ -127,42 +124,21 @@ spec:
- name: caCertPath
value: $(params.caCertPath)
- name: check-data-keys
image: quay.io/konflux-ci/release-service-utils@sha256:5546fa78d3c88d7b6a2e8cff8902f7757f00541d0bbaf113b9f293133894afa3
image: quay.io/konflux-ci/release-service-utils@sha256:282c23415a9995ab3fd6eb79dd314e84aed9b51e96ac3caaa34562c08eb7cc51
computeResources:
limits:
memory: 64Mi
requests:
memory: 64Mi # was exiting with code 137 when set to 32Mi
cpu: 10m
env:
- name: "SCHEMA_FILE"
value: "$(params.schema)"
script: |
#!/usr/bin/env bash
set -ex

if [ -f "/mnt/trusted-ca/ca-bundle.crt" ]; then
export SSL_CERT_FILE="/mnt/trusted-ca/ca-bundle.crt"
fi

if [ ! -f "$(params.dataDir)/$(params.dataPath)" ] ; then
echo "No data JSON was provided."
exit 1
fi

schema="${SCHEMA_FILE/\.git\///}"
if ! curl -sL --fail-with-body --retry 3 --retry-delay 5 --retry-all-errors "$schema" -o /tmp/schema ; then
echo "Failed to download schema file: $schema"
exit 1
fi

# We want this to output the json without expansion
# shellcheck disable=SC2016
jq --argjson systems '$(params.systems)' '.systems += $systems' \
"$(params.dataDir)/$(params.dataPath)" > "/tmp/systems"
mv "/tmp/systems" "$(params.dataDir)/$(params.dataPath)"

check-jsonschema --output-format=text --schemafile "/tmp/schema" "$(params.dataDir)/$(params.dataPath)"
- name: PARAM_DATA_DIR
value: $(params.dataDir)
- name: PARAM_DATA_PATH
value: $(params.dataPath)
- name: PARAM_SYSTEMS
value: $(params.systems)
command: ["/home/scripts/python/tasks/managed/check_data_keys.py"]
- name: create-trusted-artifact
computeResources:
limits:
Expand Down
15 changes: 0 additions & 15 deletions tasks/managed/check-data-keys/tests/mocks.sh
View file
Open in desktop

This file was deleted.

8 changes: 0 additions & 8 deletions tasks/managed/check-data-keys/tests/pre-apply-task-hook.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,6 @@
#!/usr/bin/env bash

TASK_PATH="$1"
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )

# Inject mocks.sh into the task's first step
yq -i '.spec.steps[1].script = load_str("'"$SCRIPT_DIR"'/mocks.sh") + .spec.steps[1].script' "$TASK_PATH"

# Add RBAC so that the SA executing the tests can retrieve configMap
kubectl apply -f .github/resources/crd_rbac.yaml

# Create a configMap with the schema to be used by the task
kubectl delete configmap check-data-keys-schema --ignore-not-found
kubectl create configmap check-data-keys-schema --from-file=dataKeys="$SCRIPT_DIR/../../../../schema/dataKeys.json"
Loading
Loading