Skip to content

Commit 3346fcc

Browse files
scohebclaude
scoheb
and
claude
committed
feat(HUM-2061): optimize filter-already-released-advisory-rpms for speed and reliability
- Refactor managed and internal filter tasks for large advisory handling - Add oras authentication and credential setup for OCI artifact access - Fix argument-too-long errors with large RPM lists using file-based processing - Optimize purl-to-advisory mapping build performance - Treat unpublished advisory RPMs as unreleased (correct filtering logic) - Restore advisory as authoritative source of truth for release state - Add internal pipeline for filter task orchestration - Add unit tests for all-released, no-advisories, and partial-release scenarios - Update CI config and local test runner for filter task test support Original commits: 0e2e9cf, 518504c, 2136628, ab7b1d9, ac9eb61, 2f02b82, 0feb4cc, d8a9469, 11da144, a2dbcc1, 694d187, 3801a22 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent b99b3de commit 3346fcc

12 files changed

Lines changed: 520 additions & 232 deletions

File tree

‎.github/workflows/tekton_task_tests.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ jobs:
3838
taskName=$(basename "${dir}")
3939
taskFile="${dir}/${taskName}.yaml"
4040
if [ -f "${taskFile}" ] \
41-
&& grep -q "name: use-trusted-artifact\|name: create-trusted-artifact" "${taskFile}"
41+
&& grep -q "name: use-trusted-artifact\|name: create-trusted-artifact\|oras push" "${taskFile}"
4242
then
4343
echo "Found a trusted artifacts compatible task: ${taskFile}"
4444
trustedArtifactsBasedTasks+=($dir)

‎pipelines/internal/filter-already-released-advisory-rpms/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,5 +11,6 @@ It returns lists of unreleased RPMs and RPMs found in advisories for digest vali
1111
| origin | The origin workspace where the release CR comes from | No | - |
1212
| advisory_secret_name | The name of the secret that contains the advisory GitLab metadata | No | - |
1313
| internalRequestPipelineRunName | Name of the PipelineRun that requested this pipeline | No | - |
14+
| ociStorage | The OCI repository to store results artifact | No | - |
1415
| taskGitUrl | The url to the git repo where the release-service-catalog tasks to be used are stored | Yes | https://github.com/konflux-ci/release-service-catalog.git |
1516
| taskGitRevision | The revision in the taskGitUrl repo to be used | No | - |

‎pipelines/internal/filter-already-released-advisory-rpms/filter-already-released-advisory-rpms.yaml‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,9 @@ spec:
2323
- name: internalRequestPipelineRunName
2424
type: string
2525
description: Name of the PipelineRun that requested this pipeline
26+
- name: ociStorage
27+
type: string
28+
description: The OCI repository to store results artifact
2629
- name: taskGitUrl
2730
type: string
2831
description: The url to the git repo where the release-service-catalog tasks to be used are stored
@@ -50,13 +53,13 @@ spec:
5053
value: $(params.advisory_secret_name)
5154
- name: internalRequestPipelineRunName
5255
value: $(params.internalRequestPipelineRunName)
56+
- name: ociStorage
57+
value: $(params.ociStorage)
5358
results:
5459
- name: result
5560
value: $(tasks.filter-already-released-advisory-rpms-task.results.result)
56-
- name: unreleased_rpms
57-
value: $(tasks.filter-already-released-advisory-rpms-task.results.unreleased_rpms)
58-
- name: in_advisory_rpms
59-
value: $(tasks.filter-already-released-advisory-rpms-task.results.in_advisory_rpms)
61+
- name: filter_results_artifact
62+
value: $(tasks.filter-already-released-advisory-rpms-task.results.filter_results_artifact)
6063
- name: advisory_url
6164
value: $(tasks.filter-already-released-advisory-rpms-task.results.advisory_url)
6265
- name: advisory_internal_url

‎scripts/run-local-tests.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -277,8 +277,8 @@ classify_tasks() {
277277
continue
278278
fi
279279

280-
# Check if task supports Trusted Artifacts (uses TA step actions)
281-
if grep -q "name: use-trusted-artifact\|name: create-trusted-artifact" "$task_file"; then
280+
# Check if task needs OCI registry (uses TA step actions or oras push)
281+
if grep -q "name: use-trusted-artifact\|name: create-trusted-artifact\|oras push" "$task_file"; then
282282
trusted_artifacts_tasks+=("$item")
283283
else
284284
pvc_tasks+=("$item")

‎tasks/internal/filter-already-released-advisory-rpms-task/README.md‎

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,13 @@ RPMs found in advisories (for digest validation by the calling task).
66

77
## Parameters
88

9-
| Name | Description | Optional | Default value |
10-
|--------------------------------|-----------------------------------------------------------------------|----------|---------------|
11-
| transformedSnapshot | Base64 string of gzipped JSON array of RPM entries with purls | No | - |
12-
| origin | The origin workspace for the release CR | No | - |
13-
| advisory_secret_name | Name of the secret containing advisory GitLab metadata | No | - |
14-
| internalRequestPipelineRunName | Name of the PipelineRun that requested this task | No | - |
15-
| caTrustConfigMapName | The name of the ConfigMap to read CA bundle data from | Yes | trusted-ca |
16-
| caTrustConfigMapKey | The name of the key in the ConfigMap that contains the CA bundle data | Yes | ca-bundle.crt |
9+
| Name | Description | Optional | Default value |
10+
|-------------------------------------------------|--------------------------------------------------------------------------------------------|----------|-----------------------------------------------------|
11+
| transformedSnapshot | Base64 string of gzipped JSON array of RPM entries with purls | No | - |
12+
| origin | The origin workspace for the release CR | No | - |
13+
| advisory_secret_name | Name of the secret containing advisory GitLab metadata | No | - |
14+
| internalRequestPipelineRunName | Name of the PipelineRun that requested this task | No | - |
15+
| ociStorage | The OCI repository to store results artifact | No | - |
16+
| trusted_artifacts_dockerconfig_json_secret_name | The name of the secret that contains the dockerconfig json for trusted artifact operations | Yes | quay-token-konflux-release-trusted-artifacts-secret |
17+
| caTrustConfigMapName | The name of the ConfigMap to read CA bundle data from | Yes | trusted-ca |
18+
| caTrustConfigMapKey | The name of the key in the ConfigMap that contains the CA bundle data | Yes | ca-bundle.crt |

‎tasks/internal/filter-already-released-advisory-rpms-task/filter-already-released-advisory-rpms-task.yaml‎

Lines changed: 141 additions & 58 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,13 @@ spec:
2424
- name: internalRequestPipelineRunName
2525
description: Name of the PipelineRun that requested this task
2626
type: string
27+
- name: ociStorage
28+
description: The OCI repository to store results artifact
29+
type: string
30+
- name: trusted_artifacts_dockerconfig_json_secret_name
31+
type: string
32+
description: The name of the secret that contains the dockerconfig json for trusted artifact operations
33+
default: quay-token-konflux-release-trusted-artifacts-secret
2734
- name: caTrustConfigMapName
2835
type: string
2936
description: The name of the ConfigMap to read CA bundle data from
@@ -39,10 +46,8 @@ spec:
3946
description: The name of the InternalRequest PipelineRun
4047
- name: internalRequestTaskRunName
4148
description: The name of the InternalRequest TaskRun
42-
- name: unreleased_rpms
43-
description: RPMs not found in any advisory (gzipped base64 JSON array)
44-
- name: in_advisory_rpms
45-
description: RPMs found in advisories, need digest validation (gzipped base64 JSON array)
49+
- name: filter_results_artifact
50+
description: OCI artifact reference containing filter results (unreleased_rpms, in_advisory_rpms JSON files)
4651
- name: advisory_url
4752
description: URL of the latest matching advisory when all components are already released
4853
- name: advisory_internal_url
@@ -52,6 +57,11 @@ spec:
5257
secret:
5358
secretName: $(params.advisory_secret_name)
5459
defaultMode: 0444
60+
- name: trusted-artifacts-dockerconfig-secret
61+
secret:
62+
optional: true
63+
secretName: $(params.trusted_artifacts_dockerconfig_json_secret_name)
64+
defaultMode: 0444
5565
- name: trusted-ca
5666
configMap:
5767
name: $(params.caTrustConfigMapName)
@@ -79,13 +89,23 @@ spec:
7989
volumeMounts:
8090
- name: advisory-secret
8191
mountPath: /mnt/advisory_secret
92+
- name: trusted-artifacts-dockerconfig-secret
93+
mountPath: /mnt/trusted_artifacts_dockerconfig
8294
env:
8395
- name: TRANSFORMED_SNAPSHOT
8496
value: "$(params.transformedSnapshot)"
8597
script: |
8698
#!/usr/bin/env bash
8799
set -eo pipefail
88100
101+
# Setup OCI registry credentials for trusted artifacts
102+
# AppSRE clusters do not enable credentials-init in Tekton
103+
TA_DOCKERCONFIG_JSON_PATH="/mnt/trusted_artifacts_dockerconfig/.dockerconfigjson"
104+
if [ -f "${TA_DOCKERCONFIG_JSON_PATH}" ] && [ -s "${TA_DOCKERCONFIG_JSON_PATH}" ]; then
105+
mkdir -p ~/.docker/
106+
cat "${TA_DOCKERCONFIG_JSON_PATH}" > ~/.docker/config.json
107+
fi
108+
89109
GITLAB_HOST="$(cat /mnt/advisory_secret/gitlab_host)"
90110
ACCESS_TOKEN="$(cat /mnt/advisory_secret/gitlab_access_token)"
91111
GIT_AUTHOR_NAME="$(cat /mnt/advisory_secret/git_author_name)"
@@ -144,94 +164,152 @@ spec:
144164
145165
if [[ -z "$EXISTING_ADVISORIES" ]]; then
146166
echo "No existing advisories found. All RPMs are unreleased."
167+
168+
# Create output directory for OCI artifact
169+
RESULTS_DIR=/tmp/filter-results
170+
mkdir -p "$RESULTS_DIR"
171+
147172
# All RPMs are unreleased
148-
UNRELEASED=$(echo "$TRANSFORMED_SNAPSHOT_JSON" | gzip -c | base64 -w 0)
149-
IN_ADVISORY=$(echo "[]" | gzip -c | base64 -w 0)
150-
echo -n "$UNRELEASED" > "$(results.unreleased_rpms.path)"
151-
echo -n "$IN_ADVISORY" > "$(results.in_advisory_rpms.path)"
173+
echo "$TRANSFORMED_SNAPSHOT_JSON" | jq -c '.' > "$RESULTS_DIR/unreleased_rpms.json"
174+
echo "[]" > "$RESULTS_DIR/in_advisory_rpms.json"
175+
176+
# Create tarball
177+
RESULTS_TAR="$RESULTS_DIR/filter-results.tar.gz"
178+
tar -czf "$RESULTS_TAR" -C "$RESULTS_DIR" unreleased_rpms.json in_advisory_rpms.json
179+
180+
# Push to OCI storage
181+
OCI_STORAGE="$(params.ociStorage)"
182+
OCI_TAG="filter-results-$(date +%s)"
183+
OCI_REF="${OCI_STORAGE}:${OCI_TAG}"
184+
185+
echo "Pushing filter results to OCI storage: ${OCI_REF}"
186+
ORAS_OPTS="--annotation=quay.expires-after=1d"
187+
(cd "$RESULTS_DIR" && oras push ${ORAS_OPTS} \
188+
--registry-config <(select-oci-auth "${OCI_STORAGE}") \
189+
"${OCI_REF}" "filter-results.tar.gz:application/gzip")
190+
191+
# Get digest for the artifact reference
192+
DIGEST=$(oras manifest fetch "${OCI_REF}" \
193+
--registry-config <(select-oci-auth "${OCI_STORAGE}") \
194+
--descriptor | jq -r '.digest')
195+
ARTIFACT_REF="oci:${OCI_STORAGE}@${DIGEST}"
196+
197+
echo "Filter results artifact: ${ARTIFACT_REF}"
198+
echo -n "${ARTIFACT_REF}" > "$(results.filter_results_artifact.path)"
152199
echo -n "" > "$(results.advisory_url.path)"
153200
echo -n "" > "$(results.advisory_internal_url.path)"
154201
echo -n "Success" > "$(results.result.path)"
155202
exit 0
156203
fi
157204
158-
# === Phase 4: Extract all purls from all advisories (batch) ===
205+
# === Phase 4: Extract all purls from all advisories ===
159206
echo "Extracting purls from advisories..."
160-
ALL_ADVISORY_PURLS_FILE=/tmp/all_advisory_purls.json
161-
echo "[]" > "$ALL_ADVISORY_PURLS_FILE"
207+
PURL_MAPS_FILE=/tmp/purl_advisory_maps.jsonl
208+
: > "$PURL_MAPS_FILE"
162209
163-
LATEST_ADVISORY_FILE=""
164210
for ADVISORY_SUBDIR in $EXISTING_ADVISORIES; do
165211
ADVISORY_FILE="${ADVISORY_BASE_DIR}/${ADVISORY_SUBDIR}/advisory.yaml"
166212
if [[ -f "$ADVISORY_FILE" ]]; then
167-
ADVISORY_PURLS=$(yq -o=json \
213+
yq -o=json \
168214
'.spec.content.artifacts // [] | [.[].purl] | map(select(. != null))' \
169-
"$ADVISORY_FILE")
170-
MERGED=$(jq --argjson new "$ADVISORY_PURLS" \
171-
'. + $new | unique' "$ALL_ADVISORY_PURLS_FILE")
172-
echo "$MERGED" > "$ALL_ADVISORY_PURLS_FILE"
173-
174-
if [[ -z "$LATEST_ADVISORY_FILE" ]]; then
175-
if [[ "$(jq 'length' <<< "$ADVISORY_PURLS")" -gt 0 ]]; then
176-
LATEST_ADVISORY_FILE="$ADVISORY_FILE"
177-
fi
178-
fi
215+
"$ADVISORY_FILE" \
216+
| jq -c --arg adv "$ADVISORY_FILE" \
217+
'reduce .[] as $p ({}; .[$p] = $adv)' \
218+
>> "$PURL_MAPS_FILE"
179219
fi
180220
done
181221
182-
echo "Total unique purls from advisories: $(jq 'length' "$ALL_ADVISORY_PURLS_FILE")"
222+
PURL_TO_ADVISORY_MAP_FILE=/tmp/purl_to_advisory_map.json
223+
if [[ -s "$PURL_MAPS_FILE" ]]; then
224+
jq -s 'reduce .[] as $m ({}; . * $m)' \
225+
"$PURL_MAPS_FILE" > "$PURL_TO_ADVISORY_MAP_FILE"
226+
else
227+
echo "{}" > "$PURL_TO_ADVISORY_MAP_FILE"
228+
fi
229+
rm -f "$PURL_MAPS_FILE"
230+
231+
echo "Total unique purls from advisories: $(jq 'keys | length' "$PURL_TO_ADVISORY_MAP_FILE")"
183232
184-
# === Phase 5: Categorize RPMs ===
233+
# === Phase 5: Categorize RPMs and track source advisories ===
185234
echo "Categorizing RPMs by advisory presence..."
186235
187236
SNAPSHOT_FILE=/tmp/snapshot.json
188237
echo "$TRANSFORMED_SNAPSHOT_JSON" > "$SNAPSHOT_FILE"
189238
190239
UNRELEASED_FILE=/tmp/unreleased.json
191240
IN_ADVISORY_FILE=/tmp/in_advisory.json
192-
echo "[]" > "$UNRELEASED_FILE"
193-
echo "[]" > "$IN_ADVISORY_FILE"
194-
195-
while IFS= read -r entry; do
196-
entry_file=/tmp/current_entry.json
197-
echo "$entry" > "$entry_file"
198-
199-
purl=$(jq -r '.purl' "$entry_file")
241+
ADVISORY_SET_FILE=/tmp/advisory_set.json
200242
201-
IN_ADVISORY=$(jq --arg purl "$purl" \
202-
'map(select(. == $purl)) | length > 0' "$ALL_ADVISORY_PURLS_FILE")
243+
jq --slurpfile map "$PURL_TO_ADVISORY_MAP_FILE" '
244+
. as $entries | $map[0] as $m |
245+
reduce $entries[] as $e (
246+
{unreleased: [], in_advisory: [], advisories: {}};
247+
if $m[$e.purl] then
248+
.in_advisory += [$e]
249+
| .advisories[$m[$e.purl]] = true
250+
else
251+
.unreleased += [$e]
252+
end
253+
) | {
254+
unreleased,
255+
in_advisory,
256+
advisories: (.advisories | keys)
257+
}
258+
' "$SNAPSHOT_FILE" > /tmp/categorized.json
203259
204-
if [[ "$IN_ADVISORY" == "true" ]]; then
205-
# RPM found in advisory - add to in_advisory list for digest validation
206-
UPDATED=$(jq --slurpfile e "$entry_file" '. + $e' "$IN_ADVISORY_FILE")
207-
echo "$UPDATED" > "$IN_ADVISORY_FILE"
208-
else
209-
# RPM not in any advisory - unreleased
210-
UPDATED=$(jq --slurpfile e "$entry_file" '. + $e' "$UNRELEASED_FILE")
211-
echo "$UPDATED" > "$UNRELEASED_FILE"
212-
fi
213-
214-
rm -f "$entry_file"
215-
done < <(jq -c '.[]' "$SNAPSHOT_FILE")
260+
jq '.unreleased' /tmp/categorized.json > "$UNRELEASED_FILE"
261+
jq '.in_advisory' /tmp/categorized.json > "$IN_ADVISORY_FILE"
262+
jq '.advisories' /tmp/categorized.json > "$ADVISORY_SET_FILE"
263+
rm -f /tmp/categorized.json
216264
217265
# === Phase 6: Output results ===
218266
UNRELEASED_COUNT=$(jq 'length' "$UNRELEASED_FILE")
219267
IN_ADVISORY_COUNT=$(jq 'length' "$IN_ADVISORY_FILE")
268+
ADVISORY_SET_COUNT=$(jq 'length' "$ADVISORY_SET_FILE")
269+
220270
echo "Unreleased RPMs: $UNRELEASED_COUNT"
221271
echo "In-advisory RPMs (need digest validation): $IN_ADVISORY_COUNT"
272+
echo "Unique advisories containing in-advisory RPMs: $ADVISORY_SET_COUNT"
222273
223-
# Encode results
224-
UNRELEASED_ENCODED=$(jq -c '.' "$UNRELEASED_FILE" | gzip -c | base64 -w 0)
225-
IN_ADVISORY_ENCODED=$(jq -c '.' "$IN_ADVISORY_FILE" | gzip -c | base64 -w 0)
274+
# Create output directory for OCI artifact
275+
RESULTS_DIR=/tmp/filter-results
276+
mkdir -p "$RESULTS_DIR"
226277
227-
echo -n "$UNRELEASED_ENCODED" > "$(results.unreleased_rpms.path)"
228-
echo -n "$IN_ADVISORY_ENCODED" > "$(results.in_advisory_rpms.path)"
278+
# Write JSON files (compact format to save space)
279+
jq -c '.' "$UNRELEASED_FILE" > "$RESULTS_DIR/unreleased_rpms.json"
280+
jq -c '.' "$IN_ADVISORY_FILE" > "$RESULTS_DIR/in_advisory_rpms.json"
229281
230-
# Set advisory URL if all RPMs were in advisories
231-
if [[ "$UNRELEASED_COUNT" -eq 0 ]] && [[ -n "$LATEST_ADVISORY_FILE" ]]; then
232-
echo "All RPMs found in advisories."
233-
ADVISORY_TYPE=$(yq -r '.spec.type' "$LATEST_ADVISORY_FILE")
234-
ADVISORY_NAME=$(yq -r '.metadata.name' "$LATEST_ADVISORY_FILE")
282+
# Create tarball
283+
RESULTS_TAR="$RESULTS_DIR/filter-results.tar.gz"
284+
tar -czf "$RESULTS_TAR" -C "$RESULTS_DIR" unreleased_rpms.json in_advisory_rpms.json
285+
286+
# Push to OCI storage
287+
OCI_STORAGE="$(params.ociStorage)"
288+
OCI_TAG="filter-results-$(date +%s)"
289+
OCI_REF="${OCI_STORAGE}:${OCI_TAG}"
290+
291+
echo "Pushing filter results to OCI storage: ${OCI_REF}"
292+
ORAS_OPTS="--annotation=quay.expires-after=1d"
293+
(cd "$RESULTS_DIR" && oras push ${ORAS_OPTS} \
294+
--registry-config <(select-oci-auth "${OCI_STORAGE}") \
295+
"${OCI_REF}" "filter-results.tar.gz:application/gzip")
296+
297+
# Get digest for the artifact reference
298+
DIGEST=$(oras manifest fetch "${OCI_REF}" \
299+
--registry-config <(select-oci-auth "${OCI_STORAGE}") \
300+
--descriptor | jq -r '.digest')
301+
ARTIFACT_REF="oci:${OCI_STORAGE}@${DIGEST}"
302+
303+
echo "Filter results artifact: ${ARTIFACT_REF}"
304+
echo -n "${ARTIFACT_REF}" > "$(results.filter_results_artifact.path)"
305+
306+
# Set advisory URL only if ALL RPMs are in advisories AND they all came from the SAME SINGLE advisory
307+
if [[ "$UNRELEASED_COUNT" -eq 0 ]] && [[ "$ADVISORY_SET_COUNT" -eq 1 ]]; then
308+
SINGLE_ADVISORY_FILE=$(jq -r '.[0]' "$ADVISORY_SET_FILE")
309+
echo "All RPMs found in the same single advisory: ${SINGLE_ADVISORY_FILE}"
310+
311+
ADVISORY_TYPE=$(yq -r '.spec.type' "$SINGLE_ADVISORY_FILE")
312+
ADVISORY_NAME=$(yq -r '.metadata.name' "$SINGLE_ADVISORY_FILE")
235313
236314
if [[ "${GIT_REPO}" == *"/rhtap-release/"* ]]; then
237315
ADVISORY_URL_PREFIX="https://access.stage.redhat.com/errata"
@@ -240,11 +318,16 @@ spec:
240318
fi
241319
242320
LATEST_ADVISORY_URL="${ADVISORY_URL_PREFIX}/${ADVISORY_TYPE}-${ADVISORY_NAME}"
243-
LATEST_ADVISORY_INTERNAL_URL="${GIT_REPO//\.git/}/-/raw/main/${LATEST_ADVISORY_FILE}"
321+
LATEST_ADVISORY_INTERNAL_URL="${GIT_REPO//\.git/}/-/raw/main/${SINGLE_ADVISORY_FILE}"
244322
245323
echo -n "$LATEST_ADVISORY_URL" > "$(results.advisory_url.path)"
246324
echo -n "$LATEST_ADVISORY_INTERNAL_URL" > "$(results.advisory_internal_url.path)"
247325
else
326+
if [[ "$ADVISORY_SET_COUNT" -gt 1 ]]; then
327+
echo "RPMs found in multiple advisories ($ADVISORY_SET_COUNT). Not setting advisory_url."
328+
echo "Advisories:"
329+
jq -r '.[]' "$ADVISORY_SET_FILE"
330+
fi
248331
echo -n "" > "$(results.advisory_url.path)"
249332
echo -n "" > "$(results.advisory_internal_url.path)"
250333
fi

0 commit comments

Comments
 (0)