Skip to content

Commit 98f0c76

Browse files
scohebclaude
scoheb
and
claude
committed
feat(HUM-2061): optimize filter-advisory-rpms
Refactor managed and internal filter tasks for large advisory handling with oras auth and credential setup. Fix arg-too-long errors using file-based processing. Optimize purl-to-advisory mapping performance. Treat unpublished advisory RPMs as unreleased. Restore advisory as authoritative source of truth. Add internal pipeline and unit tests. Update CI config and local test runner. Original commits: 0e2e9cf, 518504c, 2136628, ab7b1d9, ac9eb61, 2f02b82, 0feb4cc, d8a9469, 11da144, a2dbcc1, 694d187, 3801a22 Signed-off-by: Scott Hebert <shebert@redhat.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent b99b3de commit 98f0c76

12 files changed

Lines changed: 520 additions & 232 deletions

File tree

‎.github/workflows/tekton_task_tests.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ jobs:
3838
taskName=$(basename "${dir}")
3939
taskFile="${dir}/${taskName}.yaml"
4040
if [ -f "${taskFile}" ] \
41-
&& grep -q "name: use-trusted-artifact\|name: create-trusted-artifact" "${taskFile}"
41+
&& grep -q "name: use-trusted-artifact\|name: create-trusted-artifact\|oras push" "${taskFile}"
4242
then
4343
echo "Found a trusted artifacts compatible task: ${taskFile}"
4444
trustedArtifactsBasedTasks+=($dir)

‎pipelines/internal/filter-already-released-advisory-rpms/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,5 +11,6 @@ It returns lists of unreleased RPMs and RPMs found in advisories for digest vali
1111
| origin | The origin workspace where the release CR comes from | No | - |
1212
| advisory_secret_name | The name of the secret that contains the advisory GitLab metadata | No | - |
1313
| internalRequestPipelineRunName | Name of the PipelineRun that requested this pipeline | No | - |
14+
| ociStorage | The OCI repository to store results artifact | No | - |
1415
| taskGitUrl | The url to the git repo where the release-service-catalog tasks to be used are stored | Yes | https://github.com/konflux-ci/release-service-catalog.git |
1516
| taskGitRevision | The revision in the taskGitUrl repo to be used | No | - |

‎pipelines/internal/filter-already-released-advisory-rpms/filter-already-released-advisory-rpms.yaml‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,9 @@ spec:
2323
- name: internalRequestPipelineRunName
2424
type: string
2525
description: Name of the PipelineRun that requested this pipeline
26+
- name: ociStorage
27+
type: string
28+
description: The OCI repository to store results artifact
2629
- name: taskGitUrl
2730
type: string
2831
description: The url to the git repo where the release-service-catalog tasks to be used are stored
@@ -50,13 +53,13 @@ spec:
5053
value: $(params.advisory_secret_name)
5154
- name: internalRequestPipelineRunName
5255
value: $(params.internalRequestPipelineRunName)
56+
- name: ociStorage
57+
value: $(params.ociStorage)
5358
results:
5459
- name: result
5560
value: $(tasks.filter-already-released-advisory-rpms-task.results.result)
56-
- name: unreleased_rpms
57-
value: $(tasks.filter-already-released-advisory-rpms-task.results.unreleased_rpms)
58-
- name: in_advisory_rpms
59-
value: $(tasks.filter-already-released-advisory-rpms-task.results.in_advisory_rpms)
61+
- name: filter_results_artifact
62+
value: $(tasks.filter-already-released-advisory-rpms-task.results.filter_results_artifact)
6063
- name: advisory_url
6164
value: $(tasks.filter-already-released-advisory-rpms-task.results.advisory_url)
6265
- name: advisory_internal_url

‎scripts/run-local-tests.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -277,8 +277,8 @@ classify_tasks() {
277277
continue
278278
fi
279279

280-
# Check if task supports Trusted Artifacts (uses TA step actions)
281-
if grep -q "name: use-trusted-artifact\|name: create-trusted-artifact" "$task_file"; then
280+
# Check if task needs OCI registry (uses TA step actions or oras push)
281+
if grep -q "name: use-trusted-artifact\|name: create-trusted-artifact\|oras push" "$task_file"; then
282282
trusted_artifacts_tasks+=("$item")
283283
else
284284
pvc_tasks+=("$item")

‎tasks/internal/filter-already-released-advisory-rpms-task/README.md‎

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,13 @@ RPMs found in advisories (for digest validation by the calling task).
66

77
## Parameters
88

9-
| Name | Description | Optional | Default value |
10-
|--------------------------------|-----------------------------------------------------------------------|----------|---------------|
11-
| transformedSnapshot | Base64 string of gzipped JSON array of RPM entries with purls | No | - |
12-
| origin | The origin workspace for the release CR | No | - |
13-
| advisory_secret_name | Name of the secret containing advisory GitLab metadata | No | - |
14-
| internalRequestPipelineRunName | Name of the PipelineRun that requested this task | No | - |
15-
| caTrustConfigMapName | The name of the ConfigMap to read CA bundle data from | Yes | trusted-ca |
16-
| caTrustConfigMapKey | The name of the key in the ConfigMap that contains the CA bundle data | Yes | ca-bundle.crt |
9+
| Name | Description | Optional | Default value |
10+
|-------------------------------------------------|--------------------------------------------------------------------------------------------|----------|-----------------------------------------------------|
11+
| transformedSnapshot | Base64 string of gzipped JSON array of RPM entries with purls | No | - |
12+
| origin | The origin workspace for the release CR | No | - |
13+
| advisory_secret_name | Name of the secret containing advisory GitLab metadata | No | - |
14+
| internalRequestPipelineRunName | Name of the PipelineRun that requested this task | No | - |
15+
| ociStorage | The OCI repository to store results artifact | No | - |
16+
| trusted_artifacts_dockerconfig_json_secret_name | The name of the secret that contains the dockerconfig json for trusted artifact operations | Yes | quay-token-konflux-release-trusted-artifacts-secret |
17+
| caTrustConfigMapName | The name of the ConfigMap to read CA bundle data from | Yes | trusted-ca |
18+
| caTrustConfigMapKey | The name of the key in the ConfigMap that contains the CA bundle data | Yes | ca-bundle.crt |

‎tasks/internal/filter-already-released-advisory-rpms-task/filter-already-released-advisory-rpms-task.yaml‎

Lines changed: 141 additions & 58 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,13 @@ spec:
2424
- name: internalRequestPipelineRunName
2525
description: Name of the PipelineRun that requested this task
2626
type: string
27+
- name: ociStorage
28+
description: The OCI repository to store results artifact
29+
type: string
30+
- name: trusted_artifacts_dockerconfig_json_secret_name
31+
type: string
32+
description: The name of the secret that contains the dockerconfig json for trusted artifact operations
33+
default: quay-token-konflux-release-trusted-artifacts-secret
2734
- name: caTrustConfigMapName
2835
type: string
2936
description: The name of the ConfigMap to read CA bundle data from
@@ -39,10 +46,8 @@ spec:
3946
description: The name of the InternalRequest PipelineRun
4047
- name: internalRequestTaskRunName
4148
description: The name of the InternalRequest TaskRun
42-
- name: unreleased_rpms
43-
description: RPMs not found in any advisory (gzipped base64 JSON array)
44-
- name: in_advisory_rpms
45-
description: RPMs found in advisories, need digest validation (gzipped base64 JSON array)
49+
- name: filter_results_artifact
50+
description: OCI artifact reference containing filter results (unreleased_rpms, in_advisory_rpms JSON files)
4651
- name: advisory_url
4752
description: URL of the latest matching advisory when all components are already released
4853
- name: advisory_internal_url
@@ -52,6 +57,11 @@ spec:
5257
secret:
5358
secretName: $(params.advisory_secret_name)
5459
defaultMode: 0444
60+
- name: trusted-artifacts-dockerconfig-secret
61+
secret:
62+
optional: true
63+
secretName: $(params.trusted_artifacts_dockerconfig_json_secret_name)
64+
defaultMode: 0444
5565
- name: trusted-ca
5666
configMap:
5767
name: $(params.caTrustConfigMapName)
@@ -79,13 +89,23 @@ spec:
7989
volumeMounts:
8090
- name: advisory-secret
8191
mountPath: /mnt/advisory_secret
92+
- name: trusted-artifacts-dockerconfig-secret
93+
mountPath: /mnt/trusted_artifacts_dockerconfig
8294
env:
8395
- name: TRANSFORMED_SNAPSHOT
8496
value: "$(params.transformedSnapshot)"
8597
script: |
8698
#!/usr/bin/env bash
8799
set -eo pipefail
88100
101+
# Setup OCI registry credentials for trusted artifacts
102+
# AppSRE clusters do not enable credentials-init in Tekton
103+
TA_DOCKERCONFIG_JSON_PATH="/mnt/trusted_artifacts_dockerconfig/.dockerconfigjson"
104+
if [ -f "${TA_DOCKERCONFIG_JSON_PATH}" ] && [ -s "${TA_DOCKERCONFIG_JSON_PATH}" ]; then
105+
mkdir -p ~/.docker/
106+
cat "${TA_DOCKERCONFIG_JSON_PATH}" > ~/.docker/config.json
107+
fi
108+
89109
GITLAB_HOST="$(cat /mnt/advisory_secret/gitlab_host)"
90110
ACCESS_TOKEN="$(cat /mnt/advisory_secret/gitlab_access_token)"
91111
GIT_AUTHOR_NAME="$(cat /mnt/advisory_secret/git_author_name)"
@@ -144,94 +164,152 @@ spec:
144164
145165
if [[ -z "$EXISTING_ADVISORIES" ]]; then
146166
echo "No existing advisories found. All RPMs are unreleased."
167+
168+
# Create output directory for OCI artifact
169+
RESULTS_DIR=/tmp/filter-results
170+
mkdir -p "$RESULTS_DIR"
171+
147172
# All RPMs are unreleased
148-
UNRELEASED=$(echo "$TRANSFORMED_SNAPSHOT_JSON" | gzip -c | base64 -w 0)
149-
IN_ADVISORY=$(echo "[]" | gzip -c | base64 -w 0)
150-
echo -n "$UNRELEASED" > "$(results.unreleased_rpms.path)"
151-
echo -n "$IN_ADVISORY" > "$(results.in_advisory_rpms.path)"
173+
echo "$TRANSFORMED_SNAPSHOT_JSON" | jq -c '.' > "$RESULTS_DIR/unreleased_rpms.json"
174+
echo "[]" > "$RESULTS_DIR/in_advisory_rpms.json"
175+
176+
# Create tarball
177+
RESULTS_TAR="$RESULTS_DIR/filter-results.tar.gz"
178+
tar -czf "$RESULTS_TAR" -C "$RESULTS_DIR" unreleased_rpms.json in_advisory_rpms.json
179+
180+
# Push to OCI storage
181+
OCI_STORAGE="$(params.ociStorage)"
182+
OCI_TAG="filter-results-$(date +%s)"
183+
OCI_REF="${OCI_STORAGE}:${OCI_TAG}"
184+
185+
echo "Pushing filter results to OCI storage: ${OCI_REF}"
186+
ORAS_OPTS="--annotation=quay.expires-after=1d"
187+
(cd "$RESULTS_DIR" && oras push ${ORAS_OPTS} \
188+
--registry-config <(select-oci-auth "${OCI_STORAGE}") \
189+
"${OCI_REF}" "filter-results.tar.gz:application/gzip")
190+
191+
# Get digest for the artifact reference
192+
DIGEST=$(oras manifest fetch "${OCI_REF}" \
193+
--registry-config <(select-oci-auth "${OCI_STORAGE}") \
194+
--descriptor | jq -r '.digest')
195+
ARTIFACT_REF="oci:${OCI_STORAGE}@${DIGEST}"
196+
197+
echo "Filter results artifact: ${ARTIFACT_REF}"
198+
echo -n "${ARTIFACT_REF}" > "$(results.filter_results_artifact.path)"
152199
echo -n "" > "$(results.advisory_url.path)"
153200
echo -n "" > "$(results.advisory_internal_url.path)"
154201
echo -n "Success" > "$(results.result.path)"
155202
exit 0
156203
fi
157204
158-
# === Phase 4: Extract all purls from all advisories (batch) ===
205+
# === Phase 4: Extract all purls from all advisories ===
159206
echo "Extracting purls from advisories..."
160-
ALL_ADVISORY_PURLS_FILE=/tmp/all_advisory_purls.json
161-
echo "[]" > "$ALL_ADVISORY_PURLS_FILE"
207+
PURL_MAPS_FILE=/tmp/purl_advisory_maps.jsonl
208+
: > "$PURL_MAPS_FILE"
162209
163-
LATEST_ADVISORY_FILE=""
164210
for ADVISORY_SUBDIR in $EXISTING_ADVISORIES; do
165211
ADVISORY_FILE="${ADVISORY_BASE_DIR}/${ADVISORY_SUBDIR}/advisory.yaml"
166212
if [[ -f "$ADVISORY_FILE" ]]; then
167-
ADVISORY_PURLS=$(yq -o=json \
213+
yq -o=json \
168214
'.spec.content.artifacts // [] | [.[].purl] | map(select(. != null))' \
169-
"$ADVISORY_FILE")
170-
MERGED=$(jq --argjson new "$ADVISORY_PURLS" \
171-
'. + $new | unique' "$ALL_ADVISORY_PURLS_FILE")
172-
echo "$MERGED" > "$ALL_ADVISORY_PURLS_FILE"
173-
174-
if [[ -z "$LATEST_ADVISORY_FILE" ]]; then
175-
if [[ "$(jq 'length' <<< "$ADVISORY_PURLS")" -gt 0 ]]; then
176-
LATEST_ADVISORY_FILE="$ADVISORY_FILE"
177-
fi
178-
fi
215+
"$ADVISORY_FILE" \
216+
| jq -c --arg adv "$ADVISORY_FILE" \
217+
'reduce .[] as $p ({}; .[$p] = $adv)' \
218+
>> "$PURL_MAPS_FILE"
179219
fi
180220
done
181221
182-
echo "Total unique purls from advisories: $(jq 'length' "$ALL_ADVISORY_PURLS_FILE")"
222+
PURL_TO_ADVISORY_MAP_FILE=/tmp/purl_to_advisory_map.json
223+
if [[ -s "$PURL_MAPS_FILE" ]]; then
224+
jq -s 'reduce .[] as $m ({}; . * $m)' \
225+
"$PURL_MAPS_FILE" > "$PURL_TO_ADVISORY_MAP_FILE"
226+
else
227+
echo "{}" > "$PURL_TO_ADVISORY_MAP_FILE"
228+
fi
229+
rm -f "$PURL_MAPS_FILE"
230+
231+
echo "Total unique purls from advisories: $(jq 'keys | length' "$PURL_TO_ADVISORY_MAP_FILE")"
183232
184-
# === Phase 5: Categorize RPMs ===
233+
# === Phase 5: Categorize RPMs and track source advisories ===
185234
echo "Categorizing RPMs by advisory presence..."
186235
187236
SNAPSHOT_FILE=/tmp/snapshot.json
188237
echo "$TRANSFORMED_SNAPSHOT_JSON" > "$SNAPSHOT_FILE"
189238
190239
UNRELEASED_FILE=/tmp/unreleased.json
191240
IN_ADVISORY_FILE=/tmp/in_advisory.json
192-
echo "[]" > "$UNRELEASED_FILE"
193-
echo "[]" > "$IN_ADVISORY_FILE"
194-
195-
while IFS= read -r entry; do
196-
entry_file=/tmp/current_entry.json
197-
echo "$entry" > "$entry_file"
198-
199-
purl=$(jq -r '.purl' "$entry_file")
241+
ADVISORY_SET_FILE=/tmp/advisory_set.json
200242
201-
IN_ADVISORY=$(jq --arg purl "$purl" \
202-
'map(select(. == $purl)) | length > 0' "$ALL_ADVISORY_PURLS_FILE")
243+
jq --slurpfile map "$PURL_TO_ADVISORY_MAP_FILE" '
244+
. as $entries | $map[0] as $m |
245+
reduce $entries[] as $e (
246+
{unreleased: [], in_advisory: [], advisories: {}};
247+
if $m[$e.purl] then
248+
.in_advisory += [$e]
249+
| .advisories[$m[$e.purl]] = true
250+
else
251+
.unreleased += [$e]
252+
end
253+
) | {
254+
unreleased,
255+
in_advisory,
256+
advisories: (.advisories | keys)
257+
}
258+
' "$SNAPSHOT_FILE" > /tmp/categorized.json
203259
204-
if [[ "$IN_ADVISORY" == "true" ]]; then
205-
# RPM found in advisory - add to in_advisory list for digest validation
206-
UPDATED=$(jq --slurpfile e "$entry_file" '. + $e' "$IN_ADVISORY_FILE")
207-
echo "$UPDATED" > "$IN_ADVISORY_FILE"
208-
else
209-
# RPM not in any advisory - unreleased
210-
UPDATED=$(jq --slurpfile e "$entry_file" '. + $e' "$UNRELEASED_FILE")
211-
echo "$UPDATED" > "$UNRELEASED_FILE"
212-
fi
213-
214-
rm -f "$entry_file"
215-
done < <(jq -c '.[]' "$SNAPSHOT_FILE")
260+
jq '.unreleased' /tmp/categorized.json > "$UNRELEASED_FILE"
261+
jq '.in_advisory' /tmp/categorized.json > "$IN_ADVISORY_FILE"
262+
jq '.advisories' /tmp/categorized.json > "$ADVISORY_SET_FILE"
263+
rm -f /tmp/categorized.json
216264
217265
# === Phase 6: Output results ===
218266
UNRELEASED_COUNT=$(jq 'length' "$UNRELEASED_FILE")
219267
IN_ADVISORY_COUNT=$(jq 'length' "$IN_ADVISORY_FILE")
268+
ADVISORY_SET_COUNT=$(jq 'length' "$ADVISORY_SET_FILE")
269+
220270
echo "Unreleased RPMs: $UNRELEASED_COUNT"
221271
echo "In-advisory RPMs (need digest validation): $IN_ADVISORY_COUNT"
272+
echo "Unique advisories containing in-advisory RPMs: $ADVISORY_SET_COUNT"
222273
223-
# Encode results
224-
UNRELEASED_ENCODED=$(jq -c '.' "$UNRELEASED_FILE" | gzip -c | base64 -w 0)
225-
IN_ADVISORY_ENCODED=$(jq -c '.' "$IN_ADVISORY_FILE" | gzip -c | base64 -w 0)
274+
# Create output directory for OCI artifact
275+
RESULTS_DIR=/tmp/filter-results
276+
mkdir -p "$RESULTS_DIR"
226277
227-
echo -n "$UNRELEASED_ENCODED" > "$(results.unreleased_rpms.path)"
228-
echo -n "$IN_ADVISORY_ENCODED" > "$(results.in_advisory_rpms.path)"
278+
# Write JSON files (compact format to save space)
279+
jq -c '.' "$UNRELEASED_FILE" > "$RESULTS_DIR/unreleased_rpms.json"
280+
jq -c '.' "$IN_ADVISORY_FILE" > "$RESULTS_DIR/in_advisory_rpms.json"
229281
230-
# Set advisory URL if all RPMs were in advisories
231-
if [[ "$UNRELEASED_COUNT" -eq 0 ]] && [[ -n "$LATEST_ADVISORY_FILE" ]]; then
232-
echo "All RPMs found in advisories."
233-
ADVISORY_TYPE=$(yq -r '.spec.type' "$LATEST_ADVISORY_FILE")
234-
ADVISORY_NAME=$(yq -r '.metadata.name' "$LATEST_ADVISORY_FILE")
282+
# Create tarball
283+
RESULTS_TAR="$RESULTS_DIR/filter-results.tar.gz"
284+
tar -czf "$RESULTS_TAR" -C "$RESULTS_DIR" unreleased_rpms.json in_advisory_rpms.json
285+
286+
# Push to OCI storage
287+
OCI_STORAGE="$(params.ociStorage)"
288+
OCI_TAG="filter-results-$(date +%s)"
289+
OCI_REF="${OCI_STORAGE}:${OCI_TAG}"
290+
291+
echo "Pushing filter results to OCI storage: ${OCI_REF}"
292+
ORAS_OPTS="--annotation=quay.expires-after=1d"
293+
(cd "$RESULTS_DIR" && oras push ${ORAS_OPTS} \
294+
--registry-config <(select-oci-auth "${OCI_STORAGE}") \
295+
"${OCI_REF}" "filter-results.tar.gz:application/gzip")
296+
297+
# Get digest for the artifact reference
298+
DIGEST=$(oras manifest fetch "${OCI_REF}" \
299+
--registry-config <(select-oci-auth "${OCI_STORAGE}") \
300+
--descriptor | jq -r '.digest')
301+
ARTIFACT_REF="oci:${OCI_STORAGE}@${DIGEST}"
302+
303+
echo "Filter results artifact: ${ARTIFACT_REF}"
304+
echo -n "${ARTIFACT_REF}" > "$(results.filter_results_artifact.path)"
305+
306+
# Set advisory URL only if ALL RPMs are in advisories AND they all came from the SAME SINGLE advisory
307+
if [[ "$UNRELEASED_COUNT" -eq 0 ]] && [[ "$ADVISORY_SET_COUNT" -eq 1 ]]; then
308+
SINGLE_ADVISORY_FILE=$(jq -r '.[0]' "$ADVISORY_SET_FILE")
309+
echo "All RPMs found in the same single advisory: ${SINGLE_ADVISORY_FILE}"
310+
311+
ADVISORY_TYPE=$(yq -r '.spec.type' "$SINGLE_ADVISORY_FILE")
312+
ADVISORY_NAME=$(yq -r '.metadata.name' "$SINGLE_ADVISORY_FILE")
235313
236314
if [[ "${GIT_REPO}" == *"/rhtap-release/"* ]]; then
237315
ADVISORY_URL_PREFIX="https://access.stage.redhat.com/errata"
@@ -240,11 +318,16 @@ spec:
240318
fi
241319
242320
LATEST_ADVISORY_URL="${ADVISORY_URL_PREFIX}/${ADVISORY_TYPE}-${ADVISORY_NAME}"
243-
LATEST_ADVISORY_INTERNAL_URL="${GIT_REPO//\.git/}/-/raw/main/${LATEST_ADVISORY_FILE}"
321+
LATEST_ADVISORY_INTERNAL_URL="${GIT_REPO//\.git/}/-/raw/main/${SINGLE_ADVISORY_FILE}"
244322
245323
echo -n "$LATEST_ADVISORY_URL" > "$(results.advisory_url.path)"
246324
echo -n "$LATEST_ADVISORY_INTERNAL_URL" > "$(results.advisory_internal_url.path)"
247325
else
326+
if [[ "$ADVISORY_SET_COUNT" -gt 1 ]]; then
327+
echo "RPMs found in multiple advisories ($ADVISORY_SET_COUNT). Not setting advisory_url."
328+
echo "Advisories:"
329+
jq -r '.[]' "$ADVISORY_SET_FILE"
330+
fi
248331
echo -n "" > "$(results.advisory_url.path)"
249332
echo -n "" > "$(results.advisory_internal_url.path)"
250333
fi

0 commit comments

Comments
 (0)