Skip to content

Commit 16715dc

Browse files
authored
keeping-up-with-latest: examples and demos (#34)
Add some examples and demos to go along with the Keeping Up With Latest workshop.
1 parent cb4c5f1 commit 16715dc

7 files changed

Lines changed: 398 additions & 0 deletions

File tree

keeping-up-with-latest/README.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Keeping Up With Latest
2+
3+
This directory contains demos and examples you can refer to during the Keeping
4+
Up With Latest workshop.
Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# Cleanup
2+
3+
Examples of how you could implement cleanup and garbage collection policies to
4+
control the amount of data stored in registry caches and mirrors.
5+
6+
## List Unused Files in Artifactory
7+
8+
List files in an Artifactory repository that have not been downloaded in X
9+
number of days. This could serve as the input for an automated cleanup process.
10+
11+
Export your Artifactory URL and token.
12+
13+
```bash
14+
export ARTIFACTORY_TOKEN="YOUR_AUTH_TOKEN_HERE"
15+
export ARTIFACTORY_URL="https://foo.jfrog.io/artifactory"
16+
```
17+
18+
Run the script with the name of the repository and the number of days as
19+
arguments.
20+
21+
For instance, list files in `my-repository-cache` that haven't been downloaded
22+
in the last 30 days.
23+
24+
```bash
25+
./artifactory-list-unused-files.sh my-repository-cache 30
26+
```
27+
28+
The output will look like this:
29+
30+
```
31+
my-repository-cache/python/3.13.0/list.manifest.json
32+
my-repository-cache/python/sha256__d23d952e9faa8d2884fd71d4473f65951cb5e0fb41f91d7d0db063bf77f7c56b/manifest.json
33+
my-repository-cache/python/sha256__d23d952e9faa8d2884fd71d4473f65951cb5e0fb41f91d7d0db063bf77f7c56b/sha256__549e1c85c06388e2c7f9783321f2902bc8059e56cf82e364dbdcb73f162cb621
34+
my-repository-cache/python/sha256__d23d952e9faa8d2884fd71d4473f65951cb5e0fb41f91d7d0db063bf77f7c56b/sha256__f5daf33d6ac5236e5a2b1d816d772663ee229eaee093e67092e40aecc022bb1f
35+
```
Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
#!/bin/bash
2+
3+
set -euo pipefail
4+
5+
artifactory_url=${ARTIFACTORY_URL:?Must set ARTIFACTORY_URL}
6+
token=${ARTIFACTORY_TOKEN:?Must set ARTIFACTORY_TOKEN}
7+
repository="${1:?Must provide repository as the first argument}"
8+
threshold_days="${2:?Must provide threshold in days as the second argument}"
9+
10+
# Use the search API to discover files in the repository
11+
files=$(curl "${artifactory_url}/api/search/aql" \
12+
-sSf \
13+
-X POST \
14+
-H "Authorization: Bearer ${token}" \
15+
-H "Content-Type: text/plain" \
16+
-d "items.find({\"repo\": {\"\$eq\":\"${repository}\"},\"type\":{\"\$eq\":\"file\"}}).include(\"repo\", \"path\", \"name\")" \
17+
| jq -rc '.results[]'
18+
)
19+
20+
# Get the stats for each file and use the lastDownloaded timestamp to
21+
# find files that haven't been downloaded since X number of days ago
22+
threshold=$((($(date +%s)-((24*60*60)*threshold_days))))
23+
while read -r f; do
24+
repo=$(jq -r '.repo' <<<"${f}")
25+
path=$(jq -r '.path' <<<"${f}")
26+
name=$(jq -r '.name' <<<"${f}")
27+
28+
# Ignore any files under .jfrog
29+
if [[ "${path}" == ".jfrog" ]] || [[ "${path}" == .jfrog/* ]]; then
30+
continue
31+
fi
32+
33+
# Ignore any files that were downloaded more recently than the threshold
34+
lastDownloaded=$(curl -sSf -H "Authorization: Bearer ${token}" "${artifactory_url}/api/storage/${repo}/${path}/${name}?stats" | jq -r .lastDownloaded)
35+
if [[ $((lastDownloaded/1000)) -gt ${threshold} ]]; then
36+
continue
37+
fi
38+
39+
# Print the file
40+
echo "${path}/${name}"
41+
done <<<"${files}"
Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
# Diff
2+
3+
Demos and examples of how to compare images and use the comparison to inform
4+
update decisions.
5+
6+
## Chainctl Image Diff
7+
8+
Run `./demo.sh` for a demonstration of how to use `chainctl image diff`.
9+
10+
## Custom Scripts
11+
12+
One of the downsides of `chainctl image diff` is that it requires access to the
13+
Chainguard console APIs, which you may not have as a developer.
14+
15+
It is possible to replicate most of its functionality with some relatively simple
16+
scripting.
17+
18+
### diff-vulns.sh
19+
20+
Scans both images with `grype` and compares the vulnerabilities identified.
21+
22+
#### Requirements
23+
24+
- `crane`
25+
- `grype`
26+
- `jq`
27+
28+
#### Usage
29+
30+
```bash
31+
./diff-vulns.sh cgr.dev/chainguard/python:latest-dev cgr.dev/chainguard/python:latest
32+
```
33+
34+
The output will look something like this.
35+
36+
```json
37+
{
38+
"added": [],
39+
"removed": [
40+
{
41+
"id": "CVE-2024-58251",
42+
"severity": "Low"
43+
},
44+
{
45+
"id": "CVE-2025-46394",
46+
"severity": "Low"
47+
}
48+
]
49+
}
50+
```
51+
52+
### diff-sboms.sh
53+
54+
Scans both images with `syft` and compares the packages in each.
55+
56+
#### Requirements
57+
58+
- `crane`
59+
- `jq`
60+
- `syft`
61+
62+
#### Usage
63+
64+
```bash
65+
./diff-sboms.sh cgr.dev/your.org/python:3.12.1 cgr.dev/your.org/python:3.12.2
66+
```
67+
68+
The output will look something like this.
69+
70+
```json
71+
{
72+
"added": [
73+
{
74+
"purl": "pkg:apk/chainguard/libxcrypt@4.4.36-r4?arch=aarch64&distro=chainguard-20230214",
75+
"name": "libxcrypt",
76+
"version": "4.4.36-r4",
77+
"type": "apk"
78+
}
79+
],
80+
"removed": [],
81+
"changed": [
82+
{
83+
"name": "python-3.12",
84+
"type": "apk",
85+
"current": {
86+
"version": "3.12.2-r8",
87+
"reference": "pkg:apk/chainguard/python-3.12@3.12.2-r8?arch=aarch64&distro=chainguard-20230214"
88+
},
89+
"previous": {
90+
"version": "3.12.1-r1",
91+
"reference": "pkg:apk/chainguard/python-3.12@3.12.1-r1?arch=aarch64&distro=chainguard-20230214"
92+
}
93+
}
94+
]
95+
}
96+
```
97+
98+
You can specify the types of packages to include in the comparison as additional
99+
arguments.
100+
101+
```bash
102+
./diff-sboms.sh cgr.dev/your.org/python:3.12.1 cgr.dev/your.org/python:3.12.2 apk python
103+
```
Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
#! env bash
2+
. ../../base.sh
3+
4+
ORGANIZATION="cs-ttt-demo.dev"
5+
6+
tmpdir=$(mktemp -d)
7+
trap "rm -rf ${tmpdir}" EXIT
8+
diff_file="${tmpdir}/diff.json"
9+
10+
clear
11+
banner "Get the digest for the current version of python:latest"
12+
pe "DIGEST_NEW=\$(chainctl image history --parent ${ORGANIZATION} python:latest -o json | jq -r '.[0].digest')"
13+
14+
banner "Get the digest for the previous version of python:latest"
15+
pe "DIGEST_OLD=\$(chainctl image history --parent ${ORGANIZATION} python:latest -o json | jq -r '.[1].digest')"
16+
17+
banner "Run chainctl image diff to compare the two images."
18+
pe "chainctl image diff cgr.dev/${ORGANIZATION}/python@\${DIGEST_OLD} cgr.dev/${ORGANIZATION}/python@\${DIGEST_NEW} > ${diff_file}"
19+
pe "jq -r '.' ${diff_file}"
20+
21+
banner "You could use the diff to decide when to update an image. For instance, check if it removes any vulnerabilities."
22+
pe "jq -e '(.vulnerabilities.removed? | length) > 0' ${diff_file}"
23+
24+
banner "And/or, ignore updates that only change packages versions and don't add or remove any packages."
25+
pe "jq -e '(.packages.added? + .packages.removed? | length) == 0' ${diff_file}"
Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
#!/bin/bash
2+
3+
set -euo pipefail
4+
5+
usage() {
6+
echo "Usage: $0 <image1> <image2> [artifact-type...]"
7+
echo "Scans two container images with syft and diffs the packages."
8+
echo "Outputs JSON with 'added', 'removed' and 'changed' packages."
9+
echo "Optional artifact types filter packages (e.g., apk deb rpm python)."
10+
exit 1
11+
}
12+
13+
if [ $# -lt 2 ]; then
14+
usage
15+
fi
16+
17+
# Check we have the required tools installed to run the script
18+
cmds="
19+
crane
20+
jq
21+
syft
22+
"
23+
missing_cmds=""
24+
for cmd in ${cmds}; do
25+
if ! command -v "${cmd}" &> /dev/null; then
26+
missing_cmds+="${cmd} "
27+
fi
28+
done
29+
if [[ -n "${missing_cmds}" ]]; then
30+
echo "Missing required commands: ${missing_cmds}" >&2
31+
exit 1
32+
fi
33+
34+
# Resolve the images to a digest so we can be sure which images we're scanning
35+
image1=$(crane digest --full-ref "${1}")
36+
image2=$(crane digest --full-ref "${2}")
37+
38+
shift 2
39+
40+
# Remaining arguments are artifact types. Convert them to a json array for jq.
41+
artifact_types='[]'
42+
if [ $# -gt 0 ]; then
43+
artifact_types=$(printf '"%s"\n' "${@}" | jq -s .)
44+
fi
45+
46+
# Save output files to a tempdir
47+
tmpdir=$(mktemp -d)
48+
trap "rm -rf ${tmpdir}" EXIT
49+
sbom1="${tmpdir}/image1-sbom.json"
50+
sbom2="${tmpdir}/image2-sbom.json"
51+
52+
# Generate SBOMs for each image with syft
53+
echo "Generating SBOM for $image1 with syft..." >&2
54+
syft "$image1" -o syft-json > "$sbom1"
55+
56+
echo "Generating SBOM for $image2 with syft..." >&2
57+
syft "$image2" -o syft-json > "$sbom2"
58+
59+
# Diff the sboms with jq
60+
echo "Calculating diff..." >&2
61+
jq -n -s \
62+
--slurpfile sbom1 "$sbom1" \
63+
--slurpfile sbom2 "$sbom2" \
64+
--argjson artifact_types "$artifact_types" '
65+
# Extract packages from a syft-json SBOM
66+
def extract_packages(sbom):
67+
sbom[0].artifacts |
68+
(if ($artifact_types | length) > 0 then map(select(.type as $t | $artifact_types | index($t))) else . end) |
69+
map(
70+
{
71+
purl: .purl,
72+
name: .name,
73+
version: .version,
74+
type: .type
75+
}
76+
) |
77+
unique_by(.purl) |
78+
sort_by(.purl);
79+
80+
# Create a normalized purl we can compare by removing the version.
81+
def normalized_purl(purl):
82+
purl | gsub("@[^?]*";"@0.0.0");
83+
84+
# Identify that packages that have been added, removed and changed between the
85+
# two lists of packages.
86+
def diff_packages(old; new):
87+
{
88+
added: (
89+
new |
90+
map(select(normalized_purl(.purl) as $p | (old | map(normalized_purl(.purl))) | index($p) | not))
91+
),
92+
removed: (
93+
old |
94+
map(select(normalized_purl(.purl) as $np | (new | map(normalized_purl(.purl))) | index($np) | not))
95+
),
96+
changed: ([
97+
new[] as $new_pkg |
98+
old[] as $old_pkg |
99+
select(normalized_purl($new_pkg.purl) == normalized_purl($old_pkg.purl) and $new_pkg.version != $old_pkg.version) |
100+
{
101+
name: $new_pkg.name,
102+
type: $new_pkg.type,
103+
current: {
104+
version: $new_pkg.version,
105+
reference: $new_pkg.purl
106+
},
107+
previous: {
108+
version: $old_pkg.version,
109+
reference: $old_pkg.purl
110+
}
111+
}
112+
])
113+
};
114+
115+
diff_packages(extract_packages($sbom1); extract_packages($sbom2))
116+
'

0 commit comments

Comments
 (0)