ci: notice file issue #626 updates #5126
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adds a script (parseCrates.sh) that parses the output of cargo-deny into a NOTICES file that outputs crate names, licenses, and urls those resepctive crates. The script is used by piping the output of cargo deny with the --layout=crate option to the script, and redirecting the output to the NOTICES file.
metaclips
requested changes
Jun 14, 2023
Comment on lines
+26
to
+68
| for line in $INPUT | ||
| do | ||
|
|
||
| # We have the closing bracket, set OPEN_BRACKET to 0, remove any leading commas from | ||
| # licensing, and build output for this library & licensing | ||
| # NOTE: Important this check is before check for '[', otherwise crates w/o license | ||
| # are presented incorrectly | ||
| if [[ $line == *"]"* ]]; then | ||
| IFS=$' ' | ||
| OPEN_BRACKET=$(expr 0) # reset OPEN_BRACKET status | ||
| LICENSE_STRING="${LICENSE_STRING#,}" # remove any leading comma | ||
| LICENSE_STRING=$(echo $LICENSE_STRING | sed 's/ //g') # remove leading spaces from LICENSES | ||
| LICENSE_STRING=$(echo $LICENSE_STRING | sed 's/,/, /g') # add space after commas in LICENSES | ||
| FILEOUT+=$(printf "\n%1s %-32s %1s %-55s %1s %-80s %1s" "|" "$PROJECT" "|" "$LICENSE_STRING" "|" "$URL" "|" ) | ||
| LICENSE_STRING="" # reset LICENSE_STRING | ||
| IFS=$'\n' | ||
| continue # skip to next iteration/line | ||
| fi | ||
|
|
||
| # closing bracket, lets subtract from OPEN_BRACKET | ||
| if [[ $line == *"licenses ["* ]]; then | ||
| OPEN_BRACKET=$(expr 1) | ||
| continue # skip to next iteration/line | ||
| fi | ||
|
|
||
| # Keep appending license names until we hit closing bracket | ||
| if [[ $OPEN_BRACKET -eq 1 ]]; then | ||
| LICENSE_STRING+=", $line" | ||
| continue # skip to next iteration/line | ||
| fi | ||
|
|
||
| # This line contains our library, version, and URL | ||
| if [[ $OPEN_BRACKET -eq 0 ]] && [[ ${#line} -gt 3 ]]; then | ||
| IFS=$' ' | ||
| read -r PROJECT VERSION URL <<<"$line" | ||
| PROJECT=$(echo "$PROJECT" | sed 's/"//g') | ||
| VERSION=$(echo "$VERSION" | sed 's/"//g') | ||
| URL=$(echo "$URL" | sed 's/"//g') | ||
| URL="${URL/registry+}" # remove 'registry+' from start of URL | ||
| IFS=$'\n' | ||
| fi | ||
|
|
||
| done |
There was a problem hiding this comment.
- JQ has a feature to get all keys. The command below should retrieve all Licence, Crate name and URL
INPUT=$(jq "." $1)
regex="(.*) [0-9]+\.[0-9]+\.[0-9]+ \((.*)\+(.*)\)"
jq --raw-output "keys[]" <<< $INPUT |
while IFS= read -r key; do
license=$(jq --arg key "$key" --raw-output '.[$key].licenses | join(", ")' <<< $INPUT)
if [[ $key =~ $regex ]]; then
echo "Package name = ${BASH_REMATCH[1]}"
echo "Pulled from registry or path = ${BASH_REMATCH[2]}"
echo "URL of crate = ${BASH_REMATCH[3]}"
echo "License = \"$license\""
fi
break
done- If crate is pulled from path, we should ignore them as they are either examples or Ockam crates
There was a problem hiding this comment.
Hi, sorry for the late reply. I'm going to finish this up. Do I submit a new PR? Sorry this is my first time submitting a PR.
There was a problem hiding this comment.
not sure which is easier for you, but any (updating this PR with a new commit, or creating a new PR) works fine.
|
Hello @kat-coding just wanted to check in on this PR, are you still working on it? |
Open
|
@metaclips I pulled this PR branch locally and made some changes but I'm not sure how to push them here? Should I push to my own fork and make a new PR? |
@Wryhder yes please, create a new PR. |
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Current behavior
Currently the NOTICE file is out of date with no automated way to update that easily.
Proposed changes
Added job to all.yml to run the parseCrates.sh script. It will check if the notice file needs updating and fail if it's out of date. Cargo-deny can be piped into the parseCrates.sh file and it will create an updated NOTICE file in a similar format to the current version.
Checks