1
+ # go-mkopensource
2
+
1
3
## Command line tool Docs
2
- * [ go-mkopensource] ( /cmd/go-mkopensource/README.md )
3
- * [ js-mkopensource] ( /cmd/js-mkopensource/README.md )
4
+
5
+ - [ go-mkopensource] ( /cmd/go-mkopensource/README.md )
6
+ - [ js-mkopensource] ( /cmd/js-mkopensource/README.md )
4
7
5
8
## Building
6
9
@@ -18,87 +21,87 @@ package is good at detecting the licenses in a file
18
21
## Design
19
22
20
23
There are many existing packages to do license detection, such as
21
- [ go-license-detector] [ ] or GitHub's [ licensee] [ ] . The reason these
24
+ [ go-license-detector] [ ] or GitHub's [ licensee] [ ] . The reason these
22
25
are not used is that they are meant to be _ informative_ , they provide
23
26
"best effort" identification of the license.
24
27
25
- ` go-mkopensource ` isn't meant to just be _ informative _ , it is meant to
26
- be used for _ compliance _ , if it has any reason at all to be even a
28
+ ` go-mkopensource ` isn't meant to just be informative , it is meant to
29
+ be used for compliance; if it has any reason at all to be even a
27
30
little skeptical of a result, rather than returning its best guess, it
28
- blows up in your face, asking a human to verify the result.
31
+ asks a human to verify the result.
29
32
30
33
[ go-license-detector ] : https://github.com/go-enry/go-license-detector
31
34
[ licensee ] : https://github.com/licensee/licensee
32
35
33
36
## License scanning scripts
34
37
35
- Folder ` /build-aux ` contains scripts to scan licenses for Go, Python
36
- and Node.Js. Script will generate both ` DEPENDENCY_LICENSES.md ` and
38
+ Folder ` /build-aux ` contains scripts to scan licenses for Go, Python
39
+ and Node.Js. Script will generate both ` DEPENDENCY_LICENSES.md ` and
37
40
` DEPENDENCIES.md `
38
41
39
- The following environment variables are used to configure the
42
+ The following environment variables are used to configure the
40
43
application behaviour.
41
44
42
- * ` APPLICATION ` : Required. Name of the application being scanned.
45
+ - ` APPLICATION ` : Required. Name of the application being scanned.
43
46
It's used in the header of the license files.
44
47
45
- * ` APPLICATION_TYPE ` : Required. Where will the application being
46
- scanned run.
47
- ` internal ` is used for anything running on Ambassador Labs servers,
48
- and ` external ` for anything that's deployed to customer machines.
48
+ - ` APPLICATION_TYPE ` : Required. Where will the application being
49
+ scanned run.
50
+ ` internal ` is used for anything running on Ambassador Labs servers,
51
+ and ` external ` for anything that's deployed to customer machines.
49
52
50
- * ` BUILD_HOME ` Required. Location of the root folder of the repo to
53
+ - ` BUILD_HOME ` Required. Location of the root folder of the repo to
51
54
scan.
52
55
53
- * ` BUILD_TMP ` : Required. Folder to use for storing temporary files.
56
+ - ` BUILD_TMP ` : Required. Folder to use for storing temporary files.
54
57
55
- * ` GIT_TOKEN ` Required. Git token with permissions to pull
58
+ - ` GIT_TOKEN ` Required. Git token with permissions to pull
56
59
repositories
57
60
58
- * ` GO_IMAGE ` Required. Image to use for generating Go
61
+ - ` GO_IMAGE ` Required. Image to use for generating Go
59
62
dependencies.
60
63
61
- * ` PYTHON_PACKAGES ` : Optional. List of requirement.txt files to scan.
62
- Paths should be relative to ` BUILD_HOME ` .
64
+ - ` PYTHON_PACKAGES ` : Optional. List of requirement.txt files to scan.
65
+ Paths should be relative to ` BUILD_HOME ` .
63
66
Example:
64
67
65
68
` export PYTHON_PACKAGES="./python/requirements.txt ./builder/requirements.txt" `
66
69
67
- * ` PYTHON_IMAGE ` : Required. Image to use for generating Python
70
+ - ` PYTHON_IMAGE ` : Required. Image to use for generating Python
68
71
dependencies.
69
72
70
- * ` NPM_PACKAGES ` : Optional. List of package.json and package-lock.json
71
- files to scan. Paths should be relative to ` BUILD_HOME ` .
73
+ - ` NPM_PACKAGES ` : Optional. List of package.json and package-lock.json
74
+ files to scan. Paths should be relative to ` BUILD_HOME ` .
72
75
Example:
73
76
74
77
` export NPM_PACKAGES="./tools/sandbox/grpc_web/package.json ./tools/sandbox/grpc_web/package-lock.json" `
75
78
76
- * ` EXCLUDED_PKG ` : Optional. Semicolon separated list of npm packages names that we want to exclude for the validation.
77
- * Important * : it will restrict the output to the packages (package@version) from being reported in DEPENDENCIES.md and DEPENDENCY_LICENSES.md,
79
+ - ` EXCLUDED_PKG ` : Optional. Semicolon separated list of npm packages names that we want to exclude for the validation.
80
+ _ Important _ : it will restrict the output to the packages (package@version) from being reported in DEPENDENCIES.md and DEPENDENCY_LICENSES.md,
78
81
before to use it, confirm if it is absolutely necessary.
79
-
82
+
80
83
Example:
81
84
82
85
` export EXCLUDED_PKG="[email protected] ;internal-2" `
83
86
84
- * ` NODE_IMAGE ` : Required when ` NPM_PACKAGES ` is defined. Version
87
+ - ` NODE_IMAGE ` : Required when ` NPM_PACKAGES ` is defined. Version
85
88
of Node.JS to use when running npm dependency scan. Only valid
86
- version numbers (X.Y.Z) are allowed.
89
+ version numbers (X.Y.Z) are allowed.
87
90
Example:
88
91
89
92
` NODE_IMAGE=node:14.13.1-alpine `
90
93
91
- * ` SCRIPTS_HOME ` : Required. Location where ` go-mkopensource ` repo is
92
- checked out, relative to ` BUILD_HOME `
94
+ - ` SCRIPTS_HOME ` : Required. Location where ` go-mkopensource ` repo is
95
+ checked out, relative to ` BUILD_HOME `
93
96
94
- To update license information files, set the environment variables
97
+ To update license information files, set the environment variables
95
98
described above and run ` build-aux/generate.sh `
96
99
97
100
Tips:
98
101
99
- * Add a .dockerignore at the root of your repository and exclude any files that are not necessary for generating depdency
100
- information, including DEPENDENCIES.md, DEPENDENCY_INFORMATION.md and temporary folders. See the
101
- [ saas_app .dockerignore] ( https://github.com/datawire/saas_app/blob/master/.dockerignore ) for a list of suggested exclussions.
102
+ - Add a .dockerignore at the root of your repository and exclude any files that are not necessary for generating depdency
103
+ information, including DEPENDENCIES.md, DEPENDENCY_INFORMATION.md and temporary folders. See the
104
+ [ saas_app .dockerignore] ( https://github.com/datawire/saas_app/blob/master/.dockerignore ) for a list of suggested exclussions.
102
105
103
106
## When scanning fails
104
107
@@ -124,7 +127,7 @@ fatal: 2 license-detection errors:
124
127
```
125
128
126
129
A quick look at the package reveals that it uses an Apache License, but adds extra text at the top of the actual LICENSE
127
- file indicating that it also uses files from golang/encoding/json. We know that golang uses a 3-clause BSD license. So we consult the [ SPDX License List] ( https://spdx.org/licenses/ ) to get the canonical
130
+ file indicating that it also uses files from golang/encoding/json. We know that golang uses a 3-clause BSD license. So we consult the [ SPDX License List] ( https://spdx.org/licenses/ ) to get the canonical
128
131
identifiers for the licenses, and add them to an ` unparsable-packages.yaml ` file to our build system
129
132
with the following contents:
130
133
@@ -146,28 +149,29 @@ the list of all the packages in the file "vendor/modules.txt" using the command
146
149
# 26 18.24 go: downloading github.com/containerd/cgroups v0.0.0-20200531161412-0dbf7f05ba59
147
150
# 26 18.28 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da
148
151
# 26 36.85 github.com/datawire/saas_app/internal/pkg/kubernetes imports
149
- # 26 36.85 k8s.io/client-go/rest imports
150
- # 26 36.85 k8s.io/apimachinery/pkg/util/clock: no required module provides package k8s.io/apimachinery/pkg/util/clock; to add it:
151
- # 26 36.85 go get k8s.io/apimachinery/pkg/util/clock
152
+ # 26 36.85 k8s.io/client-go/rest imports
153
+ # 26 36.85 k8s.io/apimachinery/pkg/util/clock: no required module provides package k8s.io/apimachinery/pkg/util/clock; to add it:
154
+ # 26 36.85 go get k8s.io/apimachinery/pkg/util/clock
152
155
# 26 36.85 /scripts/go-mkopensource: fatal: ["go" "mod" "vendor"]: exit status 1
153
156
# 26 ERROR: executor failed running [/bin/sh -c /scripts/scan-go.sh]: exit code: 1
154
157
```
155
158
156
- Now the scanner is smart enough to follow the indications of the "go mod vendor" install the dependencies, and then
159
+ Now the scanner will follow the indications of the "go mod vendor", install the dependencies, and then
157
160
get the list of packages from the file ''vendor/modules.txt"
158
161
159
- Sometimes it isn't possible to install the dependecies sugested by the "go mod vendor" command.
162
+ Sometimes it isn't possible to install the dependencies suggested by the "go mod vendor" command.
160
163
The scanner will complain with the message "Error installing dependency". In this case the project will require human intervention to solve the problem.
161
164
162
- Another reason that may cause a failure is the use of our proprietary packages, like ` github.com/datawire/telepresence-pro/rpc/proconnector ` .
163
- The error will message will be similar to this one:
165
+ Another reason that may cause a failure is the use of our proprietary packages, like ` github.com/datawire/telepresence-pro/rpc/proconnector ` .
166
+ The error will message will be similar to this one:
164
167
165
168
``` bash
166
169
/scripts/go-mkopensource: fatal: 1 license-detection errors:
167
170
1. Package " github.com/datawire/telepresence-pro/rpc/proconnector" : could not identify a license for all sources (had no global LICENSE file)
168
171
```
169
172
170
- Fo exclude these packages, add a yaml file like this
173
+ To exclude these packages, add a yaml file like this:
174
+
171
175
``` yaml
172
176
- github.com/datawire/telepresence-pro/rpc/userdaemon
173
177
- github.com/datawire/telepresence-pro/rpc/proconnector
@@ -180,19 +184,17 @@ And pass it to the generate.sh script using the argument `--proprietary-packages
180
184
./generate.sh" --proprietary-packages proprietary-packages.yaml;
181
185
` ` `
182
186
183
- # ## Remember to always create a ticket!
184
- When a problem arise, remember to always create a ticket so that the problem can be fixed. This will help all users
185
- of the `go-mkopensource` tool and in many cases also make the owner of the failing component aware of the problem.
186
-
187
187
# # Dependabot PRs
188
188
189
189
When dependabot creates a PR, it's possible that license scanning will fail due so several factors :
190
+
190
191
1. A Go package is unavailable in the new version of a module
191
192
2. Dependency information is out of date.
192
193
193
194
To reduce friction merging dependabot PRs, there is an action (`save-dependabot-changes`) that will update a PR created by dependabot.
194
195
195
196
Use the action in a workflow as follows :
197
+
196
198
` ` ` yaml
197
199
name: "Verify licenses"
198
200
on: push
@@ -207,7 +209,7 @@ jobs:
207
209
id: changed-by-dependabot
208
210
uses: datawire/go-mkopensource/actions/[email protected]
209
211
with:
210
- branches_to_skip: ' master'
212
+ branches_to_skip: " master"
211
213
- name: Abort if dependencies changed
212
214
if: steps.changed-by-dependabot.outputs.license_information_committed == 'true'
213
215
run: |
@@ -216,10 +218,11 @@ jobs:
216
218
# Continue with other steps
217
219
` ` `
218
220
219
- *Notes*:
220
- - The GitHub token GITHUB_TOKEN should have at least `contents:write` and `actions : write`
221
+ **Notes**:
222
+
223
+ - The GitHub token GITHUB_TOKEN should have at least `contents:write` and `actions : write`
221
224
[privileges](https://docs.github.com/en/actions/security-guides/automatic-token-authentication#permissions-for-the-github_token).
222
- - The workflow that invokes the action should have a `workflow_dispatch`
225
+ - The workflow that invokes the action should have a `workflow_dispatch`
223
226
[trigger](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch).
224
227
225
228
# ## Testing changes to the `save-dependabot-changes` action
@@ -231,5 +234,5 @@ to point to the new version of the action.
231
234
uses: datawire/go-mkopensource/actions/save-dependabot-changes@<VERSION>
232
235
` ` `
233
236
234
- Note : If you want to test your changes before they are complete, you could use a branch in the action reference, and
235
- update it to a tag once you're ready to release.
237
+ Note : If you want to test your changes before they are complete, you could use a branch in the action reference, and
238
+ update it to a tag once you're ready to release.
0 commit comments