Skip to content

Commit 827c020

Browse files
itxiaohu001huyongfeng
itxiaohu001
and
huyongfeng
authored
Support exporting SBOM in spdx format (#24)
* Support the output of SBOM list in spdx format Co-authored-by: huyongfeng <[email protected]>
1 parent 7b4c7ca commit 827c020

File tree

17 files changed

+516
-40
lines changed

17 files changed

+516
-40
lines changed

‎.github/README.md

+11-12
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,6 @@
44
<img alt="logo" src="../logo.svg">
55
</p>
66
<h1 align="center" style="margin: 30px 0 30px; font-weight: bold;">OpenSCA-Cli</h1>
7-
87
<p align="center">
98
<a href="https://github.com/XmirrorSecurity/OpenSCA-cli/blob/master/LICENSE"><img src="https://img.shields.io/github/license/XmirrorSecurity/OpenSCA-cli?style=flat-square"></a>
109
<a href="https://github.com/XmirrorSecurity/OpenSCA-cli/releases"><img src="https://img.shields.io/github/v/release/XmirrorSecurity/OpenSCA-cli?style=flat-square"></a>
@@ -77,18 +76,18 @@ opensca-cli -db db.json -path ${project_path}
7776

7877
**You can either configure the parameters in configuration files or input the parameters in the command-line. When the two conflict with each other, the input parameters will be prioritized.**
7978

80-
| PARAMETER | TYPE | DESCRIPTION | SAMPLE |
81-
| ---------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- |
82-
| `config` | `string` | Set the configuration file path, when the program runs, the parameter of the configuration file will be used as the startup parameters. If the configuration parameter conflicts with the command-line input parameter, the latter will be taken. | `-config config.json` |
83-
| `path` | `string` | Set the file or directory path to be detected. | `-path ./foo` |
84-
| `url` | `string` | Check the vulnerabilities from the cloud vulnerability database, set the address of the cloud service. It needs to be used with the `token` parameter. | `-url https://opensca.xmirror.cn` |
85-
| `token` | `string` | Cloud service verification. You have to apply for it on the cloud service platform and use it with the `url` parameter. | `-token xxxxxxx` |
86-
| `cache` | `bool` | This option is recommended. It can cache the downloaded files, for example, the `.pom` file, and save your time when detecting the same component next time. The downloaded files are saved in `.cache` under the same directory as opensca-cli. | `-cache` |
87-
| `vuln` | `bool` | Show the vulnerabilities info only. Using this parameter, the component hierarchical architecture will **NOT** be included in the result. | `-vuln` |
88-
| `out` | `string` | Set the output file. The result defaults to json format. | `-out output.json` |
79+
| PARAMETER | TYPE | DESCRIPTION | SAMPLE |
80+
| ---------- | -------- | ------------------------------------------------------------ | --------------------------------- |
81+
| `config` | `string` | Set the configuration file path, when the program runs, the parameter of the configuration file will be used as the startup parameters. If the configuration parameter conflicts with the command-line input parameter, the latter will be taken. | `-config config.json` |
82+
| `path` | `string` | Set the file or directory path to be detected. | `-path ./foo` |
83+
| `url` | `string` | Check the vulnerabilities from the cloud vulnerability database, set the address of the cloud service. It needs to be used with the `token` parameter. | `-url https://opensca.xmirror.cn` |
84+
| `token` | `string` | Cloud service verification. You have to apply for it on the cloud service platform and use it with the `url` parameter. | `-token xxxxxxx` |
85+
| `cache` | `bool` | This option is recommended. It can cache the downloaded files, for example, the `.pom` file, and save your time when detecting the same component next time. The downloaded files are saved in `.cache` under the same directory as opensca-cli. | `-cache` |
86+
| `vuln` | `bool` | Show the vulnerabilities info only. Using this parameter, the component hierarchical architecture will **NOT** be included in the result. | `-vuln` |
87+
| `out` | `string` | Set the output file. The result defaults to json format.Support the output of SBOM list in spdx format. | `-out output.json` |
8988
| `db` | `string` | Set the local vulnerability database file. It helps when you prefer to use your own vulnerability database. The format of the vulnerability database is shown below. If the cloud and local vulnerability databases are both set, the result of detection will merge both. | `-db db.json` |
90-
| `progress` | `bool` | Show the progress bar. | `-progress` |
91-
| `dedup` | `bool` | Same result deduplication | `-dedup` |
89+
| `progress` | `bool` | Show the progress bar. | `-progress` |
90+
| `dedup` | `bool` | Same result deduplication | `-dedup` |
9291

9392
------
9493

‎README.md

+1-2
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,6 @@
22
<img alt="logo" src="./logo.svg">
33
</p>
44
<h1 align="center" style="margin: 30px 0 30px; font-weight: bold;">OpenSCA-Cli</h1>
5-
65
<p align="center">
76
<a href="https://github.com/XmirrorSecurity/OpenSCA-cli/blob/master/LICENSE"><img src="https://img.shields.io/github/license/XmirrorSecurity/OpenSCA-cli?style=flat-square"></a>
87
<a href="https://github.com/XmirrorSecurity/OpenSCA-cli/releases"><img src="https://img.shields.io/github/v/release/XmirrorSecurity/OpenSCA-cli?style=flat-square"></a>
@@ -84,7 +83,7 @@ opensca-cli -db db.json -path ${project_path}
8483
| `token` | `string` | 云服务验证 `token`，需要在云服务平台申请，与 `url` 参数一起使用 | `-token xxxxxxx` |
8584
| `cache` | `bool` | 建议开启，缓存下载的文件(例如 `.pom` 文件)，重复检测相同组件时会节省时间，下载的文件会保存到工具所在目录的.cache 目录下 | `-cache` |
8685
| `vuln` | `bool` | 结果仅保留有漏洞信息的组件，使用该参数将不会保留组件层级结构 | `-vuln` |
87-
| `out` | `string` | 将检测结果保存到指定文件，根据后缀生成不同格式的文件，默认为 `json` 格式 | `-out output.json` |
86+
| `out` | `string` | 将检测结果保存到指定文件，根据后缀生成不同格式的文件，默认为 `json` 格式；；支持以`spdx`格式展示`sbom`清单只需更换相应输出文件后缀即可 | `-out output.json` |
8887
| `db` | `string` | 指定本地漏洞库文件，希望使用自己漏洞库时可用，漏洞库文件为 `json` 格式，具体格式会在之后给出;若同时使用云端漏洞库与本地漏洞库，漏洞查询结果取并集 | `-db db.json` |
8988
| `progress` | `bool` | 显示进度条 | `-progress` |
9089
| `dedup` | `bool` | 相同组件去重 | `-dedup` |

‎analyzer/engine/archive.go

+2-1
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,8 @@ import (
2828
// checkFile 检测是否为可检测的文件
2929
func (e Engine) checkFile(filename string) bool {
3030
for _, analyzer := range e.Analyzers {
31-
if analyzer.CheckFile(filename) {
31+
if analyzer.CheckFile(filename) ||
32+
filter.CheckLicense(filename) {
3233
return true
3334
}
3435
}

‎analyzer/engine/parse.go

+64
Original file line numberDiff line numberDiff line change
@@ -7,16 +7,25 @@ package engine
77

88
import (
99
"path"
10+
"regexp"
1011
"strings"
1112
"util/filter"
1213
"util/model"
1314
)
1415

16+
// copyright匹配优先级
17+
const (
18+
low = iota
19+
mid
20+
high
21+
)
22+
1523
// parseDependency 解析依赖
1624
func (e Engine) parseDependency(dirRoot *model.DirTree, depRoot *model.DepTree) *model.DepTree {
1725
if depRoot == nil {
1826
depRoot = model.NewDepTree(nil)
1927
}
28+
var copyrightMess = make(map[string]string)
2029
for _, analyzer := range e.Analyzers {
2130
// 遍历目录树获取要检测的文件
2231
files := []*model.FileInfo{}
@@ -30,11 +39,22 @@ func (e Engine) parseDependency(dirRoot *model.DirTree, depRoot *model.DepTree)
3039
for _, f := range n.Files {
3140
if analyzer.CheckFile(f.Name) {
3241
files = append(files, f)
42+
} else if filter.CheckLicense(f.Name) {
43+
if _, ok := copyrightMess[path.Dir(f.Name)]; !ok {
44+
// 记录解析到的copyrigh信息
45+
copyrightMess[path.Dir(f.Name)] = parseCopyright(f)
46+
}
3347
}
3448
}
3549
}
3650
// 从文件中解析依赖树
3751
for _, d := range analyzer.ParseFiles(files) {
52+
p := path.Dir(d.Path)
53+
if _, ok := copyrightMess[p]; ok {
54+
// 将copyright信息加入与其同一文件目录的依赖节点中
55+
d.CopyrightText = copyrightMess[p]
56+
delete(copyrightMess, p)
57+
}
3858
depRoot.Children = append(depRoot.Children, d)
3959
d.Parent = depRoot
4060
if d.Name != "" && !strings.ContainsAny(d.Vendor+d.Name, "${}") && d.Version.Ok() {
@@ -99,3 +119,47 @@ func (e Engine) parseDependency(dirRoot *model.DirTree, depRoot *model.DepTree)
99119
}
100120
return depRoot
101121
}
122+
123+
// 从文件中提取copyright信息
124+
func parseCopyright(f *model.FileInfo) string {
125+
matchLevel := map[int]string{}
126+
ct := string(f.Data)
127+
if len(ct) == 0 {
128+
return ""
129+
}
130+
pras := strings.Split(ct, "\n\n")
131+
re := regexp.MustCompile(`^\d{4}$|^\d{4}-\d{4}$|^\(c\)$`)
132+
for _, pra := range pras {
133+
if !strings.Contains(strings.ToLower(pra), "copyright") {
134+
continue
135+
}
136+
lines := strings.Split(pra, "\n")
137+
line := strings.TrimSpace(lines[0])
138+
if len(lines) == 0 {
139+
continue
140+
}
141+
tks := strings.Fields(line)
142+
if len(tks) == 0 {
143+
continue
144+
}
145+
if strings.EqualFold("copyright", tks[0]) {
146+
if re.MatchString(tks[1]) {
147+
matchLevel[high] = line
148+
}
149+
matchLevel[mid] = line
150+
}
151+
for _, l := range lines {
152+
if strings.HasPrefix(strings.TrimSpace(strings.ToLower(l)), "copyright") {
153+
matchLevel[low] = strings.TrimSpace(l)
154+
break
155+
}
156+
}
157+
158+
}
159+
for i := high; i >= low; i-- {
160+
if matchLevel[i] != "" {
161+
return matchLevel[i]
162+
}
163+
}
164+
return ""
165+
}

‎analyzer/golang/gomod.go

+2
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ func parseGomod(dep *model.DepTree, file *model.FileInfo) {
2020
sub := model.NewDepTree(dep)
2121
sub.Name = strings.Trim(match[1], `'"`)
2222
sub.Version = model.NewVersion(match[2])
23+
sub.HomePage = "https://" + sub.Name
2324
}
2425
}
2526

@@ -40,6 +41,7 @@ func parseGosum(dep *model.DepTree, file *model.FileInfo) {
4041
sub := model.NewDepTree(dep)
4142
sub.Name = strings.Trim(match[1], `'"`)
4243
sub.Version = model.NewVersion(match[2])
44+
sub.HomePage = sub.Name
4345
exist[sub.Name] = struct{}{}
4446
}
4547
}

‎analyzer/javascript/package_json.go

+16-9
Original file line numberDiff line numberDiff line change
@@ -22,11 +22,13 @@ import (
2222

2323
// package.json 文件结构
2424
type PkgJson struct {
25-
Name string `json:"name"`
26-
Version string `json:"version"`
27-
License string `json:"license"`
28-
DevDeps map[string]string `json:"devDependencies"`
29-
Deps map[string]string `json:"dependencies"`
25+
Name string `json:"name"`
26+
Version string `json:"version"`
27+
License string `json:"license"`
28+
DevDeps map[string]string `json:"devDependencies"`
29+
Deps map[string]string `json:"dependencies"`
30+
HomePage string `json:"homepage"`
31+
Repository map[string]string `json:"repository,omitempty"`
3032
}
3133

3234
// npm下载文件结构
@@ -41,13 +43,15 @@ func parsePackage(root *model.DepTree, file *model.FileInfo, simulation bool) (d
4143
pkg := PkgJson{}
4244
if err := json.Unmarshal(file.Data, &pkg); err != nil {
4345
logs.Error(err)
44-
return
4546
}
4647
if pkg.Name != "" {
4748
root.Name = pkg.Name
4849
}
49-
root.Version = model.NewVersion(pkg.Version)
50+
if pkg.Version != "" {
51+
root.Version = model.NewVersion(pkg.Version)
52+
}
5053
root.AddLicense(pkg.License)
54+
root.HomePage = pkg.HomePage
5155
// 依赖列表map[name]version
5256
depMap := map[string]string{}
5357
for name, version := range pkg.DevDeps {
@@ -81,7 +85,7 @@ func parsePackage(root *model.DepTree, file *model.FileInfo, simulation bool) (d
8185
}
8286
for !q.Empty() {
8387
node := q.Pop().(*model.DepTree)
84-
for _, sub := range npmSimulation(node) {
88+
for _, sub := range npmSimulation(node, exist) {
8589
if _, ok := exist[sub.Name]; !ok {
8690
bar.Npm.Add(1)
8791
exist[sub.Name] = struct{}{}
@@ -93,7 +97,7 @@ func parsePackage(root *model.DepTree, file *model.FileInfo, simulation bool) (d
9397
}
9498

9599
// npmSimulation 模拟npm获取详细依赖信息
96-
func npmSimulation(dep *model.DepTree) (subDeps []*model.DepTree) {
100+
func npmSimulation(dep *model.DepTree, exist map[string]struct{}) (subDeps []*model.DepTree) {
97101
subDeps = []*model.DepTree{}
98102
dep.Language = language.JavaScript
99103
// 获取依赖数据
@@ -149,6 +153,9 @@ func npmSimulation(dep *model.DepTree) (subDeps []*model.DepTree) {
149153
}
150154
sort.Strings(names)
151155
for _, name := range names {
156+
if _, ok := exist[name]; ok {
157+
continue
158+
}
152159
sub := model.NewDepTree(dep)
153160
sub.Name = name
154161
sub.Version = model.NewVersion(info.Deps[name])

‎analyzer/javascript/package_lock.go

+1-1
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ func parsePackageLock(root *model.DepTree, file *model.FileInfo, direct []string
4545
}{}
4646
if err := json.Unmarshal(file.Data, &lock); err != nil {
4747
logs.Error(err)
48-
return
48+
//return
4949
}
5050
if lock.Name != "" {
5151
root.Name = lock.Name

‎analyzer/php/composer.go

+9-2
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,8 @@ type Composer struct {
2626
License string `json:"license"`
2727
Require map[string]string `json:"require"`
2828
RequireDev map[string]string `json:"require-dev"`
29+
HomePage string `json:"homepage"`
30+
Support map[string]string `json:"support"`
2931
}
3032

3133
type ComposerRepo struct {
@@ -47,6 +49,8 @@ func parseComposer(root *model.DepTree, file *model.FileInfo, simulation bool) (
4749
if composer.Name != "" {
4850
root.Name = composer.Name
4951
}
52+
root.HomePage = composer.HomePage
53+
root.DownloadLocation = composer.Support["source"]
5054
// add license
5155
if composer.License != "" {
5256
root.AddLicense(composer.License)
@@ -84,7 +88,7 @@ func parseComposer(root *model.DepTree, file *model.FileInfo, simulation bool) (
8488
}
8589
for !q.Empty() {
8690
node := q.Pop().(*model.DepTree)
87-
for _, sub := range composerSimulation(node) {
91+
for _, sub := range composerSimulation(node, exist) {
8892
if _, ok := exist[sub.Name]; !ok {
8993
bar.Composer.Add(1)
9094
exist[sub.Name] = struct{}{}
@@ -96,7 +100,7 @@ func parseComposer(root *model.DepTree, file *model.FileInfo, simulation bool) (
96100
}
97101

98102
// composerSimulation composer simulation
99-
func composerSimulation(dep *model.DepTree) (subDeps []*model.DepTree) {
103+
func composerSimulation(dep *model.DepTree, exist map[string]struct{}) (subDeps []*model.DepTree) {
100104
subDeps = []*model.DepTree{}
101105
dep.Language = language.Php
102106
data := cache.LoadCache(dep.Dependency)
@@ -148,6 +152,9 @@ func composerSimulation(dep *model.DepTree) (subDeps []*model.DepTree) {
148152
if strings.EqualFold(name, "php") {
149153
continue
150154
}
155+
if _, ok := exist[name]; ok {
156+
continue
157+
}
151158
sub := model.NewDepTree(dep)
152159
sub.Name = name
153160
sub.Version = model.NewVersion(requires[name])

‎analyzer/php/composer_lock.go

+9-4
Original file line numberDiff line numberDiff line change
@@ -8,16 +8,19 @@ package php
88
import (
99
"encoding/json"
1010
"sort"
11+
"strings"
1112
"util/logs"
1213
"util/model"
1314
)
1415

1516
// composer.lock
1617
type ComposerLock struct {
1718
Pkgs []struct {
18-
Name string `json:"name"`
19-
Version string `json:"version"`
20-
Require map[string]string `json:"require"`
19+
Name string `json:"name"`
20+
Version string `json:"version"`
21+
Require map[string]string `json:"require"`
22+
HomePage string `json:"homepage"`
23+
Source map[string]string `json:"source"`
2124
} `json:"packages"`
2225
}
2326

@@ -26,7 +29,7 @@ func parseComposerLock(root *model.DepTree, file *model.FileInfo, direct []strin
2629
lock := ComposerLock{}
2730
if err := json.Unmarshal(file.Data, &lock); err != nil {
2831
logs.Error(err)
29-
return
32+
//return
3033
}
3134
// 记录尚无Parent的依赖
3235
depMap := map[string]*model.DepTree{}
@@ -37,6 +40,8 @@ func parseComposerLock(root *model.DepTree, file *model.FileInfo, direct []strin
3740
dep.Name = cps.Name
3841
dep.Version = model.NewVersion(cps.Version)
3942
dep.Expand = cps.Require
43+
dep.HomePage = cps.HomePage
44+
dep.DownloadLocation = strings.ReplaceAll(cps.Source["url"], ".git", "")
4045
depMap[cps.Name] = dep
4146
directMap[cps.Name] = dep
4247
}

‎analyzer/python/pipfile.go

+13-3
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ package python
22

33
import (
44
"encoding/json"
5+
"strings"
56
"util/logs"
67
"util/model"
78

@@ -20,12 +21,12 @@ func parsePipfile(root *model.DepTree, file *model.FileInfo) {
2021
for name, version := range pip.Packages {
2122
dep := model.NewDepTree(root)
2223
dep.Name = name
23-
dep.Version = model.NewVersion(version)
24+
dep.Version = model.NewVersion(formatVer(version))
2425
}
2526
for name, version := range pip.DevPackages {
2627
dep := model.NewDepTree(root)
2728
dep.Name = name
28-
dep.Version = model.NewVersion(version)
29+
dep.Version = model.NewVersion(formatVer(version))
2930
}
3031
}
3132

@@ -49,8 +50,17 @@ func parsePipfileLock(root *model.DepTree, file *model.FileInfo) {
4950
if v != "" {
5051
dep := model.NewDepTree(root)
5152
dep.Name = n
52-
dep.Version = model.NewVersion(v)
53+
dep.Version = model.NewVersion(formatVer(v))
5354
}
5455
}
5556
return
5657
}
58+
59+
// 后续使用其他办法确定版本号
60+
func formatVer(v string) string {
61+
res := strings.ReplaceAll(v, "==", "")
62+
res = strings.ReplaceAll(res, "~=", "")
63+
res = strings.ReplaceAll(res, ">=", "")
64+
res = strings.ReplaceAll(res, "<=", "")
65+
return res
66+
}

‎analyzer/python/setup.go

+2-2
Original file line numberDiff line numberDiff line change
@@ -56,15 +56,15 @@ func parseSetup(root *model.DepTree, file *model.FileInfo) {
5656
logs.Warn(err)
5757
}
5858
root.Name = dep.Name
59-
root.Version = model.NewVersion(dep.Version)
59+
root.Version = model.NewVersion(formatVer(dep.Version))
6060
root.Licenses = append(root.Licenses, dep.License)
6161
for _, pkg := range [][]string{dep.Packages, dep.InstallRequires, dep.Requires} {
6262
for _, p := range pkg {
6363
index := strings.IndexAny(p, "=<>")
6464
sub := model.NewDepTree(root)
6565
if index > -1 {
6666
sub.Name = p[:index]
67-
sub.Version = model.NewVersion(p[index:])
67+
sub.Version = model.NewVersion(formatVer(p[index:]))
6868
} else {
6969
sub.Name = p
7070
}

0 commit comments

Comments
 (0)