feat(suse): add cvrf CVE feed#423
Conversation
| References []Reference `xml:"References>Reference"` | ||
| ProductStatuses []Status `xml:"ProductStatuses>Status"` | ||
| CVSSScoreSets ScoreSet `xml:"CVSSScoreSets>ScoreSet" json:",omitempty"` | ||
| CVSSScoreSets []ScoreSet `xml:"CVSSScoreSets>ScoreSet" json:",omitempty"` |
There was a problem hiding this comment.
As we discussed here — #401 (comment) — we cannot simply change the variable type.
Therefore #401 is a blocking PR.
There was a problem hiding this comment.
@DmitriyLewen yes, we agree this will fail if we don't use the latest commit, but we've got a workaround for it. We're good for merging this.
There was a problem hiding this comment.
At premium side, we're not using the cvss field at all, so for our previous branches we want to copy the old tracker code and run it as premium tracker to support backward compatibility, but in later branches we'll consume the cvss scores.
| } | ||
| if sets, ok := cache[cveID]; ok { | ||
| if len(sets) > 0 { | ||
| cv.Vulnerabilities[i].CVSSScoreSets = sets |
There was a problem hiding this comment.
You overwrite Scores, but should we merge scores from both sources?
There was a problem hiding this comment.
We don't want to get scores from the advisory feed, we'll capture only from the cve-cvrf feed.
| return scoreSetsFromCVE12(doc.Vuln.CVSSScoreSets), nil | ||
| } | ||
|
|
||
| func (c Config) mergeCVEDetailsFromCVEFeed(cv *Cvrf, cache map[string][]ScoreSet) { |
There was a problem hiding this comment.
Usually for vuln-list-update we try not to modify the raw data from sources.
This is important so that users can view the original files, their history, etc.
After merging, it would be difficult to trace the source of a CVSS Score.
Perhaps we could save the CVSS files from http://ftp.suse.com/pub/projects/security/cvrf-cve/ separately and merge them in trivy-db instead?
5785017 to
a6bdcde
Compare
01e7a62 to
321e998
Compare
DmitriyLewen
left a comment
There was a problem hiding this comment.
Hello @manojkrishna-nomula
Thanks!
left a comments.
Can you also update PR description - write why we need to add cvrf CVE feed.
| name: "positive test", | ||
| appFs: afero.NewMemMapFs(), | ||
| archiveDir: "testdata/cvrf-cve", | ||
| expectedFile: "/tmp/cvrf/suse-cves/2014/CVE-2014-6271.json", |
There was a problem hiding this comment.
Can you keep and compare golden files (like for cvrf tests)?
| var ( | ||
| target = flag.String("target", "", "update target (nvd, alpine, alpine-unfixed, redhat, redhat-oval, "+ | ||
| "redhat-csaf-vex, debian, ubuntu, amazon, oracle-oval, suse-cvrf, photon, arch-linux, glad, cwe, osvdev, mariner, kevc, wolfi, "+ | ||
| "redhat-csaf-vex, debian, ubuntu, amazon, oracle-oval, suse-cvrf, suse-cvrf-cve, photon, arch-linux, glad, cwe, osvdev, mariner, kevc, wolfi, "+ |
There was a problem hiding this comment.
Can you add suse-cvrf-cve into README.md?
| CVE string `xml:"CVE"` | ||
| Description string `xml:"Notes>Note"` | ||
| Threats []Threat `xml:"Threats>Threat"` | ||
| References []Reference `xml:"References>Reference"` | ||
| ProductStatuses []Status `xml:"ProductStatuses>Status"` | ||
| CVSSScoreSets CVSSScoreSets `xml:"CVSSScoreSets" json:",omitempty"` |
There was a problem hiding this comment.
IIUC we need to keep CVE advisories to see all CVSS scores (cvrf doesn't have V3 and V4 scores).
Do we need other fields? (we can reduce the size of the files to parse only the needed fields)
| } | ||
| } | ||
|
|
||
| func (c Config) Update() error { |
There was a problem hiding this comment.
I think we can unify the logic.
Something like this:
diff --git a/main.go b/main.go
index 05102f5..3f372e9 100644
--- a/main.go
+++ b/main.go
@@ -32,7 +32,7 @@ import (
"github.com/aquasecurity/vuln-list-update/rootio"
"github.com/aquasecurity/vuln-list-update/seal"
susecvrf "github.com/aquasecurity/vuln-list-update/suse/cvrf"
- susecvrfcve "github.com/aquasecurity/vuln-list-update/suse/cvrf-cve"
+ susecvrfcve "github.com/aquasecurity/vuln-list-update/suse/cvrfcve"
"github.com/aquasecurity/vuln-list-update/ubuntu"
"github.com/aquasecurity/vuln-list-update/utils"
"github.com/aquasecurity/vuln-list-update/wolfi"
diff --git a/suse/cvrf-cve/cvrf_cve.go b/suse/cvrf-cve/cvrf_cve.go
deleted file mode 100644
index dc51890..0000000
--- a/suse/cvrf-cve/cvrf_cve.go
+++ /dev/null
@@ -1,150 +0,0 @@
-package cvrfcve
-
-import (
- "archive/tar"
- "bytes"
- "compress/bzip2"
- "compress/gzip"
- "encoding/xml"
- "errors"
- "fmt"
- "io"
- "log"
- "path/filepath"
- "regexp"
- "strings"
- "unicode/utf8"
-
- "github.com/spf13/afero"
- "golang.org/x/xerrors"
-
- "github.com/aquasecurity/vuln-list-update/utils"
-)
-
-const (
- cvrfCVEArchiveURL = "http://ftp.suse.com/pub/projects/security/cvrf-cve.tar.bz2"
- cvrfDir = "cvrf"
- suseCVEDir = "suse-cves"
- retries = 5
-)
-
-var fileRegexp = regexp.MustCompile(`^cvrf-(CVE-\d{4}-\d+)\.xml$`)
-
-type Config struct {
- VulnListDir string
- URL string
- AppFs afero.Fs
-}
-
-func NewConfig() Config {
- return Config{
- VulnListDir: utils.VulnListDir(),
- URL: cvrfCVEArchiveURL,
- AppFs: afero.NewOsFs(),
- }
-}
-
-func (c Config) Update() error {
- log.Print("Fetching SUSE CVE CVRF archive...")
-
- // The SUSE server is sometimes unstable, so download the whole archive into
- // memory before processing. Streaming directly from the HTTP response would
- // make it hard to distinguish a mid-transfer disconnection (which surfaces
- // as a truncated tar) from a legitimate parse error. The archive is only a
- // few hundred MB, which fits comfortably in memory on CI runners.
- body, err := utils.FetchURL(c.URL, "", retries)
- if err != nil {
- return xerrors.Errorf("failed to download SUSE CVE CVRF archive: %w", err)
- }
-
- var decompressed io.Reader
- switch {
- case strings.HasSuffix(c.URL, ".tar.bz2"):
- decompressed = bzip2.NewReader(bytes.NewReader(body))
- case strings.HasSuffix(c.URL, ".tar.gz"):
- // Go's compress/bzip2 lacks a Writer, so tests use .tar.gz instead.
- gr, err := gzip.NewReader(bytes.NewReader(body))
- if err != nil {
- return xerrors.Errorf("failed to decompress gzip: %w", err)
- }
- defer gr.Close()
- decompressed = gr
- default:
- return xerrors.Errorf("unsupported archive format: %s", c.URL)
- }
- tr := tar.NewReader(decompressed)
-
- for {
- hdr, err := tr.Next()
- switch {
- case errors.Is(err, io.EOF):
- return nil
- case err != nil:
- return xerrors.Errorf("failed to read tar entry: %w", err)
- case hdr.Typeflag != tar.TypeReg:
- continue
- }
-
- filename := filepath.Base(hdr.Name)
- if !strings.HasSuffix(filename, ".xml") {
- continue
- }
- if fileRegexp.FindStringSubmatch(filename) == nil {
- continue
- }
-
- data, err := io.ReadAll(tr)
- if err != nil {
- return xerrors.Errorf("failed to read tar entry data: %w", err)
- }
-
- if len(data) == 0 {
- log.Printf("empty CVE CVRF xml: %s", filename)
- continue
- }
-
- if !utf8.Valid(data) {
- log.Printf("invalid UTF-8: %s", filename)
- data = []byte(strings.ToValidUTF8(string(data), ""))
- }
-
- var cv Cvrf
- if err = xml.Unmarshal(data, &cv); err != nil {
- return xerrors.Errorf("failed to decode SUSE CVE CVRF XML (%s): %w", filename, err)
- }
-
- cveID := extractCVEID(cv)
- if cveID == "" {
- log.Printf("invalid CVE CVRF document ID: %s", cv.Tracking.ID)
- continue
- }
-
- if err = c.saveCVEPerYear(cveID, cv); err != nil {
- return xerrors.Errorf("failed to save SUSE CVE CVRF: %w", err)
- }
- }
-}
-
-func extractCVEID(cv Cvrf) string {
- for _, v := range cv.Vulnerabilities {
- cve := strings.TrimSpace(v.CVE)
- if cve != "" {
- return cve
- }
- }
- return strings.TrimSpace(cv.Title)
-}
-
-func (c Config) saveCVEPerYear(cveID string, data Cvrf) error {
- s := strings.Split(cveID, "-")
- if len(s) < 3 {
- return nil
- }
-
- yearDir := filepath.Join(c.VulnListDir, cvrfDir, suseCVEDir, s[1])
- fileName := fmt.Sprintf("%s.json", cveID)
- if err := utils.WriteJSON(c.AppFs, yearDir, fileName, data); err != nil {
- return xerrors.Errorf("failed to write file: %w", err)
- }
- return nil
-}
diff --git a/suse/cvrf/cvrf.go b/suse/cvrf/cvrf.go
index 16e9325..87062cd 100644
--- a/suse/cvrf/cvrf.go
+++ b/suse/cvrf/cvrf.go
@@ -1,23 +1,17 @@
package cvrf
import (
- "archive/tar"
- "bytes"
- "compress/bzip2"
- "compress/gzip"
"encoding/xml"
- "errors"
"fmt"
- "io"
"log"
"path/filepath"
"regexp"
"strings"
- "unicode/utf8"
"github.com/spf13/afero"
"golang.org/x/xerrors"
+ "github.com/aquasecurity/vuln-list-update/suse/cvrfarchive"
"github.com/aquasecurity/vuln-list-update/utils"
)
@@ -47,81 +41,21 @@ func NewConfig() Config {
func (c Config) Update() error {
log.Print("Fetching SUSE CVRF archive...")
- // The SUSE server is sometimes unstable, so download the whole archive into
- // memory before processing. Streaming directly from the HTTP response would
- // make it hard to distinguish a mid-transfer disconnection (which surfaces
- // as a truncated tar) from a legitimate parse error. The archive is only a
- // few hundred MB, which fits comfortably in memory on CI runners.
- body, err := utils.FetchURL(c.URL, "", retries)
- if err != nil {
- return xerrors.Errorf("failed to download CVRF archive: %w", err)
- }
-
- var decompressed io.Reader
- switch {
- case strings.HasSuffix(c.URL, ".tar.bz2"):
- // The upstream archive is .tar.bz2, which is the only format used in production.
- decompressed = bzip2.NewReader(bytes.NewReader(body))
- case strings.HasSuffix(c.URL, ".tar.gz"):
- // Go's compress/bzip2 lacks a Writer, so tests use .tar.gz instead.
- gr, err := gzip.NewReader(bytes.NewReader(body))
- if err != nil {
- return xerrors.Errorf("failed to decompress gzip: %w", err)
- }
- defer gr.Close()
- decompressed = gr
- default:
- return xerrors.Errorf("unsupported archive format: %s", c.URL)
- }
- tr := tar.NewReader(decompressed)
-
- for {
- hdr, err := tr.Next()
- switch {
- case errors.Is(err, io.EOF):
- return nil
- case err != nil:
- return xerrors.Errorf("failed to read tar entry: %w", err)
- case hdr.Typeflag != tar.TypeReg:
- continue
- }
-
- filename := filepath.Base(hdr.Name)
- // archive contains non-XML files (e.g. LICENSE), so skip them
- if !strings.HasSuffix(filename, ".xml") {
- continue
- }
- match := fileRegexp.FindStringSubmatch(filename)
- if match == nil {
- continue
- }
+ return cvrfarchive.Walk(c.URL, retries, fileRegexp, func(e cvrfarchive.Entry) error {
+ match := fileRegexp.FindStringSubmatch(e.Filename)
osName := match[1]
- data, err := io.ReadAll(tr)
- if err != nil {
- return xerrors.Errorf("failed to read tar entry data: %w", err)
- }
-
- if len(data) == 0 {
- log.Printf("empty CVRF xml: %s", filename)
- continue
- }
-
- if !utf8.Valid(data) {
- log.Printf("invalid UTF-8: %s", filename)
- data = []byte(strings.ToValidUTF8(string(data), ""))
- }
-
var cv Cvrf
- if err = xml.Unmarshal(data, &cv); err != nil {
- return xerrors.Errorf("failed to decode SUSE XML (%s): %w", filename, err)
+ if err := xml.Unmarshal(e.Data, &cv); err != nil {
+ return xerrors.Errorf("failed to decode SUSE XML (%s): %w", e.Filename, err)
}
dir := filepath.Join(cvrfDir, suseDir, osName)
- if err = c.saveCvrfPerYear(dir, cv.Tracking.ID, cv); err != nil {
+ if err := c.saveCvrfPerYear(dir, cv.Tracking.ID, cv); err != nil {
return xerrors.Errorf("failed to save CVRF: %w", err)
}
- }
+ return nil
+ })
}
func (c Config) saveCvrfPerYear(dirName string, cvrfID string, data Cvrf) error {
diff --git a/suse/cvrfarchive/archive.go b/suse/cvrfarchive/archive.go
new file mode 100644
index 0000000..deb4758
--- /dev/null
+++ b/suse/cvrfarchive/archive.go
@@ -0,0 +1,103 @@
+// Package cvrfarchive walks SUSE CVRF tar archives published at
+// http://ftp.suse.com/pub/projects/security/. It hides the
+// download/decompress/tar plumbing so feed-specific code only needs
+// to handle XML decoding and persistence.
+package cvrfarchive
+
+import (
+ "archive/tar"
+ "bytes"
+ "compress/bzip2"
+ "compress/gzip"
+ "errors"
+ "io"
+ "log"
+ "path/filepath"
+ "regexp"
+ "strings"
+ "unicode/utf8"
+
+ "golang.org/x/xerrors"
+
+ "github.com/aquasecurity/vuln-list-update/utils"
+)
+
+// Entry is a single XML document extracted from a SUSE CVRF archive.
+// Data is guaranteed to be valid UTF-8 (invalid bytes are stripped).
+type Entry struct {
+ Filename string
+ Data []byte
+}
+
+// Walk downloads a SUSE CVRF archive from url, decompresses it
+// (bzip2 or gzip, detected by the URL suffix) and invokes handler
+// for every .xml entry whose base name matches nameRegexp.
+// Non-regular tar entries, empty files and non-XML files are skipped;
+// invalid UTF-8 byte sequences are stripped from the data.
+func Walk(url string, retries int, nameRegexp *regexp.Regexp, handler func(Entry) error) error {
+ // The SUSE server is sometimes unstable, so download the whole archive into
+ // memory before processing. Streaming directly from the HTTP response would
+ // make it hard to distinguish a mid-transfer disconnection (which surfaces
+ // as a truncated tar) from a legitimate parse error. The archive is only a
+ // few hundred MB, which fits comfortably in memory on CI runners.
+ body, err := utils.FetchURL(url, "", retries)
+ if err != nil {
+ return xerrors.Errorf("failed to download archive: %w", err)
+ }
+
+ decompressed, err := decompress(url, body)
+ if err != nil {
+ return err
+ }
+
+ tr := tar.NewReader(decompressed)
+ for {
+ hdr, err := tr.Next()
+ switch {
+ case errors.Is(err, io.EOF):
+ return nil
+ case err != nil:
+ return xerrors.Errorf("failed to read tar entry: %w", err)
+ case hdr.Typeflag != tar.TypeReg:
+ continue
+ }
+
+ filename := filepath.Base(hdr.Name)
+ if !strings.HasSuffix(filename, ".xml") {
+ continue
+ }
+ if nameRegexp != nil && !nameRegexp.MatchString(filename) {
+ continue
+ }
+
+ data, err := io.ReadAll(tr)
+ if err != nil {
+ return xerrors.Errorf("failed to read tar entry data: %w", err)
+ }
+ if len(data) == 0 {
+ log.Printf("empty xml: %s", filename)
+ continue
+ }
+ if !utf8.Valid(data) {
+ log.Printf("invalid UTF-8: %s", filename)
+ data = []byte(strings.ToValidUTF8(string(data), ""))
+ }
+
+ if err := handler(Entry{Filename: filename, Data: data}); err != nil {
+ return err
+ }
+ }
+}
+
+func decompress(url string, body []byte) (io.Reader, error) {
+ switch {
+ case strings.HasSuffix(url, ".tar.bz2"):
+ // The upstream archive is .tar.bz2, which is the only format used in production.
+ return bzip2.NewReader(bytes.NewReader(body)), nil
+ case strings.HasSuffix(url, ".tar.gz"):
+ // Go's compress/bzip2 lacks a Writer, so tests use .tar.gz instead.
+ return gzip.NewReader(bytes.NewReader(body))
+ default:
+ return nil, xerrors.Errorf("unsupported archive format: %s", url)
+ }
+}
diff --git a/suse/cvrfcve/cvrfcve.go b/suse/cvrfcve/cvrfcve.go
new file mode 100644
index 0000000..367100d
--- /dev/null
+++ b/suse/cvrfcve/cvrfcve.go
@@ -0,0 +1,65 @@
+package cvrfcve
+
+import (
+ "encoding/xml"
+ "fmt"
+ "log"
+ "path/filepath"
+ "regexp"
+ "strings"
+
+ "github.com/spf13/afero"
+ "golang.org/x/xerrors"
+
+ "github.com/aquasecurity/vuln-list-update/suse/cvrfarchive"
+ "github.com/aquasecurity/vuln-list-update/utils"
+)
+
+const (
+ cvrfCVEArchiveURL = "http://ftp.suse.com/pub/projects/security/cvrf-cve.tar.bz2"
+ cvrfDir = "cvrf"
+ suseCVEDir = "suse-cves"
+ retries = 5
+)
+
+var fileRegexp = regexp.MustCompile(`^cvrf-(CVE-\d{4}-\d+)\.xml$`)
+
+type Config struct {
+ VulnListDir string
+ URL string
+ AppFs afero.Fs
+}
+
+func NewConfig() Config {
+ return Config{
+ VulnListDir: utils.VulnListDir(),
+ URL: cvrfCVEArchiveURL,
+ AppFs: afero.NewOsFs(),
+ }
+}
+
+func (c Config) Update() error {
+ log.Print("Fetching SUSE CVE CVRF archive...")
+
+ return cvrfarchive.Walk(c.URL, retries, fileRegexp, func(e cvrfarchive.Entry) error {
+ // ID is taken from the file name, which is already validated by fileRegexp.
+ cveID := fileRegexp.FindStringSubmatch(e.Filename)[1]
+
+ var cv Cvrf
+ if err := xml.Unmarshal(e.Data, &cv); err != nil {
+ return xerrors.Errorf("failed to decode SUSE CVE CVRF XML (%s): %w", e.Filename, err)
+ }
+
+ return c.saveCVEPerYear(cveID, cv)
+ })
+}
+
+func (c Config) saveCVEPerYear(cveID string, data Cvrf) error {
+ year := strings.Split(cveID, "-")[1]
+ yearDir := filepath.Join(c.VulnListDir, cvrfDir, suseCVEDir, year)
+ fileName := fmt.Sprintf("%s.json", cveID)
+ if err := utils.WriteJSON(c.AppFs, yearDir, fileName, data); err != nil {
+ return xerrors.Errorf("failed to write file: %w", err)
+ }
+ return nil
+}
diff --git a/suse/cvrf-cve/cvrf_cve_test.go b/suse/cvrfcve/cvrfcve_test.go
similarity index 93%
rename from suse/cvrf-cve/cvrf_cve_test.go
rename to suse/cvrfcve/cvrfcve_test.go
index cfdac90..56ba325 100644
--- a/suse/cvrf-cve/cvrf_cve_test.go
+++ b/suse/cvrfcve/cvrfcve_test.go
@@ -13,7 +13,7 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
- cvrfcve "github.com/aquasecurity/vuln-list-update/suse/cvrf-cve"
+ "github.com/aquasecurity/vuln-list-update/suse/cvrfcve"
)
// createArchive creates a tar.gz archive from the given directory.
@@ -44,13 +44,13 @@ func TestConfig_Update(t *testing.T) {
{
name: "positive test",
appFs: afero.NewMemMapFs(),
- archiveDir: "testdata/cvrf-cve",
+ archiveDir: "testdata/cvrfcve",
expectedFile: "/tmp/cvrf/suse-cves/2014/CVE-2014-6271.json",
},
{
name: "broken XML",
appFs: afero.NewMemMapFs(),
- archiveDir: "testdata/broken-cvrf-cve",
+ archiveDir: "testdata/broken-cvrfcve",
expectedErrorMsg: "failed to decode SUSE CVE CVRF XML",
},
}
diff --git a/suse/cvrf-cve/testdata/broken-cvrf-cve/cvrf-CVE-2014-6271.xml b/suse/cvrfcve/testdata/broken-cvrfcve/cvrf-CVE-2014-6271.xml
similarity index 100%
rename from suse/cvrf-cve/testdata/broken-cvrf-cve/cvrf-CVE-2014-6271.xml
rename to suse/cvrfcve/testdata/broken-cvrfcve/cvrf-CVE-2014-6271.xml
diff --git a/suse/cvrf-cve/testdata/cvrf-cve/cvrf-CVE-1234-12345.xml b/suse/cvrfcve/testdata/cvrfcve/cvrf-CVE-1234-12345.xml
similarity index 100%
rename from suse/cvrf-cve/testdata/cvrf-cve/cvrf-CVE-1234-12345.xml
rename to suse/cvrfcve/testdata/cvrfcve/cvrf-CVE-1234-12345.xml
diff --git a/suse/cvrf-cve/testdata/cvrf-cve/cvrf-CVE-2014-6271.xml b/suse/cvrfcve/testdata/cvrfcve/cvrf-CVE-2014-6271.xml
similarity index 100%
rename from suse/cvrf-cve/testdata/cvrf-cve/cvrf-CVE-2014-6271.xml
rename to suse/cvrfcve/testdata/cvrfcve/cvrf-CVE-2014-6271.xml
diff --git a/suse/cvrf-cve/types.go b/suse/cvrfcve/types.go
similarity index 100%
rename from suse/cvrf-cve/types.go
rename to suse/cvrfcve/types.go
c8b6644 to
4bb48fa
Compare
| // resulting JSON tree stays small (the upstream archive is ~250 MB | ||
| // compressed / ~10 GB uncompressed; trimming brings the persisted tree | ||
| // down by two orders of magnitude). |
There was a problem hiding this comment.
nit:
I think we don't need to write upstream sizes to avoid confusion (also the size will increase).
| // resulting JSON tree stays small (the upstream archive is ~250 MB | |
| // compressed / ~10 GB uncompressed; trimming brings the persisted tree | |
| // down by two orders of magnitude). | |
| // resulting JSON tree stays small. |
|
PR description:
221 MB after removing some fields: ➜ cvrf du -sh ./suse-cves/
221M ./suse-cves/ |
4bb48fa to
0c165b2
Compare
Add a new `suse-cvrf-cve` target that fetches the per-CVE CVRF documents SUSE publishes at /pub/projects/security/cvrf-cve/ and writes them to cvrf/suse-cves/<year>/CVE-<id>.json. The regular SUSE CVRF advisory feed omits CVSS v3/v4 scores; the per-CVE feed is the only source that exposes them, so we persist these documents to make the scores available to downstream consumers (e.g. trivy-db). To keep run time reasonable on GitHub-hosted runners, fetch the bundled cvrf-cve.tar.bz2 archive once and stream-extract it, mirroring upstream PR aquasecurity#437 for the regular SUSE CVRF feed. This avoids tens of thousands of individual HTTP requests, which previously pushed the job past the 6h GitHub Actions limit. - New package suse/cvrfarchive that encapsulates the download/decompress/tar walking logic shared by the CVRF and CVE CVRF feeds; suse/cvrf is refactored to use it as well, eliminating the duplicated archive plumbing. - New package suse/cvrfcve (the previous suse/cvrf-cve was renamed to drop the hyphen in line with Go package conventions). The persisted schema is trimmed to the fields needed for CVSS lookup (Title / Tracking ID + dates / CVE / Threats / References / CVSSScoreSets); ProductTree, ProductStatuses, DocumentNotes and RevisionHistory are dropped, which keeps the resulting JSON tree small. - Tests build a tar.gz archive at runtime via tar.Writer.AddFS and compare the output against committed golden files, matching the pattern used by suse/cvrf. - Register the suse-cvrf-cve target in main.go and document it in README.md. - Add a SUSE CVE CVRF step to .github/workflows/update.yml. Co-authored-by: Cursor <cursoragent@cursor.com>
0c165b2 to
0e6ca8b
Compare
|
Sorry for the late review. The premise is correct: the advisory CVRF feed (cvrf.tar.bz2) only carries CVSS v2, not v3. But SUSE also publishes an advisory-level CSAF feed (https://ftp.suse.com/pub/projects/security/csaf.tar.bz2), which covers everything we need from a single feed:
So why maintain two SUSE sources (cvrf + cvrf-cve) instead of switching the existing one over to CSAF? The per-CVE feed adds a ~241MB mirror, drops the advisory linkage, and largely duplicates what we already fetch. trivy-db already depends on |
|
@knqyf263 Thanks for the detailed review — this is very helpful. You're right on the facts:
We did not evaluate switching the existing I'm happy to pivot this work: close or replace the cvrf-cve approach and prototype
If you prefer, I can open a follow-up PR for CSAF and revert the cvrf-cve feed from this one — let me know what you'd like. |
The cvrf feed which contains the advisories are missing the scores of the cve's data related to the advisory, so to fetch the cve data we're using cvrf-cve feed provided by the suse to get the cve details.
The new directory is 221MB
The regular SUSE CVRF advisory feed at
http://ftp.suse.com/pub/projects/security/cvrf/(consumed by the existingsuse-cvrftarget) does not publish CVSS v3 / v4 scores — only the v2 vector, where present. SUSE exposes the v3/v4 scores in a separate per-CVE CVRF feed athttp://ftp.suse.com/pub/projects/security/cvrf-cve/, and this PR adds a newsuse-cvrf-cvetarget that mirrors those documents intocvrf/suse-cves/<year>/CVE-<id>.jsonso downstream consumers (e.g. trivy-db) can look up the scores.suse/cvrfarchivepackage that encapsulates the download / decompress / tar-walk logic. Both feeds (suse/cvrfandsuse/cvrfcve) now share this code; the duplicated archive plumbing insuse/cvrf/cvrf.gohas been removed.suse/cvrfcvepackage that registers assuse-cvrf-cve(directory renamed fromsuse/cvrf-cveto drop the hyphen, matching Go package conventions).Title,Tracking.{ID,InitialReleaseDate,CurrentReleaseDate}, and per-vulnerabilityCVE/Threats/References/CVSSScoreSets. The bulkyProductTree,ProductStatuses,DocumentNotesandRevisionHistoryblocks are dropped, which keeps the persisted JSON tree small (the upstream archive is ~250 MB compressed / ~10 GB uncompressed; trimming brings the resultingcvrf/suse-cves/tree down by roughly two orders of magnitude).cvrf-cve.tar.bz2once and stream-extracts it, mirroring the fix applied tosuse-cvrfin fix(suse): download CVRF archive instead of individual files #437.main.goregisters the new target andREADME.mddocuments it.update.ymlgains a singleSUSE CVE CVRFstep.