[ANE-1639] add metadata to vendored dependencies (#1455)

* add metadata to archive

* refactor

* format

* update docs

* add test

* update fossa-deps example

* cr nits

* fix test

* fix spacing

* remove newline

Author: jagonalez

Changelog.md

@@ -1,5 +1,9 @@
 # FOSSA CLI Changelog
 
+## 3.9.30
+
+- Vendored Dependencies: add support for metadata (description, and homepage) for dependencies. ([#1455](https://github.com/fossas/fossa-cli/pull/1455))
+
 ## 3.9.29
 - install scripts: Surface curl errors and display http status code correctly. ([#1456](https://github.com/fossas/fossa-cli/pull/1456))
 - Update jar-callgraph version to 1.0.2 [#1454](https://github.com/fossas/fossa-cli/pull/1454)

docs/features/vendored-dependencies.md

@@ -17,6 +17,13 @@ vendored-dependencies:
 - name: Django
   path: vendor/Django-3.4.16.zip # path can be either a file or a folder.
   version: "3.4.16" # revision will be set to the MD5 hash of the filepath if left unspecified.
+# You can also provide a description and/or homepage. These values populate metadata fields in reports in the FOSSA web UI.
+- name: Winston
+  path: vendor/winston.zip
+  version: "5.0.0-alpha"
+  metadata:
+    description: "winston archive"
+    homepage: "https://winston-project.com"
 ```
 
 The path to a vendored dependency can either be a path to an archive or a path to a directory.
@@ -27,6 +34,8 @@ If the version is not specified, FOSSA CLI calculates the version by generating
 
 Note: When parsed, YAML considers text that could be a decimal number (such as 1.0 or 2.0) to be a number, not a string. This means that we'd parse the version 1.0 as 1. This probably isn't what you want. To avoid this, surround your version with quotes, as in "1.0".
 
+You can also optionally add metadata fields ("description" and "homepage") to populate these fields in the FOSSA web UI (these fields can be displayed when generating reports).
+
 We also support json-formatted dependencies:
 
 ```json
@@ -63,7 +72,11 @@ We also support json-formatted dependencies:
     }, {
       "name": "winston",
       "path": "vendor/winston.tar.gz",
-      "version": "5.0.0-alpha"
+      "version": "5.0.0-alpha",
+      "metadata": {
+        "description": "winston archive",
+        "homepage": "https://winston-project.com/homepage"
+      }
     }
   ],
   "remote-dependencies": [

docs/references/files/fossa-deps.md

@@ -1,6 +1,6 @@
 # `fossa-deps`
 
-`fossa-deps` file is a file named `fossa-deps.{yaml, yml, json}` at the root of the project. It can be used to provide manual and vendor dependencies. 
+`fossa-deps` file is a file named `fossa-deps.{yaml, yml, json}` at the root of the project. It can be used to provide manual and vendor dependencies.
 
 By default, the `fossa-deps.{yaml, yml, json}` file at the root of the project is used. However, if the `--fossa-deps-file` flag is present, then the provided `<name-of-file>.{yaml, yaml, json}` file will be used instead.
 
@@ -10,7 +10,7 @@ For more details on specifying a fossa-deps file, please refer to the [subcomman
 
 ### `referenced-dependencies:`
 
-Denotes listing of dependencies, which are to be analyzed in conjunction with the analysis. 
+Denotes listing of dependencies, which are to be analyzed in conjunction with the analysis.
 
 - `type`: Type of dependency. (Required)
 - `name`: Name of the dependency. It should be the same name as listed in dependencies registry. (Required)
@@ -25,20 +25,20 @@ referenced-dependencies:
   version: 2.1.7
 ```
 
-For more details, please refer to the [feature](../../features/manual-dependencies.md) walk through. 
+For more details, please refer to the [feature](../../features/manual-dependencies.md) walk through.
 
 ### `custom-dependencies:`
 
-Denotes listing of dependencies, which can't be automatically discovered or identified but are to be stubbed and included in the analysis. 
+Denotes listing of dependencies, which can't be automatically discovered or identified but are to be stubbed and included in the analysis.
 
 - `name`: Name of the dependency. (Required)
 - `version`: Revision of the dependency. (Required)
 - `license`: License of the dependency. (Required)
 - `metadata.homepage`: Homepage of the dependency. This metadata is used to enrich reporting provided in FOSSA's web interface.
 - `metadata.description`: Description of the dependency. This metadata is used to enrich reporting provided in FOSSA's web interface.
 
-Example: 
-```yaml 
+Example:
+```yaml
 - name: foo-wrapper
   version: 1.2.3
   license: MIT
@@ -47,7 +47,7 @@ Example:
     description: Provides foo and a helpful interface around foo-like tasks.
 ```
 
-For more details, please refer to the [feature](../../features/manual-dependencies.md) walk through. 
+For more details, please refer to the [feature](../../features/manual-dependencies.md) walk through.
 
 ### `remote-dependencies:`
 
@@ -62,29 +62,34 @@ Denotes listing of dependencies, whose source code is to be downloaded from prov
 > Combined length of url and version has upper bound. It depends on your organization identifier. You can
 find your organization identifier in FOSSA Webapp, by going to any project's "settings" page, and retrieving
 numeric value from project's locator. For example, project locator of `custom+123/some-project-id`, means
-`123` is your organization identifier. 
+`123` is your organization identifier.
 
 > Combined length of `url`, `version`, and your `organizaion id` must be less than `241`.
 
-For more details, please refer to the [feature](../../features/manual-dependencies.md) walk through. 
+For more details, please refer to the [feature](../../features/manual-dependencies.md) walk through.
 
 ### `vendored-dependencies:`
 
 Denotes listing of files or directories, which are to be archived and uploaded to FOSSA backend for license scanning.
 
 - `name`: Name of the dependency (Required)
 - `path`: Local path to a file, or a directory (Required)
-- `version`: Revision of the dependency. If not specified, the md5 hash of the file path will be used. 
+- `version`: Revision of the dependency. If not specified, the md5 hash of the file path will be used.
+- `metadata.homepage`: Homepage of the dependency. This metadata is used to enrich reporting provided in FOSSA's web interface.
+- `metadata.description`: Description of the dependency. This metadata is used to enrich reporting provided in FOSSA's web interface.
 
 ```yaml
 vendored-dependencies:
 - name: Django
   path: vendor/Django-3.4.16.zip
   version: 3.4.16
+  metadata:
+    homepage: https://djangoproject.com
+    description: Django
 ```
 > Note: License scanning currently operates by uploading the files at the specified path to a secure S3 bucket. All files that do not contain licenses are then removed after 2 weeks.
 
-For more details, please refer to the [feature](../../features/vendored-dependencies.md) walk through. 
+For more details, please refer to the [feature](../../features/vendored-dependencies.md) walk through.
 
 ## Errors in the `fossa-deps` file
 

docs/references/files/fossa-deps.schema.json

@@ -212,6 +212,19 @@
                 "version": {
                     "type": "string",
                     "description": "Version of the dependency. This will be the version associated with this vendored dependency in FOSSA's dashboard"
+                },
+                "metadata": {
+                    "type": "object",
+                    "properties": {
+                        "description": {
+                            "type": "string",
+                            "description": "Description of the dependency (if any)"
+                        },
+                        "homepage": {
+                            "type": "string",
+                            "description": "Homepage of the dependency. This should be web address."
+                        }
+                    }
                 }
             },
             "required": [

docs/walkthroughs/conan.md

@@ -1,8 +1,8 @@
 # Custom Integration with Conan Package Manager
 
-Conan is a dependency and package manager for C and C++ languages. It is free and open-source, works on all 
-platforms (Windows, Linux, OSX, FreeBSD, Solaris, etc.), and can be used to develop for all targets, including 
-embedded, mobile (iOS, Android), and bare metal. It also integrates with all build systems like CMake, 
+Conan is a dependency and package manager for C and C++ languages. It is free and open-source, works on all
+platforms (Windows, Linux, OSX, FreeBSD, Solaris, etc.), and can be used to develop for all targets, including
+embedded, mobile (iOS, Android), and bare metal. It also integrates with all build systems like CMake,
 Visual Studio (MSBuild), Makefiles, etc., including proprietary ones.
 
 ## Prerequisite
@@ -12,8 +12,8 @@ Visual Studio (MSBuild), Makefiles, etc., including proprietary ones.
 
 ## Integration
 
-This integration uses the `conan graph info` command to retrieve the dependency graph and source code for all dependencies. From this data, it generates [fossa-deps](./../references/files/fossa-deps.md) file with 
-[vendor-dependencies](./../features/vendored-dependencies.md) and [custom-dependencies](../features/manual-dependencies.md).
+This integration uses the `conan graph info` command to retrieve the dependency graph and source code for all dependencies. From this data, it generates [fossa-deps](./../references/files/fossa-deps.md) file with
+[vendored-dependencies](./../features/vendored-dependencies.md) and [custom-dependencies](../features/manual-dependencies.md).
 
 To use this integration,
 1. Download [make_fossa_deps_conan.py](./make_fossa_deps_conan.py) python script, and place it in the same directory as `conanfile.txt` or `conanfile.py.`
@@ -32,7 +32,7 @@ In this approach, `make_fossa_deps_conan.py` does the followings:
 
 ### Limitations
 
-This integration method uses [vendor-dependencies](./../features/vendored-dependencies.md) and [custom-dependencies](../features/manual-dependencies.md)
+This integration method uses [vendored-dependencies](./../features/vendored-dependencies.md) and [custom-dependencies](../features/manual-dependencies.md)
 functionalities, and as such, it does not provide the following,
 
 - Security functionalities (FOSSA will not be able to identify vulnerabilities, only licensing and copyright issues)
@@ -92,11 +92,11 @@ script is supplied as a potential option if you want to start using FOSSA for Co
 This integration example uses the `conan graph info` command with `--format json` and
 `-c tools.build:download_source=True` option, which are only available in Conan v2 (`v2.0.0+`).
 
-#### 3. I want to use a custom profile or provide additional options. 
+#### 3. I want to use a custom profile or provide additional options.
 
 You can provide any additional [`conan graph info`](https://docs.conan.io/2.0/reference/commands/graph/info.html) options (except `--format` or `-f`)
 
-To do so, provide options to the Python script. For example, 
+To do so, provide options to the Python script. For example,
 
 ```bash
 >> python3 make_fossa_deps_conan.py -s compiler=gcc
@@ -116,4 +116,3 @@ that declared license is always used.
 
 - [Conan Package Manager](https://docs.conan.io)
 - [Conan graph command](https://docs.conan.io/2.0/reference/commands/graph/info.html)
-

integration-test/Analysis/LicenseScannerSpec.hs

@@ -46,6 +46,7 @@ vendoredDep =
     "recursive-archive-test"
     "vendor/foo.tar.gz"
     (Just "0.0.1")
+    Nothing
 
 -- the contents of the archive look like this. The `FOO_LICENSE` files contain an MIT license.
 -- `bar_apache.rb` and `something.rb` contain an Apache-2.0 license.

spectrometer.cabal

@@ -236,6 +236,7 @@ library
     App.Fossa.Container.Sources.Podman
     App.Fossa.Container.Sources.Registry
     App.Fossa.Container.Test
+    App.Fossa.DependencyMetadata
     App.Fossa.DumpBinaries
     App.Fossa.EmbeddedBinary
     App.Fossa.FirstPartyScan

src/App/Fossa/ArchiveUploader.hs

@@ -9,6 +9,7 @@ import App.Fossa.VendoredDependency (
   arcToLocator,
   compressFile,
   dedupVendoredDeps,
+  getMetadata,
   hashFile,
  )
 import App.Types (ComponentUploadFileType (..), DependencyRebuild)
@@ -52,7 +53,7 @@ compressAndUpload ::
   Path Abs Dir ->
   VendoredDependency ->
   m Archive
-compressAndUpload arcDir tmpDir VendoredDependency{..} = context "compressing and uploading vendored deps" $ do
+compressAndUpload arcDir tmpDir dep@VendoredDependency{..} = context "compressing and uploading vendored deps" $ do
   logSticky $ "Compressing '" <> vendoredName <> "' at '" <> vendoredPath <> "'"
   compressedFile <- sendIO $ compressFile tmpDir arcDir (toString vendoredPath)
 
@@ -66,7 +67,7 @@ compressAndUpload arcDir tmpDir VendoredDependency{..} = context "compressing an
   res <- uploadArchive signedURL compressedFile
   logDebug . pretty $ (decodeUtf8 @Text res)
 
-  pure $ Archive vendoredName depVersion
+  pure . uncurry (Archive vendoredName depVersion) $ (getMetadata dep)
 
 -- archiveUploadSourceUnit receives a list of vendored dependencies, a root path, and API settings.
 -- Using this information, it uploads each vendored dependency and queues a build for the dependency.

src/App/Fossa/DependencyMetadata.hs

@@ -0,0 +1,20 @@
+module App.Fossa.DependencyMetadata (
+  DependencyMetadata (..),
+) where
+
+import Data.Aeson (FromJSON (..), withObject, (.:?))
+import Data.Aeson.Extra (forbidMembers)
+import Data.Text (Text)
+
+data DependencyMetadata = DependencyMetadata
+  { depDescription :: Maybe Text
+  , depHomepage :: Maybe Text
+  }
+  deriving (Eq, Ord, Show)
+
+instance FromJSON DependencyMetadata where
+  parseJSON = withObject "metadata" $ \obj ->
+    DependencyMetadata
+      <$> obj .:? "description"
+      <*> obj .:? "homepage"
+      <* forbidMembers "metadata" ["url"] obj

src/App/Fossa/FirstPartyScan.hs

@@ -99,7 +99,7 @@ firstPartyScanMain ::
 firstPartyScanMain base cfg org = do
   runFirstPartyScans <- shouldRunFirstPartyScans cfg org
   manualDeps <- findAndReadFossaDepsFile base
-  let vdep = VendoredDependency "first-party" "." Nothing
+  let vdep = VendoredDependency "first-party" "." Nothing Nothing
       uploadKind = orgFileUpload org
   pathFilters <- mergePathFilters base manualDeps (licenseScanPathFilters $ vendoredDeps cfg)
   case runFirstPartyScans of

src/App/Fossa/Init/fossa-deps.yml

@@ -19,8 +19,8 @@
 #
 # # referenced-dependencies
 # # --
-# # Denotes listing of dependencies, which are to be analyzed in 
-# # conjunction with the analysis. This is used for packages that 
+# # Denotes listing of dependencies, which are to be analyzed in
+# # conjunction with the analysis. This is used for packages that
 # # originate from a known package manager.
 # #
 # # Possible dependency type include:
@@ -66,13 +66,13 @@
 #
 # # custom-dependencies
 # # --
-# # Denotes listing of dependencies, which can't be automatically discovered 
-# # or identified but are to be stubbed and included in the analysis. This is used 
-# # to add a unique dependency to your dependency graph 
+# # Denotes listing of dependencies, which can't be automatically discovered
+# # or identified but are to be stubbed and included in the analysis. This is used
+# # to add a unique dependency to your dependency graph
 # # and the metadata you wish to associate with it.
 # #
 # # Learn more: https://github.com/fossas/fossa-cli/blob/master/docs/features/manual-dependencies.md#custom-dependencies
-# # 
+# #
 # custom-dependencies:
 # - name: foo-wrapper                                     # Name of the dependency. (Required)
 #   version: 1.2.3                                        # Revision of the dependency. (Required)
@@ -85,8 +85,8 @@
 #
 # # remote-dependencies
 # # --
-# # Denotes listing of dependencies, whose source code is to be downloaded 
-# # from provided URL, and analyzed for license scanning in FOSSA backend. The following archive types 
+# # Denotes listing of dependencies, whose source code is to be downloaded
+# # from provided URL, and analyzed for license scanning in FOSSA backend. The following archive types
 # # are supported: *.zip, *.tar, *.tar.gz, *.tar.bz2, *.tar.xz.
 # #
 # # Learn more: https://github.com/fossas/fossa-cli/blob/master/docs/references/files/fossa-deps.md#remote-dependencies
@@ -103,13 +103,16 @@
 #
 # # vendored-dependencies
 # # --
-# # Denotes listing of files or directories, which are to be 
+# # Denotes listing of files or directories, which are to be
 # # either, license scanned by fossa-cli, or and uploaded to FOSSA backend for
 # # license scanning.
 # #
 # # Learn more: https://github.com/fossas/fossa-cli/blob/master/docs/features/vendored-dependencies.md
 # #
 # vendored-dependencies:
-# - name: Django                    # Name of the dependency (Required)
-#   path: vendor/Django-3.4.16.zip  # Local path to a archive file, or a directory (Required)
-#   version: 3.4.16                 # Revision of the dependency. If not specified, the md5 hash of the file path will be used.
+# - name: Django                               # Name of the dependency (Required)
+#   path: vendor/Django-3.4.16.zip             # Local path to a archive file, or a directory (Required)
+#   version: 3.4.16                            # Revision of the dependency. If not specified, the md5 hash of the file path will be used.
+#   metadata:                                  # Metadata of the dependency (Optional)
+#     description: Django                      # Description of the dependency, this will be shown in FOSSA UI, and generated reports. (Optional)
+#     homepage: https://www.djangoproject.com  # Homepage of the dependency, this will be shown in FOSSA UI, and generated reports. (Optional)

src/App/Fossa/LicenseScanner.hs

@@ -23,6 +23,7 @@ import App.Fossa.VendoredDependency (
   compressFile,
   dedupVendoredDeps,
   forceVendoredToArchive,
+  getMetadata,
   hashFile,
   skippedDepsDebugLog,
   vendoredDependencyScanModeToDependencyRebuild,
@@ -327,7 +328,7 @@ uploadVendoredDep ::
   VendoredDependency ->
   LicenseSourceUnit ->
   m (Maybe Archive)
-uploadVendoredDep baseDir VendoredDependency{..} licenseSourceUnit = do
+uploadVendoredDep baseDir dep@VendoredDependency{..} licenseSourceUnit = do
   depVersion <- case vendoredVersion of
     Nothing -> sendIO $ withSystemTempDir "fossa-temp" (calculateVendoredHash baseDir vendoredPath)
     Just version -> pure version
@@ -337,7 +338,7 @@ uploadVendoredDep baseDir VendoredDependency{..} licenseSourceUnit = do
   logSticky $ "Uploading license results for '" <> vendoredName <> "' to secure S3 bucket"
   uploadLicenseScanResult signedURL licenseSourceUnit
 
-  pure $ Just $ Archive vendoredName depVersion
+  pure . Just . uncurry (Archive vendoredName depVersion) $ (getMetadata dep)
 
 -- | licenseScanSourceUnit receives a list of vendored dependencies, a root path, and API settings.
 -- Using this information, it license scans each vendored dependency, uploads the license scan results and then queues a build for the dependency.