[ANE-1809] - Partial distroless container support (#1448)

* Upgrade deps. Add fingerprint-lib as a dependency to millhone.

* Initial subcommand for jar searching.

* Can fingerprint jars in a container tarfile.

* Better error handling.

* Add config for millhone for where to output traces.

* Output layer information.

* Expose a way to get the stderr from a program when using the exec effect.

* Run millhone jar analyze as part of container scanning.

* Apply suggestions from code review (Jess)

Co-authored-by: Jessica Black <[email protected]>

* New module.

* Address PR requests from Jess.

* Update to latest lib-fingerprint.

* Fix fingerprints name.

* Parse Millhone's output properly.

* Output the observation kind so the CLI can just pass observations through.

* Create new output build type. Have the container analyzer include jar results.

* A type for LayerPaths.

* Add layer path to layers.

* Use container jar output from millhone to augment Container scans.

* Remove unneeded log statement.

* Fix warning.

* Hex encode millhone jar observation output.

* Refactor analyze native to fetch the organization only once.

* Implement fallback from sparkle to core.

* Rename Makefile target.

* Fix warning.

* Apply suggestions from code review

Co-authored-by: Jessica Black <[email protected]>

* Rename analyze_jar to analyze_container.

* Update comment, log message.

* Update changelog.

* Fix format.

* Output better messaging if Jar analysis fails.

* Update millhone cli command.

* Blurb mentioning Jar analysis.

* Remove bad underscore.

* CLI-side tests.

* Millhone test.

* Test against abstract JSON.

* fmt

* WIP: Cli rough-cut but we need Core support to make this work.

* Tests for JIC distroless container.

* Fix millhone test.

* Add a small npm project to the container.

* Add docs.

* Update changelog.

* Make paths on windows work for test.

* Update container scanning TOC.

* Add informational messaging when no system info can be found.

---------

Co-authored-by: Jessica Black <[email protected]>

Author: csasarak

Changelog.md

@@ -1,5 +1,9 @@
 # FOSSA CLI Changelog
 
+## 3.9.28
+
+- Container Scanning: Distroless containers will now return results for non-system dependencies. ([#1448](https://github.com/fossas/fossa-cli/pull/1448))
+
 ## 3.9.27
 
 - Tar: Move to the upstream Haskell tar library. FOSSA CLI should now work more reliably when unpacking containers for analysis. ([#1452](https://github.com/fossas/fossa-cli/pull/1452))

Makefile

@@ -26,7 +26,7 @@ build-cargo:
 # 	make test ARGS="Node.PackageLockV3"
 test: test-cargo test-cabal
 
-test-cabal:
+test-cabal: build-cargo
 ifdef ARGS
 	cabal test unit-tests --test-show-details=streaming --test-option=--format=checks --test-option=--times --test-option=--color --test-option=--match --test-option="$(ARGS)"
 else

docs/README.md

@@ -99,7 +99,8 @@ Concept guides explain the nuances behind how basic FOSSA primitives work. If yo
   - [From Docker Engine](./references/subcommands/container/scanner.md#2-from-docker-engine)
   - [From Container Registries](./references/subcommands/container/scanner.md#3-from-registries)
 - [Supported Container Package Managers](./references/subcommands/container/scanner.md#supported-container-package-managers)
-  - [Container Jar File Analysis](./references/subcommands/container/scanner.md#container-jar-analysis)
+ - [Container Jar File Analysis](./references/subcommands/container/scanner.md#container-jar-analysis)
+ - [Distroless Containers](./references/subcommands/container/scanner.md#distroless-containers)
 - [Viewing Detected Projects](./references/subcommands/container/scanner.md#view-detected-projects)
 - [Configuring Container Analysis Targets](./references/subcommands/container/scanner.md#utilize-analysis-target-configuration)
 - [Integrating Container Scanning in CI](./walkthroughs/container-scanning-generic-ci.md)

docs/references/subcommands/container/scanner.md

@@ -9,6 +9,8 @@
     - [3) From registries](#3-from-registries)
   - [Container image analysis](#container-image-analysis)
     - [Container Jar analysis](#container-jar-analysis)
+    - [Distroless Containers](#distroless-containers)
+    - [Supported Container Package Managers](#supported-container-package-managers)
     - [View detected projects](#view-detected-projects)
       - [Command output](#command-output)
     - [Utilize analysis target configuration](#utilize-analysis-target-configuration)
@@ -213,6 +215,11 @@ It will then report them to FOSSA which will try to match the Jar files to the p
 This process relies on there being a back-end that can perform that analysis.
 SaaS customers should have this functionality available but on-prem customers may need to contact FOSSA support to have it enabled.
 
+### Distroless Containers
+
+Container images where FOSSA cannot detect an operating system are supported but in a more limited way than images where FOSSA can.
+These container images will not support reporting system deps (APK, DPKG, and RPM) but can support the other forms of analyses listed in the table below.
+
 ### Supported Container Package Managers
 The following package managers are supported in container scanning:
 

extlib/millhone/src/cmd/analyze_container.rs

@@ -193,35 +193,49 @@ mod tests {
 
     use super::*;
 
-    const MILLHONE_OUT: &str = r#"{
+    const MILLHONE_OUT: &str = r#"
+{
   "discovered_jars": {
-    "blobs/sha256/5c079c30beb013e4b2f7729b6bdce6fba57941d28f20db985333fc1dd969f018": [
+    "blobs/sha256/61aed1a8baa251dee118b9ab203c1e420f0eda0a9b3f9322d67d235dd27a12ee": [
       {
         "kind": "v1.discover.binary.jar",
-        "path": "inner_directory/commons-email2-jakarta-2.0.0-M1.jar",
+        "path": "jackson-annotations-2.17.1.jar",
         "fingerprints": {
-          "v1.mavencentral.jar": "6bzpyKql6Q+UxKQgm14pcP4wHGo=",
-          "v1.raw.jar": "QA4SAurtJeo+lx1Vqve5uQYnvDKLFx1NgsSuoWKi8pw=",
-          "sha_256": "MuEcK3nOFuySTTg4HOJi3vvTpI9bYspfMHa9AK2merQ=",
-          "v1.class.jar": "2wRGbMGyGRwEXqNm53h1YK8OO879kvzDxazmJXiAcfI="
+          "sha_256": "/MrYLhMXLA5DhNtxV3IZybhjHAgg9LGNqqVwFvtmHHY=",
+          "v1.mavencentral.jar": "/KfvYZLJrQXQe8UNqZG/k3qErzo=",
+          "v1.class.jar": "t2Btr6rNrvzghM5Nud2uldRGVjw0/n5rK9j0xooQQyk=",
+          "v1.raw.jar": "wjGJk8cvY4tpKcUC5r8YuO15Wfv+rVuyWANBYCUIeDs="
         }
       }
     ],
-    "blobs/sha256/9733ccc395133a067f01ee6e380003d80fe9f443673e0f992ae6a4a7860a872c": [],
-    "blobs/sha256/61aed1a8baa251dee118b9ab203c1e420f0eda0a9b3f9322d67d235dd27a12ee": [
+    "blobs/sha256/0e4613a3c620a37d93aca05039001fb5a6063c9d9cfb0935e3aa984025f31198": [
       {
         "kind": "v1.discover.binary.jar",
-        "path": "jackson-annotations-2.17.1.jar",
+        "path": "slf4j-ext-2.0.0.jar",
         "fingerprints": {
-          "v1.mavencentral.jar": "/KfvYZLJrQXQe8UNqZG/k3qErzo=",
-          "sha_256": "/MrYLhMXLA5DhNtxV3IZybhjHAgg9LGNqqVwFvtmHHY=",
-          "v1.class.jar": "t2Btr6rNrvzghM5Nud2uldRGVjw0/n5rK9j0xooQQyk=",
-          "v1.raw.jar": "wjGJk8cvY4tpKcUC5r8YuO15Wfv+rVuyWANBYCUIeDs="
+          "v1.mavencentral.jar": "WO8bdGURkfUQyqK6rJ4miP9caEU=",
+          "v1.class.jar": "PexFkKDUkwq7Do2Pt3HVPjMBRfqj/Zzp+nK5D6LfPF4=",
+          "sha_256": "bWERAhXZlaGR2AxgVJDRTAlbOtHqLIVOpmncfLIUKj0=",
+          "v1.raw.jar": "7CEXDIU3h2Vcj6lmy4gbuh4KsMVCNgUZTCj9VA4VoV8="
+        }
+      }
+    ],
+    "blobs/sha256/632e84390ad558f9db0524f5e38a0af3e79c623a46bdce8a5e6a1761041b9850": [],
+    "blobs/sha256/054f94aa7ce72b59cd6abac5462f77f0645b2f1a7b17e55d8f847a6da58c90db": [
+      {
+        "kind": "v1.discover.binary.jar",
+        "path": "inner_directory/commons-email2-jakarta-2.0.0-M1.jar",
+        "fingerprints": {
+          "v1.class.jar": "2wRGbMGyGRwEXqNm53h1YK8OO879kvzDxazmJXiAcfI=",
+          "v1.raw.jar": "QA4SAurtJeo+lx1Vqve5uQYnvDKLFx1NgsSuoWKi8pw=",
+          "sha_256": "MuEcK3nOFuySTTg4HOJi3vvTpI9bYspfMHa9AK2merQ=",
+          "v1.mavencentral.jar": "6bzpyKql6Q+UxKQgm14pcP4wHGo="
         }
       }
     ]
   }
-}"#;
+}
+"#;
 
     #[test]
     fn it_finds_expected_output() {

integration-test/Container/AnalysisSpec.hs

@@ -48,10 +48,10 @@ registrySourceAnalysis = do
   aroundAll (runAnalyze registrySourceCfg) $ do
     describe "Container analysis from registry source" $ do
       it "Has the correct OS" $
-        \res -> res.imageData.imageOs `shouldBe` "alpine"
+        \scan -> scan.imageData.imageOs `shouldBe` Just "alpine"
       it "Has the correct OS release version" $
-        \res -> res.imageData.imageOsRelease `shouldBe` "3.19.1"
+        \scan -> scan.imageData.imageOsRelease `shouldBe` Just "3.19.1"
       it "Has the expected image tag" $
-        \res -> res.imageTag `shouldBe` "public.ecr.aws/docker/library/alpine"
+        \scan -> scan.imageTag `shouldBe` "public.ecr.aws/docker/library/alpine"
       it "Has at least one layer" $
-        \res -> res.imageData.imageLayers `shouldSatisfy` (not . null)
+        \scan -> scan.imageData.imageLayers `shouldSatisfy` (not . null)

src/App/Fossa/Container/Sources/Discovery.hs

@@ -59,11 +59,15 @@ import Types (
   FoundTargets (FoundTargets, ProjectWithoutTargets),
  )
 
-layerAnalyzers :: AnalyzeStaticTaskEffs sig m => OsInfo -> Bool -> [DiscoverFunc m]
-layerAnalyzers os onlySystemDeps =
+layerAnalyzers :: AnalyzeStaticTaskEffs sig m => Maybe OsInfo -> Bool -> [DiscoverFunc m]
+layerAnalyzers os onlySystemDeps = do
+  -- For true distroless container support the linux builds need to be able to produce a result without os info.
+  -- Not having OS info will mean we miss any system deps in the container.
+  -- It's still useful to allow this though because we can do jar in container analysis as well as regular project analysis.
+  let systemDeps = maybe [] osDepsAnalyzers os
   if onlySystemDeps
-    then osDepsAnalyzers os
-    else osDepsAnalyzers os ++ managedDepsDiscoveryF
+    then systemDeps
+    else systemDeps ++ managedDepsDiscoveryF
 
 osDepsAnalyzers :: AnalyzeStaticTaskEffs sig m => OsInfo -> [DiscoverFunc m]
 osDepsAnalyzers osInfo =

src/App/Fossa/Container/Sources/DockerArchive.hs

@@ -62,7 +62,7 @@ import Data.FileTree.IndexFileTree (SomeFileTree, fixedVfsRoot)
 import Data.Flag (Flag, fromFlag)
 import Data.Foldable (traverse_)
 import Data.Map qualified as Map
-import Data.Maybe (listToMaybe, mapMaybe)
+import Data.Maybe (isNothing, listToMaybe, mapMaybe)
 import Data.String.Conversion (ToString (toString))
 import Data.Text (Text)
 import Data.Text.Extra (breakOnEndAndRemove, showT)
@@ -122,8 +122,13 @@ analyzeFromDockerArchive systemDepsOnly filters withoutDefaultFilters tarball =
   let imageBaseLayer = baseLayer image
       baseDigest = layerDigest imageBaseLayer
   osInfo <-
-    context "Retrieving OS Information" $
-      runTarballReadFSIO baseFs tarball getOsInfo
+    context "Retrieving OS Information"
+      . warnThenRecover @Text "Could not retrieve OS info"
+      $ runTarballReadFSIO baseFs tarball getOsInfo
+
+  when (isNothing osInfo) $
+    logInfo "No image system information detected. System dependencies will not be included with this scan."
+
   baseUnits <-
     context "Analyzing From Base Layer" $
       analyzeLayer systemDepsOnly filters withoutDefaultFilters capabilities osInfo baseFs tarball
@@ -133,8 +138,8 @@ analyzeFromDockerArchive systemDepsOnly filters withoutDefaultFilters tarball =
         ContainerScan
           { imageData =
               ( ContainerScanImage
-                  (nameId osInfo)
-                  (version osInfo)
+                  (nameId <$> osInfo)
+                  (version <$> osInfo)
                   layers
               )
           , imageDigest
@@ -185,7 +190,7 @@ analyzeLayer ::
   AllFilters ->
   Flag WithoutDefaultFilters ->
   Int ->
-  OsInfo ->
+  Maybe OsInfo ->
   SomeFileTree TarEntryOffset ->
   Path Abs File ->
   m [SourceUnit]
@@ -228,7 +233,7 @@ runAnalyzers ::
   , Has AtomicCounter sig m
   ) =>
   Bool ->
-  OsInfo ->
+  Maybe OsInfo ->
   AllFilters ->
   Flag WithoutDefaultFilters ->
   m ()
@@ -330,7 +335,10 @@ listTargetsFromDockerArchive tarball = do
 
   logInfo "Analyzing Base Layer"
   baseFs <- context "Building Base Layer FS" $ mkFsFromChangeset $ baseLayer image
-  osInfo <- context "Retrieving OS Information" $ runTarballReadFSIO baseFs tarball getOsInfo
+  osInfo <-
+    context "Retrieving OS Information"
+      . warnThenRecover @Text "Could not retrieve OS info"
+      $ runTarballReadFSIO baseFs tarball getOsInfo
   context "Analyzing From Base Layer" $ listTargetLayer capabilities osInfo baseFs tarball "Base Layer"
 
   when (hasOtherLayers image) $ do
@@ -346,7 +354,7 @@ listTargetLayer ::
   , Has Telemetry sig m
   ) =>
   Int ->
-  OsInfo ->
+  Maybe OsInfo ->
   SomeFileTree TarEntryOffset ->
   Path Abs File ->
   Text ->

src/Container/Types.hs

@@ -100,8 +100,8 @@ instance ToJSON ContainerScan where
   toJSON scan = object ["image" .= imageData scan]
 
 data ContainerScanImage = ContainerScanImage
-  { imageOs :: Text
-  , imageOsRelease :: Text
+  { imageOs :: Maybe Text
+  , imageOsRelease :: Maybe Text
   , imageLayers :: [ContainerScanImageLayer]
   }
   deriving (Show, Eq, Ord)

test/App/Fossa/Container/AnalyzeNativeSpec.hs

@@ -27,11 +27,7 @@ import Effect.ReadFS (ReadFSIOC, runReadFSIO)
 import Path (Dir, File, Rel, mkRelDir, mkRelFile, (</>))
 import Path.IO (getCurrentDir)
 import Path.Internal (Path)
-import Srclib.Types (
-  Locator (Locator),
-  SourceUnit (sourceUnitBuild),
-  SourceUnitBuild (buildImports),
- )
+import Srclib.Types (Locator (..), SourceUnit (..), SourceUnitBuild (..), SourceUnitDependency (..), textToOriginPath)
 import Test.Effect (
   expectFatal',
   handleDiag,
@@ -51,10 +47,7 @@ import Test.MockDockerEngineApi (
   runMockApi,
  )
 import Type.Operator (type ($))
-import Types (
-  DiscoveredProjectType (SetuptoolsProjectType),
-  TargetFilter (TypeDirTarget, TypeTarget),
- )
+import Types (DiscoveredProjectType (SetuptoolsProjectType), GraphBreadth (..), TargetFilter (TypeDirTarget, TypeTarget))
 
 spec :: Spec
 spec = do
@@ -194,8 +187,8 @@ jarsInContainerSpec :: Spec
 jarsInContainerSpec = describe "Jars in Containers" $ do
   currDir <- runIO getCurrentDir
   let imageArchivePath = currDir </> jarsInContainerImage
-      baseLayerId = "sha256:9733ccc395133a067f01ee6e380003d80fe9f443673e0f992ae6a4a7860a872c"
-      otherLayerId = "sha256:5c079c30beb013e4b2f7729b6bdce6fba57941d28f20db985333fc1dd969f018"
+      baseLayerId = "sha256:61aed1a8baa251dee118b9ab203c1e420f0eda0a9b3f9322d67d235dd27a12ee"
+      otherLayerId = "sha256:632e84390ad558f9db0524f5e38a0af3e79c623a46bdce8a5e6a1761041b9850"
 
   it' "Reads and merges the layers correctly" $ do
     ContainerScan{imageData = ContainerScanImage{imageLayers}} <- analyzeFromDockerArchive False mempty (toFlag' False) imageArchivePath
@@ -209,5 +202,41 @@ jarsInContainerSpec = describe "Jars in Containers" $ do
     -- The CLI only passes observations along without inspecting them.
     -- So this test just checks that the number of them that we expect are there.
     -- More specific tests for observation content are in Millhone.
-    (length <$> Map.lookup baseLayerId observationsMap) `shouldBe'` Just 0
+    (length <$> Map.lookup baseLayerId observationsMap) `shouldBe'` Just 1
     (length <$> Map.lookup otherLayerId observationsMap) `shouldBe'` Just 2
+
+  it' "Discovers non-system/non-JIC dependencies" $ do
+    ContainerScan{imageData = ContainerScanImage{imageLayers}} <- analyzeFromDockerArchive False mempty (toFlag' False) imageArchivePath
+
+    let srcUnitsMap = Map.fromList $ map (\layer -> (layerId layer, srcUnits layer)) imageLayers
+        depLocator =
+          Locator
+            { locatorFetcher = "npm"
+            , locatorProject = "color-name"
+            , locatorRevision = Just "2.0.0"
+            }
+        expectedSrcUnit =
+          SourceUnit
+            { sourceUnitName = toText $(mkRelDir "./")
+            , sourceUnitType = "npm"
+            , sourceUnitManifest = toText $(mkRelDir "./")
+            , sourceUnitBuild =
+                Just
+                  SourceUnitBuild
+                    { buildArtifact = "default"
+                    , buildSucceeded = True
+                    , buildImports =
+                        [depLocator]
+                    , buildDependencies =
+                        [ SourceUnitDependency
+                            { sourceDepLocator = depLocator
+                            , sourceDepImports = []
+                            }
+                        ]
+                    }
+            , sourceUnitGraphBreadth = Complete
+            , sourceUnitOriginPaths = [textToOriginPath "package-lock.json"]
+            , additionalData = Nothing
+            }
+
+    Map.lookup otherLayerId srcUnitsMap `shouldBe'` Just [expectedSrcUnit]

test/App/Fossa/Container/AnalyzeNativeUploadSpec.hs

@@ -45,7 +45,7 @@ fixtureProjectMetadata :: ProjectMetadata
 fixtureProjectMetadata = ProjectMetadata Nothing Nothing Nothing Nothing Nothing Nothing ["label-1", "label-2"] Nothing
 
 fixtureContainerScan :: ContainerScan
-fixtureContainerScan = ContainerScan (ContainerScanImage "alpine" "3.1.4" []) "some-digest" "some-tag"
+fixtureContainerScan = ContainerScan (ContainerScanImage (Just "alpine") (Just "3.1.4") []) "some-digest" "some-tag"
 
 fixtureRevision :: ProjectRevision
 fixtureRevision = ProjectRevision "some-tag" "some-digest" $ Just "master"

test/App/Fossa/Container/testdata/container_jars.Dockerfile

@@ -1,6 +1,10 @@
 # Use this DockerFile to build an image used in the jars in containers tests.
-FROM alpine:3.14
+FROM scratch
 # https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.17.1/
 COPY jackson-annotations-2.17.1.jar .
 # https://repo1.maven.org/maven2/org/apache/commons/commons-email2-jakarta/2.0.0-M1/
 COPY commons-email2-jakarta-2.0.0-M1.jar ./inner_directory/
+# https://repo1.maven.org/maven2/org/slf4j/slf4j-ext/2.0.0/
+COPY slf4j-ext-2.0.0.jar .
+# This sample project is in the current directory.
+COPY tiny-project .

test/App/Fossa/Container/testdata/jar_test_container.tar

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c7a7c625c39b39900428d0a68a2f2330991a9298d90453968de6d32b4ffba2a4
-size 6047232
+oid sha256:cf00a8861519fd2357380e66e060f9080d5e3ec99e4b8f7feeeaf6b11bbfbb62
+size 197120

test/App/Fossa/Container/testdata/tiny-project/package-lock.json

@@ -0,0 +1,14 @@
+{
+  "name": "tiny-project",
+  "version": "1.0.0",
+  "description": "",
+  "main": "index.js",
+  "scripts": {
+    "test": "echo \"Error: no test specified\" && exit 1"
+  },
+  "author": "",
+  "license": "ISC",
+  "dependencies": {
+    "color-name": "^2.0.0"
+  }
+}