Skip to content

Commit b5f97eb

Browse files
jssblckjssblck
authored
Reachability: custom jars (#1382)
1 parent 6775077 commit b5f97eb

File tree

20 files changed

+359
-84
lines changed

20 files changed

+359
-84
lines changed

‎Changelog.md

+28-25
Original file line numberDiff line numberDiff line change
@@ -1,51 +1,54 @@
11
# FOSSA CLI Changelog
22

3+
## v3.9.8
4+
- Reachability: Users may now provide custom locations for the JAR files emitted by projects and used for reachability analysis ([#1382](https://github.com/fossas/fossa-cli/pull/1382)).
5+
36
## v3.9.7
47
- Add preflight permission checks to validate token type, subscription type, project permissions, and release group permissions [#1383](https://github.com/fossas/fossa-cli/pull/1383)
58

69
## v3.9.6
710
- Add debug logs for build warnings in `analyze` commands [#1386](https://github.com/fossas/fossa-cli/pull/1386)
811

912
## v3.9.5
10-
- Maven: Fix hanging maven analysis [#1381](https://github.com/fossas/fossa-cli/pull/1381)
13+
- Maven: Fix hanging maven analysis ([#1381](https://github.com/fossas/fossa-cli/pull/1381)).
1114

1215
## v3.9.4
13-
- Reachability: Includes reachability analysis in scan summary [#1379](https://github.com/fossas/fossa-cli/pull/1379)
16+
- Reachability: Includes reachability analysis in scan summary ([#1379](https://github.com/fossas/fossa-cli/pull/1379)).
1417

1518
## v3.9.3
16-
- Update error structure [#1364](https://github.com/fossas/fossa-cli/pull/1364)
19+
- Update error structure ([#1364](https://github.com/fossas/fossa-cli/pull/1364)).
1720

1821
## v3.9.2
19-
- Maven: Adds reachability analysis [#1372](https://github.com/fossas/fossa-cli/pull/1377)
20-
- Gradle: Adds reachability analysis [#1377](https://github.com/fossas/fossa-cli/pull/1377)
22+
- Maven: Adds reachability analysis ([#1372](https://github.com/fossas/fossa-cli/pull/1377)).
23+
- Gradle: Adds reachability analysis ([#1377](https://github.com/fossas/fossa-cli/pull/1377)).
2124

2225
## v3.9.1
23-
- `--detect-dynamic`: Safely ignores scenarios in ldd output parsing where we run into not found error. ([#1376](https://github.com/fossas/fossa-cli/pull/1376))
26+
- `--detect-dynamic`: Safely ignores scenarios in ldd output parsing where we run into not found error ([#1376](https://github.com/fossas/fossa-cli/pull/1376)).
2427

2528
## v3.9.0
26-
- Emits a warning instead of an error when no analysis targets are found ([#1375](https://github.com/fossas/fossa-cli/pull/1375))
29+
- Emits a warning instead of an error when no analysis targets are found ([#1375](https://github.com/fossas/fossa-cli/pull/1375)).
2730

2831
## 3.8.37
2932

30-
- Container Scans: Bugfix for some registry scans that fail with an STM error. ([#1370](https://github.com/fossas/fossa-cli/pull/1370))
33+
- Container Scans: Bugfix for some registry scans that fail with an STM error. ([#1370](https://github.com/fossas/fossa-cli/pull/1370)).
3134

3235
## v3.8.36
33-
- `fossa feedback`: Allow users to provide feedback on their cli experience ([#1368](https://github.com/fossas/fossa-cli/pull/1368))
34-
- Add preflight checks to validate API key, connection to FOSSA app, and ability to write to temp directory in relevant commands
36+
- `fossa feedback`: Allow users to provide feedback on their cli experience ([#1368](https://github.com/fossas/fossa-cli/pull/1368)).
37+
- Add preflight checks to validate API key, connection to FOSSA app, and ability to write to temp directory in relevant commands.
3538

3639

3740
## v3.8.35
3841
- Running `fossa analyze --detect-vendored` no longer fails if there are no detected vendored dependencies ([#1373](https://github.com/fossas/fossa-cli/pull/1373)).
3942

4043
## v3.8.34
41-
- Add color and update formatting in cli help commands ([#1367](https://github.com/fossas/fossa-cli/pull/1367))
44+
- Add color and update formatting in cli help commands ([#1367](https://github.com/fossas/fossa-cli/pull/1367)).
4245

4346
## v3.8.33
44-
- Removes warnings and tracebacks to stderr [#1358](https://github.com/fossas/fossa-cli/pull/1358)
47+
- Removes warnings and tracebacks to stderr ([#1358](https://github.com/fossas/fossa-cli/pull/1358)).
4548

4649
## v3.8.32
4750

48-
- Options: Add a `--static-only-analysis` option. ([#1362](https://github.com/fossas/fossa-cli/pull/1362))
51+
- Options: Add a `--static-only-analysis` option ([#1362](https://github.com/fossas/fossa-cli/pull/1362)).
4952

5053
## v3.8.31
5154

@@ -55,33 +58,33 @@
5558

5659
## v3.8.30
5760

58-
- Fix an issue with long-option syntax for older versions of `sbt` ([#1356](https://github.com/fossas/fossa-cli/pull/1356))
59-
- Debug: add more logging for debugging missing dependencies. ([#1360](https://github.com/fossas/fossa-cli/pull/1360))
61+
- Fix an issue with long-option syntax for older versions of `sbt` ([#1356](https://github.com/fossas/fossa-cli/pull/1356)).
62+
- Debug: add more logging for debugging missing dependencies ([#1360](https://github.com/fossas/fossa-cli/pull/1360)).
6063

6164
## v3.8.29
62-
- Prevents showing SCM warnings in fossa analyze, test, and report [#1354](https://github.com/fossas/fossa-cli/pull/1354)
63-
- Pathfinder: Pathfinder has been deprecated and removed. ([#1350](https://github.com/fossas/fossa-cli/pull/1350))
65+
- Prevents showing SCM warnings in fossa analyze, test, and report ([#1354](https://github.com/fossas/fossa-cli/pull/1354)).
66+
- Pathfinder: Pathfinder has been deprecated and removed ([#1350](https://github.com/fossas/fossa-cli/pull/1350)).
6467

6568
## v3.8.28
66-
- VSI: no longer reports paths inside of extracted archives with the `!_fossa.virtual_!` literal [#1345](https://github.com/fossas/fossa-cli/pull/1345)
69+
- VSI: no longer reports paths inside of extracted archives with the `!_fossa.virtual_!` literal ([#1345](https://github.com/fossas/fossa-cli/pull/1345)).
6770

6871
## v3.8.27
69-
- Maven: Fix a bug that broke maven analysis if the build directory was in a non-standard location ([#1343](https://github.com/fossas/fossa-cli/pull/1343))
72+
- Maven: Fix a bug that broke maven analysis if the build directory was in a non-standard location ([#1343](https://github.com/fossas/fossa-cli/pull/1343)).
7073

7174
## v3.8.26
72-
- Maven: add support for maven submodule filtering [#1339](https://github.com/fossas/fossa-cli/pull/1339)
75+
- Maven: add support for maven submodule filtering ([#1339](https://github.com/fossas/fossa-cli/pull/1339)).
7376

7477
## v3.8.25
75-
- Maven: add support for maven scope filtering ([#1331](https://github.com/fossas/fossa-cli/pull/1331))
76-
- `fossa init`: adds new `fossa init` command which creates `.fossa.yml.example`, and `fossa-deps.yml.example` file. ([#1323](https://github.com/fossas/fossa-cli/pull/1323))
78+
- Maven: add support for maven scope filtering ([#1331](https://github.com/fossas/fossa-cli/pull/1331)).
79+
- `fossa init`: adds new `fossa init` command which creates `.fossa.yml.example`, and `fossa-deps.yml.example` file. ([#1323](https://github.com/fossas/fossa-cli/pull/1323)).
7780

7881
## v3.8.24
7982

80-
- Python: use `pip` to determine transitive dependencies for setuptool projects that contain a req*.txt or setup.py file. ([#1334](https://github.com/fossas/fossa-cli/pull/1334))
81-
- Container Scanning: warn and exclude rpm packages that are missing attributes. ([#1335](https://github.com/fossas/fossa-cli/pull/1335))
83+
- Python: use `pip` to determine transitive dependencies for setuptool projects that contain a req*.txt or setup.py file. ([#1334](https://github.com/fossas/fossa-cli/pull/1334)).
84+
- Container Scanning: warn and exclude rpm packages that are missing attributes ([#1335](https://github.com/fossas/fossa-cli/pull/1335)).
8285

8386
## v3.8.23
84-
- Custom License Scans: Support full-file uploads for custom license scans ([#1333](https://github.com/fossas/fossa-cli/pull/1333))
87+
- Custom License Scans: Support full-file uploads for custom license scans ([#1333](https://github.com/fossas/fossa-cli/pull/1333)).
8588

8689
## v3.8.22
8790
- path: adds path dependency scanning functionality. ([#1327](https://github.com/fossas/fossa-cli/pull/1327))

‎docs/features/vuln_reachability.md

+50-7
Original file line numberDiff line numberDiff line change
@@ -2,16 +2,16 @@
22

33
### What is Reachability?
44

5-
Reachability Analysis is a security offering designed to enhance FOSSA's security analysis by providing context on vulnerable packages. It alleviates the constraints of traditional CVE assessments through the static analysis of application and dependency code, confirming the presence of vulnerable call paths.
5+
Reachability Analysis is a security offering designed to enhance FOSSA's security analysis by providing context on vulnerable packages. It alleviates the constraints of traditional CVE assessments through the static analysis of application and dependency code, confirming the presence of vulnerable call paths.
66

77
### Limitations
88

9-
- Reachability currently supports all Maven and Gradle projects dynamically analyzed by FOSSA CLI.
9+
- Reachability currently supports all Maven and Gradle projects dynamically analyzed by FOSSA CLI.
1010
- The target jar of the project must exist, prior to the analysis. If the jar artifact is not present, or FOSSA CLI fails to
1111
associate this jar with project, FOSSA CLI will not perform reachability analysis.
1212
- Reachability requires that `java` is present in PATH, and `java` version must be greater than `1.8` (jdk8+).
1313

14-
For example,
14+
For example,
1515
- if you are using maven, you should run `mvn package` to ensure jar artifact exists, prior to running `fossa analyze`
1616
- if you are using gradle, you should run `gradlew build` to ensure jar artifact exists, prior to running `fossa analyze`
1717

@@ -23,6 +23,50 @@ For Maven projects, FOSSA CLI performs an analysis to infer dependencies. If FOS
2323

2424
For Gradle projects, FOSSA CLI invokes `./gradlew -I jsonpaths.gradle jsonPaths`. Where [jsonpaths.gradle](./../../scripts/jarpaths.gradle) is gradle script, which uses `java` plugin, and `jar` task associated with gradle to infer path of the built jar file. If neither of those are present, FOSSA CLI won't be able to identify jar artifacts for analysis.
2525

26+
### Custom JAR locations
27+
28+
For both Gradle and Maven, it is possible to choose different locations for output JAR files, or to move them before FOSSA CLI runs.
29+
For these cases, you can configure the locations directly.
30+
31+
First, run `fossa list-targets` inside your project. For example:
32+
```sh
33+
❯ fossa list-targets
34+
Found project: gradle@./
35+
Found target: gradle@./::app
36+
```
37+
38+
This output specifies that a Gradle project was found at the root of the current directory (at `./`).
39+
Note for which project(s) you wish to provide the paths to the built JAR files, we'll need these later.
40+
41+
Now, create a `.fossa.yml` file if one does not already exist ([more information here](../references/files/fossa-yml.md)).
42+
Add the following keys to it:
43+
```yml
44+
reachability:
45+
jvmOutputs:
46+
```
47+
48+
Inside the `jvmOutputs` object, create a map that describes, for each project path (noted down earlier) the paths to the JAR files that the project output.
49+
50+
For example, let's say that the Gradle project at the root of my scan directory (at `./`) produces a JAR file named `app.jar`, and that I configured Gradle such that when it builds the project that `app.jar` file is also written to the root of my scan directory (so its path is `./app.jar`, relative to the scan root).
51+
My config file would look like this:
52+
53+
```yml
54+
reachability:
55+
jvmOutputs:
56+
'./':
57+
- './app.jar'
58+
```
59+
60+
You can also provide the absolute paths to make this more clear, if desired. For example, if my project was at `~/projects/example-project` and I wanted to use absolute paths in the config file, the file would look like this:
61+
```yml
62+
reachability:
63+
jvmOutputs:
64+
'/Users/me/projects/example-project':
65+
- '/Users/me/projects/example-project/app.jar'
66+
```
67+
68+
With this configuration, when FOSSA CLI analyzes a Maven or Gradle project at the specified path (in this case `/Users/me/projects/example-project`) instead of attempting to query the build tool for the location of the built JAR files it will instead use the values provided for that path (in this case `/Users/me/projects/example-project/app.jar`).
69+
2670
### How do I debug reachability from `fossa-cli`?
2771

2872
```bash
@@ -75,12 +119,12 @@ cat fossa.debug.json | jq '.bundleReachabilityEndpoint'
75119
}
76120
```
77121

78-
FOSSA CLI uses [jar-callgraph-1.0.0.jar](../../scripts/jar-callgraph-1.0.0.jar) to infer call path edges.
122+
FOSSA CLI uses [jar-callgraph-1.0.0.jar](../../scripts/jar-callgraph-1.0.0.jar) to infer call path edges.
79123
FOSSA CLI uses `java -jar jar-callgraph-1.0.0.jar ./path/to/your/build.jar` command to record edges from
80124
the your target jar. If you are running into issues with reachability, please confirm that you can execute
81125
`java -jar jar-callgraph-1.0.0.jar ./path/to/your/build.jar` on your environment.
82126

83-
<!--
127+
<!--
84128
## How do I debug reachability from endpoint?
85129

86130
```bash
@@ -109,7 +153,7 @@ Likewise, you can also inspect analysis done in datadog, by looking at logs asso
109153
performs reachability analysis as part of provided build (all variants of provided builds).
110154
-->
111155

112-
## F.A.Q.
156+
## F.A.Q.
113157

114158
1. What data from my codebase is uploaded to endpoint?
115159

@@ -174,4 +218,3 @@ Reachability analysis
174218
| skipped (partial graph) | Project has partial dependency graph (e.g. missing transitive dependencies) |
175219
| skipped (not supported) | Project is not supported for reachability analysis |
176220
| skipped (no dependency analysis) | Project's dependencies were not analyzed, so reachability cannot be computed |
177-

‎docs/references/files/fossa-yml.v3.schema.json

+18-1
Original file line numberDiff line numberDiff line change
@@ -397,6 +397,23 @@
397397
"$ref": "#/$defs/grepDefinition"
398398
}
399399
},
400+
"reachability": {
401+
"type": "object",
402+
"description": "Controls the Reachability computation functionality",
403+
"properties": {
404+
"jvmOutputs": {
405+
"type": "object",
406+
"description": "Manually specify the list of JAR files output by each discovered Maven or Gradle project",
407+
"additionalProperties": {
408+
"type": "array",
409+
"items": {
410+
"type": "string",
411+
"description": "The path to a JAR file output by the project"
412+
}
413+
}
414+
}
415+
}
416+
},
400417
"ignoreOrgWideCustomLicenseScanConfigs": {
401418
"type": "boolean",
402419
"default": false,
@@ -477,4 +494,4 @@
477494
"required": [
478495
"version"
479496
]
480-
}
497+
}

‎integration-test/Analysis/FixtureUtils.hs

+5-5
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ module Analysis.FixtureUtils (
1010
TestC,
1111
performDiscoveryAndAnalyses,
1212
getArtifact,
13-
testRunnerWithLogger,
13+
testRunner,
1414
withResult,
1515
) where
1616

@@ -122,8 +122,8 @@ type TestC m =
122122
$ StackC
123123
$ IgnoreTelemetryC m
124124

125-
testRunnerWithLogger :: TestC IO a -> FixtureEnvironment -> IO (Result a)
126-
testRunnerWithLogger f env =
125+
testRunner :: TestC IO a -> FixtureEnvironment -> IO (Result a)
126+
testRunner f env =
127127
f
128128
& runExecIOWithinEnv env
129129
& runReadFSIO
@@ -160,10 +160,10 @@ performDiscoveryAndAnalyses targetDir AnalysisTestFixture{..} = do
160160
_ <- sendIO $ runCmd environment buildCmd
161161

162162
-- Perform discovery
163-
discoveryResult <- sendIO $ testRunnerWithLogger (discover targetDir) environment
163+
discoveryResult <- sendIO $ testRunner (discover targetDir) environment
164164
withResult discoveryResult $ \_ dps ->
165165
for dps $ \dp -> do
166-
analysisResult <- sendIO $ testRunnerWithLogger (ignoreDebug $ analyzeProject (projectBuildTargets dp) (projectData dp)) environment
166+
analysisResult <- sendIO $ testRunner (ignoreDebug $ analyzeProject (projectBuildTargets dp) (projectData dp)) environment
167167
withResult analysisResult $ \_ dr -> pure (dp, dr)
168168
where
169169
runCmd :: FixtureEnvironment -> Maybe (Command) -> IO ()

‎integration-test/Reachability/UploadSpec.hs

+34-4
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33

44
module Reachability.UploadSpec (spec) where
55

6-
import Analysis.FixtureUtils (FixtureEnvironment (..), TestC, testRunnerWithLogger, withResult)
6+
import Analysis.FixtureUtils (FixtureEnvironment (..), TestC, testRunner, withResult)
77
import App.Fossa.Analyze.Project (ProjectResult (..))
88
import App.Fossa.Analyze.Types (
99
DiscoveredProjectIdentifier (..),
@@ -15,15 +15,18 @@ import App.Fossa.Reachability.Types (
1515
CallGraphAnalysis (..),
1616
ContentRef (..),
1717
ParsedJar (..),
18+
ReachabilityConfig (..),
1819
SourceUnitReachability (..),
1920
)
2021
import App.Fossa.Reachability.Upload (
2122
analyzeForReachability,
2223
callGraphOf,
2324
onlyFoundUnits,
2425
)
26+
import Control.Carrier.Reader (ReaderC, runReader)
2527
import Data.ByteString.Lazy qualified as LB
2628
import Data.Foldable (for_)
29+
import Data.Map qualified as Map
2730
import Data.String.Conversion (toText)
2831
import Data.Text (Text)
2932
import Data.Text.Encoding qualified as TL
@@ -42,6 +45,7 @@ import Path (
4245
import Path.IO qualified as PIO
4346
import Test.Hspec (Spec, describe, it, runIO, shouldBe)
4447
import Text.RawString.QQ (r)
48+
import Type.Operator (type ($))
4549
import Types (
4650
DiscoveredProjectType (MavenProjectType),
4751
GraphBreadth (..),
@@ -60,7 +64,10 @@ java21 :: FixtureEnvironment
6064
java21 = NixEnv ["jdk21"]
6165

6266
run :: FixtureEnvironment -> TestC IO a -> IO (Result a)
63-
run env act = testRunnerWithLogger act env
67+
run env act = testRunner act env
68+
69+
runConf :: FixtureEnvironment -> ReachabilityConfig -> (ReaderC ReachabilityConfig $ TestC IO) a -> IO (Result a)
70+
runConf env conf act = testRunner (runReader conf act) env
6471

6572
spec :: Spec
6673
spec = describe "Reachability" $ do
@@ -86,7 +93,7 @@ spec = describe "Reachability" $ do
8693
it "should retrieve call graph" $ do
8794
let (dpi, dps) = mavenCompleteScan projDir
8895
let expected = SourceUnitReachabilityFound dpi (Success [] (mavenCompleteScanUnit projDir jarFile))
89-
resp <- run java8 $ callGraphOf dps
96+
resp <- runConf java8 mempty $ callGraphOf dps
9097
withResult resp $ \_ res -> res `shouldBe` expected
9198

9299
describe "analyzeForReachability" $ do
@@ -96,12 +103,32 @@ spec = describe "Reachability" $ do
96103
it "should return analyzed reachability unit" $ do
97104
let (_, dps) = mavenCompleteScan projDir
98105
let expected = [mavenCompleteScanUnit projDir jarFile]
99-
analyzed <- run java8 $ analyzeForReachability [dps]
106+
analyzed <- runConf java8 mempty $ analyzeForReachability [dps]
107+
withResult analyzed $ \_ analyzed' -> (onlyFoundUnits analyzed') `shouldBe` expected
108+
109+
describe "user provided jar" $ do
110+
projDir <- (</> sampleMavenProjectMissingOutputJarDir) <$> runIO PIO.getCurrentDir
111+
jarFile <- (</> sampleMavenProjectJar) <$> runIO PIO.getCurrentDir
112+
113+
it "should fail to analyze reachability when jar is missing" $ do
114+
let (_, dps) = mavenCompleteScan projDir
115+
let expected = [mavenEmptyScanUnit projDir]
116+
analyzed <- runConf java8 mempty $ analyzeForReachability [dps]
117+
withResult analyzed $ \_ analyzed' -> (onlyFoundUnits analyzed') `shouldBe` expected
118+
119+
it "should succeed analyzing the project when a missing jar is manually specified" $ do
120+
let conf = ReachabilityConfig $ Map.fromList [(projDir, [jarFile])]
121+
let (_, dps) = mavenCompleteScan projDir
122+
let expected = [mavenCompleteScanUnit projDir jarFile]
123+
analyzed <- runConf java8 conf $ analyzeForReachability [dps]
100124
withResult analyzed $ \_ analyzed' -> (onlyFoundUnits analyzed') `shouldBe` expected
101125

102126
sampleMavenProjectDir :: Path Rel Dir
103127
sampleMavenProjectDir = $(mkRelDir "test/Reachability/testdata/maven-default/")
104128

129+
sampleMavenProjectMissingOutputJarDir :: Path Rel Dir
130+
sampleMavenProjectMissingOutputJarDir = $(mkRelDir "test/Reachability/testdata/maven-default-missing-jar/")
131+
105132
sampleMavenProjectJar :: Path Rel File
106133
sampleMavenProjectJar = $(mkRelFile "test/Reachability/testdata/maven-default/target/project-1.0.0.jar")
107134

@@ -117,6 +144,9 @@ mavenCompleteScanUnit projDir jarFile =
117144
(ContentRaw sampleJarParsedContent')
118145
]
119146

147+
mavenEmptyScanUnit :: Path Abs Dir -> SourceUnitReachability
148+
mavenEmptyScanUnit projDir = mkReachabilityUnit projDir []
149+
120150
mkDiscoveredProjectScan :: DiscoveredProjectType -> Path Abs Dir -> GraphBreadth -> (DiscoveredProjectIdentifier, DiscoveredProjectScan)
121151
mkDiscoveredProjectScan projectType dir breadth =
122152
( dpi

‎src/App/Fossa/Analyze.hs

+1-1
Original file line numberDiff line numberDiff line change
@@ -400,7 +400,7 @@ analyze cfg = Diag.context "fossa-analyze" $ do
400400
(False, _) -> traverse (withPathDependencyNudge includeAll) filteredProjects
401401
logDebug $ "Filtered projects with path dependencies: " <> pretty (show filteredProjects')
402402

403-
reachabilityUnitsResult <- analyzeForReachability projectScans
403+
reachabilityUnitsResult <- Diag.context "reachability analysis" . runReader (Config.reachabilityConfig cfg) $ analyzeForReachability projectScans
404404
let reachabilityUnits = onlyFoundUnits reachabilityUnitsResult
405405

406406
let analysisResult = AnalysisScanResult projectScans vsiResults binarySearchResults manualSrcUnits dynamicLinkedResults maybeLernieResults reachabilityUnitsResult

0 commit comments

Comments
 (0)