Merge pull request #85 from secure-software-engineering/develop

Restructure, bump and improve project modules

Author: oshando

README.md

@@ -1,35 +1,35 @@
 # SWAN (Security methods for WeAkNess detection)
 
-What is SWAN?
--------------
-SWAN is a machine-learning approach used to detect of security-relevant methods (SRM) in Java programs.
+SWAN is a machine-learning approach that detects security-relevant methods (SRM) in Java programs.
 SWAN should be used in combination with other static analyses tools and it helps the users to create a set of relevant methods required as an input for static analyses, e.g. taint- and type-state analysis.
 The tool currently detects four types of security relevant methods, namely: source, sink, sanitizer, and authentication methods.
-The detected methods are further categorized according to relevant vulnerabilities from the [Common Weakness Enumeration (CWE)](https://cwe.mitre.org/). The following CWEs are currently supported: [CWE78 OS Command Injection](https://cwe.mitre.org/data/definitions/78.html), [CWE79 Cross-site Scripting](https://cwe.mitre.org/data/definitions/79.html), [CWE89 SQL Injection](https://cwe.mitre.org/data/definitions/89.html), [CWE306 Missing Authentication](https://cwe.mitre.org/data/definitions/306.html), [CWE601 Open Redirect](https://cwe.mitre.org/data/definitions/601.html), [CWE862 Missing Authorisation](https://cwe.mitre.org/data/definitions/862.html), and
+SWAN also labels methods as relevant for 7 [Common Weakness Enumeration (CWE)](https://cwe.mitre.org/), namely: [CWE78 OS Command Injection](https://cwe.mitre.org/data/definitions/78.html), [CWE79 Cross-site Scripting](https://cwe.mitre.org/data/definitions/79.html), [CWE89 SQL Injection](https://cwe.mitre.org/data/definitions/89.html), [CWE306 Missing Authentication](https://cwe.mitre.org/data/definitions/306.html), [CWE601 Open Redirect](https://cwe.mitre.org/data/definitions/601.html), [CWE862 Missing Authorisation](https://cwe.mitre.org/data/definitions/862.html), and
 [CWE863 Incorrect Authorisation](https://cwe.mitre.org/data/definitions/863.html).
 
-The project contains the following modules:
-* **swan-pipeline**: core machine learning implementation for SWAN with components for data collection and preparation, feature engineering and model selection phases
-* **swan-assist**: IntelliJ plugin provides GUI support for SWAN and enables active machine learning.
-* **swan-javadoc-exporter**: Doclet exports doc comments to XML files so that they can be analyzed by the Natural Language Processing (NLP) module
-* **doc-coverage-doclet**: Doclet calculates the software documentation coverage of Java programs based on the presence of doc comments for classes, methods, and other objects.
+The project is divided into two main components: the command line tool [<code>**swan-cmd**</code>](https://github.com/secure-software-engineering/swan/tree/master/swan-cmd) and the IntelliJ plugin [<code>**dev-assist**</code>](https://github.com/secure-software-engineering/swan/tree/master/dev-assist) that provides a GUI for SWAN. 
+
+1. <code>**swan-cmd**</code> is the command line implementation for SWAN with components for data collection, feature engineering, model selection and SRM prediction. The command line tool uses the following Maven modules:
+    - <code>**training-data-jars**</code> contains dependencies from which the training examples are extracted.
+    - Java Doclets to process and export software documentation
+      - <code>**coverage-doclet**</code> calculates the software documentation coverage of Java programs based on the presence of doc comments for classes, methods, and other objects.
+      - <code>**xml-doclet**</code> exports doc comments to XML files so that they can be analyzed by the Natural Language Processing (NLP) module
+2. <code>**dev-assist**</code> provides GUI support for SWAN and enables active machine learning.
+
 
 How do I get started with SWAN?
 -------------
-The easiest way to get started with SWAN is to use the pre-built binary from the newest release. To run SWAN, we provide a path to the Java project to be analyzed (JAR files or compiled classes) as well an output directory where SWAN will export its results.
-
-After downloading the necessary files from the most recent release, SWAN can be executed on the command line with the following command:
+To run SWAN, you will need to provide a path to the Java project to be analyzed (JAR files or compiled classes) as well an output directory where SWAN will export its results. The easiest way to get started with SWAN is to use the pre-built binary from the newest release. After downloading the necessary files from the most recent release, SWAN can be executed on the command line with the following command:
 
-<code>java -jar swan-&lt;swan-version&gt;.jar-test <project-path> -output <output-directory></code>
+<code>**java -jar swan-cmd-3.x.x.jar -test** */path/to/project/files* **-o** */output/directory* </code>
 
-This command runs the application and exports the detected security-relevant methods to a JSON file in the provided output directory. The available command line options can be found in the Wiki or by using the <code>-help</code> command line option.
+This command runs the application and exports the detected security-relevant methods to a JSON file in the provided output directory. This command uses the following default settings: training dataset <code>-in [dataset](/swan-cmd/src/main/resources/dataset)</code>, code features <code>-f code</code>, and the MEKA toolkit <code>-t meka</code>. The remaining default options are found in [CLIRunner](/swan-cmd/src/main/java/de/fraunhofer/iem/swan/cli/CliRunner.java). The available command line options can be found in the Wiki or by using the <code>-help</code> command line option.
 
 How do I build SWAN?
 -------------
-If you downloaded SWAN as a compressed release (e.g. .zip or .tar.gz), you can use <code>mvn package</code> to package the project. The commands provided above can then be used to run the generated JAR file. Alternatively, you can import the project directly into your IDE from the repository and package the project via the terminal or the Maven plugin in your IDE.
+If you cloned the project or downloaded SWAN as a compressed release (e.g. .zip or .tar.gz), you can use <code>mvn package</code> to package the project. The commands provided above can then be used to run the generated JAR file. Alternatively, you can import the project directly into your IDE from the repository and package the project via the terminal or the Maven plugin in your IDE.
 
 
 Contributors
 -------------
-The following persons have contributed to SWAN: Goran Piskachev ([email protected].de), Lisa Nguyen ([email protected]), Oshando Johnson ([email protected]), Eric Bodden ([email protected])
+The following persons have contributed to SWAN: Goran Piskachev (gpiskach@amazon.de), Lisa Nguyen ([email protected]), Oshando Johnson (oshando.johnson@iem.fraunhofer.de), Eric Bodden ([email protected]).
 

doc-coverage-doclet/pom.xml renamed to coverage-doclet/pom.xml

@@ -4,9 +4,13 @@
          xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
     <modelVersion>4.0.0</modelVersion>
 
-    <groupId>de.fraunhofer.iem</groupId>
+    <parent>
+        <groupId>de.fraunhofer.iem</groupId>
+        <artifactId>swan</artifactId>
+        <version>3.3.1</version>
+    </parent>
+
     <artifactId>coverage-doclet</artifactId>
-    <version>1.1</version>
 
     <properties>
         <maven.compiler.source>11</maven.compiler.source>

doc-coverage-doclet/src/main/java/de/fraunhofer/iem/coveragedoclet/CoverageDoclet.java renamed to coverage-doclet/src/main/java/de/fraunhofer/iem/coveragedoclet/CoverageDoclet.java

@@ -1,60 +1,32 @@
-# dev-assist
-IntelliJ IDEA Plug-in for the Security methods for WeAkNess detection (SWAN) tool.
-
-Description: 
--------------
-SWAN_Assist provides a GUI support for SWAN. The user is able to interact with the learning process by giving feedback on the methods of interest. 
-The tool helps users that write static analyses to create list of SWAN for their specific Java libraries. 
-Moreover, users can manually inspect the proper usage of the methods detected by SWAN. 
-
-
-## Downloading the Project
-
-The project can be downloaded using either of the following methods:
-
-##### Method 1: Cloning the Project 
-1) Select the **File>Project from Version Control>Git** option, enter the repository’s URL and then select **Clone** to import the project. The project will contain the following directories: ``swan_core`` (SWAN core application), ``swan_assist`` (IntelliJ Plugin) and ``swan_datasets`` (datasets for the research paper).
-2) To configure the project settings and modules, go to **File>Project Structure**. 
-3) For **Project SDK**, select the corresponding Java SDK version.
-4) Select **Modules** from the left panel and remove the existing module that was automatically created.
-5) Click the **Add** button and then **Import Module** to create the SWAN Core module. Follow the steps in the [Setting up the Project Modules](https://github.com/secure-software-engineering/swan/tree/master/swan_assist#setting-up-the-project-modules) section to finish configuring the core module as well as the plugin module.
-
-		
-##### Method 2: Downloading Project ZIP
-1) Download and extract the project resources from GitHub.
-2) In Intellij, use the **File>Project from Existing Resources** to import the project modules. This can also be done from the IntelliJ start screen.  
-3) Follow the steps in the [Setting up the Project Modules](https://github.com/secure-software-engineering/swan/tree/master/swan_assist#setting-up-the-project-modules) section to finish configuring the core module as well as the plugin module.
-
-## Setting up the Project Modules
-
-##### SWAN Core
-1) In the window that appears, open the ``/swan_core`` directory of the project. 
-2) Select the **Import module from external Model** radio button and also select **Maven**. 
-3) The default settings in the dialogs that appear can be used. 
-4) Close the **Project Settings** dialog so that IntelliJ will index the new project module.
-
-##### SWAN Assist
-1) Return to the **Project Structure** dialog and Select **Modules** from the left panel.
-2) Click the **Add** button and then **Import Module**. 
-3) In the window that appears, open the ``/swan_assist`` directory. Select the **Import module from external Model** radio button and also select **Gradle**. 
-4) The default settings in the dialogs that appear can be used. The plugin module should now be indexed. 
-
-The core and plugin modules should now be imported. 
-
-## Running the Plugin
-
-The plugin uses ``swan_core`` dependency from [Maven Central](https://mvnrepository.com/artifact/de.upb.cs.swt/swan_core). If the version in the plugin's ``build.gradle`` file is not available on Maven Central, perform the following steps:
-1) Run the Maven ``install`` command of the ``swan_core`` project from the console or using the Maven Plugin.
-2) Add ``mavenLocal()`` in the ``repositories`` section of the ``build.gradle`` file. The locally installed library can now be use by the plugin.
-
-##### To run the plugin:
-
-1) Select the **Run Configuration** drop down menu and select **Edit Configurations** or from the **Run** menu, select **Edit Configurations**. 
-2) Click the **Add** button and select **Gradle**. 
-3) Select the **swan_assist** Gradle module that was just created and enter ``:runIde`` as the value for **Tasks** - this task will run the plugin in a new instance of IntelliJ. The plugin can also be executed using the Gradle Plugin in IntelliJ: Open the Gradle Tool Window, expand the ``intellij`` task and double click on ``runIde``. The other tasks can be used as necessary.
-4) When the new instance of IntelliJ launches, use the open option to select the project found in ``/test-project`` directory. You may need to set a project SDK, if one isn’t automatically configured for the project. 
-
-Logs for the plugin will appear in the initial instance of IntelliJ.
-
+## Dev-Assist IntelliJ Plugin
+
+Dev-Assist is an IntelliJ IDEA plugin that uses [SWAN](https://github.com/secure-software-engineering/swan) to detect security-relevant methods (SRMs) that are required when configuring static analysis tools. After automatically detecting SRMs, the plugin can be used to adjust the list of security-relevant methods and also generate the tainit-flow specifications required to run the taint analysis tool [SecuCheck](https://github.com/secure-software-engineering/secucheck/).
+
+The plugin works with IntelliJ IDEA 2022.2 and higher. 
+
+### Plugin Features
+The plugin has the following main features which are accessible in the plugin's tool window and from the editor:
+
+- Detect security-relevant methods in Java programs with SWAN's machine learning approach
+- Update security-relevant methods list using method dialog
+  - Import existing SRM list
+  - Add new SRMs from the editor
+  - Update existing method (SRM labels, data-in/data-out and meta properties)
+  - Delete SRMs
+  - Filter SRM list
+  - Expand/collapse method list
+  - Export updated SRM list
+- Generate [*fluent*TQL](https://github.com/secure-software-engineering/secucheck/tree/master) taint-flow specifications necessary to configure SecuCheck in order to detect vulnerabilities
+- Run SecuCheck and displays results using Qodana 
+
+### Installation
+
+To install the plugin in IntelliJ IDEA:
+- Download the [latest](https://github.com/secure-software-engineering/swan/releases) plugin archive file (ZIP or JAR) 
+- Open the IDE settings and select <kbd>Plugins</kbd>
+- On the <kbd>Plugins</kbd> page, click <kbd>Gear</kbd> icon and then click <kbd>Install plugin from disk...</kbd>.
+- Select the Dev-Assist plugin archive file and select <kbd>OK</kbd>
+- Click <kbd>OK</kbd> to apply the changes.
+- Restart the IDE to complete the installation
 
 

doc-coverage-doclet/src/main/java/de/fraunhofer/iem/coveragedoclet/CoverageReport.java renamed to coverage-doclet/src/main/java/de/fraunhofer/iem/coveragedoclet/CoverageReport.java

@@ -13,7 +13,8 @@ repositories {
 
 dependencies {
     implementation 'com.googlecode.json-simple:json-simple:1.1'
-    implementation 'de.fraunhofer.iem:swan:3.2.2'
+    implementation 'de.fraunhofer.iem:swan:3.3.1'
+    implementation 'de.fraunhofer.iem:swan-cmd:3.3.1'
     implementation 'org.slf4j:slf4j-api:1.7.29'
     implementation 'org.slf4j:slf4j-simple:1.7.29'
     implementation 'org.apache.commons:commons-lang3:3.0'
@@ -32,7 +33,7 @@ patchPluginXml {
             "<br>- filter list" +
             "<br>- import and export configuration file"
     sinceBuild = "221.*"
-    untilBuild = "233.*"
+    untilBuild = "242.*"
 }
 
 runIde {

doc-coverage-doclet/src/main/java/de/fraunhofer/iem/coveragedoclet/Main.java renamed to coverage-doclet/src/main/java/de/fraunhofer/iem/coveragedoclet/Main.java

@@ -0,0 +1,45 @@
+package de.fraunhofer.iem.devassist.actions;
+
+import com.intellij.ide.util.PropertiesComponent;
+import com.intellij.openapi.actionSystem.AnAction;
+import com.intellij.openapi.actionSystem.AnActionEvent;
+import com.intellij.openapi.actionSystem.CommonDataKeys;
+import com.intellij.openapi.progress.PerformInBackgroundOption;
+import com.intellij.openapi.progress.ProgressManager;
+import com.intellij.openapi.project.Project;
+import de.fraunhofer.iem.devassist.analysis.SecucheckBackgroundTask;
+import de.fraunhofer.iem.devassist.comm.SecucheckNotifier;
+import de.fraunhofer.iem.devassist.util.Constants;
+import org.jetbrains.annotations.NotNull;
+
+import java.io.File;
+import java.util.Objects;
+
+public class RunSecucheckAction extends AnAction {
+
+    @Override
+    public void actionPerformed(@NotNull AnActionEvent e) {
+        final Project project = e.getRequiredData(CommonDataKeys.PROJECT);
+
+        if (!PropertiesComponent.getInstance(project).isTrueValue(Constants.SWAN_SETTINGS)) {
+            e.getActionManager().getAction("Dev_Assist.SettingsAction").actionPerformed(e);
+            //TODO Run SWAN if the tool has been configured
+        } else {
+            runSecucheck(project);
+        }
+    }
+
+    public void runSecucheck(Project project) {
+
+        File outputFolder = new File(Objects.requireNonNull(PropertiesComponent.getInstance(project).getValue(Constants.OUTPUT_DIRECTORY)));
+
+        if (!outputFolder.exists())
+            outputFolder.mkdir();
+
+        ProgressManager.getInstance().run(new SecucheckBackgroundTask(project, "Running Taint Analysis", true,
+                PerformInBackgroundOption.ALWAYS_BACKGROUND));
+
+        SecucheckNotifier publisher = project.getMessageBus().syncPublisher(SecucheckNotifier.START_SECUCHECK_PROCESS_TOPIC);
+        publisher.launchSecuCheck();
+    }
+}

dev-assist/README.md

@@ -64,7 +64,7 @@ public void actionPerformed(AnActionEvent anActionEvent) {
         }
 
         if (!PropertiesComponent.getInstance(project).isTrueValue(Constants.SWAN_SETTINGS)) {
-            anActionEvent.getActionManager().getAction("SWAN_Assist.SettingsAction").actionPerformed(anActionEvent);
+            anActionEvent.getActionManager().getAction("Dev_Assist.SettingsAction").actionPerformed(anActionEvent);
             //TODO Run SWAN if the tool has been configured
         } else {
             runSwan(project);

dev-assist/build.gradle

@@ -64,7 +64,6 @@ public void actionPerformed(AnActionEvent anActionEvent) {
 
             HashMap<String, String> settings = dialog.getParameters();
 
-            System.out.println("PPP: "+settings);
             for (String property : settings.keySet())
                 PropertiesComponent.getInstance(project).setValue(property, settings.get(property));
         }

dev-assist/src/main/java/de/fraunhofer/iem/devassist/actions/RunSecucheckAction.java

@@ -1,18 +1,24 @@
 package de.fraunhofer.iem.devassist.analysis;
 
+import com.intellij.ide.util.PropertiesComponent;
 import com.intellij.openapi.progress.PerformInBackgroundOption;
 import com.intellij.openapi.progress.ProgressIndicator;
 import com.intellij.openapi.progress.Task;
 import com.intellij.openapi.project.Project;
 import com.intellij.openapi.util.NlsContexts;
 import com.intellij.util.messages.MessageBus;
+import de.fraunhofer.iem.aidevassist.sa.cli.AIDevAssistCli;
+import de.fraunhofer.iem.aidevassist.sa.cli.CliRunner;
 import de.fraunhofer.iem.devassist.comm.SecucheckNotifier;
+import de.fraunhofer.iem.devassist.util.Constants;
 import org.jetbrains.annotations.NotNull;
 import org.jetbrains.annotations.Nullable;
+import picocli.CommandLine;
 
 import java.time.LocalDateTime;
 import java.time.format.DateTimeFormatter;
 import java.util.HashMap;
+import java.util.Objects;
 import java.util.concurrent.TimeUnit;
 
 public class SecucheckBackgroundTask extends Task.Backgroundable {
@@ -31,24 +37,25 @@ public void run(@NotNull ProgressIndicator indicator) {
 
         long start = System.currentTimeMillis();
 
-        //TODO add implementation to run SecuCheck
-        indicator.setText("Generating fluentTQL Specifications");
-
         indicator.setText("Configuring SecuCheck");
 
-        try {
-            TimeUnit.SECONDS.sleep(2);
-        } catch (InterruptedException e) {
-            throw new RuntimeException(e);
-        }
-        indicator.setText("Configuring analysis");
+        String[] args = new String[]{
+                "--analysis", "0",
+                Objects.requireNonNull(PropertiesComponent.getInstance(project).getValue(Constants.SOURCE_DIRECTORY)),
+                "--output", Objects.requireNonNull(PropertiesComponent.getInstance(project).getValue(Constants.OUTPUT_DIRECTORY)),
+                "--inclusion", "*",
+                "--entry", "*"};
+
+        CliRunner cliRunner = new CliRunner(args);
+        CommandLine.ParseResult cmd = new CommandLine(cliRunner).parseArgs(args);
+
+        indicator.setText("Running analysis");
 
-        try {
-            TimeUnit.SECONDS.sleep(2);
-        } catch (InterruptedException e) {
-            throw new RuntimeException(e);
+        if (cmd.errors().isEmpty()) {
+            AIDevAssistCli cli = new AIDevAssistCli();
+            cli.run(cliRunner.createOptions());
+            PropertiesComponent.getInstance(project).setValue(Constants.LAST_SARIF_FILE, cli.getResults().getResultFile());
         }
-        indicator.setText("Exporting analysis results");
 
         MessageBus messageBus = project.getMessageBus();
         SecucheckNotifier publisher = messageBus.syncPublisher(SecucheckNotifier.END_SECUCHECK_PROCESS_TOPIC);