fixes in main file, pom files, updated readme and a new guide on how to compile newly added code

Author: zkaoudi

README.md

@@ -184,6 +184,7 @@ You can see examples on how to start using Wayang [here](guides/wayang-examples.
 ## Contributing
 Before submitting a PR, please take a look on how to contribute with Apache Wayang contributing guidelines [here](CONTRIBUTING.md).
 
+There is also a guide on how to compile your code [here](guides/develop-in-Wayang.md).
 ## Authors
 The list of [contributors](https://github.com/apache/incubator-wayang/graphs/contributors).
 

guides/develop-in-Wayang.md

@@ -0,0 +1,42 @@
+<!--
+
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+
+-->
+This tutorial shows users how to compile their code within Wayang using **maven**.
+
+## Compile the module you modified
+Within the root directory of Wayang, compile only the module you modified for faster compilation:
+```shell
+./mvnw clean install -DskipTests -pl <modified_module>
+```
+Important: before making a Pull Request make sure all modules compile and all tests are passing: 
+```shell
+./mvnw clean install
+```
+##Package the project
+```shell
+./mvnw clean package -pl :wayang-assembly -Pdistribution
+```
+
+## Execute your code
+Before executing your code, make sure the required environment variables are set correctly (see [tutorial.md](tutorial.md))
+```shell
+cd wayang-assembly/target/
+tar -xvf apache-wayang-assembly-0.7.1-SNAPSHOT-incubating-dist.tar.gz
+cd wayang-0.7.1-SNAPSHOT
+./bin/wayang-submit org.apache.wayang.<main_class> <parameters>
+```

guides/develop-with-Wayang.md

@@ -23,27 +23,27 @@ This tutorial shows users how to import Wayang in their Java project using the m
         <dependency>
             <groupId>org.apache.wayang</groupId>
             <artifactId>wayang-core</artifactId>
-            <version>0.6.0</version>
+            <version>0.7.1</version>
         </dependency>
         <dependency>
             <groupId>org.apache.wayang</groupId>
             <artifactId>wayang-basic</artifactId>
-            <version>0.6.0</version>
+            <version>0.7.1</version>
         </dependency>
         <dependency>
             <groupId>org.apache.wayang</groupId>
             <artifactId>wayang-java</artifactId>
-            <version>0.6.0</version>
+            <version>0.7.1</version>
         </dependency>
         <dependency>
             <groupId>org.apache.wayang</groupId>
             <artifactId>wayang-spark_2.12</artifactId>
-            <version>0.6.0</version>
+            <version>0.7.1</version>
         </dependency>
         <dependency>
             <groupId>org.apache.wayang</groupId>
             <artifactId>wayang-api-scala-java_2.12</artifactId>
-            <version>0.6.0</version>
+            <version>0.7.1</version>
         </dependency> 
 ```
 

pom.xml

@@ -391,7 +391,7 @@
 
             - Prevents thrid-party snapshot dependencies in projects
         -->
-        <!-- TODO: This is actually automatically done by the maven-release-plugin:prepare goal Therefore if could be removed -->
+        <!-- TODO: This is actually automatically done by the maven-release-plugin:prepare goal Therefore it could be removed -->
         <profile>
             <id>pre-release</id>
             <build>
@@ -1208,8 +1208,6 @@
                     </execution>
                 </executions>
                 <configuration>
-                    <!-- Right now this would fail the build as not all files have Apache headers -->
-                    <!-- TODO: Enable asap -->
                     <useMavenDefaultExcludes>true</useMavenDefaultExcludes>
                     <!--
                        Make rat output the files with missing licensed directly into the

wayang-assembly/pom.xml

@@ -84,6 +84,11 @@
       <artifactId>wayang-postgres</artifactId>
       <version>0.7.1-SNAPSHOT</version>
     </dependency>
+    <dependency>
+      <groupId>org.apache.wayang</groupId>
+      <artifactId>wayang-ml4all</artifactId>
+      <version>0.7.1-SNAPSHOT</version>
+    </dependency>
   </dependencies>
 
   <build>

wayang-ml4all/README.md

@@ -0,0 +1,56 @@
+# ML4all: scalable ML system for everyone
+
+
+ML4all is a system that frees users from the burden of machine learning algorithm selection and low-level implementation details.
+It uses a new abstraction that is capable of solving most ML tasks and provides a cost-based optimizer on top of the proposed abstraction for choosing the best gradient descent algorithm in a given setting.
+Our results show that ML4all is more than two orders of magnitude faster than state-of-the-art systems and can process large datasets that were not possible before.
+
+More details can be found in our dedicated [SIGMOD publication](https://dl.acm.org/citation.cfm?id=3064042) and 
+in Wayang's core [system paper](https://sigmodrecord.org/publications/sigmodRecord/2309/pdfs/05_Systems_Beedkar.pdf).
+
+## Abstraction
+ML4all abstracts most ML algorithms with seven operators:
+
+- (1) `Transform` receives a data point to transform
+(e.g., normalize it) and outputs a new data point.
+
+- (2) `Stage` initializes all the required global param-
+eters (e.g., centroids for the k-means algorithm).
+
+- (3) `Compute` performs user-defined computations
+on the input data point and returns a new data
+point. For example, it can compute the nearest cen-
+troid for each input data point.
+
+- (4) `Update` updates the global parameters based on
+a user-defined formula. For example, it can update
+the new centroids based on the output computed by
+the Compute operator.
+
+- (5) `Sample` takes as input the size of the desired
+sample and the data points to sample from and re-
+turns a reduced set of sampled data points.
+
+- (6) `Converge` specifies a function that outputs
+a convergence dataset required for determining
+whether the iterations should continue or stop.
+
+- (7) `Loop` specifies the stopping condition on the
+convergence dataset.
+
+Similar to MapReduce, where users need to implement a map and reduce function, users of ML4all wishing to develop their own algorithm should implement the above interfaces.
+The interfaces can be found in `org.apache.wayang.ml4all.abstraction.api`.
+
+Examples for KMeans clustering and stochastic gradient descent can be found in `org.apache.wayang.ml4all.algorithms`.
+
+## Example runs
+- Kmeans:
+
+```shell
+./bin/wayang-submit org.apache.wayang.ml4all.examples.RunKMeans java,spark file:///Users/zoi/Work/WAYANG/forked/incubator-wayang/wayang-ml4all/src/main/resources/input/USCensus1990-sample.input 3 68 0 1
+```
+
+- SGD:
+```shell
+./bin/wayang-submit org.apache.wayang.ml4all.examples.RunSGD spark file:///Users/zoi/Work/WAYANG/forked/incubator-wayang/wayang-ml4all/src/main/resources/input/adult.zeros.input 100827 123 10 0.001
+```