Release 0.1.0 (#5)

ZhengshuaiPENG · web-flow · commit 4f0d3fbcbba7 · 2023-03-21T22:37:46.000+08:00
* bump version to release 0.1.0

* update readme

* minor update
diff --git a/README.md b/README.md
@@ -3,23 +3,26 @@
 ## ZenML
 
 **ZenML** stands for **Zen Metrics Language**. It is a **YAML-based descriptive language** that is used
-by [Kyligence Zen](https://zen.kyligence.io/)
-to define metrics, dimensions, and the underlying relational datasets. Think of ZenML as a semantic layer that promotes
-technical data into business data, like a bridge that enables non-technical users to understand, explore, and get
-insight out of traditional data warehouses. ZenML encourages the separation of data modeling and data visualization, and
-facilitates a central definition of business data language for all downstream data consumers.
+by [Kyligence Zen](https://zen.kyligence.io/) to define `metrics`, `dimensions`, and the `underlying relational datasets`. Think of ZenML as a semantic layer that promotes technical
+data into business data, like a bridge that enables non-technical users to understand, explore, and get insight out of
+traditional data warehouses. ZenML encourages the separation of data modeling and data visualization, and facilitates a
+central definition of business data language for all downstream data consumers.
 
-Kyligence Zen uses ZenML to store all the metric definitions and related data models that user creates. And a variety of
-data services are then provided around ZenML, including
+Kyligence Zen uses ZenML to store all metric definitions and their relations and mappings to relational data models.
+With this metadata, Kyligence Zen is able to automatically translate all kinds of business requests and tasks into SQL
+executions. A variety of data services are then provided around ZenML, including
 
-- SQL based data queries and BI integrations, via open APIS like JDBC / ODBC / RestAPI
-- Metrics templating of business domain know-how
-- Goal management, progress tracking, and alerting
+- SQL based metric queries and BI integrations, via open APIs like JDBC / ODBC / RestAPI
+- Metric templates of business domain know-hows
 - Causal analysis, anomaly detection
+- Goal management, progress tracking, and alerting
 - Metrics data linage
 - And more...
 
-![arch](docs/images/architecture.png)
+- You can also maintain ZenML/YAML files in git for version control, and publish (or rollback to) a certain version of
+  metric definitions to Kyligence Zen service for business operation.
+
+More details please refer to [ZenML Introduction](https://zen-docs.kyligence.io/en/appendix/zenml-reference)
 
 ## Kyligence Zen
 
@@ -38,12 +41,27 @@ Try out Kyligence Zen today and take control of your metrics like never before!
 Visit [zen.kyligence.io](https://zen.kyligence.io/) to get started
 for free.
 
-## Prerequisite
+## Kyligence ZenML Toolkit
+
+Kyligence ZenML Toolkit is a toolkit which help users to extract metrics metadata from any type of file, includes but
+not only:
+
+- **Tableau (Implemented)**
+    - workbook file (`.twb`)
+    - datasource file(`.tds`)
+- **Excel file(coming soon)**
+    - `.xlsx`
+- **SQL file(coming soon)**
+    - `.sql`
+
+![arch](docs/images/architecture.png)
+
+### Prerequisite
 
 - Supported OS: `Linux` / `MacOS`
 - Supported JDK version: `JDK17`
 
-## Download
+### Download
 
 Please go to [Release Page](https://github.com/Kyligence/zen-ml-toolkit/releases) to download.
 
@@ -63,7 +81,7 @@ After downloaded package, extract the tar file
 $ tar -zxvf Kyligence-ZenML-Toolkit-{version}.tar.gz
 ```
 
-## Usage
+### Usage
 
 in Terminal
 
@@ -76,18 +94,19 @@ $ sh ./bin/zen.sh -i <arg> -o <arg>
                      generated
 ```
 
-## Example
+### Example
 
-### Convert tableau tds file to ZenML file
+#### Convert tableau tds file to ZenML file
 
 ```
-$ cd Kyligence-ZenML-Toolkit-{version}
-$ sh ./bin/zen.sh -i ./samples/tableau/superstore.sh -o ./samples/tableau/
+# please replace ${Kyligence-ZenML-Toolkit-{version}} with absolute path
+$ cd ${Kyligence-ZenML-Toolkit-{version}}
+$ sh ./bin/zen.sh -i ${Kyligence-ZenML-Toolkit-{version}}/samples/superstore.sh -o ${Kyligence-ZenML-Toolkit-{version}}/samples/
 ```
 
 You will get a `superstore.zen.yml` file in folder `Kyligence-ZenML-Toolkit-{version}/samples`
 
-### Import ZenML to Kyligence Zen
+#### Import ZenML to Kyligence Zen
 
 1. Login into [Kyligence Zen](https://zen.kyligence.io/)
 2. Go to **Data**, click **New**, Choose **Table**
@@ -107,6 +126,9 @@ You will get a `superstore.zen.yml` file in folder `Kyligence-ZenML-Toolkit-{ver
 
 Welcome to contribute, you can convert any files which contains metrics metadata to a ZenML file.
 
+- Please note that this project use **JDK17**
+- For ZenML spec, please refer to  [ZenML Introduction](https://zen-docs.kyligence.io/en/appendix/zenml-reference)
+
 ## Contributors
 
 <a href="https://github.com/Kyligence/zen-ml-toolkit/graphs/contributors">
diff --git a/build/bin/zen.sh b/build/bin/zen.sh
@@ -1,5 +1,4 @@
 #!/bin/bash
-
 #
 # Licensed to the Apache Software Foundation (ASF) under one or more
 # contributor license agreements.  See the NOTICE file distributed with
@@ -17,6 +16,7 @@
 # limitations under the License.
 #
 
+
 function main() {
   # init
   source $(cd -P -- "$(dirname -- "$0")" && pwd -P)/../sbin/header.sh "$@"
diff --git a/build/sbin/bootstrap.sh b/build/sbin/bootstrap.sh
@@ -1,5 +1,4 @@
 #!/bin/bash
-
 #
 # Licensed to the Apache Software Foundation (ASF) under one or more
 # contributor license agreements.  See the NOTICE file distributed with
diff --git a/build/sbin/check-1000-os.sh b/build/sbin/check-1000-os.sh
@@ -1,7 +1,4 @@
-
-
-#!/usr/bin/env bash
-
+#!/bin/bash
 #
 # Licensed to the Apache Software Foundation (ASF) under one or more
 # contributor license agreements.  See the NOTICE file distributed with
diff --git a/build/sbin/check-1100-java.sh b/build/sbin/check-1100-java.sh
@@ -1,6 +1,4 @@
-
-#!/usr/bin/env bash
-
+#!/bin/bash
 #
 # Licensed to the Apache Software Foundation (ASF) under one or more
 # contributor license agreements.  See the NOTICE file distributed with
@@ -18,6 +16,7 @@
 # limitations under the License.
 #
 
+
 #title=Checking Java Version
 
 source $(cd -P -- "$(dirname -- "$0")" && pwd -P)/header.sh
diff --git a/build/sbin/check-env.sh b/build/sbin/check-env.sh
@@ -1,5 +1,4 @@
-#!/usr/bin/env bash
-
+#!/bin/bash
 #
 # Licensed to the Apache Software Foundation (ASF) under one or more
 # contributor license agreements.  See the NOTICE file distributed with
@@ -17,6 +16,7 @@
 # limitations under the License.
 #
 
+
 source $(cd -P -- "$(dirname -- "$0")" && pwd -P)/header.sh
 
 mkdir -p ${ZEN_HOME}/logs
diff --git a/build/sbin/header.sh b/build/sbin/header.sh
@@ -1,5 +1,4 @@
 #!/bin/bash
-
 #
 # Licensed to the Apache Software Foundation (ASF) under one or more
 # contributor license agreements.  See the NOTICE file distributed with
@@ -17,6 +16,7 @@
 # limitations under the License.
 #
 
+
 # source me
 
 function isValidJavaVersion() {
diff --git a/build/sbin/rotate-log.sh b/build/sbin/rotate-log.sh
@@ -1,5 +1,4 @@
 #!/bin/bash
-
 #
 # Licensed to the Apache Software Foundation (ASF) under one or more
 # contributor license agreements.  See the NOTICE file distributed with
@@ -17,6 +16,7 @@
 # limitations under the License.
 #
 
+
 source $(cd -P -- "$(dirname -- "$0")" && pwd -P)/header.sh
 
 function checkFileOccupied() {
diff --git a/build/sbin/setenv.sh b/build/sbin/setenv.sh
@@ -1,5 +1,4 @@
 #!/bin/bash
-
 #
 # Licensed to the Apache Software Foundation (ASF) under one or more
 # contributor license agreements.  See the NOTICE file distributed with
@@ -16,6 +15,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+mitations under the License.
+#
 
 source $(cd -P -- "$(dirname -- "$0")" && pwd -P)/header.sh $@
 
diff --git a/build/scripts/download-jdk.sh b/build/scripts/download-jdk.sh
@@ -1,5 +1,4 @@
 #!/bin/bash
-
 #
 # Licensed to the Apache Software Foundation (ASF) under one or more
 # contributor license agreements.  See the NOTICE file distributed with
@@ -17,6 +16,7 @@
 # limitations under the License.
 #
 
+
 #JDK Download Page: https://jdk.java.net/archive/
 
 root_dir=$(cd -P -- "$(dirname -- "$0")/../.." && pwd -P)
diff --git a/build/scripts/package.sh b/build/scripts/package.sh
@@ -1,5 +1,4 @@
 #!/bin/bash
-
 #
 # Licensed to the Apache Software Foundation (ASF) under one or more
 # contributor license agreements.  See the NOTICE file distributed with
@@ -17,6 +16,7 @@
 # limitations under the License.
 #
 
+
 echo "Begin to package..."
 # project root dir absolute path
 
@@ -81,6 +81,11 @@ cp ${root_dir}/build/sbin/* sbin
 
 ## 6. others
 mkdir logs
+mkidr samples
+cp -r ${root_dir}/samples/*  samples
+mkdir docs
+cp -r ${root_dir}/docs/*  docs
+cp ${root_dir}/README.md ./
 
 ## 7. download jdk
 sh ${root_dir}/build/scripts/download-jdk.sh
diff --git a/pom.xml b/pom.xml
@@ -10,7 +10,7 @@
     </parent>
     <groupId>io.kyligence</groupId>
     <artifactId>zen-ml-toolkit</artifactId>
-    <version>0.1.0-SNAPSHOT</version>
+    <version>0.1.0</version>
     <name>zen-ml-toolkit</name>
     <description>zen-ml-toolkit</description>
     <properties>
diff --git a/src/main/java/io/kyligence/zenml/toolkit/ZenMlToolkitCLI.java b/src/main/java/io/kyligence/zenml/toolkit/ZenMlToolkitCLI.java
@@ -64,6 +64,7 @@ private void printUsage(OptionsHelper optionsHelper) {
     private void initOptionValues(OptionsHelper optionsHelper) {
         if (optionsHelper.getOptions().length != 2) {
             printUsage(optionsHelper);
+            log.error("Illegal arguments, please check the help usage");
             throw new RuntimeException("Illegal arguments, please check the help usage");
         }
 
@@ -72,10 +73,12 @@ private void initOptionValues(OptionsHelper optionsHelper) {
 
         if (StringUtils.isBlank(srcFilePath)) {
             printUsage(optionsHelper);
+            log.error("Source file path is empty, please check the help usage");
             throw new RuntimeException("Source file path is empty, please check the help usage");
         }
         if (StringUtils.isBlank(destDirPath)) {
             printUsage(optionsHelper);
+            log.error("Output directory path is empty, please check the help usage");
             throw new RuntimeException("Output directory path is empty, please check the help usage");
         }
 
@@ -93,6 +96,7 @@ protected void execute(OptionsHelper optionsHelper) throws Exception {
             ZenGenerator generator = new ZenGenerator();
             generator.generateZenMetrics(srcFilePath, destDirPath);
         } catch (IOException e) {
+            log.error("Failed to write metrics to file. please check the details: ", e);
             throw new RuntimeException("Failed to write metrics to file. please check the details: ", e);
         }
     }
diff --git a/src/main/java/io/kyligence/zenml/toolkit/entry/ZenGenerator.java b/src/main/java/io/kyligence/zenml/toolkit/entry/ZenGenerator.java
@@ -21,6 +21,7 @@
 import io.kyligence.zenml.toolkit.converter.ConverterFactory;
 import io.kyligence.zenml.toolkit.converter.FileType;
 import io.kyligence.zenml.toolkit.converter.MetricsConverter;
+import io.kyligence.zenml.toolkit.metrics.MetricSpec;
 import io.kyligence.zenml.toolkit.metrics.Metrics;
 import io.kyligence.zenml.toolkit.utils.YamlUtils;
 import lombok.extern.slf4j.Slf4j;
@@ -30,6 +31,7 @@
 import java.io.IOException;
 import java.nio.file.Path;
 import java.nio.file.Paths;
+import java.util.List;
 
 @Slf4j
 public class ZenGenerator {
@@ -40,6 +42,14 @@ public void generateZenMetrics(String srcPath, String destDir) throws IOExceptio
 
         log.info("Begin to convert metrics from {}", srcPath);
         Metrics metrics = converter.convert2Metrics(srcPath);
+
+        List<MetricSpec> metricSpecs = metrics.getMetrics();
+        log.info("{} metrics extracted",metricSpecs.size() );
+        for(MetricSpec ms : metricSpecs){
+            log.info("    - Metrics Name: {}, Expression:{}", ms.getName(), ms.getExpression());
+        }
+
+
         String destPath = getFullOutputPath(srcPath, destDir);
 
         log.info("Begin to write metrics to path: {}", destPath);

Original file line number	Diff line number	Diff line change
`@@ -1,5 +1,4 @@`
`1`	`1`	`#!/bin/bash`
`2`		`-`
`3`	`2`	`#`
`4`	`3`	`# Licensed to the Apache Software Foundation (ASF) under one or more`
`5`	`4`	`# contributor license agreements. See the NOTICE file distributed with`
`@@ -17,6 +16,7 @@`
`17`	`16`	`# limitations under the License.`
`18`	`17`	`#`
`19`	`18`
	`19`	`+`
`20`	`20`	`function main() {`
`21`	`21`	`# init`
`22`	`22`	`source $(cd -P -- "$(dirname -- "$0")" && pwd -P)/../sbin/header.sh "$@"`
Original file line number	Diff line number	Diff line change
`@@ -1,7 +1,4 @@`
`1`		`-`
`2`		`-`
`3`		`-#!/usr/bin/env bash`
`4`		`-`
	`1`	`+#!/bin/bash`
`5`	`2`	`#`
`6`	`3`	`# Licensed to the Apache Software Foundation (ASF) under one or more`
`7`	`4`	`# contributor license agreements. See the NOTICE file distributed with`
Original file line number	Diff line number	Diff line change
`@@ -1,6 +1,4 @@`
`1`		`-`
`2`		`-#!/usr/bin/env bash`
`3`		`-`
	`1`	`+#!/bin/bash`
`4`	`2`	`#`
`5`	`3`	`# Licensed to the Apache Software Foundation (ASF) under one or more`
`6`	`4`	`# contributor license agreements. See the NOTICE file distributed with`
`@@ -18,6 +16,7 @@`
`18`	`16`	`# limitations under the License.`
`19`	`17`	`#`
`20`	`18`
	`19`	`+`
`21`	`20`	`#title=Checking Java Version`
`22`	`21`
`23`	`22`	`source $(cd -P -- "$(dirname -- "$0")" && pwd -P)/header.sh`