Skip to content

tutorial.md

Latest commit

 

History

History
153 lines (118 loc) · 4.52 KB

tutorial.md

File metadata and controls

153 lines (118 loc) · 4.52 KB

This tutorial will show users how to run the WordCount example locally with Wayang.

Clone repository

git clone https://github.com/apache/incubator-wayang.git

Create binaries

Running following commands to build Wayang and generate the tar.gz

cd incubator-wayang
./mvnw clean package -pl :wayang-assembly -Pdistribution

Then you can find the wayang-assembly-1.1.1-SNAPSHOT-dist.tar.gz under wayang-assembly/target directory.

Prepare the environment

Wayang

tar -xvf wayang-assembly-1.1.1-SNAPSHOT-dist.tar.gz
cd wayang-1.1.1-SNAPSHOT

In linux

echo "export WAYANG_HOME=$(pwd)" >> ~/.bashrc
echo "export PATH=${PATH}:${WAYANG_HOME}/bin" >> ~/.bashrc
source ~/.bashrc

In MacOS

echo "export WAYANG_HOME=$(pwd)" >> ~/.zshrc
echo "export PATH=${PATH}:${WAYANG_HOME}/bin" >> ~/.zshrc
source ~/.zshrc

Others

  • You need to install Apache Spark version 3 or higher. Don’t forget to set the SPARK_HOME environment variable.
  • You need to install Apache Hadoop version 3 or higher. Don’t forget to set the HADOOP_HOME environment variable.

Run the program

To execute the WordCount example with Apache Wayang, you need to execute your program with the 'wayang-submit' command:

cd wayang-1.1.1-SNAPSHOT
./bin/wayang-submit org.apache.wayang.apps.wordcount.Main java file://$(pwd)/README.md

If you're using Java 17, add the following JVM flags:

Update your wayang-submit (wayang-assembly/target/wayang-1.0.1-SNAPSHOT/bin/wayang-submit) script (or command) with:

eval "$RUNNER \
  --add-exports=java.base/sun.nio.ch=ALL-UNNAMED \
  --add-opens=java.base/java.nio=ALL-UNNAMED \
  --add-opens=java.base/java.lang=ALL-UNNAMED \
  --add-opens=java.base/java.util=ALL-UNNAMED \
  --add-opens=java.base/java.io=ALL-UNNAMED \
  --add-opens=java.base/java.lang.reflect=ALL-UNNAMED \
  --add-opens=java.base/java.util.concurrent=ALL-UNNAMED \
  --add-opens=java.base/java.net=ALL-UNNAMED \
  --add-opens=java.base/java.lang.invoke=ALL-UNNAMED \
  $FLAGS -cp \"${WAYANG_CLASSPATH}\" $CLASS ${ARGS}"

Then you should be able to see outputs like this:

img.png

Running SQL Queries in Wayang

Wayang provides support for executing SQL queries using its SQL API.

Example

SELECT * FROM my_table;
### Steps to execute SQL queries

1. Configure the Wayang Calcite model using the property:
   `wayang.calcite.model`

2. Create a `SqlContext` instance:

```java
SqlContext context = new SqlContext(configuration);
Example configuration:

```java
configuration.setProperty(
    "wayang.calcite.model",
    "{ \"version\": \"1.0\", \"defaultSchema\": \"MY_SCHEMA\", \"schemas\": [] }"
);

---

## 👉 Step 2 — Improve SQL execution step

Below this:

```md
3. Execute your SQL query:
```java
Collection<Record> result = context.executeSql("SELECT * FROM my_table");

for (Record record : result) {
    System.out.println(record);
}
### Example: Running a simple SQL query

Wayang allows executing SQL queries using the `SqlContext` API.

```java
import org.apache.wayang.api.sql.context.SqlContext;
import org.apache.wayang.core.api.Configuration;
import org.apache.wayang.basic.data.Record;

import java.util.Collection;

public class Example {
    public static void main(String[] args) throws Exception {
        Configuration configuration = new Configuration();

        configuration.setProperty(
            "wayang.calcite.model",
            "{ \"version\": \"1.0\", \"defaultSchema\": \"MY_SCHEMA\", \"schemas\": [] }"
        );

        SqlContext context = new SqlContext(configuration);

        Collection<Record> result = context.executeSql("SELECT * FROM my_table");

        for (Record record : result) {
            System.out.println(record);
        }
    }
}