Skip to content

tutorial.md

Latest commit

 

History

History
124 lines (97 loc) · 3.69 KB

tutorial.md

File metadata and controls

124 lines (97 loc) · 3.69 KB

This tutorial will show users how to run the WordCount example locally with Wayang.

Clone repository

git clone https://github.com/apache/incubator-wayang.git

Create binaries

Running following commands to build Wayang and generate the tar.gz

cd incubator-wayang
./mvnw clean package -pl :wayang-assembly -Pdistribution

Then you can find the wayang-assembly-1.1.1-SNAPSHOT-dist.tar.gz under wayang-assembly/target directory.

Prepare the environment

Wayang

tar -xvf wayang-assembly-1.1.1-SNAPSHOT-dist.tar.gz
cd wayang-1.1.1-SNAPSHOT

In linux

echo "export WAYANG_HOME=$(pwd)" >> ~/.bashrc
echo "export PATH=${PATH}:${WAYANG_HOME}/bin" >> ~/.bashrc
source ~/.bashrc

In MacOS

echo "export WAYANG_HOME=$(pwd)" >> ~/.zshrc
echo "export PATH=${PATH}:${WAYANG_HOME}/bin" >> ~/.zshrc
source ~/.zshrc

Others

  • You need to install Apache Spark version 3 or higher. Don’t forget to set the SPARK_HOME environment variable.
  • You need to install Apache Hadoop version 3 or higher. Don’t forget to set the HADOOP_HOME environment variable.

Run the program

To execute the WordCount example with Apache Wayang, you need to execute your program with the 'wayang-submit' command:

cd wayang-1.1.1-SNAPSHOT
./bin/wayang-submit org.apache.wayang.apps.wordcount.Main java file://$(pwd)/README.md

If you're using Java 17, add the following JVM flags:

Update your wayang-submit (wayang-assembly/target/wayang-1.0.1-SNAPSHOT/bin/wayang-submit) script (or command) with:

eval "$RUNNER \
  --add-exports=java.base/sun.nio.ch=ALL-UNNAMED \
  --add-opens=java.base/java.nio=ALL-UNNAMED \
  --add-opens=java.base/java.lang=ALL-UNNAMED \
  --add-opens=java.base/java.util=ALL-UNNAMED \
  --add-opens=java.base/java.io=ALL-UNNAMED \
  --add-opens=java.base/java.lang.reflect=ALL-UNNAMED \
  --add-opens=java.base/java.util.concurrent=ALL-UNNAMED \
  --add-opens=java.base/java.net=ALL-UNNAMED \
  --add-opens=java.base/java.lang.invoke=ALL-UNNAMED \
  $FLAGS -cp \"${WAYANG_CLASSPATH}\" $CLASS ${ARGS}"

Then you should be able to see outputs like this:

img.png

Running SQL Queries in Wayang

Wayang provides support for executing SQL queries using its SQL API.

Example

SELECT * FROM my_table;
### Steps to execute SQL queries

1. Configure the Wayang Calcite model using the property:
   `wayang.calcite.model`

2. Create a `SqlContext` instance:

```java
SqlContext context = new SqlContext(configuration);
Example configuration:

```java
configuration.setProperty(
    "wayang.calcite.model",
    "{ \"version\": \"1.0\", \"defaultSchema\": \"MY_SCHEMA\", \"schemas\": [] }"
);

---

## 👉 Step 2 — Improve SQL execution step

Below this:

```md
3. Execute your SQL query:
```java
Collection<Record> result = context.executeSql("SELECT * FROM my_table");

for (Record record : result) {
    System.out.println(record);
}