Getting started task

[ELMERWANG]

Hi,

I am quite new to stream mining and I am trying to get the streamDM running. However when I finished setting up all the dependencies I got some issues reported below.

Expected behavior

I want to obtain the confusion matrix by using the example commands.

Observed behavior

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/03/09 18:40:50 INFO SparkContext: Running Spark version 2.1.0
21/03/09 18:40:50 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/03/09 18:40:50 WARN Utils: Your hostname, localhost.localdomain resolves to a loopback address: 127.0.0.1; using 192.168.122.1 instead (on interface virbr0)
21/03/09 18:40:50 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
21/03/09 18:40:50 INFO SecurityManager: Changing view acls to: shunyao
21/03/09 18:40:50 INFO SecurityManager: Changing modify acls to: shunyao
21/03/09 18:40:50 INFO SecurityManager: Changing view acls groups to:
21/03/09 18:40:50 INFO SecurityManager: Changing modify acls groups to:
21/03/09 18:40:50 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(shunyao); groups with view permissions: Set(); users with modify permissions: Set(shunyao); groups with modify permissions: Set()
21/03/09 18:40:50 INFO Utils: Successfully started service 'sparkDriver' on port 41123.
21/03/09 18:40:50 INFO SparkEnv: Registering MapOutputTracker
21/03/09 18:40:50 INFO SparkEnv: Registering BlockManagerMaster
21/03/09 18:40:50 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
21/03/09 18:40:50 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
21/03/09 18:40:50 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-8d18da96-7b2c-48c9-bbd2-b49a29b16274
21/03/09 18:40:50 INFO MemoryStore: MemoryStore started with capacity 413.9 MB
21/03/09 18:40:50 INFO SparkEnv: Registering OutputCommitCoordinator
21/03/09 18:40:51 INFO Utils: Successfully started service 'SparkUI' on port 4040.
21/03/09 18:40:51 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.122.1:4040
21/03/09 18:40:51 INFO SparkContext: Added JAR file:/home/shunyao/Downloads/streamDM-0.2/scripts/../target/scala-2.10/streamdm-spark-streaming-_2.10-0.2.jar at spark://192.168.122.1:41123/jars/streamdm-spark-streaming-_2.10-0.2.jar with timestamp 1615333251155
21/03/09 18:40:51 INFO Executor: Starting executor ID driver on host localhost
21/03/09 18:40:51 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 45261.
21/03/09 18:40:51 INFO NettyBlockTransferService: Server created on 192.168.122.1:45261
21/03/09 18:40:51 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
21/03/09 18:40:51 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.122.1, 45261, None)
21/03/09 18:40:51 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.122.1:45261 with 413.9 MB RAM, BlockManagerId(driver, 192.168.122.1, 45261, None)
21/03/09 18:40:51 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.122.1, 45261, None)
21/03/09 18:40:51 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.122.1, 45261, None)
Exception in thread "main" java.lang.Exception: Problem creating instance of class: EvaluatePrequential
at com.github.javacliparser.ClassOption.cliStringToObject(ClassOption.java:139)
at org.apache.spark.streamdm.streamDMJob$.main(streamDMJob.scala:43)
at org.apache.spark.streamdm.streamDMJob.main(streamDMJob.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.IllegalArgumentException: Problems with option: streamReader
at com.github.javacliparser.ClassOption.setValueViaCLIString(ClassOption.java:60)
at com.github.javacliparser.AbstractOption.resetToDefault(AbstractOption.java:90)
at com.github.javacliparser.AbstractClassOption.(AbstractClassOption.java:84)
at com.github.javacliparser.AbstractClassOption.(AbstractClassOption.java:63)
at com.github.javacliparser.ClassOption.(ClassOption.java:34)
at org.apache.spark.streamdm.tasks.EvaluatePrequential.(EvaluatePrequential.scala:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at com.github.javacliparser.ClassOption.cliStringToObject(ClassOption.java:137)
... 11 more
Caused by: java.lang.Exception: Class not found: FileReader
at com.github.javacliparser.ClassOption.cliStringToObject(ClassOption.java:132)
at com.github.javacliparser.ClassOption.setValueViaCLIString(ClassOption.java:57)
... 22 more
21/03/09 18:40:51 INFO SparkContext: Invoking stop() from shutdown hook
21/03/09 18:40:51 INFO SparkUI: Stopped Spark web UI at http://192.168.122.1:4040
21/03/09 18:40:51 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
21/03/09 18:40:51 INFO MemoryStore: MemoryStore cleared
21/03/09 18:40:51 INFO BlockManager: BlockManager stopped
21/03/09 18:40:51 INFO BlockManagerMaster: BlockManagerMaster stopped
21/03/09 18:40:51 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
21/03/09 18:40:51 INFO SparkContext: Successfully stopped SparkContext
21/03/09 18:40:51 INFO ShutdownHookManager: Shutdown hook called
21/03/09 18:40:51 INFO ShutdownHookManager: Deleting directory /tmp/spark-6f0a2908-b3f5-477c-bccc-6302be0e18f8

Steps to reproduce the issue

I tried different OS: macOS, Kali, Ubuntu and CentOS 7.

These steps are what I did and followed:
Step 1: Download & install the Java openjdk8.
Step 2: Download & install the Spark (I tried 2.3.2 and 2.1.0 with Hadoop or without Hadoop)
Step 3: Download & install Scala (from https://www.scala-lang.org/files/archive/scala-2.11.8.deb)
Step 4: Download & install SBT (from https://sbt.bintray.com/native-packages/sbt/0.13.15/sbt-0.13.15.tgz)
Step 5: Download StreamDM from Github.
Step 6: Change the SPARK_HOME to the Spark path.
Step 7: In the streamDM directory, run sbt package
Step 8: In the /scripts, run ./spark.sh or ./spark.sh "EvaluatePrequential -l (SGDLearner -l 0.01 -o LogisticLoss -r ZeroRegularizer) –s (FileReader –k 100 –d 60 –f ../data/mydata)"
Result: Errors occurred.

Command line
sbt package
./spark.sh
./spark.sh "EvaluatePrequential -l (SGDLearner -l 0.01 -o LogisticLoss -r ZeroRegularizer) –s (FileReader –k 100 –d 60 –f ../data/mydata)"

Infrastructure details (Contains the versions that I tried)

• Java Version: 8 | 11
• Scala Version: 2.10.2 | 2.11.8 | 2.11.11
• Spark version: 2.1.0 | 2.3.2
• OS version: macOS, Ubuntu 20.04, Kali, CentOS 7
• Cluster mode or local mode? Local

For the Spark, do I need to download with Hadoop or without Hadoop or the file name only contains spark-x.x.x.tgz?
Such as:

• spark-2.3.2-bin-hadoop2.7.tgz
• spark-2.3.2-bin-without-hadoop.tgz
• spark-2.3.2.tgz

Is anything wrong with my setup? I produced the same error on different OS.
Please help, thanks in advance :)

[zhangjiajin]

@abifet
@hmgomes

[hmgomes]

Hi @ELMERWANG
It looks like a problem with SGDLearner. Can you run it using a Hoeffding tree?

./spark.sh "EvaluatePrequential -l (trees.HoeffdingTree) -s (FileReader -f ../data/mydata.arff -k 450 -i 45500)" 1> results_ht_mydata.csv 2> log_ht_mydata.log

Btw, if you are new to stream mining, I suggest this open book to get started

[ELMERWANG]

Hi @hmgomes,

Thank you for your reply, I have tried the command that you provide, however the error now changed to:

Error message:
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/03/21 19:46:27 INFO SparkContext: Running Spark version 2.1.0
21/03/21 19:46:27 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/03/21 19:46:27 WARN Utils: Your hostname, localhost.localdomain resolves to a loopback address: 127.0.0.1; using 192.168.122.1 instead (on interface virbr0)
21/03/21 19:46:27 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
21/03/21 19:46:27 INFO SecurityManager: Changing view acls to: shunyao
21/03/21 19:46:27 INFO SecurityManager: Changing modify acls to: shunyao
21/03/21 19:46:27 INFO SecurityManager: Changing view acls groups to:
21/03/21 19:46:27 INFO SecurityManager: Changing modify acls groups to:
21/03/21 19:46:27 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(shunyao); groups with view permissions: Set(); users with modify permissions: Set(shunyao); groups with modify permissions: Set()
21/03/21 19:46:28 INFO Utils: Successfully started service 'sparkDriver' on port 45035.
21/03/21 19:46:28 INFO SparkEnv: Registering MapOutputTracker
21/03/21 19:46:28 INFO SparkEnv: Registering BlockManagerMaster
21/03/21 19:46:28 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
21/03/21 19:46:28 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
21/03/21 19:46:28 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-deb8bb4e-c0e0-450e-b68d-3f96548bece1
21/03/21 19:46:28 INFO MemoryStore: MemoryStore started with capacity 413.9 MB
21/03/21 19:46:28 INFO SparkEnv: Registering OutputCommitCoordinator
21/03/21 19:46:28 INFO Utils: Successfully started service 'SparkUI' on port 4040.
21/03/21 19:46:28 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.122.1:4040
21/03/21 19:46:28 INFO SparkContext: Added JAR file:/home/shunyao/Downloads/streamDM-0.2/scripts/../target/scala-2.10/streamdm-spark-streaming-_2.10-0.2.jar at spark://192.168.122.1:45035/jars/streamdm-spark-streaming-_2.10-0.2.jar with timestamp 1616370388511
21/03/21 19:46:28 INFO Executor: Starting executor ID driver on host localhost
21/03/21 19:46:28 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38860.
21/03/21 19:46:28 INFO NettyBlockTransferService: Server created on 192.168.122.1:38860
21/03/21 19:46:28 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
21/03/21 19:46:28 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.122.1, 38860, None)
21/03/21 19:46:28 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.122.1:38860 with 413.9 MB RAM, BlockManagerId(driver, 192.168.122.1, 38860, None)
21/03/21 19:46:28 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.122.1, 38860, None)
21/03/21 19:46:28 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.122.1, 38860, None)
Exception in thread "main" java.lang.Exception: Problem creating instance of class: EvaluatePrequential
at com.github.javacliparser.ClassOption.cliStringToObject(ClassOption.java:139)
at org.apache.spark.streamdm.streamDMJob$.main(streamDMJob.scala:43)
at org.apache.spark.streamdm.streamDMJob.main(streamDMJob.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.IllegalArgumentException: Problems with option: streamReader
at com.github.javacliparser.ClassOption.setValueViaCLIString(ClassOption.java:60)
at com.github.javacliparser.AbstractOption.resetToDefault(AbstractOption.java:90)
at com.github.javacliparser.AbstractClassOption.(AbstractClassOption.java:84)
at com.github.javacliparser.AbstractClassOption.(AbstractClassOption.java:63)
at com.github.javacliparser.ClassOption.(ClassOption.java:34)
at org.apache.spark.streamdm.tasks.EvaluatePrequential.(EvaluatePrequential.scala:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at com.github.javacliparser.ClassOption.cliStringToObject(ClassOption.java:137)
... 11 more
Caused by: java.lang.Exception: Class not found: FileReader
at com.github.javacliparser.ClassOption.cliStringToObject(ClassOption.java:132)
at com.github.javacliparser.ClassOption.setValueViaCLIString(ClassOption.java:57)
... 22 more
21/03/21 19:46:28 INFO SparkContext: Invoking stop() from shutdown hook
21/03/21 19:46:28 INFO SparkUI: Stopped Spark web UI at http://192.168.122.1:4040
21/03/21 19:46:28 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
21/03/21 19:46:28 INFO MemoryStore: MemoryStore cleared
21/03/21 19:46:28 INFO BlockManager: BlockManager stopped
21/03/21 19:46:28 INFO BlockManagerMaster: BlockManagerMaster stopped
21/03/21 19:46:28 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
21/03/21 19:46:28 INFO SparkContext: Successfully stopped SparkContext
21/03/21 19:46:28 INFO ShutdownHookManager: Shutdown hook called
21/03/21 19:46:28 INFO ShutdownHookManager: Deleting directory /tmp/spark-d340aa5a-9e9b-4d68-a6fd-b24a0276338d

And the result file is empty...

[hmgomes]

Hi @ELMERWANG

That is interesting. Can you try using Spark 2.4.5?

Cheers,
Heitor

[ELMERWANG]

Hi @hmgomes,

Thanks, may I ask which version should I download? With or without hadoop?

[ ] spark-2.4.5-bin-hadoop2.6.tgz 2020-02-02 20:27 220M

[ ] spark-2.4.5-bin-hadoop2.7.tgz 2020-02-02 20:27 222M

[ ] spark-2.4.5-bin-without-hadoop-scala-2.12.tgz 2020-02-02 20:27 139M

[ ] spark-2.4.5-bin-without-hadoop.tgz 2020-02-02 20:27 160M

[ ] spark-2.4.5.tgz 2020-02-02 20:27 15M

[LYZ1125]

@ELMERWANG Hi,I met the same problem as yours. Do you solve your problem?I want to get some suggestions to solve these problems. Thank you!

Infrastructure details
Java Version: 8
Scala Version: 2.11.8
Spark version: 2.3.2
OS version: Ubuntu
sbt version:0.13.18

When I run these Command lines
sbt package
./spark.sh

I get some errors:

java.lang.ArrayIndexOutOfBoundsException: 4
	at org.apache.spark.streamdm.core.DenseInstance.add(DenseInstance.scala:84)
	at org.apache.spark.streamdm.core.DenseInstance.add(DenseInstance.scala:26)
	at org.apache.spark.streamdm.classifiers.SGDLearner$$anonfun$train$1$$anonfun$3.apply(SGDLearner.scala:91)
	at org.apache.spark.streamdm.classifiers.SGDLearner$$anonfun$train$1$$anonfun$3.apply(SGDLearner.scala:85)
	at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
	at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
	at scala.collection.Iterator$class.foreach(Iterator.scala:893)
	at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
	at org.apache.spark.InterruptibleIterator.foldLeft(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce$class.aggregate(TraversableOnce.scala:214)
	at org.apache.spark.InterruptibleIterator.aggregate(InterruptibleIterator.scala:28)
	at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$21.apply(RDD.scala:1122)
	at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$21.apply(RDD.scala:1122)
	at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
	at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:109)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2021-03-30 06:25:41 ERROR Executor:91 - Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.ArrayIndexOutOfBoundsException: 4
	at org.apache.spark.streamdm.core.DenseInstance.add(DenseInstance.scala:84)
	at org.apache.spark.streamdm.core.DenseInstance.add(DenseInstance.scala:26)
	at org.apache.spark.streamdm.classifiers.SGDLearner$$anonfun$train$1$$anonfun$3.apply(SGDLearner.scala:91)
	at org.apache.spark.streamdm.classifiers.SGDLearner$$anonfun$train$1$$anonfun$3.apply(SGDLearner.scala:85)
	at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
	at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
	at scala.collection.Iterator$class.foreach(Iterator.scala:893)
	at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
	at org.apache.spark.InterruptibleIterator.foldLeft(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce$class.aggregate(TraversableOnce.scala:214)
	at org.apache.spark.InterruptibleIterator.aggregate(InterruptibleIterator.scala:28)
	at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$21.apply(RDD.scala:1122)
	at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$21.apply(RDD.scala:1122)
	at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
	at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:109)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

after I ran ./spark.sh "EvaluatePrequential -l (SGDLearner -l 0.01 -o LogisticLoss -r ZeroRegularizer) –s (FileReader –k 100 –d 60 –f ../data/mydata)",I got the follewing result:

/home/yuanzhanglin/local/spark-2.3.2/conf/spark-env.sh: line 71: syntax error near unexpected token `newline'
/home/yuanzhanglin/local/spark-2.3.2/conf/spark-env.sh: line 71: `<!--主节点IP-->'
2021-03-30 06:28:28 WARN  Utils:66 - Your hostname, ubuntu resolves to a loopback address: 127.0.1.1; using 192.168.65.128 instead (on interface ens33)
2021-03-30 06:28:28 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address
2021-03-30 06:28:32 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2021-03-30 06:28:32 INFO  SparkContext:54 - Running Spark version 2.3.2
2021-03-30 06:28:32 INFO  SparkContext:54 - Submitted application: streamDM
2021-03-30 06:28:32 INFO  SecurityManager:54 - Changing view acls to: yuanzhanglin
2021-03-30 06:28:32 INFO  SecurityManager:54 - Changing modify acls to: yuanzhanglin
2021-03-30 06:28:32 INFO  SecurityManager:54 - Changing view acls groups to: 
2021-03-30 06:28:32 INFO  SecurityManager:54 - Changing modify acls groups to: 
2021-03-30 06:28:32 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yuanzhanglin); groups with view permissions: Set(); users  with modify permissions: Set(yuanzhanglin); groups with modify permissions: Set()
2021-03-30 06:28:33 INFO  Utils:54 - Successfully started service 'sparkDriver' on port 44865.
2021-03-30 06:28:33 INFO  SparkEnv:54 - Registering MapOutputTracker
2021-03-30 06:28:33 INFO  SparkEnv:54 - Registering BlockManagerMaster
2021-03-30 06:28:33 INFO  BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2021-03-30 06:28:33 INFO  BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2021-03-30 06:28:33 INFO  DiskBlockManager:54 - Created local directory at /tmp/blockmgr-beb224af-932d-4d9f-9695-42a4a64c2287
2021-03-30 06:28:33 INFO  MemoryStore:54 - MemoryStore started with capacity 366.3 MB
2021-03-30 06:28:33 INFO  SparkEnv:54 - Registering OutputCommitCoordinator
2021-03-30 06:28:33 INFO  log:192 - Logging initialized @6306ms
2021-03-30 06:28:33 INFO  Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
2021-03-30 06:28:33 INFO  Server:419 - Started @6448ms
2021-03-30 06:28:33 INFO  AbstractConnector:278 - Started ServerConnector@21a5fd96{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2021-03-30 06:28:33 INFO  Utils:54 - Successfully started service 'SparkUI' on port 4040.
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@42a9a63e{/jobs,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5bd1ceca{/jobs/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@30c31dd7{/jobs/job,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@596df867{/jobs/job/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@c1fca1e{/stages,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@241a53ef{/stages/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@344344fa{/stages/stage,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@615f972{/stages/stage/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@285f09de{/stages/pool,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@73393584{/stages/pool/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@31500940{/storage,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1827a871{/storage/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@48e64352{/storage/rdd,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7249dadf{/storage/rdd/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4362d7df{/environment,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@66238be2{/environment/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1c25b8a7{/executors,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@200606de{/executors/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@750fe12e{/executors/threadDump,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@f8908f6{/executors/threadDump/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3e587920{/static,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@77b7ffa4{/,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5ed190be{/api,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@41382722{/jobs/job/kill,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7dac3fd8{/stages/stage/kill,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://192.168.65.128:4040
2021-03-30 06:28:34 INFO  SparkContext:54 - Added JAR file:/home/yuanzhanglin/streamDM/scripts/../target/scala-2.11/streamdm-spark-streaming-_2.11-0.2.jar at spark://192.168.65.128:44865/jars/streamdm-spark-streaming-_2.11-0.2.jar with timestamp 1617110914021
2021-03-30 06:28:34 INFO  Executor:54 - Starting executor ID driver on host localhost
2021-03-30 06:28:34 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37021.
2021-03-30 06:28:34 INFO  NettyBlockTransferService:54 - Server created on 192.168.65.128:37021
2021-03-30 06:28:34 INFO  BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2021-03-30 06:28:34 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, 192.168.65.128, 37021, None)
2021-03-30 06:28:34 INFO  BlockManagerMasterEndpoint:54 - Registering block manager 192.168.65.128:37021 with 366.3 MB RAM, BlockManagerId(driver, 192.168.65.128, 37021, None)
2021-03-30 06:28:34 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, 192.168.65.128, 37021, None)
2021-03-30 06:28:34 INFO  BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, 192.168.65.128, 37021, None)
2021-03-30 06:28:34 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3703bf3c{/metrics/json,null,AVAILABLE,@Spark}
Exception in thread "main" java.lang.Exception: Class not found: or
	at com.github.javacliparser.ClassOption.cliStringToObject(ClassOption.java:132)
	at org.apache.spark.streamdm.streamDMJob$.main(streamDMJob.scala:55)
	at org.apache.spark.streamdm.streamDMJob.main(streamDMJob.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2021-03-30 06:28:34 INFO  SparkContext:54 - Invoking stop() from shutdown hook
2021-03-30 06:28:34 INFO  AbstractConnector:318 - Stopped Spark@21a5fd96{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2021-03-30 06:28:34 INFO  SparkUI:54 - Stopped Spark web UI at http://192.168.65.128:4040
2021-03-30 06:28:34 INFO  MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2021-03-30 06:28:34 INFO  MemoryStore:54 - MemoryStore cleared
2021-03-30 06:28:34 INFO  BlockManager:54 - BlockManager stopped
2021-03-30 06:28:34 INFO  BlockManagerMaster:54 - BlockManagerMaster stopped
2021-03-30 06:28:34 INFO  OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2021-03-30 06:28:34 INFO  SparkContext:54 - Successfully stopped SparkContext
2021-03-30 06:28:34 INFO  ShutdownHookManager:54 - Shutdown hook called
2021-03-30 06:28:34 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-27030bd4-3077-4318-8fec-a42efb3c4dbb
2021-03-30 06:28:34 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-d29803f8-d90b-491c-82e5-cc2599ac2e0f

[ELMERWANG]

@ELMERWANG Hi,I met the same problem as yours. Do you solve your problem?I want to get some suggestions to solve these problems. Thank you!

Infrastructure details
Java Version: 8
Scala Version: 2.11.8
Spark version: 2.3.2
OS version: Ubuntu
sbt version:0.13.18

When I run these Command lines
sbt package
./spark.sh

I get some errors:

java.lang.ArrayIndexOutOfBoundsException: 4
	at org.apache.spark.streamdm.core.DenseInstance.add(DenseInstance.scala:84)
	at org.apache.spark.streamdm.core.DenseInstance.add(DenseInstance.scala:26)
	at org.apache.spark.streamdm.classifiers.SGDLearner$$anonfun$train$1$$anonfun$3.apply(SGDLearner.scala:91)
	at org.apache.spark.streamdm.classifiers.SGDLearner$$anonfun$train$1$$anonfun$3.apply(SGDLearner.scala:85)
	at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
	at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
	at scala.collection.Iterator$class.foreach(Iterator.scala:893)
	at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
	at org.apache.spark.InterruptibleIterator.foldLeft(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce$class.aggregate(TraversableOnce.scala:214)
	at org.apache.spark.InterruptibleIterator.aggregate(InterruptibleIterator.scala:28)
	at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$21.apply(RDD.scala:1122)
	at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$21.apply(RDD.scala:1122)
	at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
	at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:109)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2021-03-30 06:25:41 ERROR Executor:91 - Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.ArrayIndexOutOfBoundsException: 4
	at org.apache.spark.streamdm.core.DenseInstance.add(DenseInstance.scala:84)
	at org.apache.spark.streamdm.core.DenseInstance.add(DenseInstance.scala:26)
	at org.apache.spark.streamdm.classifiers.SGDLearner$$anonfun$train$1$$anonfun$3.apply(SGDLearner.scala:91)
	at org.apache.spark.streamdm.classifiers.SGDLearner$$anonfun$train$1$$anonfun$3.apply(SGDLearner.scala:85)
	at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
	at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
	at scala.collection.Iterator$class.foreach(Iterator.scala:893)
	at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
	at org.apache.spark.InterruptibleIterator.foldLeft(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce$class.aggregate(TraversableOnce.scala:214)
	at org.apache.spark.InterruptibleIterator.aggregate(InterruptibleIterator.scala:28)
	at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$21.apply(RDD.scala:1122)
	at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$21.apply(RDD.scala:1122)
	at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
	at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:109)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

after I ran ./spark.sh "EvaluatePrequential -l (SGDLearner -l 0.01 -o LogisticLoss -r ZeroRegularizer) –s (FileReader –k 100 –d 60 –f ../data/mydata)",I got the follewing result:

/home/yuanzhanglin/local/spark-2.3.2/conf/spark-env.sh: line 71: syntax error near unexpected token `newline'
/home/yuanzhanglin/local/spark-2.3.2/conf/spark-env.sh: line 71: `<!--主节点IP-->'
2021-03-30 06:28:28 WARN  Utils:66 - Your hostname, ubuntu resolves to a loopback address: 127.0.1.1; using 192.168.65.128 instead (on interface ens33)
2021-03-30 06:28:28 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address
2021-03-30 06:28:32 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2021-03-30 06:28:32 INFO  SparkContext:54 - Running Spark version 2.3.2
2021-03-30 06:28:32 INFO  SparkContext:54 - Submitted application: streamDM
2021-03-30 06:28:32 INFO  SecurityManager:54 - Changing view acls to: yuanzhanglin
2021-03-30 06:28:32 INFO  SecurityManager:54 - Changing modify acls to: yuanzhanglin
2021-03-30 06:28:32 INFO  SecurityManager:54 - Changing view acls groups to: 
2021-03-30 06:28:32 INFO  SecurityManager:54 - Changing modify acls groups to: 
2021-03-30 06:28:32 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yuanzhanglin); groups with view permissions: Set(); users  with modify permissions: Set(yuanzhanglin); groups with modify permissions: Set()
2021-03-30 06:28:33 INFO  Utils:54 - Successfully started service 'sparkDriver' on port 44865.
2021-03-30 06:28:33 INFO  SparkEnv:54 - Registering MapOutputTracker
2021-03-30 06:28:33 INFO  SparkEnv:54 - Registering BlockManagerMaster
2021-03-30 06:28:33 INFO  BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2021-03-30 06:28:33 INFO  BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2021-03-30 06:28:33 INFO  DiskBlockManager:54 - Created local directory at /tmp/blockmgr-beb224af-932d-4d9f-9695-42a4a64c2287
2021-03-30 06:28:33 INFO  MemoryStore:54 - MemoryStore started with capacity 366.3 MB
2021-03-30 06:28:33 INFO  SparkEnv:54 - Registering OutputCommitCoordinator
2021-03-30 06:28:33 INFO  log:192 - Logging initialized @6306ms
2021-03-30 06:28:33 INFO  Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
2021-03-30 06:28:33 INFO  Server:419 - Started @6448ms
2021-03-30 06:28:33 INFO  AbstractConnector:278 - Started ServerConnector@21a5fd96{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2021-03-30 06:28:33 INFO  Utils:54 - Successfully started service 'SparkUI' on port 4040.
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@42a9a63e{/jobs,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5bd1ceca{/jobs/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@30c31dd7{/jobs/job,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@596df867{/jobs/job/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@c1fca1e{/stages,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@241a53ef{/stages/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@344344fa{/stages/stage,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@615f972{/stages/stage/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@285f09de{/stages/pool,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@73393584{/stages/pool/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@31500940{/storage,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1827a871{/storage/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@48e64352{/storage/rdd,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7249dadf{/storage/rdd/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4362d7df{/environment,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@66238be2{/environment/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1c25b8a7{/executors,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@200606de{/executors/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@750fe12e{/executors/threadDump,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@f8908f6{/executors/threadDump/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3e587920{/static,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@77b7ffa4{/,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5ed190be{/api,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@41382722{/jobs/job/kill,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7dac3fd8{/stages/stage/kill,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://192.168.65.128:4040
2021-03-30 06:28:34 INFO  SparkContext:54 - Added JAR file:/home/yuanzhanglin/streamDM/scripts/../target/scala-2.11/streamdm-spark-streaming-_2.11-0.2.jar at spark://192.168.65.128:44865/jars/streamdm-spark-streaming-_2.11-0.2.jar with timestamp 1617110914021
2021-03-30 06:28:34 INFO  Executor:54 - Starting executor ID driver on host localhost
2021-03-30 06:28:34 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37021.
2021-03-30 06:28:34 INFO  NettyBlockTransferService:54 - Server created on 192.168.65.128:37021
2021-03-30 06:28:34 INFO  BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2021-03-30 06:28:34 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, 192.168.65.128, 37021, None)
2021-03-30 06:28:34 INFO  BlockManagerMasterEndpoint:54 - Registering block manager 192.168.65.128:37021 with 366.3 MB RAM, BlockManagerId(driver, 192.168.65.128, 37021, None)
2021-03-30 06:28:34 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, 192.168.65.128, 37021, None)
2021-03-30 06:28:34 INFO  BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, 192.168.65.128, 37021, None)
2021-03-30 06:28:34 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3703bf3c{/metrics/json,null,AVAILABLE,@Spark}
Exception in thread "main" java.lang.Exception: Class not found: or
	at com.github.javacliparser.ClassOption.cliStringToObject(ClassOption.java:132)
	at org.apache.spark.streamdm.streamDMJob$.main(streamDMJob.scala:55)
	at org.apache.spark.streamdm.streamDMJob.main(streamDMJob.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2021-03-30 06:28:34 INFO  SparkContext:54 - Invoking stop() from shutdown hook
2021-03-30 06:28:34 INFO  AbstractConnector:318 - Stopped Spark@21a5fd96{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2021-03-30 06:28:34 INFO  SparkUI:54 - Stopped Spark web UI at http://192.168.65.128:4040
2021-03-30 06:28:34 INFO  MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2021-03-30 06:28:34 INFO  MemoryStore:54 - MemoryStore cleared
2021-03-30 06:28:34 INFO  BlockManager:54 - BlockManager stopped
2021-03-30 06:28:34 INFO  BlockManagerMaster:54 - BlockManagerMaster stopped
2021-03-30 06:28:34 INFO  OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2021-03-30 06:28:34 INFO  SparkContext:54 - Successfully stopped SparkContext
2021-03-30 06:28:34 INFO  ShutdownHookManager:54 - Shutdown hook called
2021-03-30 06:28:34 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-27030bd4-3077-4318-8fec-a42efb3c4dbb
2021-03-30 06:28:34 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-d29803f8-d90b-491c-82e5-cc2599ac2e0f

Unfortunately, no luck on fixing it, sorry..

[hmgomes]

Hi @hmgomes,

Thanks, may I ask which version should I download? With or without hadoop?

[ ] spark-2.4.5-bin-hadoop2.6.tgz 2020-02-02 20:27 220M

[ ] spark-2.4.5-bin-hadoop2.7.tgz 2020-02-02 20:27 222M

[ ] spark-2.4.5-bin-without-hadoop-scala-2.12.tgz 2020-02-02 20:27 139M

[ ] spark-2.4.5-bin-without-hadoop.tgz 2020-02-02 20:27 160M

[ ] spark-2.4.5.tgz 2020-02-02 20:27 15M

Hi @ELMERWANG

I have the version with Hadoop
"spark-2.4.5-bin-hadoop2.7"

Regards,
Heitor

[hmgomes]

Hi @LYZ1125

Spark version: 2.3.2
Try spark 2.4.5

Regards,
Heitor