Description
Describe the bug
Exception in thread "main" java.lang.AbstractMethodError
To Reproduce
./gradlew -q sparkSubmit -Dmain=com.salesforce.hw.OpTitanicSimple -Dargs="
pwd/src/main/resources/TitanicDataset/TitanicPassengersTrainData.csv"
Expected behavior
Successfully run this example
Logs or screenshots
Exception in thread "main" java.lang.AbstractMethodError
at org.apache.spark.ml.param.Params$class.$init$(params.scala:868)
at com.salesforce.op.stages.impl.tuning.Splitter.(Splitter.scala:47)
at com.salesforce.op.stages.impl.tuning.DataSplitter.(DataSplitter.scala:62)
at com.salesforce.op.stages.impl.tuning.DataSplitter$.apply(DataSplitter.scala:51)
at com.salesforce.op.stages.impl.classification.BinaryClassificationModelSelector$.withTrainValidationSplit$default$1(BinaryClassificationModelSelector.scala:211)
at com.salesforce.hw.OpTitanicSimple$.main(OpTitanicSimple.scala:135)
at com.salesforce.hw.OpTitanicSimple.main(OpTitanicSimple.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
FAILURE: Build failed with an exception.
- What went wrong:
Execution failed for task ':sparkSubmit'.
Process 'command '/home/user/spark_workspace/spark-project/bin/spark-submit'' finished with non-zero exit value 1
-
Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. -
Get more help at https://help.gradle.org
BUILD FAILED in 4s
Additional context
I am using version 0.7.0. My spark version is 2.4.5, and my hadoop version is 2.7.7. From what I saw on other issues, the cause of this error seems to be incompatible spark version, however my spark version should be correct.
I have tried using spark 2.4.7, but the results were the same.
I did run ./gradlew compileTestScala installDist in helloworld folder and TransmogrifAI folder before submitting the example, and I am under helloworld folder when I submitted the command.
I tried submitting it using $SPARK_HOME/bin/spark-submit --class com.salesforce.hw.titanic.OpTitanic build/libs/transmogrifai-helloworld-0.0.1-all.jar --run-type train --model-location /tmp/titanic-model --read-location Passenger=pwd/src/main/resources/TitanicDataset/TitanicPassengersTrainData.csv command, but the results were similar.
My java version is 1.8.0_161, and I have set JAVA_HOME correctly.
I did modifiy gradle/wrapper/gradle-wrapper.properties, so that I could use a local gradle of version 5.2.1. Also I have modified the repositories in build.gradle to two faster mirrors. I am not sure if this is related to this problem.
I am running this example on a pseudo-cluster on a freshly configured CentOS7 virtual machine. Is this problem caused by the lack of some necessary libraries?