azkaban-plugins does not recognize the directory of HDFS-version-azkaban-3.90.0 #301
Description
Hello, I use the Azkaban plugin to perform the task of spark on yarn, And I configured the spark plugin and introduced all the environment information and dependencies,But still report an error. Azkaban can't find my HDFS directory correctly.
It prefixes my HDFS directory with my Azkaban installation directory.
Hope to get your help!
Thank you very much!
The error report is as follows:
java.lang.Exception: Job set up failed: execution jar is suppose to be in this folder, but the folder doesn't exist: /apps/azkaban/exec-server/bin/executions/10/hdfs:///spark-jar/test
at azkaban.jobExecutor.ProcessJob.handleError(ProcessJob.java:434)
at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:208)
at azkaban.jobtype.AbstractHadoopJavaProcessJob.run(AbstractHadoopJavaProcessJob.java:50)
at azkaban.jobtype.HadoopSparkJob.run(HadoopSparkJob.java:307)
at azkaban.execapp.JobRunner.runJob(JobRunner.java:830)
at azkaban.execapp.JobRunner.doRun(JobRunner.java:607)
at azkaban.execapp.JobRunner.run(JobRunner.java:568)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: execution jar is suppose to be in this folder, but the folder doesn't exist: /apps/azkaban/exec-server/bin/executions/10/hdfs:///spark-jar/test
at azkaban.jobtype.HadoopJobUtils.resolveExecutionJarName(HadoopJobUtils.java:319)
at azkaban.jobtype.HadoopSparkJob.executionJarHelper(HadoopSparkJob.java:219)
at azkaban.jobtype.HadoopSparkJob.testableGetMainArguments(HadoopSparkJob.java:193)
at azkaban.jobtype.HadoopSparkJob.getMainArguments(HadoopSparkJob.java:406)
at azkaban.jobExecutor.JavaProcessJob.createCommandLine(JavaProcessJob.java:73)
at azkaban.jobExecutor.JavaProcessJob.getCommandList(JavaProcessJob.java:62)
at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:206)
... 10 more
22-10-2020 15:51:17 CST ray_spark_test ERROR - Job run failed!
java.lang.Exception: Job set up failed: execution jar is suppose to be in this folder, but the folder doesn't exist: /apps/azkaban/exec-server/bin/executions/10/hdfs:///spark-jar/test
at azkaban.jobExecutor.ProcessJob.handleError(ProcessJob.java:434)
at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:208)
at azkaban.jobtype.AbstractHadoopJavaProcessJob.run(AbstractHadoopJavaProcessJob.java:50)
at azkaban.jobtype.HadoopSparkJob.run(HadoopSparkJob.java:307)
at azkaban.execapp.JobRunner.runJob(JobRunner.java:830)
at azkaban.execapp.JobRunner.doRun(JobRunner.java:607)
at azkaban.execapp.JobRunner.run(JobRunner.java:568)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: execution jar is suppose to be in this folder, but the folder doesn't exist: /apps/azkaban/exec-server/bin/executions/10/hdfs:///spark-jar/test
at azkaban.jobtype.HadoopJobUtils.resolveExecutionJarName(HadoopJobUtils.java:319)
at azkaban.jobtype.HadoopSparkJob.executionJarHelper(HadoopSparkJob.java:219)
at azkaban.jobtype.HadoopSparkJob.testableGetMainArguments(HadoopSparkJob.java:193)
at azkaban.jobtype.HadoopSparkJob.getMainArguments(HadoopSparkJob.java:406)
at azkaban.jobExecutor.JavaProcessJob.createCommandLine(JavaProcessJob.java:73)
at azkaban.jobExecutor.JavaProcessJob.getCommandList(JavaProcessJob.java:62)
at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:206)
... 10 more