Mango submit fails with parquet class not found

[SatyaGsk]

I am using mango-distribution-0.0.1 on Hadoop 2.6.0-cdh5.14.4/spark version 2.2.0/Scala version 2.11.8
Mango submit fails with parquet class not found. I tried to pass parquet class in CLI but it not helping as shown by messages below. I also included how data files layout on HDFS.

[sm@bluedata750 bin]$ ./mango-submit --packages org.apache.parquet:parquet-hadoop:1.8.2 /user/sm/hg19.17.2bit -genes /user/sm/ensGene.bb -reads /user/sm/chr17.7500000-7515000.sam.adam -variants /user/sm/chr17.adam -show_genotypes -discover Using spark-submit=/usr/bin/spark2-submit
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/parquet/hadoop/metadata/CompressionCodecName
at org.bdgenomics.utils.cli.ParquetArgs$class.$init$(ParquetArgs.scala:40)
at org.bdgenomics.mango.cli.VizReadsArgs.(VizReads.scala:252)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at org.bdgenomics.utils.cli.Args4j$.apply(Args4j.scala:34)
at org.bdgenomics.mango.cli.VizReads$.apply(VizReads.scala:196)
at org.bdgenomics.utils.cli.BDGCommandCompanion$class.main(BDGCommand.scala:33)
at org.bdgenomics.mango.cli.VizReads$.main(VizReads.scala:125)
at org.bdgenomics.mango.cli.VizReads.main(VizReads.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.parquet.hadoop.metadata.CompressionCodecName
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 21 more
[sm@bluedata750 bin]$ ./mango-submit /user/sm/hg19.17.2bit -genes /user/sm/ensGene.bb -reads /user/sm/chr17.7500000-7515000.sam.adam -variants /user/sm/chr17.adam -show_genotypes -discover
Using spark-submit=/usr/bin/spark2-submit
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/parquet/hadoop/metadata/CompressionCodecName
at org.bdgenomics.utils.cli.ParquetArgs$class.$init$(ParquetArgs.scala:40)
at org.bdgenomics.mango.cli.VizReadsArgs.(VizReads.scala:252)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at org.bdgenomics.utils.cli.Args4j$.apply(Args4j.scala:34)
at org.bdgenomics.mango.cli.VizReads$.apply(VizReads.scala:196)
at org.bdgenomics.utils.cli.BDGCommandCompanion$class.main(BDGCommand.scala:33)
at org.bdgenomics.mango.cli.VizReads$.main(VizReads.scala:125)
at org.bdgenomics.mango.cli.VizReads.main(VizReads.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.parquet.hadoop.metadata.CompressionCodecName
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 21 more
[sm@bluedata750 bin]$ hadoop fs -ls /user/sm
Found 7 items
drwxrwx---+ - sm supergroup 0 2018-10-30 21:27 /user/sm/.sparkStaging
-rw-rw----+ 3 sm supergroup 5440756334 2018-10-30 21:03 /user/sm/LN44765.bed
drwxrwx---+ - sm supergroup 0 2018-11-09 07:19 /user/sm/chr17.7500000-7515000.sam.adam
drwxrwx---+ - sm supergroup 0 2018-10-30 21:27 /user/sm/chr17.adam
-rw-rw----+ 3 sm supergroup 91866 2018-10-17 10:28 /user/sm/chr17.vcf
-rw-rw----+ 3 sm supergroup 3344732 2018-11-09 07:28 /user/sm/ensGene.bb
-rw-rw----+ 3 sm supergroup 21252941 2018-11-09 07:21 /user/sm/hg19.17.2bit

[akmorrow13]

Please try

./mango-submit --packages org.apache.parquet:parquet-hadoop:1.8.2 -- /user/sm/hg19.17.2bit -genes /user/sm/ensGene.bb -reads /user/sm/chr17.7500000-7515000.sam.adam -variants /user/sm/chr17.adam -show_genotypes