java.lang.NoClassDefFoundError: Could not initialize class io.circe.Decoder$ after Shading

[clintoncheang]

Hi all, I tried to shade shapeless when I did a spark-submit using sbt-assembly. In my Spark Assembly scala file, I added assembly / assemblyShadeRules := Seq(ShadeRule.rename("shapeless.**" -> "new_shapeless.@1").inAll). But when I submit it to Spark 3.1.2, I got this error above. I am wondering am I adding the Shading correctly?

[dmykhailov]

Same here. Have you found any solution so far?

[joan38]

Same here when I try to upgrade to Spark 3.x. Works fine on Spark 2.4.4

[mattharder91]

Possibly related to this issue:

typelevel/cats#3628

my issue was that circe had a different version of cats than spark 3

shading should help:

assembly / assemblyShadeRules := Seq( ShadeRule.rename("shapeless.**" -> "new_shapeless.@1").inAll, ShadeRule.rename("cats.kernel.**" -> s"new_cats.kernel.@1").inAll ),