Spark2.1.0 niekompatybilne wersje Jackson 2.7.6

Question

Próbuję uruchomić prosty przykład zapłonową w IntelliJ, ale pojawia się błąd tak:

Exception in thread "main" java.lang.ExceptionInInitializerError 
at org.apache.spark.SparkContext.withScope(SparkContext.scala:701) 
at org.apache.spark.SparkContext.textFile(SparkContext.scala:819) 
at spark.test$.main(test.scala:19) 
at spark.test.main(test.scala) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:498) 
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) 
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.7.6 
at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64) 
at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19) 
at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:730) 
at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82) 
at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala) 

mam spróbować zaktualizować zależność Jackson, ale to wydaje się nie działać, zrobić to:

libraryDependencies += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7" 
libraryDependencies += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7" 

ale wciąż pojawiają się te same komunikaty o błędach, może ktoś mi pomóc naprawić ten błąd?

Oto iskra przykładowy kod:

object test { 
def main(args: Array[String]): Unit = { 
    if (args.length < 1) { 
     System.err.println("Usage: <file>") 
     System.exit(1) 
    } 

    val conf = new SparkConf() 
    val sc = new SparkContext("local","wordcount",conf) 
    val line = sc.textFile(args(0)) 

    line.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_+_).collect().foreach(println) 

    sc.stop() 
    } 
} 

I tu jest moje built.sbt:

name := "testSpark2" 

version := "1.0" 

scalaVersion := "2.11.8" 


libraryDependencies += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7" 
libraryDependencies += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7" 

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-mllib_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-repl_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-streaming-flume_2.10" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-network-shuffle_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-hive_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-streaming-flume-assembly_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-mesos_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-graphx_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-catalyst_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-launcher_2.11" % "2.1.0" 

Answer 1

Spark 2.1.0 zawiera com.fasterxml.jackson.core jako przechodnia zależność. Nie musimy więc włączać go do libraryDependencies.

Ale jeśli chcesz dodać inną wersję zależności com.fasterxml.jackson.core, musisz je zastąpić. Tak:

name := "testSpark2" 

version := "1.0" 

scalaVersion := "2.11.8" 


dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7" 
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7" 
dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.8.7" 

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-mllib_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-repl_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-streaming-flume_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-network-shuffle_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-hive_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-streaming-flume-assembly_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-mesos_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-graphx_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-catalyst_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-launcher_2.11" % "2.1.0" 

więc zmienić build.sbt jak ten powyżej i będzie działać zgodnie z oczekiwaniami.

Mam nadzieję, że to pomoże!

Answer 2

FYI. W moim przypadku używam iskry i kafka-streams w aplikacji, podczas gdy kafka-streams używa com.fasterxml.jackson.core 2.8.5. Dodawanie exclude jak poniżej stała się kwestią

(Gradle)

compile (group: "org.apache.kafka", name: "kafka-streams", version: "0.11.0.0"){ 
    exclude group:"com.fasterxml.jackson.core" 
}