Spark2.1.0 incompatible Jackson versions 2.7.6

Spark2.1.0 incompatible Jackson versions 2.7.6

Asked on December 25, 2018 in Apache-spark.
Add Comment


  • 3 Answer(s)

    In the Spark version 2.1.0 contains com.fasterxml.jackson.core as transitive dependency. So, here no need to include then in libraryDependencies.

    When we need to add a different com.fasterxml.jackson.core dependencies’ version after that we need to override them. Like this:

    name := "testSpark2"
     
    version := "1.0"
     
    scalaVersion := "2.11.8"
     
    dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7"
    dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7"
    dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.8.7"
     
    libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0"
    libraryDependencies += "org.apache.spark" % "spark-mllib_2.11" % "2.1.0"
    libraryDependencies += "org.apache.spark" % "spark-repl_2.11" % "2.1.0"
    libraryDependencies += "org.apache.spark" % "spark-streaming-flume_2.11" % "2.1.0"
    libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.1.0"
    libraryDependencies += "org.apache.spark" % "spark-network-shuffle_2.11" % "2.1.0"
    libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.1.0"
    libraryDependencies += "org.apache.spark" % "spark-hive_2.11" % "2.1.0"
    libraryDependencies += "org.apache.spark" % "spark-streaming-flume-assembly_2.11" % "2.1.0"
    libraryDependencies += "org.apache.spark" % "spark-mesos_2.11" % "2.1.0"
    libraryDependencies += "org.apache.spark" % "spark-graphx_2.11" % "2.1.0"
    libraryDependencies += "org.apache.spark" % "spark-catalyst_2.11" % "2.1.0"
    libraryDependencies += "org.apache.spark" % "spark-launcher_2.11" % "2.1.0"
    
    Answered on December 25, 2018.
    Add Comment

    Note: Here we are using spark and kafka-streams in the app, while kafka-streams uses com.fasterxml.jackson.core 2.8.5. Adding exclude as below can the problem.

    (gradle)

    compile (group: "org.apache.kafka", name: "kafka-streams", version: "0.11.0.0"){
         exclude group:"com.fasterxml.jackson.core"
    }
    

     

    Answered on December 26, 2018.
    Add Comment

    Here the resolutionStrategy can be used. answer with Gradle.

    configurations {
     
        all {
            resolutionStrategy {
                force 'com.fasterxml.jackson.core:jackson-core:2.4.4', 'com.fasterxml.jackson.core:jackson-databind:2.4.4', 'com.fasterxml.jackson.core:jackson-annotations:2.4.4'
            }
        }
    }
    
    Answered on December 26, 2018.
    Add Comment


  • Your Answer

    By posting your answer, you agree to the privacy policy and terms of service.