How to run external jar functions in spark-shell

How to run external jar functions in spark-shell

Asked on January 12, 2019 in Apache-spark.
Add Comment


  • 3 Answer(s)

    Here the solution is, try by providing jars with argument.

    
    ./spark-shell --jars pathOfjarsWithCommaSeprated
    
    

    In spark-defaults.conf the following configuration is added but make sure that to remove template from end of spark-defaults

    
    spark.driver.extraClassPath pathOfJarsWithCommaSeprated
    
    
    Answered on January 12, 2019.
    Add Comment

    When spark-shell is entered, And if there is need to add a .jar to the classpath Then :require is used. as follows:

    scala> :require /path/to/file.jar
    Added '/path/to/file.jar' to classpath.
    
    Answered on January 12, 2019.
    Add Comment

    EXAMPLE

    Create a jar file of the spark application using the following command. Here, wordcount is the file name for jar file.

    jar -cvf wordcount.jar SparkWordCount*.class spark-core_2.10-1.3.0.jar/usr/local/spark/lib/spark-assembly-1.4.0-hadoop2.6.0.jar

     

     

    Submit the spark application using the following command −

    spark-submit --class SparkWordCount --master local wordcount.jar
    

    If it is executed successfully, then you will find the output given below. The OKletting in the following output is for user identification and that is the last line of the program. If you carefully read the following output, you will find different things, such as −

    • successfully started service ‘sparkDriver’ on port 42954
    • MemoryStore started with capacity 267.3 MB
    • Started SparkUI at http://192.168.1.217:4040
    • Added JAR file:/home/hadoop/piapplication/count.jar
    • ResultStage 1 (saveAsTextFile at SparkPi.scala:11) finished in 0.566 s
    • Stopped Spark web UI at http://192.168.1.217:4040
    • MemoryStore cleared

     

    When spark-shell is entered, And if there is need to add a .jar to the classpath Then :requireis used. as follows:

    1
    2
    scala> :require /path/to/file.jar
    Added '/path/to/file.jar' to classpath.
    Answered on January 13, 2019.
    Add Comment


  • Your Answer

    By posting your answer, you agree to the privacy policy and terms of service.