How to set Master address for Spark examples from command line

How to set Master address for Spark examples from command line

Asked on December 10, 2018 in Apache-spark.
Add Comment


  • 3 Answer(s)

    This could be a good solution:

    Just set the Spark master from the command-line by adding the JVM parameter:

    -Dspark.master=spark://myhost:7077
    

     

    Answered on December 10, 2018.
    Add Comment

    Here .setMaster(…) is used when creating the SparkConf:

    val conf = new SparkConf().setAppName("Simple Application")
                              .setMaster("spark://myhost:7077")
    

    For the version in Spark 2.x +:

    val spark = SparkSession.builder()
                            .appName("app_name")
                            .getOrCreate()
    

    Command line (2.x) assuming local standalone cluster.

    spark-shell --master spark://localhost:7077
    

     

    Answered on December 10, 2018.
    Add Comment

    In this simply  setMaster(String master)

    The master URL to connect to, such as local to run locally with one thread, local[4] to run locally with 4 cores, or spark://master:7077 to run on a Spark standalone cluster.

    Answered on December 10, 2018.
    Add Comment


  • Your Answer

    By posting your answer, you agree to the privacy policy and terms of service.