How to load java properties file and use in Spark ?

How to load java properties file and use in Spark ?

Asked on January 12, 2019 in Apache-spark.
Add Comment


  • 2 Answer(s)

    This could be the solution for this issue:

    props file : (mypropsfile.conf) // note: prefix your key with “spark.” else props will be ignored.

    spark.myapp.input /input/path
    spark.myapp.output /output/path
    

    launch this by

    $SPARK_HOME/bin/spark-submit --properties-file mypropsfile.conf
    

    Inside the code:

    sc.getConf.get("spark.driver.host") // localhost
    sc.getConf.get("spark.myapp.input") // /input/path
    sc.getConf.get("spark.myapp.output") // /output/path
    
    Answered on January 12, 2019.
    Add Comment

    spark.myapp.input
    spark.myapp.output

    If suppose you have a property which doesn’t start with spark:

    job.property:

    app.name=xyz

    $SPARK_HOME/bin/spark-submit --properties-file  job.property

    Spark will ignore all properties doesn’t have prefix spark. with message:

    Warning: Ignoring non-spark config property: app.name=test

    Answered on January 13, 2019.
    Add Comment


  • Your Answer

    By posting your answer, you agree to the privacy policy and terms of service.