How to change memory per node for apache spark worker
While using 1.0.0+ and using spark-shell or spark-submit, The –executor-memory option is used.
spark-shell --executor-memory 8G ...
For the version 0.9.0 and under:
While starting the shell change the memory. And need to modify the spark-shell script so that it would carry command line arguments through as arguments for the underlying java application. In particular:
OPTIONS="[email protected]" ... $FWDIR/bin/spark-class $OPTIONS org.apache.spark.repl.Main "[email protected]"
Then run our spark shell as follows:
When configuring it for a standalone jar, I set the system property programmatically before creating the spark context and pass the value in as a command line argument (I can make it shorter than the long winded system props then).
Note: When we have 2 nodes with 2GB and one with 6GB. The memory you can use will be limited to the smallest node – so here 2GB.
For the version Spark 1.1.1, to set the Max Memory of workers. in conf/spark.env.sh,
Just copy the template file when the config file is not used.
cp conf/spark-env.sh.template conf/spark-env.sh
Then make the change and don’t forget to source it
Here ipython notebook server is used to connect to spark. To increase the memory for executor.
Try this method:
from pyspark import SparkContext from pyspark.conf import SparkConf conf = SparkConf() conf.setMaster(CLUSTER_URL).setAppName('ipython-notebook').set("spark.executor.memory", "2g") sc = SparkContext(conf=conf)