How to load IPython shell with PySpark
When using version Spark < 1.2, The bin/pyspark is executed with an environmental variable IPYTHON=1.
export IPYTHON=1 /path/to/bin/pyspark
And this will also works in version Spark 1.2 and For setting Python environment for these versions is PYSPARK_DRIVER_PYTHON
export PYSPARK_DRIVER_PYTHON=ipython /path/to/bin/pyspark
In this ipython can be replaced with a path to the interpreter.
Alternatively the below command can be used.
# if you run your ipython with 2.7 version with ipython2 # whatever you use for launching ipython shell should come after '=' sign export PYSPARK_DRIVER_PYTHON=ipython2
With the SPARK_HOME directory: