Python版本不一致導致的,設置下環境變量即可。
Exception: Python in worker has different version 2.6 than that in driver 2.7, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.
python --version 發現是2.6
echo $PYSPARK_PYTHON 值為空
echo $PYSPARK_DRIVER_PYTHON 值是ipython
which ipython發現路徑是
/home/work/envs/python_2.7.13/bin/ipython
設置環境變量,問題解決:
export PYSPARK_PYTHON=/home/work/envs/python_2.7.13/bin/ipython