一、Python環境准備
遠程服務器上Python版本要與本地想匹配，這里本地使用Anaconda來進行安裝，遠程服務器之間安裝。
wget --no-check-certificate https://www.python.org/ftp/python/3.6.7/Python-3.6.7.tgz
tar -zxvf Python-3.6.7.tgz
sudo ./configure --prefix=/usr/local/python3
make & make install
建立軟連接
mv /usr/bin/python /usr/bin/python.bak
ln -s /usr/local/python3/bin/python3.6 /usr/bin/python
- 1
- 2
- 3
- 4
- 5
- 6
- 7
本地之間下載Anaconda安裝包，會自動安裝相關環境。
二、配置PyCharm
在file->setting->project interpreter中新增一個python環境，
tool->Deployment->Configuration中新建一個SFTP
Mappings中Local path路徑正斜杠要修改一下改為反斜杠，否則可能會無法識別導致報錯
接下來需要在configuration中更改運行參數
因為是遠程環境，Environment Variables中需要指明服務器端python路徑，spark路徑，java路徑。
在file->setting->project interpreter下載安裝py4j，pyspark等包，否則會報錯
以下是配置中遇到的一些常見錯誤：
去上面搜索安裝pyspark-stubs
上面錯誤表示py4j版本不兼容，重新下載想匹配的版本
spark FileNotFoundError: [WinError 2] 系統找不到指定的文件，需要查詢edit configuration中python interpreter是否選擇正確
python3安裝完成后默認沒有安裝pip，需要自己去安裝，否則會出先pip未找到的錯誤
備份原來的源：輸入命令sudo cp /etc/apt/sources.list /etc/apt/sources_init.list
輸入命令sudo vi /etc/apt/sources.list
添加如下
deb http://mirrors.aliyun.com/ubuntu/ trusty main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ trusty-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ trusty-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ trusty-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ trusty-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ trusty main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-backports main restricted universe multiverse
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install python3-pip
sudo apt-get install python-pip