Hadoop/Spark開發環境配置


修改hostname bogon 為localhost

查看ip地址

[training@bogon ~]$ sudo hostname localhost
[training@bogon ~]$ hostname 

執行結果

此時python 版本為2.7

將python版本升級至3.0及以上

一、            下載欲升級python版本 Python-3.4.5,將包放於本機與虛似機的共享目錄下,上傳至虛似機的opt目錄下

二、            解壓

 

三、閱讀README安裝步驟,根據實驗步驟向下執行

 

執行完成后出現Succssfully,恭喜你!安裝成功!

 

四、配置環境變量,讓Spark用到新版本的Python。編輯 ~training/.bashrc,添加如下內容,並在相應位置下載ipython3

五、   修改 spark-evn.sh 配置文件中的對應參數PYSPARK_PYTHON,此文件一般在 /etc/spark/conf 或者Spark 安裝目錄的conf 目錄下。

 

六、 Python和ipython都升級完成,但是spark的版本是1.6的,下面我們開始升級spark

此時我們已經將python升級到3.4.5,但是spark的版本沒有改變!!

升級spark

一、將共享文件夾中的要升級的spark版本到opt目錄下

 

二、解壓

[training@localhost opt]$ tar -zvxf spark-2.1.0-bin-hadoop2.6.tgz [training@localhost conf]$ ls /opt/ spark-2.1.0-bin-hadoop2.6.tgz spark-2.1.0-bin-hadoop2.6

三、修改新版Spark配置文件,將Hadoop、hive 配置文件拷貝到Spark配置文件目錄中:

四、安裝成功后的頁面

五、雖然下載成功了,但是對於一些日常代碼的編寫與保存不好,所以我們需要下載一個notebook

運行界面

安裝matplotlib 庫

 

測試matplotlib 庫

import matplotlib.pyplot as plt from matplotlib.sankey import Sankey fig = plt.figure(figsize=(8, 9)) ax = fig.add_subplot(1, 1, 1, xticks=[], yticks=[], title="Rankine Power Cycle: Example 8.6 from Moran and " "Shapiro\n\x22Fundamentals of Engineering Thermodynamics " "\x22, 6th ed., 2008") Hdot = [260.431, 35.078, 180.794, 221.115, 22.700, 142.361, 10.193, 10.210, 43.670, 44.312, 68.631, 10.758, 10.758, 0.017, 0.642, 232.121, 44.559, 100.613, 132.168] # MW sankey = Sankey(ax=ax, format='%.3G', unit=' MW', gap=0.5, scale=1.0/Hdot[0]) sankey.add(patchlabel='\n\nPump 1', rotation=90, facecolor='#37c959', flows=[Hdot[13], Hdot[6], -Hdot[7]], labels=['Shaft power', '', None], pathlengths=[0.4, 0.883, 0.25], orientations=[1, -1, 0]) sankey.add(patchlabel='\n\nOpen\nheater', facecolor='#37c959', flows=[Hdot[11], Hdot[7], Hdot[4], -Hdot[8]], labels=[None, '', None, None], pathlengths=[0.25, 0.25, 1.93, 0.25], orientations=[1, 0, -1, 0], prior=0, connect=(2, 1)) sankey.add(patchlabel='\n\nPump 2', facecolor='#37c959', flows=[Hdot[14], Hdot[8], -Hdot[9]], labels=['Shaft power', '', None], pathlengths=[0.4, 0.25, 0.25], orientations=[1, 0, 0], prior=1, connect=(3, 1)) sankey.add(patchlabel='Closed\nheater', trunklength=2.914, fc='#37c959', flows=[Hdot[9], Hdot[1], -Hdot[11], -Hdot[10]], pathlengths=[0.25, 1.543, 0.25, 0.25], labels=['', '', None, None], orientations=[0, -1, 1, -1], prior=2, connect=(2, 0)) sankey.add(patchlabel='Trap', facecolor='#37c959', trunklength=5.102, flows=[Hdot[11], -Hdot[12]], labels=['\n', None], pathlengths=[1.0, 1.01], orientations=[1, 1], prior=3, connect=(2, 0)) sankey.add(patchlabel='Steam\ngenerator', facecolor='#ff5555', flows=[Hdot[15], Hdot[10], Hdot[2], -Hdot[3], -Hdot[0]], labels=['Heat rate', '', '', None, None], pathlengths=0.25, orientations=[1, 0, -1, -1, -1], prior=3, connect=(3, 1)) sankey.add(patchlabel='\n\n\nTurbine 1', facecolor='#37c959', flows=[Hdot[0], -Hdot[16], -Hdot[1], -Hdot[2]], labels=['', None, None, None], pathlengths=[0.25, 0.153, 1.543, 0.25], orientations=[0, 1, -1, -1], prior=5, connect=(4, 0)) sankey.add(patchlabel='\n\n\nReheat', facecolor='#37c959', flows=[Hdot[2], -Hdot[2]], labels=[None, None], pathlengths=[0.725, 0.25], orientations=[-1, 0], prior=6, connect=(3, 0)) sankey.add(patchlabel='Turbine 2', trunklength=3.212, facecolor='#37c959', flows=[Hdot[3], Hdot[16], -Hdot[5], -Hdot[4], -Hdot[17]], labels=[None, 'Shaft power', None, '', 'Shaft power'], pathlengths=[0.751, 0.15, 0.25, 1.93, 0.25], orientations=[0, -1, 0, -1, 1], prior=6, connect=(1, 1)) sankey.add(patchlabel='Condenser', facecolor='#58b1fa', trunklength=1.764, flows=[Hdot[5], -Hdot[18], -Hdot[6]], labels=['', 'Heat rate', None], pathlengths=[0.45, 0.25, 0.883], orientations=[-1, 1, 0], prior=8, connect=(2, 0)) diagrams = sankey.finish() for diagram in diagrams: diagram.text.set_fontweight('bold') diagram.text.set_fontsize('10') for text in diagram.texts: text.set_fontsize('10') # Notice that the explicit connections are handled automatically, but the # implicit ones currently are not. The lengths of the paths and the trunks # must be adjusted manually, and that is a bit tricky.  plt.show()

運行結果

 


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM