由於需要首次手動安裝sbt,需要聯網,故將虛擬機的網絡適配器模式設置為"橋接模式",這樣就可以和互聯網相連接。
但是后面執行"spark-shell --master yarn --deploy-mode client" 命令時,無法啟動,一直停留在中間狀態不動,
如下:
[root@master test_code]# spark-shell --master yarn --deploy-mode client
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/05/07 18:07:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platfor
m... using builtin-java classes where applicable
后來突然想起來,虛擬機的網絡適配器模式沒有更改過來,重新設置為"僅主機模式" 后(這是由於,安裝Hadoop集群
時,幾個VMware虛擬機都是使用的"僅主機模式"),spark-shell 正常啟動,如下:
[root@master master]# spark-shell --master yarn --deploy-mode client
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/05/07 18:30:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/05/07 18:30:28 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
17/05/07 18:31:15 WARN metastore.ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Spark context Web UI available at http://192.168.1.200:4040
Spark context available as 'sc' (master = yarn, app id = application_1494142860645_0001).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.1.0
/_/
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_121)
Type in expressions to have them evaluated.
Type :help for more information.
scala>
BTW:
通過查看active NameNode 狀態,可以看到,集群狀態不正常,無法與其他節點通信,造成block丟失。如下WebUI 上所示:
There are 35 missing blocks. The following files may be corrupted:
blk_1073741946 /opt/hadoop/out_wordcount2/part-r-00000
blk_1073741945 /opt/hadoop/out_wordcount/part-r-00000
blk_1073741933 /opt/hadoop/input/README.txt
blk_1073741931 /hbase/MasterProcWALs/state-00000000000000000018.log
blk_1073741930 /hbase/oldWALs/slave4%2C16020%2C1489814171196.meta.1489821379742.meta
blk_1073741929 /hbase/oldWALs/slave3%2C16020%2C1489814171011.1489821375778
blk_1073741928 /hbase/oldWALs/slave4%2C16020%2C1489814171196.1489821375970
blk_1073741927 /hbase/oldWALs/slave5%2C16020%2C1489814170009.1489821374298
blk_1073741920 /hbase/data/hbase/meta/1588230740/info/8e011b40156f4eeab4e83caf63ee1d23
blk_1073741847 /hbase/data/hbase/namespace/3792ee8c4881d96201d73a19d76aa598/info/cc6c271a546248419df4c0988d191b4d
blk_1073741846 /hbase/data/hbase/namespace/3792ee8c4881d96201d73a19d76aa598/.regioninfo
blk_1073741845 /hbase/data/hbase/namespace/.tabledesc/.tableinfo.0000000001
blk_1073741968 /linkage/block_9.csv
blk_1073741967 /linkage/block_8.csv
blk_1073741839 /hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001
blk_1073741966 /linkage/block_7.csv
blk_1073741838 /hbase/data/hbase/meta/1588230740/.regioninfo
blk_1073741837 /hbase/hbase.id
blk_1073741965 /linkage/block_6.csv
blk_1073741836 /hbase/hbase.version
blk_1073741964 /linkage/block_5.csv
blk_1073741963 /linkage/block_4.csv
blk_1073741962 /linkage/block_3.csv
blk_1073741961 /linkage/block_2.csv
blk_1073741833 /tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1489568197327_0001.summary
blk_1073741960 /linkage/block_1.csv
blk_1073741832 /out/part-r-00000
blk_1073741959 /linkage/block_10.csv
blk_1073741958 /sogou/SogouQ1.txt
blk_1073741957 /tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1492443686126_0001_conf.xml
blk_1073741956 /tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1492443686126_0001-1492443717631-root-word+count-1492443749993-1-1-SUCCEEDED-default-1492443728809.jhist
blk_1073741955 /tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1492443686126_0001.summary
blk_1073741954 /opt/hadoop/out_1/part-r-00000
blk_1073741825 /word
Please check the logs or run fsck in order to identify the missing blocks. See the Hadoop FAQ for common causes and potential solutions.