同時打開了兩個SecureCRT的終端界面,其中一個已經進入了Spark-shell,在另一個SecureCRT界面中執行
"spark-shell --master yarn --deploy-mode client < cha02.scala" 時報錯,如下:
[root@master test_code]# spark-shell --master yarn --deploy-mode client <cha02.scala
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/05/07 20:41:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/05/07 20:41:37 WARN component.AbstractLifeCycle: FAILED ServerConnector@1953bc95{HTTP/1.1}{0.0.0.0:4040}: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
17/05/07 21:00:46 WARN util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
17/05/07 21:00:59 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
17/05/07 21:01:37 WARN metastore.ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Spark context Web UI available at http://192.168.1.200:4041
Spark context available as 'sc' (master = yarn, app id = application_1494142860645_0005).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.1.0
/_/
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_121)
Type in expressions to have them evaluated.
Type :help for more information.
scala>
這是由於啟動一個Spark context 時,SparkUI 默認會使用 4040 端口,當 4040 端口被占用時,則嘗試使用另外一個端口,即日志中的
Service 'SparkUI' could not bind on port 4040. Attempting port 4041.