啟動 ./spark-shell 命令報錯

當使用./spark-shell 命令報錯

Caused by: ERROR XJ040: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@476fde05, see the next exception for details.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(Unknown Source)
... 153 more
Caused by: ERROR XSDB6: Another instance of Derby may have already booted the database /usr/local/development/spark-2.1.1-bin-hadoop2.7/bin/metastore_db.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.privGetJBMSLockOnDB(Unknown Source)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.run(Unknown Source)
--------------------
Caused by: java.sql.SQLException: Unable to open a test connection to the given database. JDBC url = jdbc:derby:;databaseName=metastore_db;create=true, username = APP. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------
java.sql.SQLException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@476fde05, see the next exception for details.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
-------------------
Caused by: org.apache.hadoop.ipc.RemoteException: Cannot create directory /tmp/hive/root/5436b1aa-85e3-4512-b505-b0bdc7444e46. Name node is in safe mode.
The reported blocks 0 needs additional 9 blocks to reach the threshold 0.9990 of total blocks 9.
The number of live datanodes 0 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1327)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3895)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenode

 

拋出了很多錯誤，甚至我退出spark-shell，再次進入拋出的錯誤還不一樣，最后一個錯誤是Name node is in safe mode. 先解決這個
網上說這是hdfs的分布式文件系統的安全模式，當安全模式時文件系統中的內容不允許修改和刪除，直到安全模式結束，安全模式就是系統在檢查各個dataNode數據塊的有效性

bin/hadoop dfsadmin -safemode leave  //離開安全模式

 

用戶可以通過dfsadmin -safemode value 來操作安全模式，參數value的說明如下：

//enter - 進入安全模式

//leave - 強制NameNode離開安全模式

[root@node1 sbin]# hdfs dfsadmin -safemode leave
Safe mode is OFF

//get - 返回安全模式是否開啟的信息

[root@node1 sbin]# hdfs dfsadmin -safemode get
Safe mode is ON

 

//wait - 等待，一直到安全模式結束。

當離開安全模式再次spark-shell,拋出異常

Caused by: org.apache.derby.iapi.error.StandardException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@476fde05, see the next exception for details.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(Unknown Source)
... 153 more
Caused by: org.apache.derby.iapi.error.StandardException: Another instance of Derby may have already booted the database /usr/local/development/spark-2.1.1-bin-hadoop2.7/bin/metastore_db.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.privGetJBMSLockOnDB(Unknown Source)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.run(Unknown Source)

 

可以看到，這個異常是說在/usr/local/development/spark-2.1.1-bin-hadoop2.7/bin/ 中已經有另一個數據庫（Derby）實例了，我一直都是使用mysql數據庫，沒用過Derby數據庫，最后想想可能是我在安裝spark的時候，連接了hive,那個時候我的hive還
配置mysql數據庫，hive默認使用Derby數據庫，所以當我啟動spark連接hive的時候，就自動生成了Derby的實例。我看了'metastore_db' 文件夾的內容是Derby數據庫的配置

解決方法：
刪除/usr/local/development/spark-2.1.1-bin-hadoop2.7/bin/ metastore_db 文件夾。

重新啟動

./spark-shell

[root@node1 bin]# ./spark-shell 
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/07/05 00:13:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/07/05 00:13:29 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
17/07/05 00:13:29 WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException
17/07/05 00:13:40 WARN metastore.ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Spark context Web UI available at http://192.168.177.120:4040
Spark context available as 'sc' (master = local[*], app id = local-1499184787668).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.1
      /_/
         
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_131)
Type in expressions to have them evaluated.
Type :help for more information.

scala> 

scala> sc
res0: org.apache.spark.SparkContext = org.apache.spark.SparkContext@178c4480