1.hbase的改的會影響器他的組件的使用,
故而, 在修改 hadoop的任何組件后, 一定要記得其它的組件也能受到影響,
一下是我在將hadoop的集群改了之后 , 再次運行hbase的時候, 就會發生異常, 原因是在連接namenode的時候,發生連接不到, 就是因為我改了之前的配置,將端口號改了,
沒有去將配置文件在hbase中進行更新, 具體問如下:
##我今天起得hbase的集群時候,發生了
1.http://node3:60010/master-status
HTTP ERROR 503
Problem accessing /master-status. Reason:
Master not ready
Powered by Jetty://
2.http://node3:60030/rs-status
The RegionServer is initializing!
在hbase-hadoop-master-node3.log中, 報異常是:
2016-10-12 09:45:04,616 DEBUG [main-EventThread] master.ActiveMasterManager: A master is now available
2016-10-12 09:59:10,484 WARN [master:node3:60000] retry.RetryInvocationHandler: Exception while invoking class org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.setSafeMode. Not retrying because failovers (15) exceeded maximum allowed (15)
java.net.ConnectException: Call From node3/192.168.8.13 to node2:8020 failed on connection exception: java.net.ConnectException: 拒絕連接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
at org.apache.hadoop.ipc.Client.call(Client.java:1351)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy12.setSafeMode(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.setSafeMode(ClientNamenodeProtocolTranslatorPB.java:561)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy13.setSafeMode(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.setSafeMode(DFSClient.java:2146)
at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:983)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hbase.util.FSUtils.isInSafeMode(FSUtils.java:426)
at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:851)
at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:435)
at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:127)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:790)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:607)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.net.ConnectException: 拒絕連接
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:547)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:642)
at org.apache.hadoop.ipc.Client$Connection.access$2600(Client.java:314)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1399)
at org.apache.hadoop.ipc.Client.call(Client.java:1318)
... 25 more
2016-10-12 10:02:04,280 WARN [master:node3:60000] retry.RetryInvocationHandler: Exception while invoking class org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.setSafeMode. Not retrying because failovers (15) exceeded maximum allowed (15)
java.net.ConnectException: Call From node3/192.168.8.13 to node1:8020 failed on connection exception: java.net.ConnectException: 拒絕連接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
at org.apache.hadoop.ipc.Client.call(Client.java:1351)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy12.setSafeMode(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.setSafeMode(ClientNamenodeProtocolTranslatorPB.java:561)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy13.setSafeMode(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.setSafeMode(DFSClient.java:2146)
at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:983)
at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:967)
at org.apache.hadoop.hbase.util.FSUtils.isInSafeMode(FSUtils.java:432)
at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:851)
at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:435)
at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:127)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:790)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:607)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.net.ConnectException: 拒絕連接
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:547)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:642)
at org.apache.hadoop.ipc.Client$Connection.access$2600(Client.java:314)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1399)
at org.apache.hadoop.ipc.Client.call(Client.java:1318)
... 21 more
2016-10-12 10:02:04,282 FATAL [master:node3:60000] master.HMaster: Unhandled exception. Starting shutdown.
java.net.ConnectException: Call From node3/192.168.8.13 to node1:8020 failed on connection exception: java.net.ConnectException: 拒絕連接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
at org.apache.hadoop.ipc.Client.call(Client.java:1351)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy12.setSafeMode(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.setSafeMode(ClientNamenodeProtocolTranslatorPB.java:561)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy13.setSafeMode(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.setSafeMode(DFSClient.java:2146)
at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:983)
at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:967)
at org.apache.hadoop.hbase.util.FSUtils.isInSafeMode(FSUtils.java:432)
at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:851)
at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:435)
at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:127)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:790)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:607)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.net.ConnectException: 拒絕連接
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:547)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:642)
at org.apache.hadoop.ipc.Client$Connection.access$2600(Client.java:314)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1399)
at org.apache.hadoop.ipc.Client.call(Client.java:1318)
... 21 more
2016-10-12 10:02:04,352 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: HMaster Aborted
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:192)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:134)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2799)
而:hbase-hadoop-regionserver-node3.log中沒有錯誤, 但是有如下:
016-10-12 09:55:51,150 INFO [main] regionserver.ShutdownHook: Installed shutdown hook thread: Shutdownhook:regionserver60020
2016-10-12 10:00:45,947 DEBUG [LruStats #0] hfile.LruBlockCache: Total=3.26 MB, free=393.42 MB, max=396.68 MB, blocks=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN
2016-10-12 10:05:45,947 DEBUG [LruStats #0] hfile.LruBlockCache: Total=3.26 MB, free=393.42 MB, max=396.68 MB, blocks=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN
2016-10-12 10:10:45,948 DEBUG [LruStats #0] hfile.LruBlockCache: Total=3.26 MB, free=393.42 MB, max=396.68 MB, blocks=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN
2016-10-12 10:15:45,948 DEBUG [LruStats #0] hfile.LruBlockCache: Total=3.26 MB, free=393.42 MB, max=396.68 MB, blocks=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN
2016-10-12 10:20:45,948 DEBUG [LruStats #0] hfile.LruBlockCache: Total=3.26 MB, free=393.42 MB, max=396.68 MB, blocks=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN
2016-10-12 10:25:45,981 DEBUG [LruStats #0] hfile.LruBlockCache: Total=3.26 MB, free=393.42 MB, max=396.68 MB, blocks=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN
2016-10-12 10:30:45,947 DEBUG [LruStats #0] hfile.LruBlockCache: Total=3.26 MB, free=393.42 MB, max=396.68 MB, blocks=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN
這是為啥, 我找了很久 , 記得以前是如下原因引起的,
1.通過hostname設置為regionserver端鏈接時指定的hostname即可 , 這個就是域名和主機不一致導致的, (hbase分布式部署問題之org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master)
--->很快我就排除了, 因為我的hadoop的集群啟動正常,故這個排除
2.時間不一致引起的,類似這個人提到(時間同步上有問題，查了一下RegionServer和Master兩台機的時間，果然不一致。由於之前把主節點的時間同步而未同步子節點的時間 ,)
我在各台機子上都就行查看,用:date;hwclock -r 進行查看 , 沒有問題,(如何 一步到位修改linux系統時間 date -s "20091112 18:30:50" &&hwclock --systohc)
3.hbase 未成功啟動 1、關閉防火牆：service iptables stop ,我有排除了, 原因是我的hadoop集群是啟動的, 沒有這個問題
最后, 我無奈去繼續找, 去看看我之前配置hbase的配置, 是我之前有其他問題, 因為我早一個多月前試了試hbase,是沒有問題,還有簡單編程的嘗試的,於是
查看搭建：分布式的hbase 的文檔,
1.上傳hbase安裝包
2.解壓
3.配置hbase集群，要修改3個文件（首先zk集群已經安裝好了）
注意：要把hadoop的hdfs-site.xml和core-site.xml 放到hbase/conf下
3.1修改hbase-env.sh
export JAVA_HOME=/usr/java/jdk1.7.0_55
//告訴hbase使用外部的zk
export HBASE_MANAGES_ZK=false
vim hbase-site.xml
<configuration>
<!-- 指定hbase在HDFS上存儲的路徑,其中：ns1是在hadoop的core-site.xml中配置的 -->
<property>
<name>hbase.rootdir</name>
<value>hdfs://ns1/hbase</value>
</property>
<!-- 指定hbase是分布式的 -->
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<!-- 指定zk的地址，多個用“,”分割 -->
<property>
<name>hbase.zookeeper.quorum</name>
<value>itcast04:2181,itcast05:2181,itcast06:2181</value>
</property>
</configuration>
vim regionservers
itcast03
itcast04
itcast05
itcast06
3.2拷貝hbase到其他節點
scp -r /itcast/hbase-0.96.2-hadoop2/ itcast02:/itcast/
scp -r /itcast/hbase-0.96.2-hadoop2/ itcast03:/itcast/
scp -r /itcast/hbase-0.96.2-hadoop2/ itcast04:/itcast/
scp -r /itcast/hbase-0.96.2-hadoop2/ itcast05:/itcast/
scp -r /itcast/hbase-0.96.2-hadoop2/ itcast06:/itcast/
4.將配置好的HBase拷貝到每一個節點並同步時間。
5.啟動所有的hbase
分別啟動zk
./zkServer.sh start
啟動hbase集群
start-dfs.sh
啟動hbase，在主節點上運行：
start-hbase.sh
6.通過瀏覽器訪問hbase管理頁面
192.168.1.201:60010
7.為保證集群的可靠性，要啟動多個HMaster
hbase-daemon.sh start master
到第三部的時候,我就發現了問題, 為啥, 原因是我之前改過那個hadoop的dfs.namenode.rpc-address.myserver.nn1的端口 <value>node1:8020</value>,
改為了
<property>
<name>dfs.namenode.rpc-address.myserver.nn2</name>
<value>node2:9000</value>
</property>
,這時候,
Call From node3/192.168.8.13 to node1:8020 failed on connection exception: java.net.ConnectException: 拒絕連接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
故,問題就在這里
配置問題的問題