淺談hbase表中數據導出導入(也就是備份)


轉自:http://blog.chinaunix.net/xmlrpc.php?r=blog/article&uid=23916356&id=3321832

最近因為生產環境hbase中某張表的數據要導出到測試環境(數據不多,大概200W條左右),如果用程序掉接口導入的話太慢,所以考慮直接用 hbase的功能來導入導出。因為此次是實驗,所以我在正式環境建了一張小表,只有兩條數據,目的是將它導入到一張新表中(空表,但是表結構一樣)
hbase(main):004:0> scan 'xyz'
ROW                   COLUMN+CELL                                              
 10000                column=cf1:val, timestamp=1345598242644, value=china     
 20000                column=cf1:val, timestamp=1345598283332, value=zengzhunzhu
                      n                                                        
2 row(s) in 0.0350 seconds
開始導出:
[hadoop@master ~]$ hbase/bin/hbase org.apache.hadoop.hbase.mapreduce.Driver expo
rt xyz file:///home/hadoop/xyz
12/08/22 10:12:07 INFO mapreduce.Export: verisons=1, starttime=0, endtime=9223372036854775807
12/08/22 10:12:08 DEBUG mapreduce.TableMapReduceUtil: New JarFinder: org.apache.hadoop.util.JarFinder.getJar not available.  Using old findContainingJar
12/08/22 10:12:08 DEBUG mapreduce.TableMapReduceUtil: New JarFinder: org.apache.hadoop.util.JarFinder.getJar not available.  Using old findContainingJar
12/08/22 10:12:08 DEBUG mapreduce.TableMapReduceUtil: New JarFinder: org.apache.hadoop.util.JarFinder.getJar not available.  Using old findContainingJar
12/08/22 10:12:08 DEBUG mapreduce.TableMapReduceUtil: New JarFinder: org.apache.hadoop.util.JarFinder.getJar not available.  Using old findContainingJar
12/08/22 10:12:08 DEBUG mapreduce.TableMapReduceUtil: New JarFinder: org.apache.hadoop.util.JarFinder.getJar not available.  Using old findContainingJar
12/08/22 10:12:08 DEBUG mapreduce.TableMapReduceUtil: New JarFinder: org.apache.hadoop.util.JarFinder.getJar not available.  Using old findContainingJar
12/08/22 10:12:08 DEBUG mapreduce.TableMapReduceUtil: New JarFinder: org.apache.hadoop.util.JarFinder.getJar not available.  Using old findContainingJar
12/08/22 10:12:08 DEBUG mapreduce.TableMapReduceUtil: New JarFinder: org.apache.hadoop.util.JarFinder.getJar not available.  Using old findContainingJar
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/hbase/lib/slf4j-log4j12-1.5.8.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/hadoop/lib/slf4j-log4j12-1.4.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
12/08/22 10:12:09 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.3-1240972, built on 02/06/2012 10:48 GMT
12/08/22 10:12:09 INFO zookeeper.ZooKeeper: Client environment:host.name=master
12/08/22 10:12:09 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_14
12/08/22 10:12:09 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
12/08/22 10:12:09 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.6.0_14/jre
12/08/22 10:12:09 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/home/hadoop/hbase/bin/../conf:/usr/java/jdk1.6.0_14/lib/tools.jar:/home/hadoop/hbase:/home/hadoop/hbase/hbase-0.92.1.jar:/home/hadoop/hbase/hbase-0.92.1-tests.jar:/home/hadoop/hbase/lib/activation-1.1.jar:/home/hadoop/hbase/lib/asm-3.1.jar:/home/hadoop/hbase/lib/avro-1.5.3.jar:/home/hadoop/hbase/lib/avro-ipc-1.5.3.jar:/home/hadoop/hbase/lib/commons-beanutils-1.7.0.jar:/home/hadoop/hbase/lib/commons-beanutils-core-1.8.0.jar:/home/hadoop/hbase/lib/commons-cli-1.2.jar:/home/hadoop/hbase/lib/commons-codec-1.4.jar:/home/hadoop/hbase/lib/commons-collections-3.2.1.jar:/home/hadoop/hbase/lib/commons-configuration-1.6.jar:/home/hadoop/hbase/lib/commons-digester-1.8.jar:/home/hadoop/hbase/lib/commons-el-1.0.jar:/home/hadoop/hbase/lib/commons-httpclient-3.1.jar:/home/hadoop/hbase/lib/commons-lang-2.5.jar:/home/hadoop/hbase/lib/commons-logging-1.1.1.jar:/home/hadoop/hbase/lib/commons-math-2.1.jar:/home/hadoop/hbase/lib/commons-net-1.4.1.jar:/home/hadoop/hbase/lib/core-3.1.1.jar:/home/hadoop/hbase/lib/guava-r09.jar:/home/hadoop/hbase/lib/hadoop-core-1.0.0.jar:/home/hadoop/hbase/lib/high-scale-lib-1.1.1.jar:/home/hadoop/hbase/lib/httpclient-4.0.1.jar:/home/hadoop/hbase/lib/httpcore-4.0.1.jar:/home/hadoop/hbase/lib/jackson-core-asl-1.5.5.jar:/home/hadoop/hbase/lib/jackson-jaxrs-1.5.5.jar:/home/hadoop/hbase/lib/jackson-mapper-asl-1.5.5.jar:/home/hadoop/hbase/lib/jackson-xc-1.5.5.jar:/home/hadoop/hbase/lib/jamon-runtime-2.3.1.jar:/home/hadoop/hbase/lib/jasper-compiler-5.5.23.jar:/home/hadoop/hbase/lib/jasper-runtime-5.5.23.jar:/home/hadoop/hbase/lib/jaxb-api-2.1.jar:/home/hadoop/hbase/lib/jaxb-impl-2.1.12.jar:/home/hadoop/hbase/lib/jersey-core-1.4.jar:/home/hadoop/hbase/lib/jersey-json-1.4.jar:/home/hadoop/hbase/lib/jersey-server-1.4.jar:/home/hadoop/hbase/lib/jettison-1.1.jar:/home/hadoop/hbase/lib/jetty-6.1.26.jar:/home/hadoop/hbase/lib/jetty-util-6.1.26.jar:/home/hadoop/hbase/lib/jruby-complete-1.6.5.jar:/home/hadoop/hbase/lib/jsp-2.1-6.1.14.jar:/home/hadoop/hbase/lib/jsp-api-2.1-6.1.14.jar:/home/hadoop/hbase/lib/libthrift-0.7.0.jar:/home/hadoop/hbase/lib/log4j-1.2.16.jar:/home/hadoop/hbase/lib/netty-3.2.4.Final.jar:/home/hadoop/hbase/lib/protobuf-java-2.4.0a.jar:/home/hadoop/hbase/lib/servlet-api-2.5-6.1.14.jar:/home/hadoop/hbase/lib/servlet-api-2.5.jar:/home/hadoop/hbase/lib/slf4j-api-1.5.8.jar:/home/hadoop/hbase/lib/slf4j-log4j12-1.5.8.jar:/home/hadoop/hbase/lib/snappy-java-1.0.3.2.jar:/home/hadoop/hbase/lib/stax-api-1.0.1.jar:/home/hadoop/hbase/lib/velocity-1.7.jar:/home/hadoop/hbase/lib/xmlenc-0.52.jar:/home/hadoop/hbase/lib/zookeeper-3.4.3.jar:/home/hadoop/hadoop/libexec/../conf:/usr/java/jdk1.6.0_14/lib/tools.jar:/home/hadoop/hadoop/libexec/..:/home/hadoop/hadoop/libexec/../hadoop-core-1.0.3.jar:/home/hadoop/hadoop/libexec/../lib/asm-3.2.jar:/home/hadoop/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/home/hadoop/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/home/hadoop/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/home/hadoop/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/home/hadoop/hadoop/libexec/../lib/commons-cli-1.2.jar:/home/hadoop/hadoop/libexec/../lib/commons-codec-1.4.jar:/home/hadoop/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/home/hadoop/hadoop/libexec/../lib/commons-configuration-1.6.jar:/home/hadoop/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/home/hadoop/hadoop/libexec/../lib/commons-digester-1.8.jar:/home/hadoop/hadoop/libexec/../lib/commons-el-1.0.jar:/home/hadoop/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/home/hadoop/hadoop/libexec/../lib/commons-io-2.1.jar:/home/hadoop/hadoop/libexec/../lib/commons-lang-2.4.jar:/home/hadoop/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/home/hadoop/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/home/hadoop/hadoop/libexec/../lib/commons-math-2.1.jar:/home/hadoop/hadoop/libexec/../lib/commons-net-1.4.1.jar:/home/hadoop/hadoop/libexec/../lib/core-3.1.1.jar:/home/hadoop/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.3.jar:/home/hadoop/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.3.jar:/home/hadoop/hadoop/libexec/../lib/hadoop-thriftfs-1.0.3.jar:/home/hadoop/hadoop/libexec/../lib/hbase-0.92.1.jar:/home/hadoop/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/home/hadoop/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/home/hadoop/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/home/hadoop/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/home/hadoop/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/home/hadoop/hadoop/libexec/../lib/jdeb-0.8.jar:/home/hadoop/hadoop/libexec/../lib/jersey-core-1.8.jar:/home/hadoop/hadoop/libexec/../lib/jersey-json-1.8.jar:/home/hadoop/hadoop/libexec/../lib/jersey-server-1.8.jar:/home/hadoop/hadoop/libexec/../lib/jets3t-0.6.1.jar:/home/hadoop/hadoop/libexec/../lib/jetty-6.1.26.jar:/home/hadoop/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/home/hadoop/hadoop/libexec/../lib/jsch-0.1.42.jar:/home/hadoop/hadoop/libexec/../lib/junit-4.5.jar:/home/hadoop/hadoop/libexec/../lib/kfs-0.2.2.jar:/home/hadoop/hadoop/libexec/../lib/log4j-1.2.15.jar:/home/hadoop/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/home/hadoop/hadoop/libexec/../lib/oro-2.0.8.jar:/home/hadoop/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/home/hadoop/hadoop/libexec/../lib/slf4j-api-1.4.3.jar:/home/hadoop/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar:/home/hadoop/hadoop/libexec/../lib/xmlenc-0.52.jar:/home/hadoop/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/home/hadoop/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar
12/08/22 10:12:09 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/home/hadoop/hadoop/libexec/../lib/native/Linux-amd64-64:/home/hadoop/hbase/lib/native/Linux-amd64-64
12/08/22 10:12:09 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
12/08/22 10:12:09 INFO zookeeper.ZooKeeper: Client environment:java.compiler=
12/08/22 10:12:09 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
12/08/22 10:12:09 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
12/08/22 10:12:09 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.9-89.ELsmp
12/08/22 10:12:09 INFO zookeeper.ZooKeeper: Client environment:user.name=hadoop
12/08/22 10:12:09 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/hadoop
12/08/22 10:12:09 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/hadoop
12/08/22 10:12:09 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=slave2:2222,slave1:2222,slave3:2222 sessionTimeout=180000 watcher=hconnection
12/08/22 10:12:09 INFO zookeeper.ClientCnxn: Opening socket connection to server /192.168.15.132:2222
12/08/22 10:12:09 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 23606@master
12/08/22 10:12:09 WARN client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: 無法定位登錄配置 occurred when trying to find JAAS configuration.
12/08/22 10:12:09 INFO client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
12/08/22 10:12:09 INFO zookeeper.ClientCnxn: Socket connection established to slave3/192.168.15.132:2222, initiating session
12/08/22 10:12:09 WARN zookeeper.ClientCnxnSocket: Connected to an old server; r-o mode will be unavailable
12/08/22 10:12:09 INFO zookeeper.ClientCnxn: Session establishment complete on server slave3/192.168.15.132:2222, sessionid = 0x33943bafeb90005, negotiated timeout = 40000
12/08/22 10:12:09 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@28bb494b; serverName=slave2,60020,1345461138645
12/08/22 10:12:09 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for .META.,,1.1028785192 is slave3:60020
12/08/22 10:12:09 DEBUG client.MetaScanner: Scanning .META. starting at row=xyz,,00000000000000 for max=10 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@28bb494b
12/08/22 10:12:09 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for xyz,,1340764906812.6aa4cb2fb4c9eb34f360953acdb1e21c. is slave2:60020
12/08/22 10:12:09 DEBUG client.MetaScanner: Scanning .META. starting at row=xyz,,00000000000000 for max=2147483647 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@28bb494b
12/08/22 10:12:09 DEBUG mapreduce.TableInputFormatBase: getSplits: split -> 0 -> slave2:,
12/08/22 10:12:10 INFO mapred.JobClient: Running job: job_201208201908_0002
12/08/22 10:12:11 INFO mapred.JobClient:  map 0% reduce 0%
12/08/22 10:12:30 INFO mapred.JobClient:  map 100% reduce 0%
12/08/22 10:12:35 INFO mapred.JobClient: Job complete: job_201208201908_0002
12/08/22 10:12:35 INFO mapred.JobClient: Counters: 18
12/08/22 10:12:35 INFO mapred.JobClient:   Job Counters
12/08/22 10:12:35 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=16203
12/08/22 10:12:35 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
12/08/22 10:12:35 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
12/08/22 10:12:35 INFO mapred.JobClient:     Launched map tasks=1
12/08/22 10:12:35 INFO mapred.JobClient:     Data-local map tasks=1
12/08/22 10:12:35 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
12/08/22 10:12:35 INFO mapred.JobClient:   File Output Format Counters
12/08/22 10:12:35 INFO mapred.JobClient:     Bytes Written=255
12/08/22 10:12:35 INFO mapred.JobClient:   FileSystemCounters
12/08/22 10:12:35 INFO mapred.JobClient:     HDFS_BYTES_READ=58
12/08/22 10:12:35 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=31370
12/08/22 10:12:35 INFO mapred.JobClient:   File Input Format Counters
12/08/22 10:12:35 INFO mapred.JobClient:     Bytes Read=0
12/08/22 10:12:35 INFO mapred.JobClient:   Map-Reduce Framework
12/08/22 10:12:35 INFO mapred.JobClient:     Map input records=2
12/08/22 10:12:35 INFO mapred.JobClient:     Physical memory (bytes) snapshot=77606912
12/08/22 10:12:35 INFO mapred.JobClient:     Spilled Records=0
12/08/22 10:12:35 INFO mapred.JobClient:     CPU time spent (ms)=1830
12/08/22 10:12:35 INFO mapred.JobClient:     Total committed heap usage (bytes)=31850496
12/08/22 10:12:35 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=488656896
12/08/22 10:12:35 INFO mapred.JobClient:     Map output records=2
12/08/22 10:12:35 INFO mapred.JobClient:     SPLIT_RAW_BYTES=58
上述紅色字體輸出已提示導出了兩條數據了,因為我有3個datanode,加上數據比較少,所以肯定只會在一台datanode上有導出文件。 如果數據很多,可能每個datanode節點都會有導出文件,至於在哪台datanode上你就需要找一下/home/hadoop目錄下有沒有xyz目 錄了。
我這找到后文件如下:
[hadoop@slave2 ~]$ cd xyz/
[hadoop@slave2 xyz]$ ls
part-m-00000  _SUCCESS
開始新建一張新表,表結構和xyz表一樣
hbase(main):001:0> create 'zzz','cf1'
0 row(s) in 2.0490 seconds
然后開始導入,這里我就利用導出文件在哪我就在哪導入了。當然你也可以拿這個part-m-00000文件到其余的datanode上導入, 順便友情提醒一下,如果導出的數據很多,你導入的時候千萬別把所有的part-m-0000*文件都放到一個目錄下開始導入,肯定會失敗的!你只能把part-m-0000*文件一個個開始導入。
[hadoop@slave2 ~]$ hbase/bin/hbase org.apache.hadoop.hbase.mapreduce.Driver impo
rt zzz file:///home/hadoop/xyz/
12/08/22 10:30:42 DEBUG mapreduce.TableMapReduceUtil: New JarFinder: org.apache.hadoop.util.JarFinder.getJar not available.  Using old findContainingJar
12/08/22 10:30:42 DEBUG mapreduce.TableMapReduceUtil: New JarFinder: org.apache.hadoop.util.JarFinder.getJar not available.  Using old findContainingJar
12/08/22 10:30:42 DEBUG mapreduce.TableMapReduceUtil: New JarFinder: org.apache.hadoop.util.JarFinder.getJar not available.  Using old findContainingJar
12/08/22 10:30:42 DEBUG mapreduce.TableMapReduceUtil: New JarFinder: org.apache.hadoop.util.JarFinder.getJar not available.  Using old findContainingJar
12/08/22 10:30:42 DEBUG mapreduce.TableMapReduceUtil: New JarFinder: org.apache.hadoop.util.JarFinder.getJar not available.  Using old findContainingJar
12/08/22 10:30:42 DEBUG mapreduce.TableMapReduceUtil: New JarFinder: org.apache.hadoop.util.JarFinder.getJar not available.  Using old findContainingJar
12/08/22 10:30:42 DEBUG mapreduce.TableMapReduceUtil: New JarFinder: org.apache.hadoop.util.JarFinder.getJar not available.  Using old findContainingJar
12/08/22 10:30:42 DEBUG mapreduce.TableMapReduceUtil: New JarFinder: org.apache.hadoop.util.JarFinder.getJar not available.  Using old findContainingJar
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/hbase/lib/slf4j-log4j12-1.5.8.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/hadoop/lib/slf4j-log4j12-1.4.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
12/08/22 10:30:44 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.3-1240972, built on 02/06/2012 10:48 GMT
12/08/22 10:30:44 INFO zookeeper.ZooKeeper: Client environment:host.name=slave2
12/08/22 10:30:44 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_14
12/08/22 10:30:44 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
12/08/22 10:30:44 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.6.0_14/jre
12/08/22 10:30:44 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/home/hadoop/hbase/bin/../conf:/usr/java/jdk1.6.0_14/lib/tools.jar:/home/hadoop/hbase:/home/hadoop/hbase/hbase-0.92.1.jar:/home/hadoop/hbase/hbase-0.92.1-tests.jar:/home/hadoop/hbase/lib/activation-1.1.jar:/home/hadoop/hbase/lib/asm-3.1.jar:/home/hadoop/hbase/lib/avro-1.5.3.jar:/home/hadoop/hbase/lib/avro-ipc-1.5.3.jar:/home/hadoop/hbase/lib/commons-beanutils-1.7.0.jar:/home/hadoop/hbase/lib/commons-beanutils-core-1.8.0.jar:/home/hadoop/hbase/lib/commons-cli-1.2.jar:/home/hadoop/hbase/lib/commons-codec-1.4.jar:/home/hadoop/hbase/lib/commons-collections-3.2.1.jar:/home/hadoop/hbase/lib/commons-configuration-1.6.jar:/home/hadoop/hbase/lib/commons-digester-1.8.jar:/home/hadoop/hbase/lib/commons-el-1.0.jar:/home/hadoop/hbase/lib/commons-httpclient-3.1.jar:/home/hadoop/hbase/lib/commons-lang-2.5.jar:/home/hadoop/hbase/lib/commons-logging-1.1.1.jar:/home/hadoop/hbase/lib/commons-math-2.1.jar:/home/hadoop/hbase/lib/commons-net-1.4.1.jar:/home/hadoop/hbase/lib/core-3.1.1.jar:/home/hadoop/hbase/lib/guava-r09.jar:/home/hadoop/hbase/lib/hadoop-core-1.0.0.jar:/home/hadoop/hbase/lib/high-scale-lib-1.1.1.jar:/home/hadoop/hbase/lib/httpclient-4.0.1.jar:/home/hadoop/hbase/lib/httpcore-4.0.1.jar:/home/hadoop/hbase/lib/jackson-core-asl-1.5.5.jar:/home/hadoop/hbase/lib/jackson-jaxrs-1.5.5.jar:/home/hadoop/hbase/lib/jackson-mapper-asl-1.5.5.jar:/home/hadoop/hbase/lib/jackson-xc-1.5.5.jar:/home/hadoop/hbase/lib/jamon-runtime-2.3.1.jar:/home/hadoop/hbase/lib/jasper-compiler-5.5.23.jar:/home/hadoop/hbase/lib/jasper-runtime-5.5.23.jar:/home/hadoop/hbase/lib/jaxb-api-2.1.jar:/home/hadoop/hbase/lib/jaxb-impl-2.1.12.jar:/home/hadoop/hbase/lib/jersey-core-1.4.jar:/home/hadoop/hbase/lib/jersey-json-1.4.jar:/home/hadoop/hbase/lib/jersey-server-1.4.jar:/home/hadoop/hbase/lib/jettison-1.1.jar:/home/hadoop/hbase/lib/jetty-6.1.26.jar:/home/hadoop/hbase/lib/jetty-util-6.1.26.jar:/home/hadoop/hbase/lib/jruby-complete-1.6.5.jar:/home/hadoop/hbase/lib/jsp-2.1-6.1.14.jar:/home/hadoop/hbase/lib/jsp-api-2.1-6.1.14.jar:/home/hadoop/hbase/lib/libthrift-0.7.0.jar:/home/hadoop/hbase/lib/log4j-1.2.16.jar:/home/hadoop/hbase/lib/netty-3.2.4.Final.jar:/home/hadoop/hbase/lib/protobuf-java-2.4.0a.jar:/home/hadoop/hbase/lib/servlet-api-2.5-6.1.14.jar:/home/hadoop/hbase/lib/servlet-api-2.5.jar:/home/hadoop/hbase/lib/slf4j-api-1.5.8.jar:/home/hadoop/hbase/lib/slf4j-log4j12-1.5.8.jar:/home/hadoop/hbase/lib/snappy-java-1.0.3.2.jar:/home/hadoop/hbase/lib/stax-api-1.0.1.jar:/home/hadoop/hbase/lib/velocity-1.7.jar:/home/hadoop/hbase/lib/xmlenc-0.52.jar:/home/hadoop/hbase/lib/zookeeper-3.4.3.jar:/home/hadoop/hadoop/libexec/../conf:/usr/java/jdk1.6.0_14/lib/tools.jar:/home/hadoop/hadoop/libexec/..:/home/hadoop/hadoop/libexec/../hadoop-core-1.0.3.jar:/home/hadoop/hadoop/libexec/../lib/asm-3.2.jar:/home/hadoop/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/home/hadoop/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/home/hadoop/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/home/hadoop/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/home/hadoop/hadoop/libexec/../lib/commons-cli-1.2.jar:/home/hadoop/hadoop/libexec/../lib/commons-codec-1.4.jar:/home/hadoop/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/home/hadoop/hadoop/libexec/../lib/commons-configuration-1.6.jar:/home/hadoop/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/home/hadoop/hadoop/libexec/../lib/commons-digester-1.8.jar:/home/hadoop/hadoop/libexec/../lib/commons-el-1.0.jar:/home/hadoop/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/home/hadoop/hadoop/libexec/../lib/commons-io-2.1.jar:/home/hadoop/hadoop/libexec/../lib/commons-lang-2.4.jar:/home/hadoop/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/home/hadoop/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/home/hadoop/hadoop/libexec/../lib/commons-math-2.1.jar:/home/hadoop/hadoop/libexec/../lib/commons-net-1.4.1.jar:/home/hadoop/hadoop/libexec/../lib/core-3.1.1.jar:/home/hadoop/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.3.jar:/home/hadoop/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.3.jar:/home/hadoop/hadoop/libexec/../lib/hadoop-thriftfs-1.0.3.jar:/home/hadoop/hadoop/libexec/../lib/hbase-0.92.1.jar:/home/hadoop/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/home/hadoop/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/home/hadoop/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/home/hadoop/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/home/hadoop/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/home/hadoop/hadoop/libexec/../lib/jdeb-0.8.jar:/home/hadoop/hadoop/libexec/../lib/jersey-core-1.8.jar:/home/hadoop/hadoop/libexec/../lib/jersey-json-1.8.jar:/home/hadoop/hadoop/libexec/../lib/jersey-server-1.8.jar:/home/hadoop/hadoop/libexec/../lib/jets3t-0.6.1.jar:/home/hadoop/hadoop/libexec/../lib/jetty-6.1.26.jar:/home/hadoop/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/home/hadoop/hadoop/libexec/../lib/jsch-0.1.42.jar:/home/hadoop/hadoop/libexec/../lib/junit-4.5.jar:/home/hadoop/hadoop/libexec/../lib/kfs-0.2.2.jar:/home/hadoop/hadoop/libexec/../lib/log4j-1.2.15.jar:/home/hadoop/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/home/hadoop/hadoop/libexec/../lib/oro-2.0.8.jar:/home/hadoop/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/home/hadoop/hadoop/libexec/../lib/slf4j-api-1.4.3.jar:/home/hadoop/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar:/home/hadoop/hadoop/libexec/../lib/xmlenc-0.52.jar:/home/hadoop/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/home/hadoop/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar
12/08/22 10:30:44 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/home/hadoop/hadoop/libexec/../lib/native/Linux-amd64-64:/home/hadoop/hbase/lib/native/Linux-amd64-64
12/08/22 10:30:44 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
12/08/22 10:30:44 INFO zookeeper.ZooKeeper: Client environment:java.compiler=
12/08/22 10:30:44 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
12/08/22 10:30:44 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
12/08/22 10:30:44 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.9-89.ELsmp
12/08/22 10:30:44 INFO zookeeper.ZooKeeper: Client environment:user.name=hadoop
12/08/22 10:30:44 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/hadoop
12/08/22 10:30:44 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/hadoop
12/08/22 10:30:44 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=slave2:2222,slave1:2222,slave3:2222 sessionTimeout=180000 watcher=hconnection
12/08/22 10:30:44 INFO zookeeper.ClientCnxn: Opening socket connection to server /192.168.15.72:2222
12/08/22 10:30:44 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 30654@slave2
12/08/22 10:30:44 WARN client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: 無法定位登錄配置 occurred when trying to find JAAS configuration.
12/08/22 10:30:44 INFO client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
12/08/22 10:30:44 INFO zookeeper.ClientCnxn: Socket connection established to slave1/192.168.15.72:2222, initiating session
12/08/22 10:30:44 WARN zookeeper.ClientCnxnSocket: Connected to an old server; r-o mode will be unavailable
12/08/22 10:30:44 INFO zookeeper.ClientCnxn: Session establishment complete on server slave1/192.168.15.72:2222, sessionid = 0x13943ba912f0007, negotiated timeout = 40000
12/08/22 10:30:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1c23f1bb; serverName=slave2,60020,1345461138645
12/08/22 10:30:44 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for .META.,,1.1028785192 is slave3:60020
12/08/22 10:30:44 DEBUG client.MetaScanner: Scanning .META. starting at row=zzz,,00000000000000 for max=10 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1c23f1bb
12/08/22 10:30:44 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for zzz,,1345602149536.dbeb5fc388bcc537d40b5602b60798ff. is slave3:60020
12/08/22 10:30:44 INFO mapreduce.TableOutputFormat: Created table instance for zzz
12/08/22 10:30:44 INFO input.FileInputFormat: Total input paths to process : 1
12/08/22 10:30:45 INFO mapred.JobClient: Running job: job_201208201908_0004
12/08/22 10:30:46 INFO mapred.JobClient:  map 0% reduce 0%
12/08/22 10:31:23 INFO mapred.JobClient: Task Id : attempt_201208201908_0004_m_000000_0, Status : FAILED
java.io.FileNotFoundException: File file:/home/hadoop/xyz/part-m-00000 does not exist.
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
        at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:796)
        at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1475)
        at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1470)
        at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.initialize(SequenceFileRecordReader.java:50)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:522)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)
12/08/22 10:31:35 INFO mapred.JobClient:  map 100% reduce 0%
12/08/22 10:31:40 INFO mapred.JobClient: Job complete: job_201208201908_0004
12/08/22 10:31:40 INFO mapred.JobClient: Counters: 19
12/08/22 10:31:40 INFO mapred.JobClient:   Job Counters
12/08/22 10:31:40 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=22927
12/08/22 10:31:40 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
12/08/22 10:31:40 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
12/08/22 10:31:40 INFO mapred.JobClient:     Rack-local map tasks=2
12/08/22 10:31:40 INFO mapred.JobClient:     Launched map tasks=2
12/08/22 10:31:40 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
12/08/22 10:31:40 INFO mapred.JobClient:   File Output Format Counters
12/08/22 10:31:40 INFO mapred.JobClient:     Bytes Written=0
12/08/22 10:31:40 INFO mapred.JobClient:   FileSystemCounters
12/08/22 10:31:40 INFO mapred.JobClient:     FILE_BYTES_READ=255
12/08/22 10:31:40 INFO mapred.JobClient:     HDFS_BYTES_READ=99
12/08/22 10:31:40 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=31054
12/08/22 10:31:40 INFO mapred.JobClient:   File Input Format Counters
12/08/22 10:31:40 INFO mapred.JobClient:     Bytes Read=255
12/08/22 10:31:40 INFO mapred.JobClient:   Map-Reduce Framework
12/08/22 10:31:40 INFO mapred.JobClient:     Map input records=2
12/08/22 10:31:40 INFO mapred.JobClient:     Physical memory (bytes) snapshot=72753152
12/08/22 10:31:40 INFO mapred.JobClient:     Spilled Records=0
12/08/22 10:31:40 INFO mapred.JobClient:     CPU time spent (ms)=260
12/08/22 10:31:40 INFO mapred.JobClient:     Total committed heap usage (bytes)=18350080
12/08/22 10:31:40 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=491810816
12/08/22 10:31:40 INFO mapred.JobClient:     Map output records=2
12/08/22 10:31:40 INFO mapred.JobClient:     SPLIT_RAW_BYTES=99
上述輸出中可以看到導入了2條記錄,但是仍然會報錯,報文件不存在,這就不知道是什么原因了。但是數據是導入進去了。
查看zzz表中的數據

hbase(main):003:0> scan 'zzz'
ROW 

                  COLUMN+CELL                                              
 10000                column=cf1:val, timestamp=1345598242644, value=china     
 20000                column=cf1:val, timestamp=1345598283332, value=zengzhunzhu
                      n                                                        
2 row(s) in 0.0410 seconds
這樣基本就完成了hbase表中的數據我們可以轉化為mapreduce任務進程開始導出導入。當然也可以這么備份的。


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM