sqoop2安裝總結

sqoop2安裝

1. 下載解壓縮

此次安裝版本為1.99.6

# Decompress Sqoop distribution tarball
tar -xvf sqoop-<version>-bin-hadoop<hadoop-version>.tar.gz


ln -s sqoop-<version>-bin-hadoop<hadoop version>.tar.gz sqoop

export SQOOP_HOME=`pwd`/sqoop
# Change working directory
cd $SQOOP_HOME

2. 配置服務

配置文件在$SQOOP_HOME/server/conf下，大多數默認配置即可。需要注意的配置是catalina.properties文件下common_loader需要正確配置hadoop庫文件，包括hdfs、mapreduce、hive等所有jar包. 我的配置為:

common.loader=${catalina.base}/lib,\ 
${catalina.base}/lib/*.jar,\ 
${catalina.home}/lib,\ 
${catalina.home}/lib/*.jar,\ 
${catalina.home}/../lib/*.jar,\ 
/opt/cloudera/parcels/CDH/lib/hadoop/*.jar,\ 
/opt/cloudera/parcels/CDH/lib/hadoop/lib/*.jar,\ 
/opt/cloudera/parcels/CDH/lib/hadoop-hdfs/*.jar,\ 
/opt/cloudera/parcels/CDH/lib/hadoop-hdfs/lib/*.jar,\ /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/*.jar,\ 
/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*.jar,\ 
/opt/cloudera/parcels/CDH/lib/hadoop-yarn/*.jar,\ 
/opt/cloudera/parcels/CDH/lib/hadoop-yarn/lib/*.jar,\ 
/opt/cloudera/parcels/CDH/lib/hive/lib/*.jar

sqoop.properties文件的org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/etc/hadoop/conf/配置hadoop
配置文件路徑，默認為/etc/hadoop/conf

3. 測試配置是否正確

運行

sqoop2-tool verify

若正確,則輸出為:

Verification was successful.
Tool class org.apache.sqoop.tools.tool.VerifyTool has finished correctly

4. 下載derby 包

自帶derby版本過低運行時會由於不兼容而出現以下錯誤：

org.apache.sqoop.common.SqoopException: JDBCREPO_0007:Unable to lease link
at org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:63)
	at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:85)
	at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:61)
	at org.apache.sqoop.repository.JdbcRepository.createOrUpgradeRepository(JdbcRepository.java:127)
	at org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:123)
	at org.apache.sqoop.tools.tool.UpgradeTool.runToolWithConfiguration(UpgradeTool.java:39)
	at org.apache.sqoop.tools.ConfiguredTool.runTool(ConfiguredTool.java:35)
	at org.apache.sqoop.tools.ToolRunner.main(ToolRunner.java:75)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.sqoop.tomcat.TomcatToolRunner.main(TomcatToolRunner.java:77)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.catalina.startup.Tool.main(Tool.java:225)
Caused by: java.sql.SQLException: No suitable driver found for 

解決辦法為下載derby最新版本，先刪除/sqoop-1.99.6-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib下的derby舊包，然后把新下載的derby目錄下的lib下的jar包拷貝到/sqoop-1.99.6-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib

5. 驗證

運行bin/sqoop2-shell,進入sqoop shell模式, 運行show version --all，若能正確輸出server版本，則安裝成功:

sqoop:000> show version --all
client version:
  Sqoop 1.99.6 source revision 07244c3915975f26f03d9e1edf09ab7d06619bb8
  Compiled by root on Wed Apr 29 10:40:43 CST 2015
server version:
  Sqoop 1.99.6 source revision 07244c3915975f26f03d9e1edf09ab7d06619bb8
  Compiled by root on Wed Apr 29 10:40:43 CST 2015
API versions:
  [v1]