來源於 https://blog.csdn.net/weixin_43241054/article/details/89036669
一、sqoop2的安裝：
安裝sqoop2前，首先安裝配置好Hadoop，sqoop2是在Hadoop的基礎上運行的。
1.sqoop2的版本：sqoop-1.99.7-bin-hadoop200.tar.gz
解壓縮即可： tar -zxvf sqoop-1.99.7-bin-hadoop200.tar.gz
2.環境變量的配置：vi /etc/profile
添加（根據自己的安裝路徑）：
export SQOOP_HOME=/data/software/sqoop-1.99.7-bin-hadoop200
export CATALINA_HOME=$SQOOP_HOME/server
export PATH=$SQOOP_HOME/bin:$PATH
3.修改Hadoop下的core-site.xml文件
添加：
特別要注意的是sqoop2是提交任務的用戶名，這里要改成你提交任務的用戶，否則會報
org.apache.hadoop.security.authorize.AuthorizationException: User: root is not allowed to impersonate root
<property>
<name>hadoop.proxyuser.sqoop2.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.sqoop2.groups</name>
<value>*</value>
</property>
4.配置文件vi conf/sqoop.properties
（1）把$HADOOP_HOME替換為自己實際路徑
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=$HADOOP_HOME/etc/hadoop
（2）將以下注釋放開
org.apache.sqoop.security.authentication.type=SIMPLE
org.apache.sqoop.security.authentication.handler=org.apache.sqoop.security.authentication.SimpleAuthenticationHandler
org.apache.sqoop.security.authentication.anonymous=true
（3）創建目錄mkdir /home/hadoop/sqoop/logs
將@LOGDIR@替換為/home/hadoop/sqoop/logs；BASEDIR替換為/home/hadoop/sqoop
5.下載mysql JDBC Driver 到sqoop/server/lib 下
cp mysql-connector-java-xxx.jar sqoop/server/lib （若安裝了hive，在/home/hadoop/Hive/apache-hive-2.3.3-bin/lib目錄下）。
6.初始化：bin/sqoop2-tool upgrade
7.啟動/關閉sqoop server：
sqoop.sh server start/stop
8.啟動交互命令：
sqoop2-shell 出現sqoop:000>
二、sqoop2的應用
1.啟動Hadoop和sqoop server，
查看進程命令jps:可以看到以下進程
2898 SecondaryNameNode
3219 NodeManager
2643 NameNode
3109 ResourceManager
2742 DataNode
5798 SqoopShell
10153 Jps
3644 SqoopJettyServer
2.設置端口
set server --host localhost --port 12000 --webapp sqoop
3.創建連接（show connector）
(1)創建MySQL連接
create link -connector generic-jdbc-connector
Creating link for connector with name generic-jdbc-connector
Please fill following values to create new link object
Name: First Link 自己起連接名
Database connection
Driver class: com.mysql.jdbc.Driver
Connection String: jdbc:mysql://localhost:3306/test test為數據庫
Username: root
Password: ******
Fetch Size: 100
Connection Properties:
There are currently 0 values in the map:
entry#
SQL Dialect
Identifier enclose: 注意 這里不能直接回車！要打一個空格符號！因為如果不打，查詢mysql表的時候會在表上加上“”，導致查詢出錯！
New link was successfully created with validation status OK and name First Link
#HDFS Link
sqoop:000> create link -connector hdfs-connector
Creating link for connector with name hdfs-connector
Please fill following values to create new link object
Name: Second Link
HDFS cluster
URI: hdfs://xiaobin:9000/
Conf directory: /home/hadoop/etc/hadoop hadoop配置文件目錄
Additional configs::
There are currently 0 values in the map:
entry#
New link was successfully created with validation status OK and name Second Link
（2）創建job
create job -f First Link -t Second Link
Creating job for links with from name First Link and to name Second Link
Please fill following values to create new job object
Name: Sqoopy 自己起job名
Database source
Schema name: test 數據庫名
Table name: emp 表名
SQL statement:
Column names:
There are currently 0 values in the list:
element#
Partition column:
Partition column nullable:
Boundary query:
Incremental read
Check column:
Last value:
Target configuration
Override null value:
Null value:
File format:
0 : TEXT_FILE
1 : SEQUENCE_FILE
2 : PARQUET_FILE
Choose: 0
Compression codec:
0 : NONE
1 : DEFAULT
2 : DEFLATE
3 : GZIP
4 : BZIP2
5 : LZO
6 : LZ4
7 : SNAPPY
8 : CUSTOM
Choose: 0
Custom codec:
Output directory: /test/sqoop Hadoop的目錄 注意必須為空，報錯Invalid input/output directory - Output directory is not empty
Append mode:
Throttling resources
Extractors:
Loaders:
Classpath configuration
Extra mapper jars:
There are currently 0 values in the list:
element#
New job was successfully created with validation status OK and name Sqoopy
（3）啟動/關閉/查看job
start/stop/status job -name Sqoopy -name也可改為-n
start job -name Sqoopy -s 加-s，顯示執行進度
（4）瀏覽器查看狀態
http://192.168.159.11:8088
三、常見錯誤
(1)org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
原因：hadoop的lib庫沒有設置環境變量，添加環境變量如下：
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib:$HADOOP_COMMON_LIB_NATIVE_DIR"
(2)Message: GENERIC_JDBC_CONNECTOR_0016:Can't fetch schema -
原因：配置job時，Schema name:配置錯誤導致的，Schema name在MySQL中為數據庫名，在Oracle中為用戶名。
（3）User: root is not allowed to impersonate root
原因：未配置文件core-site.xml，添加：
<property>
<name>hadoop.proxyuser.sqoop2.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.sqoop2.groups</name>
<value>*</value>
</property>
（4）Message: GENERIC_HDFS_CONNECTOR_0007:Invalid input/output directory - Unexpected exception
原因：文件夾沒有權限或者無此文件夾，或者文件夾不為空。
（5）Message: COMMON_0023:Unable to create new job data
原因：可能創建的job名已經存在。
(6)sqoop:000> stop job -n sql_hdfs
Exception has occurred during processing command
Exception: org.apache.sqoop.common.SqoopException Message: MAPREDUCE_0003:Can't get RunningJob instance -
原因：由於沒有啟動historyserver引起的
解決辦法：在mapred-site.xml配置文件中添加：
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop-allinone-200-123.wdcloud.locl:10020</value>
</property>
執行命令：mr-jobhistory-daemon.sh start historyserver
（7）出現以下情況：
ACCEPTED: waiting for AM container to be allocated, launched and register with RM.
這種情況可能是因為資源有限，保證硬盤使用率在90%以下。
————————————————
版權聲明：本文為CSDN博主「磨刀大神」的原創文章，遵循CC 4.0 BY-SA版權協議，轉載請附上原文出處鏈接及本聲明。
原文鏈接：https://blog.csdn.net/weixin_43241054/java/article/details/89036669