Windows(Win10)
打開cmd
首先輸入sqlplus,依次輸入用戶名、口令
C:\Users\hasee>sqlplus SQL*Plus: Release 11.2.0.1.0 Production on 星期三 3月 13 16:55:46 2019 Copyright (c) 1982, 2010, Oracle. All rights reserved. 請輸入用戶名: scott 輸入口令: 連接到: Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options
連接數據庫管理員
[oracle@CentOS7One ~]$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.1.0 Production on 星期一 3月 18 14:28:40 2019 Copyright (c) 1982, 2009, Oracle. All rights reserved. 已連接到空閑例程。
1.exp導出表
退出oracle，將scott用戶下orcl數據庫的buy_cnt_c1表導出到e:/opt/oracle_output/daochu.dmp
C:\Users\hasee>exp scott/tiger@orcl file=e:\opt\\oracle_output\daochu.dmp tables=(buy_cnt_c1) Export: Release 11.2.0.1.0 - Production on 星期三 3月 13 16:58:30 2019 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. 連接到: Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options 已導出 ZHS16GBK 字符集和 AL16UTF16 NCHAR 字符集 即將導出指定的表通過常規路徑... . . 正在導出表 BUY_CNT_C1導出了 8293 行 成功終止導出, 沒有出現警告。
8293條數據 22ms
8492033條數據18s65ms
2.imp導入表
導入前要將原來的表刪除
drop table buy_cnt_c1;
退出oracle，將e:/opt/oracle_output/daochu.dmp導入到scott用戶下orcl數據庫
C:\Users\hasee>imp scott/tiger@orcl file=e:\opt\\oracle_output\daochu.dmp Import: Release 11.2.0.1.0 - Production on 星期三 3月 13 17:03:29 2019 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. 連接到: Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options 經由常規路徑由 EXPORT:V11.02.00 創建的導出文件 已經完成 ZHS16GBK 字符集和 AL16UTF16 NCHAR 字符集中的導入 . 正在將 SCOTT 的對象導入到 SCOTT . 正在將 SCOTT 的對象導入到 SCOTT . . 正在導入表 "BUY_CNT_C1"導入了 8293 行 成功終止導入, 沒有出現警告。
8293條數據 18ms
8492033條數據 1m19s8ms
3.sqluldr2導出表
1.導出表
E:\bigdata\sqluldr2_win_jb51>sqluldr264 scott/tiger@127.0.0.1/orcl query="select * from buy_cnt_c1" head=yes file=e:\opt\oracle_output\tmp001.txt 0 rows exported at 2019-03-13 17:40:46, size 0 MB. 8293 rows exported at 2019-03-13 17:40:46, size 0 MB. output file e:\opt\oracle_output\tmp001.txt closed at 8293 rows, size 0 MB.
8293條數據 23ms
8492033條數據17s55ms
2.指定分隔符
field對字段進行分隔，record對記錄進行分隔。
ESCF是escape from，指定哪些形式的數據需要被轉義；
ESCAPE是轉義前綴
ESCT是escape to,轉義成目標字符串
\r=0x0d \n=0x0a |=0x7c ,=0x2c, \t=0x09, :=0x3a, #=0x23, "=0x22 '=0x27
sqluldr264 test/test@127.0.0.1/orcl query="select * from test" head=yes file=e:\opt\oracle_output\hex_test.txt log=e:\opt\oracle_output\hex.log field=0x7c record=0x0d ESCF=0x7c0x0d ESCAPE='\' ESCT=i
表信息如下
第一行“|a”字段里含有我們的字段分隔符，用ESCAPE即可替換
第五行含有0x0d，即換行符，不會影響我們的結果，sqluldr會將它看成字符串
導出結果為
ID|NAME 1|'ia 2|sanq 3|jieba 4|wuren 5|0x0d
4.sqlldr導入數據
首先創建表
DROP TABLE "SCOTT"."BUY_CNT_C1"; CREATE TABLE "SCOTT"."BUY_CNT_C1" ( "T_INDEX" NUMBER(19) NULL , "INNET_TIME" NUMBER(19) NULL , "UP_DATA_AMOUNT_TOTAL" FLOAT(126) NULL , "DOWN_DATA_AMOUNT_TOTAL" FLOAT(126) NULL , "PAY_CNT" NUMBER(19) NULL , "BUY_CNT" NUMBER(19) NULL , "MONTH_FEE" FLOAT(126) NULL , "CALL_DURATION" NUMBER(19) NULL , "AGE_RANGE_ID" FLOAT(126) NULL , "DAY_ACT_NUM_MEAN" FLOAT(126) NULL ) LOGGING NOCOMPRESS NOCACHE ;
編寫ctl文件
tmp_002.ctl
load data infile * into table BUY_CNT_TEST ( "T_INDEX" char terminated by ',', "INNET_TIME" char terminated by ',', "UP_DATA_AMOUNT_TOTAL" char terminated by ',', "DOWN_DATA_AMOUNT_TOTAL" char terminated by ',', "PAY_CNT" char terminated by ',', "BUY_CNT" char terminated by ',', "MONTH_FEE" char terminated by ',', "CALL_DURATION" char terminated by ',', "AGE_RANGE_ID" char terminated by ',', "DAY_ACT_NUM_MEAN" char terminated by ',' )
cmd輸入
E:\bigdata\sqluldr2_win_jb51>sqlldr userid=scott/tiger control=e:/opt/oracle_ctl/tmp002.ctl log=e:/opt/oracle_ctl/tmp001.log data=e:/opt/oracle_ctl/tmp001.csv rows=64 SQL*Loader: Release 11.2.0.1.0 - Production on 星期四 3月 14 10:50:46 2019 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. 達到提交點 - 邏輯記錄計數 64 達到提交點 - 邏輯記錄計數 128 達到提交點 - 邏輯記錄計數 192 達到提交點 - 邏輯記錄計數 256
...
達到提交點 - 邏輯記錄計數 8192 達到提交點 - 邏輯記錄計數 8256 達到提交點 - 邏輯記錄計數 8294
8294條數據 28ms
8492033條數據 2m6s12ms
Linux(CentOS7)
1.exp導出表
[oracle@CentOS7One ~]$ exp scott/tiger file=output/daochu.dmp tables='(buy_cnt_c1)' log=output/log/daochudmp.log Export: Release 11.2.0.1.0 - Production on 星期三 3月 20 09:12:08 2019 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. 連接到: Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options 已導出 AL32UTF8 字符集和 AL16UTF16 NCHAR 字符集 服務器使用 ZHS16GBK 字符集 (可能的字符集轉換) 即將導出指定的表通過常規路徑... . . 正在導出表 BUY_CNT_C1導出了 8492032 行 成功終止導出, 沒有出現警告。
8293條數據 2s
8492033條數據60s
2.imp導入表
導入前要將原來的表刪除
drop table buy_cnt_c1;
[oracle@CentOS7One ~]$ imp scott/tiger@orcl11g file=input/daoru.cmp Import: Release 11.2.0.1.0 - Production on 星期三 3月 20 10:42:51 2019 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. 連接到: Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options 經由常規路徑由 EXPORT:V11.02.00 創建的導出文件 已經完成 AL32UTF8 字符集和 AL16UTF16 NCHAR 字符集中的導入 導入服務器使用 ZHS16GBK 字符集 (可能的字符集轉換) . 正在將 SCOTT 的對象導入到 SCOTT . 正在將 SCOTT 的對象導入到 SCOTT . . 正在導入表 "BUY_CNT_C1"導入了 8492032 行 成功終止導入, 沒有出現警告。
8293條數據 48ms
8492032條數據61s
3.sqluldr2導出數據
sqluldr2_linux64_10204.bin user=scott/tiger@orcl11g query="select * from buy_cnt_c1" head=yes file=output/tmp002.csv log=output/log/tmp002.log
8293條數據 小於1s
8492032條數據 5min17s
4.sqlldr導入數據
ctl文件同上,先創建表
[oracle@CentOS7One ~]$ sqlldr userid=scott/tiger control=input/tmp002.ctl log=input/log/tmp002.log data=input/tmp001.csv rows=64 SQL*Loader: Release 11.2.0.1.0 - Production on 星期二 3月 19 16:46:56 2019 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. 達到提交點 - 邏輯記錄計數 64 達到提交點 - 邏輯記錄計數 128 達到提交點 - 邏輯記錄計數 192 達到提交點 - 邏輯記錄計數 256 達到提交點 - 邏輯記錄計數 320 ... 達到提交點 - 邏輯記錄計數 8192 達到提交點 - 邏輯記錄計數 8256 達到提交點 - 邏輯記錄計數 8294
8294條數據 47ms
8492033數據 5m27s26ms
測試
編寫測試腳本，統計腳本命令執行時間
Windows腳本如下
@echo off set /a startMS=%time:~9,2% set /a startS=%time:~6,2% set /a startM=%time:~3,2% echo %time% ::寫你的命令 set /a endMS=%time:~9,2% set /a endS=%time:~6,2% set /a endM=%time:~3,2% echo %time% set /a diffMS_=%endMS%-%startMS% set /a diffS_=%endS%-%startS% set /a diffM_=%endM%-%startM% echo cost:%diffM_% %diffS_% %diffMS_% pause
Linux腳本如下
start_time=`date --date='0 days ago' "+%Y-%m-%d %H:%M:%S"` #this is your shell script #寫你的命令
############## finish_time=`date --date='0 days ago' "+%Y-%m-%d %H:%M:%S"` duration=$(($(($(date +%s -d "$finish_time")-$(date +%s -d "$start_time"))))) echo "this shell script execution duration: $duration"
CentOS7部署在虛擬機上,所以配置較Windows差很多
8294條數據（小數據量）
8492033條數據（大數據量）
業務實操1
中移動3300萬數據的csv(6.03GB)使用sqlldr導入到oracle數據庫
表寬61列，建表語句如下，字段長度建議設置255以上
create table T_CJYX_DZXX ( id VARCHAR2(50), version VARCHAR2(15), metacategory VARCHAR2(100), indicator VARCHAR2(20), iscity VARCHAR2(20), parentid VARCHAR2(24), name VARCHAR2(255), code VARCHAR2(255), nameabbrpy VARCHAR2(255), hierarchy VARCHAR2(20), alias VARCHAR2(255), alias2 VARCHAR2(255), alias3 VARCHAR2(255), alias4 VARCHAR2(255), postcode VARCHAR2(255), cover VARCHAR2(25), longitude VARCHAR2(50), latitude VARCHAR2(26), rgt VARCHAR2(20), lft VARCHAR2(20), status VARCHAR2(25), comments VARCHAR2(255), detailaddress VARCHAR2(255), logdate VARCHAR2(50), areaid VARCHAR2(28), isdefine VARCHAR2(22), faultstatus VARCHAR2(25), restoretime VARCHAR2(50), faultdescription VARCHAR2(2555), deviceusage VARCHAR2(25), faultdetailcode VARCHAR2(20), floortype VARCHAR2(25), faultcount VARCHAR2(20), createdate VARCHAR2(50), auditdate VARCHAR2(50), createuser VARCHAR2(255), audituser VARCHAR2(255), addresstype VARCHAR2(25), marketgrade VARCHAR2(25), iscoordinate VARCHAR2(20), ttmigrate VARCHAR2(25), gjgx VARCHAR2(25), accesstype VARCHAR2(25), shape VARCHAR2(128), fillcolor VARCHAR2(255), fillwidth VARCHAR2(22), fillalpha VARCHAR2(21), fillstyle VARCHAR2(255), fillbordercolor VARCHAR2(255), area VARCHAR2(23), areaunit VARCHAR2(22), createuserphone VARCHAR2(255), modifyuser VARCHAR2(255), modifydate VARCHAR2(50), gcstatus VARCHAR2(25), pmsprjcode VARCHAR2(255), pmsprjname VARCHAR2(255), deleteuser VARCHAR2(255), deletedate VARCHAR2(50), afterdeleteuser VARCHAR2(255), afterdeletedate VARCHAR2(50) ) tablespace TBS_AHJZH_DATA pctfree 10 initrans 1 maxtrans 255 storage ( initial 64 next 8 minextents 1 maxextents unlimited );
ctl文件如下
LOAD DATA INFILE * INTO TABLE T_CJYX_DZXX REPLACE FIELDS TERMINATED BY '$' TRAILING NULLCOLS ( id, version, metacategory, indicator, iscity, parentid, name, code, nameabbrpy, hierarchy, alias, alias2, alias3, alias4, postcode, cover, longitude, latitude, rgt, lft, status, comments, detailaddress, logdate, areaid, isdefine, faultstatus, restoretime, faultdescription, deviceusage, faultdetailcode, floortype, faultcount, createdate, auditdate, createuser, audituser, addresstype, marketgrade, iscoordinate, ttmigrate, gjgx, accesstype, shape, fillcolor, fillwidth, fillalpha, fillstyle, fillbordercolor, area, areaunit, createuserphone, modifyuser, modifydate, gcstatus, pmsprjcode, pmsprjname, deleteuser, deletedate, afterdeleteuser, afterdeletedate )
運行日志如下，耗時約14小時
SQL*Loader: Release 11.2.0.1.0 - Production on 星期三 4月 3 08:46:55 2019 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. 控制文件: E:\opt\srcbigdata\di_70002_20190325.ctl 數據文件: E:\opt\srcbigdata\di_70002_20190325.csv 錯誤文件: E:\opt\srcbigdata\di_70002_20190325.bad 廢棄文件: 未作指定 (可廢棄所有記錄) 要加載的數: ALL 要跳過的數: 0 允許的錯誤: 50 綁定數組: 64 行, 最大 256000 字節 繼續: 未作指定 所用路徑: 常規 表 T_CJYX_DZXX,已加載從每個邏輯記錄 插入選項對此表 REPLACE 生效 TRAILING NULLCOLS 選項生效 列名 位置 長度 中止 包裝數據類型 ------------------------------ ---------- ----- ---- ---- --------------------- ID FIRST * $ CHARACTER VERSION NEXT * $ CHARACTER METACATEGORY NEXT * $ CHARACTER INDICATOR NEXT * $ CHARACTER ISCITY NEXT * $ CHARACTER PARENTID NEXT * $ CHARACTER NAME NEXT * $ CHARACTER CODE NEXT * $ CHARACTER NAMEABBRPY NEXT * $ CHARACTER HIERARCHY NEXT * $ CHARACTER ALIAS NEXT * $ CHARACTER ALIAS2 NEXT * $ CHARACTER ALIAS3 NEXT * $ CHARACTER ALIAS4 NEXT * $ CHARACTER POSTCODE NEXT * $ CHARACTER COVER NEXT * $ CHARACTER LONGITUDE NEXT * $ CHARACTER LATITUDE NEXT * $ CHARACTER RGT NEXT * $ CHARACTER LFT NEXT * $ CHARACTER STATUS NEXT * $ CHARACTER COMMENTS NEXT * $ CHARACTER DETAILADDRESS NEXT * $ CHARACTER LOGDATE NEXT * $ CHARACTER AREAID NEXT * $ CHARACTER ISDEFINE NEXT * $ CHARACTER FAULTSTATUS NEXT * $ CHARACTER RESTORETIME NEXT * $ CHARACTER FAULTDESCRIPTION NEXT * $ CHARACTER DEVICEUSAGE NEXT * $ CHARACTER FAULTDETAILCODE NEXT * $ CHARACTER FLOORTYPE NEXT * $ CHARACTER FAULTCOUNT NEXT * $ CHARACTER CREATEDATE NEXT * $ CHARACTER AUDITDATE NEXT * $ CHARACTER CREATEUSER NEXT * $ CHARACTER AUDITUSER NEXT * $ CHARACTER ADDRESSTYPE NEXT * $ CHARACTER MARKETGRADE NEXT * $ CHARACTER ISCOORDINATE NEXT * $ CHARACTER TTMIGRATE NEXT * $ CHARACTER GJGX NEXT * $ CHARACTER ACCESSTYPE NEXT * $ CHARACTER SHAPE NEXT * $ CHARACTER FILLCOLOR NEXT * $ CHARACTER FILLWIDTH NEXT * $ CHARACTER FILLALPHA NEXT * $ CHARACTER FILLSTYLE NEXT * $ CHARACTER FILLBORDERCOLOR NEXT * $ CHARACTER AREA NEXT * $ CHARACTER AREAUNIT NEXT * $ CHARACTER CREATEUSERPHONE NEXT * $ CHARACTER MODIFYUSER NEXT * $ CHARACTER MODIFYDATE NEXT * $ CHARACTER GCSTATUS NEXT * $ CHARACTER PMSPRJCODE NEXT * $ CHARACTER PMSPRJNAME NEXT * $ CHARACTER DELETEUSER NEXT * $ CHARACTER DELETEDATE NEXT * $ CHARACTER AFTERDELETEUSER NEXT * $ CHARACTER AFTERDELETEDATE NEXT * $ CHARACTER ROWS 參數所用的值已從 64 更改為 16 表 T_CJYX_DZXX: 33543432 行 加載成功。 由於數據錯誤, 0 行 沒有加載。 由於所有 WHEN 子句失敗, 0 行 沒有加載。 由於所有字段都為空的, 0 行 沒有加載。 為綁定數組分配的空間: 251808 字節 (16 行) 讀取 緩沖區字節數: 1048576 跳過的邏輯記錄總數: 0 讀取的邏輯記錄總數: 33543432 拒絕的邏輯記錄總數: 0 廢棄的邏輯記錄總數: 0 從 星期三 4月 03 08:46:55 2019 開始運行 在 星期三 4月 03 22:48:09 2019 處運行結束 經過時間為: 14: 01: 14.33 CPU 時間為: 00: 18: 19.89
業務實操2
建表語句
create table T_CJYX_HOMECOUNT_BACKUP ( acyc_id VARCHAR2(50), address_id VARCHAR2(50), address_name VARCHAR2(200), address_level VARCHAR2(50), check_type VARCHAR2(50), check_target_num VARCHAR2(50), check_value VARCHAR2(50), target_phone VARCHAR2(4000), notarget_phone VARCHAR2(4000), parent_id VARCHAR2(50), bcyc_id VARCHAR2(50) )
從服務器導入16G數據，表寬十一列，一億八千萬條記錄
控制文件di_00121_20190427.ctl如下
LOAD DATA INFILE * INTO TABLE T_CJYX_HOMECOUNT_BACKUP REPLACE FIELDS TERMINATED BY '$' TRAILING NULLCOLS ( acyc_id CHAR(40000), address_id CHAR(40000), address_name CHAR(40000), address_level CHAR(40000), check_type CHAR(40000), check_target_num CHAR(40000), check_value CHAR(40000), target_phone CHAR(40000), notarget_phone CHAR(40000), parent_id CHAR(40000), bcyc_id CHAR(40000) )
運行腳本
這種模式導入速度非常慢，十五小時只能導入三千萬條，但是此方案安全穩定
sqlldr userid=username/password@//10.243.5.16:1521/itgrept control=E:\opt\srcbigdata2\di_00121_20190427.ctl log=E:\opt\srcbigdata2\di_00121_20190427.log data=E:\opt\srcbigdata2\di_00121_20190427.dat
參數調優后的腳本,十五小時可以導入一億條數據
sqlldr userid=username/password@//10.243.5.16:1521/itgrept control=E:\opt\srcbigdata2\di_00121_20190427.ctl log=E:\opt\srcbigdata2\di_00121_20190427_gbk.log data=E:\opt\srcbigdata2\di_00121_20190427_gbk.dat rows=10000 readsize=20680000 bindsize=20680000
最佳性能的情況下，
報錯ORA-03113：通信通道的文件結尾
原因：sqlORA-03113 連接到數據庫的網絡中斷
1.SYSTEM中審計表aud$在數據庫編程過程中，對SQL語句的編譯過程，資源占用會較大
2.SYSAUX則進行的是AWR快照，也會進行占用較多的空間
解決方案：以sysdba的身份清除審計表，最好的方法是弄一個本地表空間進行導入
報錯：
SQL*Loader-604: 試圖提交時出錯
ORA-03135: 連接失去聯系
實際上腳本運行中途會因為各種原因中斷，使用append方式向表追加
di_00121_20190427_append.ctl如下
LOAD DATA INFILE * INTO TABLE T_CJYX_HOMECOUNT_BACKUP APPEND FIELDS TERMINATED BY '$' TRAILING NULLCOLS ( acyc_id CHAR(40000), address_id CHAR(40000), address_name CHAR(40000), address_level CHAR(40000), check_type CHAR(40000), check_target_num CHAR(40000), check_value CHAR(40000), target_phone CHAR(40000), notarget_phone CHAR(40000), parent_id CHAR(40000), bcyc_id CHAR(40000) )
在108238266行中斷
從108238266行繼續導入
運行腳本如下
sqlldr userid=username/password@//10.243.5.16:1521/itgrept control=E:\opt\srcbigdata2\di_00121_20190427_append.ctl log=E:\opt\srcbigdata2\di_00121_20190427_gbk_append.log data=E:\opt\srcbigdata2\di_00121_20190427_gbk.dat rows=100 readsize=20680000 bindsize=20680000 skip=108238266
耗時大約四十一小時
總結
exp導出和imp導入的效率要大於sqluldr2和sqlldr；
小數據量體現不出來exp和imp的優勢，效率和sqluldr2和sqlldr差不多；
大數據量exp和imp明顯優於sqluldr2和sqlldr；
硬件配置高會提高導入導出效率；
不同數據庫遷移建議使用sqluldr2和sqlldr,支持csv,txt等數據格式的導入導出；