Oracle 11g數據脫敏
前言
最近開發人員有個需求，導一份生產庫的數據到測試庫。
由於生產數據安全需要，需要並允許對導出的數據進行加密脫敏處理。
關於加密和脫敏
個人理解，
加密是通過一系列規則對數據進行處理，可以通過規則解密出原有的數據甚至被破解。
而脫敏則是按照一定規律對數據進行處理，屬於不可逆行為，會丟失原有數據內容。
加密的數據一定是已經脫敏，但是脫敏的數據不等同於加密。
此篇文章討論脫敏。
環境構造
SYS@zkm> drop table scott.test purge; Table dropped. SYS@zkm> create table scott.test as select level id from dual connect by level<=15; Table created. SYS@zkm> desc scott.test Name Null? Type ----------------------------------------- -------- ---------------------------- ID NUMBER SYS@zkm> drop table zkm.test purge; Table dropped. SYS@zkm> create table zkm.test as select * from scott.test where 1=2; Table created. SYS@zkm> desc zkm.test Name Null? Type ----------------------------------------- -------- ---------------------------- ID NUMBER SYS@zkm>
基本需求是scott.test表的數據脫敏導出后導入zkm.test表。
方法1
使用md5算法創建函數。
create or replace function fn_md5(input_string VARCHAR2) return varchar2 IS raw_input RAW(128) := UTL_RAW.CAST_TO_RAW(input_string); decrypted_raw RAW(2048); error_in_input_buffer_length EXCEPTION; BEGIN sys.dbms_obfuscation_toolkit.MD5(input => raw_input,checksum => decrypted_raw); return rawtohex(decrypted_raw); END; /
SYS@zkm> select id,fn_md5(id) string from scott.test; ID STRING ---------- -------------------------------------------------- 1 C4CA4238A0B923820DCC509A6F75849B 2 C81E728D9D4C2F636F067F89CC14862C 3 ECCBC87E4B5CE2FE28308FD9F2A7BAF3 4 A87FF679A2F3E71D9181A67B7542122C 5 E4DA3B7FBBCE2345D7772B0674A318D5 6 1679091C5A880FAF6FB5E6087EB1B2DC 7 8F14E45FCEEA167A5A36DEDD4BEA2543 8 C9F0F895FB98AB9159F51FD0297E236D 9 45C48CCE2E2D7FBDEA1AFC51C7C6AD26 10 D3D9446802A44259755D38E6D163E820 11 6512BD43D9CAA6E02C990B0A82652DCA ID STRING ---------- -------------------------------------------------- 12 C20AD4D76FE97759AA27A0C99BFF6710 13 C51CE410C124A10E0DB5E4B97FC2AF39 14 AAB3238922BCC25A6F606EB525FFDC56 15 9BF31C7FF062936A96D3C8BD1F8F2FF3 15 rows selected.
缺點很明顯，長度變長了，並且類型不再是number類型。
后續創建新表（也是一個缺點）存儲轉換后的數據，再通過數據泵導出即可。
（但是zkm.test表的id字段不能是number類型，不然是導不進去的）因此此方法對於存在字段是非字符串類型的幾乎不可行。
方法2
使用內部函數translate。
SYS@zkm> select id,translate(id,'0123456789','abcdefghijk') string from scott.test; ID STRING ---------- -------------------------------------------------- 1 b 2 c 3 d 4 e 5 f 6 g 7 h 8 i 9 j 10 ba 11 bb ID STRING ---------- -------------------------------------------------- 12 bc 13 bd 14 be 15 bf 15 rows selected.
后續創建新表（缺點）存儲轉換后的數據，再通過數據泵導出即可。
如果數字類型轉換為字符類型，則存在和方法1一樣的問題。因此數字還是轉換為數字類型即可，比如。
SYS@zkm> select id,translate(id,'0123456789','1564719874') string,to_number(translate(id,'0123456789','1564719874')) string2 from scott.test; ID STRING STRING2 ---------- --------------- ---------- 1 5 5 2 6 6 3 4 4 4 7 7 5 1 1 6 9 9 7 8 8 8 7 7 9 4 4 10 51 51 11 55 55 ID STRING STRING2 ---------- --------------- ---------- 12 56 56 13 54 54 14 57 57 15 51 51 15 rows selected.
注意，translate之后返回的類型還是字符串型的，需要to_number轉換一下。
方法3
使用expdp的REMAP_DATA參數。
具體用法說明見官方文檔：REMAP_DATA
該方法相對於方法1和方法2稍微復雜點，但是由於數據量比較大，因此用該方法最為方便。
構造一個含有中文行的字段。
SYS@zkm> create table scott.test2 (name varchar2(20)); Table created. SYS@zkm> insert into scott.test2 values('數據庫'); 1 row created. SYS@zkm> insert into scott.test2 values('綁定變量'); 1 row created. SYS@zkm> insert into scott.test2 values('執行計划'); 1 row created. SYS@zkm> commit; Commit complete. SYS@zkm> select * from scott.test2; NAME ------------------------------------------------------------ 數據庫 綁定變量 執行計划 SYS@zkm> create table zkm.test2 as select * from scott.test2 where 1=2; Table created.
使用方法1，方法2的驗證結果。
SYS@zkm> select name,fn_md5(name) string from scott.test2; NAME STRING ---------- ----------------------------------------------------------------- 數據庫 8B2F1D29A87AB8968601C2AB7D9084B9 綁定變量 FCE81E294017C008C111B61164728CFA 執行計划 A394306DB20856B8BF68FA197DC5DAD4 SYS@zkm> col name for a15 SYS@zkm> set line 500 SYS@zkm> col string for a50 SYS@zkm> select name,translate(name,'數據庫定變量執行計划綁','之后返回的類型還是字符') string from scott.test2; NAME STRING --------------- -------------------------------------------------- 數據庫 之后返 綁定變量 符回的類 執行計划 型還是字 SYS@zkm> select name,translate(name,'數據庫定變量執行計划綁','哈哈哈哈哈哈哈哈哈哈嗝') string from scott.test2; NAME STRING --------------- -------------------------------------------------- 數據庫 哈哈哈 綁定變量 嗝哈哈哈 執行計划 哈哈哈哈 SYS@zkm>
由於remap_data的使用需要創建包和包體來調用，主要考慮兩種情況，
- 字符串類型脫敏（無論是否中文）
- 數字類型
在考慮一下兩種情況，
- 類型不變
- 字符串的長度不變
創建包和包體內容如下：
create or replace package scott.pkg_remap is function fn_numeral(input_string number) return number; function f_remap_varchar(p_varchar varchar2) return varchar2; end; / create or replace package body scott.pkg_remap is function fn_numeral(input_string number) return number as begin return floor(dbms_random.value(1, 100000)); end; function f_remap_varchar(p_varchar varchar2) return varchar2 as begin return translate(p_varchar,'數據庫定變量執行計划綁','之后返回的類型還是字符'); end; end; /
SYS@zkm> create or replace package scott.pkg_remap 2 is 3 function fn_numeral(input_string number) return number; 4 function f_remap_varchar(p_varchar varchar2) return varchar2; 5 end; 6 / Package created. SYS@zkm> SYS@zkm> SYS@zkm> create or replace package body scott.pkg_remap 2 is 3 function fn_numeral(input_string number) return number as 4 begin 5 return floor(dbms_random.value(1, 100000)); 6 end; 7 function f_remap_varchar(p_varchar varchar2) return varchar2 as 8 begin 9 return translate(p_varchar,'數據庫定變量執行計划綁','之后返回的類型還是字符'); 10 end; 11 end; 12 / Package body created.
導出並導入。
其中，由於表scott.test的id字段是number類型，因此調用的是pkg_remap.fn_numeral
[oracle@oracle ~]$ expdp \'/ as sysdba\' directory=dirtmp dumpfile=test.dmp logfile=test.log cluster=n tables=scott.test remap_data=scott.test.id:scott.pkg_remap.fn_numeral reuse_dumpfiles=y Export: Release 11.2.0.4.0 - Production on Mon May 11 09:27:44 2020 Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options Starting "SYS"."SYS_EXPORT_TABLE_01": "/******** AS SYSDBA" directory=dirtmp dumpfile=test.dmp logfile=test.log cluster=n tables=scott.test remap_data=scott.test.id:scott.pkg_remap.fn_numeral reuse_dumpfiles=y Estimate in progress using BLOCKS method... Processing object type TABLE_EXPORT/TABLE/TABLE_DATA Total estimation using BLOCKS method: 64 KB Processing object type TABLE_EXPORT/TABLE/TABLE . . exported "SCOTT"."TEST" 5.125 KB 15 rows Master table "SYS"."SYS_EXPORT_TABLE_01" successfully loaded/unloaded ****************************************************************************** Dump file set for SYS.SYS_EXPORT_TABLE_01 is: /home/oracle/test.dmp Job "SYS"."SYS_EXPORT_TABLE_01" successfully completed at Mon May 11 09:27:48 2020 elapsed 0 00:00:03 [oracle@oracle ~]$ impdp \'/ as sysdba\' directory=dirtmp dumpfile=test.dmp logfile=imptest.log cluster=n remap_schema=scott:zkm TABLE_EXISTS_ACTION=truncate Import: Release 11.2.0.4.0 - Production on Mon May 11 09:27:53 2020 Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options Master table "SYS"."SYS_IMPORT_FULL_01" successfully loaded/unloaded Starting "SYS"."SYS_IMPORT_FULL_01": "/******** AS SYSDBA" directory=dirtmp dumpfile=test.dmp logfile=imptest.log cluster=n remap_schema=scott:zkm TABLE_EXISTS_ACTION=truncate Processing object type TABLE_EXPORT/TABLE/TABLE Table "ZKM"."TEST" exists and has been truncated. Data will be loaded but all dependent metadata will be skipped due to table_exists_action of truncate Processing object type TABLE_EXPORT/TABLE/TABLE_DATA . . imported "ZKM"."TEST" 5.125 KB 15 rows Job "SYS"."SYS_IMPORT_FULL_01" successfully completed at Mon May 11 09:27:55 2020 elapsed 0 00:00:02
查詢zkm.test結果如下：
SYS@zkm> select * from zkm.test; ID ---------- 87848 3399 49478 86510 85267 82478 19478 80623 2317 93170 89452 ID ---------- 571 90558 86931 68415 15 rows selected.
可以看到，數據已經脫敏成功。
導出並導入。
其中，由於表scott.test2的name字段是varchar2類型，因此調用的是pkg_remap.f_remap_varchar
[oracle@oracle ~]$ expdp \'/ as sysdba\' directory=dirtmp dumpfile=test.dmp logfile=test.log cluster=n tables=scott.test2 remap_data=scott.test2.name:scott.pkg_remap.f_remap_varchar reuse_dumpfiles=y Export: Release 11.2.0.4.0 - Production on Mon May 11 09:35:34 2020 Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options Starting "SYS"."SYS_EXPORT_TABLE_01": "/******** AS SYSDBA" directory=dirtmp dumpfile=test.dmp logfile=test.log cluster=n tables=scott.test2 remap_data=scott.test2.name:scott.pkg_remap.f_remap_varchar reuse_dumpfiles=y Estimate in progress using BLOCKS method... Processing object type TABLE_EXPORT/TABLE/TABLE_DATA Total estimation using BLOCKS method: 64 KB Processing object type TABLE_EXPORT/TABLE/TABLE . . exported "SCOTT"."TEST2" 5.039 KB 3 rows Master table "SYS"."SYS_EXPORT_TABLE_01" successfully loaded/unloaded ****************************************************************************** Dump file set for SYS.SYS_EXPORT_TABLE_01 is: /home/oracle/test.dmp Job "SYS"."SYS_EXPORT_TABLE_01" successfully completed at Mon May 11 09:35:38 2020 elapsed 0 00:00:03 [oracle@oracle ~]$ impdp \'/ as sysdba\' directory=dirtmp dumpfile=test.dmp logfile=imptest.log cluster=n remap_schema=scott:zkm TABLE_EXISTS_ACTION=truncate Import: Release 11.2.0.4.0 - Production on Mon May 11 09:35:43 2020 Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options Master table "SYS"."SYS_IMPORT_FULL_01" successfully loaded/unloaded Starting "SYS"."SYS_IMPORT_FULL_01": "/******** AS SYSDBA" directory=dirtmp dumpfile=test.dmp logfile=imptest.log cluster=n remap_schema=scott:zkm TABLE_EXISTS_ACTION=truncate Processing object type TABLE_EXPORT/TABLE/TABLE Table "ZKM"."TEST2" exists and has been truncated. Data will be loaded but all dependent metadata will be skipped due to table_exists_action of truncate Processing object type TABLE_EXPORT/TABLE/TABLE_DATA . . imported "ZKM"."TEST2" 5.039 KB 3 rows Job "SYS"."SYS_IMPORT_FULL_01" successfully completed at Mon May 11 09:35:45 2020 elapsed 0 00:00:02
查詢zkm.test2結果如下：
SYS@zkm> select * from zkm.test2; NAME ------------------------------------------------------------ 之后返 符回的類 型還是字
可以看到，數據已經脫敏成功。
方法4
oracle12c中em集成了一個功能 data masking(數據脫敏)。
由於沒有12c生產環境，因此此方法不做討論。
參考鏈接
https://blog.csdn.net/enmotech/article/details/81713790
http://blog.itpub.net/28539951/viewspace-2063896/
https://www.anbob.com/archives/1016.html