解決kettle在兩個mysql之間遷移數據時亂碼的問題 和 相關報錯 及參數調整, 速度優化

1. 亂碼問題

編輯目標數據庫的鏈接：

配置編碼參數即可。

 

2. 報錯 No operations allowed after statement closed. 

需要調整wait_timeout： set global wait_timeout=1000000；

3. net_write_timeout 參數也需要調整：set global net_write_timeout='60000'

kettle在遷移數據時，運行速度很慢，如果數量很大時，需要調整相關參數，不然運行到一半就報錯。

遷移完成之后，可以恢復相關參數。

4. kettle 加速

 

 

 原理是把 單條的insert轉換為 批量 batch insert

 To remedy this, in PDI I create a separate, specialized Database Connection I use for batch inserts. Set these two MySQL-specific options on your Database Connection:

useServerPrepStmts false
rewriteBatchedStatements true

Used together, these "fake" batch inserts on the client. Specificially, the insert statements:

INSERT INTO t (c1,c2) VALUES ('One',1);
INSERT INTO t (c1,c2) VALUES ('Two',2);
INSERT INTO t (c1,c2) VALUES ('Three',3);

will be rewritten into:

INSERT INTO t (c1,c2) VALUES ('One',1),('Two',2),('Three',3);

So that the batched rows will be inserted with one statement (and one network round-trip). With this simple change, Table Output is very fast and close to performance of the bulk loader steps.

設置后寫入速度有明顯提升。

 

另外 目標 端的mysql可以調整一下參數：

innodb_flush_log_at_trx_commit = 0

sync_binlog = 0