ETL工具之kettle的使用(一) MongoDB->Mysql

kettle簡介：

　　Kettle是一款國外開源的ETL工具，純java編寫，可以在Window、Linux、Unix上運行，數據抽取高效穩定。

　　Kettle 中文名稱叫水壺，該項目的主程序員MATT 希望把各種數據放到一個壺里，然后以一種指定的格式流出。

　　Kettle這個ETL工具集，它允許你管理來自不同數據庫的數據，通過提供一個圖形化的用戶環境來描述你想做什么，而不是你想怎么做。

　　Kettle中有兩種腳本文件，transformation和job，transformation完成針對數據的基礎轉換，job則完成整個工作流的控制

1：Kettle的下載與安裝

　　下載地址 ： https://community.hds.com/docs/DOC-1009855

　　1) 下載下來是壓縮版   解壓就可以使用  解壓后雙擊 Spoon.bat  等幾秒 即出現圖形界面工具

2：從MongoDB數據庫的otherData文檔 中 讀取數據到 Mysql數據庫t_other_data表 中

　　1)在kettle中，mongodb input組件。

　　

　　2) 編輯組件

　　3) 輸入連接信息

　　4) 寫查詢語句

　　點擊預覽 可以預覽成功說明配置無誤

　　5) MongoDB->Mysql中MongoDB的配置就ok了    接下來配置表輸出  即 mysql的配置

　　雙擊 表輸出

 

　　6) 填寫數據庫連接信息

　　7) 點擊測試 看mysql配置的是否無誤 

　　看到successfully說明 mysql配置無誤  (這里最初我配置的時候報錯  報錯信息后面單拎出來記錄)

 　　8) 配置Mysql

　　9) 點擊運行

　　10) 查看Mysql中t_other_data表中數據

 

 3：第一次在配置好Mysql后  點擊測試 Mysql是報錯的 

org.pentaho.di.core.exception.KettleDatabaseException: 
Error occurred while trying to connect to the database

Driver class 'org.gjt.mm.mysql.Driver' could not be found, make sure the 'MySQL' driver (jar file) is installed.
org.gjt.mm.mysql.Driver


    at org.pentaho.di.core.database.Database.normalConnect(Database.java:472)
    at org.pentaho.di.core.database.Database.connect(Database.java:370)
    at org.pentaho.di.core.database.Database.connect(Database.java:341)
    at org.pentaho.di.core.database.Database.connect(Database.java:331)
    at org.pentaho.di.ui.core.database.dialog.XulDatabaseExplorerController.createDatabaseNodes(XulDatabaseExplorerController.java:380)
    at org.pentaho.di.ui.core.database.dialog.XulDatabaseExplorerController.init(XulDatabaseExplorerController.java:130)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.pentaho.ui.xul.impl.AbstractXulDomContainer.invoke(AbstractXulDomContainer.java:313)
    at org.pentaho.ui.xul.impl.AbstractXulDomContainer.initialize(AbstractXulDomContainer.java:141)
    at org.pentaho.ui.xul.swt.SwtXulRunner$1.run(SwtXulRunner.java:67)
    at org.eclipse.swt.widgets.Synchronizer.syncExec(Unknown Source)
    at org.eclipse.swt.widgets.Display.syncExec(Unknown Source)
    at org.pentaho.ui.xul.swt.SwtXulRunner.initialize(SwtXulRunner.java:64)
    at org.pentaho.di.ui.core.database.dialog.XulDatabaseExplorerDialog.open(XulDatabaseExplorerDialog.java:92)
    at org.pentaho.di.ui.core.database.dialog.DataOverrideHandler.explore(DataOverrideHandler.java:86)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.pentaho.ui.xul.impl.AbstractXulDomContainer.invoke(AbstractXulDomContainer.java:313)
    at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:157)
    at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:141)
    at org.pentaho.ui.xul.swt.tags.SwtButton.access$500(SwtButton.java:43)
    at org.pentaho.ui.xul.swt.tags.SwtButton$4.widgetSelected(SwtButton.java:137)
    at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
    at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
    at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
    at org.eclipse.jface.window.Window.runEventLoop(Window.java:820)
    at org.eclipse.jface.window.Window.open(Window.java:796)
    at org.pentaho.di.ui.xul.KettleDialog.show(KettleDialog.java:80)
    at org.pentaho.di.ui.xul.KettleDialog.show(KettleDialog.java:47)
    at org.pentaho.di.ui.core.database.dialog.XulDatabaseDialog.open(XulDatabaseDialog.java:116)
    at org.pentaho.di.ui.core.database.dialog.DatabaseDialog.open(DatabaseDialog.java:61)
    at org.pentaho.di.ui.trans.step.BaseStepDialog.showDbDialogUnlessCancelledOrValid(BaseStepDialog.java:808)
    at org.pentaho.di.ui.trans.step.BaseStepDialog$EditConnectionListener.widgetSelected(BaseStepDialog.java:1515)
    at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
    at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
    at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
    at org.pentaho.di.ui.trans.steps.tableoutput.TableOutputDialog.open(TableOutputDialog.java:888)
    at org.pentaho.di.ui.spoon.delegates.SpoonStepsDelegate.editStep(SpoonStepsDelegate.java:120)
    at org.pentaho.di.ui.spoon.Spoon.editStep(Spoon.java:8662)
    at org.pentaho.di.ui.spoon.trans.TransGraph.editStep(TransGraph.java:3293)
    at org.pentaho.di.ui.spoon.trans.TransGraph.mouseDoubleClick(TransGraph.java:785)
    at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
    at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
    at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
    at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1381)
    at org.pentaho.di.ui.spoon.Spoon.waitForDispose(Spoon.java:7817)
    at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:9179)
    at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:707)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.pentaho.commons.launcher.Launcher.main(Launcher.java:92)
Caused by: org.pentaho.di.core.exception.KettleDatabaseException: 
Driver class 'org.gjt.mm.mysql.Driver' could not be found, make sure the 'MySQL' driver (jar file) is installed.
org.gjt.mm.mysql.Driver

    at org.pentaho.di.core.database.Database.connectUsingClass(Database.java:515)
    at org.pentaho.di.core.database.Database.normalConnect(Database.java:456)
    ... 66 more
Caused by: java.lang.ClassNotFoundException: org.gjt.mm.mysql.Driver
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.pentaho.di.core.database.Database.connectUsingClass(Database.java:490)
    ... 67 more

錯誤信息顯示是jar包未安裝

下載mysql-connector-java-5.0.7 jar包並放到lib目錄下 關閉圖形界面重新打開Spoon.bat即可

(有個小坑 ： 我第一次下載的版本是8依然報錯，隨后版本換成5好用了 )

下載jar包地址： http://mvnrepository.com/artifact/mysql/mysql-connector-java

思考：MongoDB的數據讀取到Mysql中是手動處理的  這在生產環境合理嗎？如何自動實時的讀取數據呢？下節總結