kettle之mongodb數據同步


需求:

1.源數據庫新增一條記錄,目標庫同時新增一條記錄;

2.源數據庫修改一條記錄,目標庫同時修改該條記錄;

示例用到三個Kettle組件

 

下面詳細說下每個組件的配置

Source:

本示例連接的是Mongodb數據庫,四個字段,ID默認為主鍵,_id會系統自動生成暫時先不管。

本節點的詳細說明,可見官網:http://wiki.pentaho.com/display/EAI/MongoDB+Input

值映射:

本步在本示例作用不大,只是為了測試效果。按照截圖上進行配置即可

MongoDbOutput:

關鍵是這一步的配置

官網上對這個tab頁的解釋是這樣的:

2.2 Selecting the write mode

The MongoDb output step provides a number of options that control what and how data is written to the target Mongo document collection. By default, data is inserted into the target collection. If the specified collection doesn't exist, it will be created before data is inserted. Selecting the Truncate option will delete any existing data in the target collection before inserting begins. Unless unique indexes are being used (see section on indexing below) then Mongo DB will allow duplicate records to be inserted. Mongo DB allows for fast bulk insert operations - the batch size can be configured using the Batch insert size field. If no value is supplied here, then the default size of 100 rows is used.

Selecting the Upsert option changes the write mode from insert to upsert (i.e. update if a match is found, otherwise insert a new record). Information on defining how records are matched can be found in the next section. Standard upsert replaces a matched record with an entire new record based on all the incoming fields specified in the Mongo document fields tab. Modifier update enables modifier ($ operators) to be used to mutate individual fields within matching documents. This type of update is fast and involves minimal network traffic; it also has the ability to update all matching documents, rather than just the first, if the Multi-update option is enabled

個人理解就是默認都不選擇時往目標集合添加數據,如果指定的集合不存在則先創建集合然后插入數據;

Truncate:插入數據之前先把集合里的所有數據刪除。(相當於數據刪除同步);

Update:修改數據,存在則修改,不存在不操作。

upsert:找到匹配項則修改,找不到匹配項則添加。

勾選上紅色圈着的選項之后,源數據修改、添加了,在目標庫里都會有相應的操作。不過還要設置下面的一步

ID為主鍵match field for update時一定要選擇Y否則運行時出錯。

同步過程最主要的就是上邊列出的幾步設置,當然如果想要再設置更強大的功能,可詳細去研究官網的API

官網API地址:http://wiki.pentaho.com/display/EAI/

示例kri文件下載地址:http://files.cnblogs.com/nyzhai/mongodbTran.rar

 


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM