docker版mongodb4.0同步數據到es7.3.2的簡單配置（mongo無安全校驗）

          網上目前能查到的mongo同步數據到es的資料最多就是用的mongo-connector，此外還有revier和logstash。全都是比較過時的，不支持高版本的es。目前能查到的只有monstache支持。版本對照參考官網：

 https://rwynn.github.io/monstache-site/start/

　　es7.3.2的搭建翻看我的上一篇，有詳細的elk7.3.2的搭建。這篇里直接用的上次搭建的es。

         首先搭建mongodb。我用的是4.0版本。首先需要配置開啟副本集，修改配置文件/etc/mongod.conf.orig：

storage:
  dbPath: /data/db
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# network interfaces
net:
  port: 27017
#開啟遠程訪問
  bindIp: 0.0.0.0


# how the process runs
processManagement:
  timeZoneInfo: /usr/share/zoneinfo

#security:

#operationProfiling:
#副本集配置內容。
replication:
  replSetName: es
  oplogSizeMB: 10240

 

此時啟動mongo的docker-compose為：

version: '3.7'

services:
  mongo:
    image: mongo:4.0
    restart: always
    container_name: mongodb
    environment:
      TZ: Asia/Shanghai       
    volumes:
      - /data/javaProject/mongo-data:/data/db
      - /data/javaProject/mongo-config/mongod.conf.orig:/etc/mongod.conf.orig
    command: --config /etc/mongod.conf.orig
    ports:
      - 12000:27017
  mongo-express:
    image: mongo-express
    container_name: mongo-express
    restart: always
    depends_on:
      - mongo
    ports:
      - 8081:8081
      TZ: Asia/Shanghai

 

　　此時啟動容器，在mongo容器中的日志看到內容是：

No logs available

 

　　此時是正常的。docker exec進入mongo容器后執行命令 mongo命令進入客戶端，執行命令：

> show dbs
2020-09-25T15:32:36.098+0800 E QUERY    [js] Error: listDatabases failed:{
    "operationTime" : Timestamp(0, 0),
    "ok" : 0,
    "errmsg" : "not master and slaveOk=false",
    "code" : 13435,
    "codeName" : "NotMasterNoSlaveOk",
    "$clusterTime" : {
        "clusterTime" : Timestamp(0, 0),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    }
} :
_getErrorWithCode@src/mongo/shell/utils.js:25:13
Mongo.prototype.getDBs@src/mongo/shell/mongo.js:139:1
shellHelper.show@src/mongo/shell/utils.js:882:13
shellHelper@src/mongo/shell/utils.js:766:15
@(shellhelp2):1:1

　　查資料說這種報錯是因為沒有初始化副本集：

rs.initiate({_id:”es”,members:[{_id:0,host:’127.0.0.1:27017’}]})
其中es要對應配置文件中的replSetName值，而且這句命令要手敲，復制執行會報解析錯誤
> use admin
switched to db admin
> rs.initiate({_id:”es”,members:[{_id:0,host:’127.0.0.1:27017’}]})
2020-09-25T15:50:35.176+0800 E QUERY    [js] SyntaxError: illegal character @(shell):1:17
手敲執行：
> rs.initiate({_id:"es",members:[{_id:0,host:'127.0.0.1:27017'}]})
{
    "ok" : 1,
    "operationTime" : Timestamp(1601020319, 1),
    "$clusterTime" : {
        "clusterTime" : Timestamp(1601020319, 1),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    }
}

　　此時創建一個數據庫，創建一個集合，創建一個文檔。然后在mongo-express可視化頁面中找數據：

在自帶的local數據庫的oplog.rs集合中可以找到剛才添加的數據已經進入了副本集

 

　　拉取monstache后，掛載配置文件monstache.config.toml到容器中，配置文件內容：

　

# connectionn settings

# connect to MongoDB using the following URL
mongo-url = "mongodb://192.168.10.20:12000"
# connect to the Elasticsearch REST API at the following node URLs
elasticsearch-urls = ["http://192.168.10.20:9200"]

# frequently required settings

# if you need to seed an index from a collection and not just listen and sync changes events
# you can copy entire collections or views from MongoDB to Elasticsearch
direct-read-namespaces = ["testdemo.*"]

# if you want to use MongoDB change streams instead of legacy oplog tailing use change-stream-namespaces
# change streams require at least MongoDB API 3.6+
# if you have MongoDB 4+ you can listen for changes to an entire database or entire deployment
# in this case you usually don't need regexes in your config to filter collections unless you target the deployment.
# to listen to an entire db use only the database name.  For a deployment use an empty string.
#change-stream-namespaces = [""]

# additional settings

# if you don't want to listen for changes to all collections in MongoDB but only a few
# e.g. only listen for inserts, updates, deletes, and drops from mydb.mycollection
# this setting does not initiate a copy, it is only a filter on the change event listener
#namespace-regex = ''
# compress requests to Elasticsearch
#gzip = true
# generate indexing statistics
#stats = true
# index statistics into Elasticsearch
#index-stats = true
# use the following PEM file for connections to MongoDB
#mongo-pem-file = ""
# disable PEM validation
#mongo-validate-pem-file = true
# use the following user name for Elasticsearch basic auth
elasticsearch-user = "elastic"
# use the following password for Elasticsearch basic auth
elasticsearch-password = "pwd"
# use 4 go routines concurrently pushing documents to Elasticsearch
elasticsearch-max-conns = 4 
# use the following PEM file to connections to Elasticsearch
#elasticsearch-pem-file = ""
# validate connections to Elasticsearch
#elastic-validate-pem-file = false
# propogate dropped collections in MongoDB as index deletes in Elasticsearch
dropped-collections = true
# propogate dropped databases in MongoDB as index deletes in Elasticsearch
dropped-databases = true
# do not start processing at the beginning of the MongoDB oplog
# if you set the replay to true you may see version conflict messages
# in the log if you had synced previously. This just means that you are replaying old docs which are already
# in Elasticsearch with a newer version. Elasticsearch is preventing the old docs from overwriting new ones.
#replay = false
# resume processing from a timestamp saved in a previous run
resume = true
# do not validate that progress timestamps have been saved
#resume-write-unsafe = false
# override the name under which resume state is saved
#resume-name = "default"
# use a custom resume strategy (tokens) instead of the default strategy (timestamps)
# tokens work with MongoDB API 3.6+ while timestamps work only with MongoDB API 4.0+
resume-strategy = 0
# exclude documents whose namespace matches the following pattern
#namespace-exclude-regex = '^mydb\.ignorecollection$'
# turn on indexing of GridFS file content
#index-files = true
# turn on search result highlighting of GridFS content
#file-highlighting = true
# index GridFS files inserted into the following collections
#file-namespaces = ["users.fs.files"]
# print detailed information including request traces
verbose = true
# enable clustering mode
cluster-name = 'docker-cluster'
# do not exit after full-sync, rather continue tailing the oplog
#exit-after-direct-reads = false

 

　　monstache掛載的配置文件monstache.config.toml需要查看官網或者阿里雲的案例，阿里雲有詳細的配置文件各個key的含義，參考：https://help.aliyun.com/knowledge_detail/171650.html

　　此時完整的docker-compose.yml文件是：

version: '3.7'

services:

  mongo:
    image: mongo:4.0
    restart: always
    container_name: mongodb
    environment:
      TZ: Asia/Shanghai       
    volumes:
      - /data/javaProject/mongo-data:/data/db
      - /data/javaProject/mongo-config/mongod.conf.orig:/etc/mongod.conf.orig
    command: --config /etc/mongod.conf.orig
    ports:
      - 12000:27017
  mongo-express:
    image: mongo-express
    container_name: mongo-express
    restart: always
    depends_on:
      - mongo
    ports:
      - 8081:8081
    environment:
      TZ: Asia/Shanghai
  monstache: 
    image: rwynn/monstache:rel6
    restart: always
    container_name: monstache
    volumes: 
      - /data/javaProject/monstache-conf/monstache.config.toml:/app/monstache.config.toml
    command: -f /app/monstache.config.toml

 

　　此時構建啟動，monstache的日志報錯顯示連接不上：

current topology: { Type: ReplicaSetNoPrimary,

　　查到說是需要mongo副本集開啟后還要初始化，可是我已經初始化過了。遂刪除mongo容器后，全部重新構建，容器啟動后，進入mongo容器重新初始化副本集。這時查看monstache的啟動日志，已經連接成功。

初始化參考：https://blog.csdn.net/jack_brandy/article/details/88887795。此次構建啟動后monstache連接成功：

ERROR 2020/09/28 03:44:32 Unable to connect to MongoDB using URL mongodb://192.168.10.20:12000: server selection error: server selection timeout, current topology: { Type: Unknown, Servers: [{ Addr: 192.168.10.20:12000, Type: RSGhost, State: Connected, Average RTT: 631124 }, ] }


INFO 2020/09/28 03:44:47 Started monstache version 6.7.0


INFO 2020/09/28 03:44:47 Go version go1.14


INFO 2020/09/28 03:44:47 MongoDB go driver v1.3.5


INFO 2020/09/28 03:44:47 Elasticsearch go driver 7.0.18


INFO 2020/09/28 03:44:47 Successfully connected to MongoDB version 4.0.20


INFO 2020/09/28 03:44:47 Successfully connected to Elasticsearch version 7.3.2


INFO 2020/09/28 03:44:47 Sending systemd READY=1


WARN 2020/09/28 03:44:47 Systemd notification not supported (i.e. NOTIFY_SOCKET is unset)


INFO 2020/09/28 03:44:47 Joined cluster docker-cluster


INFO 2020/09/28 03:44:47 Starting work for cluster docker-cluster


INFO 2020/09/28 03:44:47 Watching changes on the deployment


INFO 2020/09/28 03:44:47 Listening for events


INFO 2020/09/28 03:44:47 Resuming from timestamp {T:1601264687 I:5}


INFO 2020/09/28 03:44:47 Direct reads completed

 　　此時在mongo中新增數據庫，新增集合，文檔。登陸kibana可以查看到新增數據。同步完成。