mongodb多實例安裝及replica set配置

本文轉載自查看原文 2019-02-24 23:36 871 mongo

　　需求：為了容災，需要部署兩台服務器，便於主備切換，當主服務器掛掉之后，主輔服務器角色永遠對換，即使之前主服務器恢復也不再重新恢復之前主輔角色，直到下次發生主備切換

分析：mongodb兩種數據復制方式：1、主備，當主服務器掛掉之后，輔服務器不能夠提供寫服務，故mongodb主備方案不合適。2、replica set，但只有兩個節點時，主節點掛掉，輔節點也不能提供寫服務，與主備效果一樣，所以至少需要三個mongodb實例，但當前只有兩台服務器，所以有一台服務器要安裝有兩個mongodb實例，一台機器上不需要兩份數據拷貝，故第三個節點使用arbiter，只進行投票，不會被選為主節點，也不會拷貝數據。另外考慮到需求中主備邏輯，結合replica set特點，arbiter只能在服務器中運行，考慮到主備角色調換，故兩台服務器每台需要安裝兩個mongodb實例，其中一個為arbiter

　　兩台服務器：

　　192.168.103.5 （原始主服務器）

　　192.168.103.28 （原始輔服務器）

一、兩台服務器分別安裝mongodb

安裝步驟可參考：https://www.cnblogs.com/qq931399960/p/10425803.html

二、兩台服務器分別安裝arbiter

1、新建mongodb日志和數據庫存儲路徑

mkdir -p /home/mongo-arbiter/db
mkdir -p /home/mongo-arbiter/log

2、拷貝配置文件

cp /etc/mongod.conf /etc/mongod-arbiter.conf

3、修改配置文件

vim /etc/mongod-arbiter.conf

systemLog.path修改為/home/mongo-arbiter/log/mongod.log

Storage.dbPath修改為/home/mongo-arbiter/db

processManagement.pidFilePath修改為/var/run/mongodb/mongod-arbiter.pid

net.port修改為27018

4、修改權限

chown -R mongod:mongod /home/mongo-arbiter

5、設置arbiter開機啟動

cp /usr/lib/systemd/system/mongod.service /usr/lib/systemd/system/mongod-arbiter.service

修改開機啟動文件

vim /usr/lib/systemd/system/mongod-arbiter.service

Environment改為："OPTIONS=-f /home/mongo-arbiter/mongod.conf"

PIDFile改為：/var/run/mongodb/mongod-arbiter.pid

使配置生效

systemctl enable mongod-arbiter.service
systemctl daemon-reload

初始化用戶（四個實例的用戶要一致）

 cat ./mongdb/initUser.js | mongo --port 27018 --shell

6、登錄檢測

mongo  -uroot -pabc123 --authenticationDatabase "admin" --port 27018

正常登陸，ok

三、配置replica set

1、停止所有mongodb實例

2、生成keyfile文件

openssl rand -base64 756 > /home/replSetKey
chmod 400 /home/replSetKey
chown -R mongod:mongod /home/replSetKey

2、將上述文件拷貝到另外一台服務器/home下（路徑可自定義，只要四個實例使用相同kefile文件即可）

3、添加/修改mongodb實例配置（如下配置每個實例相同，4個mongodb實例都需要修改，如下列出的只有新加和修改的，不變的未列出）

security:
  keyFile: /home/replSetKey      ##  新加
replication:
  replSetName: mongoSet         ##  新加
net:
  bindIp: 0.0.0.0   ## 修改

4、啟動主服務器默認27017端口mongodb實例，啟動輔服務器中兩個mongodb實例

systemctl start mongod
systemctl start mongod-arbiter

5、查看防火牆是否關閉，或者27017和27018端口是否放開

未關閉防火牆或者以上端口不能訪問，則初始化replica set失敗

關閉防火牆命令

systemctl stop firewalld
systemctl disable firewalld

6、在主服務器（103.5）進入mongodb命令

mongo -u "root" -p "abc123" --authenticationDatabase "admin"

7、初始化兩個memeber的replicat set

config = {_id:"mongoSet", members:[{_id:0,host:"192.168.103.5:27017",priority:2},
　　　　　　　　　　　　　　　　　　{_id:1,host:"192.168.103.28:27017",priority:1}]}
rs.initiate(config, true)

此時可通過rs.status()狀態

8、添加arbiter仲裁節點

rs.addArb("192.168.103.28:27018")

rs.status()查看各節點狀態，如果有節點stateStr顯示not reachable/healthy，health顯示為0，則重啟該節點，再次通過rs.status()查看是否正常

到此，主備服務基本配置結束

java客戶端連接地址為：

mongodb://root:abc123@192.168.103.5:27017,192.168.103.28:27017/?authSource=mydb&replicaSet=mongoSet

四、重新配置replica set

當主備服務器切換，java客戶端連接由於配置了replicaSet，可以自動去新主節點進行業務處理，不會有問題，當主設備恢復為后，原主服務器mongodb再次成為了主節點，但其他服務已經在新主服務器上

A、mongodb和web服務不在台機器，中間有網絡傳輸，影響性能

B、此時的replica set中，主節點在輔服務器上，主服務器上只有輔節點和arbiter，當主服務器掛掉，只有一個mongodb節點，不能在提供寫服務

綜上兩點，尤其是第二點，需要重新配置replica set，登錄到mongodb主節點

mongo  -uroot -pabc123 --authenticationDatabase "admin"

或者在主服務器遠程登錄恢復的mongodb主節點

mongo -u "root" -p "abc123" --authenticationDatabase "admin" --host 192.168.103.5

重新配置

reconfig = {_id:"mongoSet","protocolVersion" : 1, members:[{_id:0,host:"192.168.103.5:27017",priority:1},
　　　　　　　　　　　　　　　　　　{_id:1,host:"192.168.103.28:27017",priority:2}]}

rs.reconfig(reconfig, true)

添加arbiter節點

rs.addArb("192.168.103.5:27018")

使用rs.status()查看狀態，如果有節點stateStr顯示not reachable/healthy，health顯示為0，則重啟該節點，再次通過rs.status()查看是否正常

PS：

1、若有節點stateStr出現not reachable/healthy，health顯示為0，則該節點可能是arbiter節點，官方不推薦將arbiter節點安裝到存在primary和slave節點的服務器上，不過當前只有兩台服務器，也只有這樣了，可以正常運行。

2、重新配置時，需要保證之前已存在的host的id是相同的

3、重新配置需要添加protocolVersion

五、使用腳本簡化操作

1、提前准備好配置文件和arbiter開機啟動service文件，需要的字段都添加上,但replica set相關字段為注釋狀態，因為打開后，所有節點都將無法進行寫操作，新建用戶后打開，並重啟mongodb

2、提前准備好kefile，兩台服務器都使用該文件

3、安裝mongodb (數據庫和日志路徑可以自定義)

#!/bin/bash
basedir=$(cd "$(dirname "$0")";pwd)
function initenv(){
  setenforce 0
  sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
}
########inst mongdb 4.0#############
function inst(){
  cp -f ./mongdb/mongod.conf /etc/
  rpm -ivh ./mongdb/*rpm
  cd /home
  mkdir -p mongo-db
  mkdir -p mongo-log
  chown -R mongod:mongod ./mongo-db
  chown -R mongod:mongod ./mongo-log

  systemctl enable mongod
  systemctl start mongod
}
function initdb(){
  cd $basedir
  cat ./mongdb/initUser.js | mongo --shell
  cat ./mongdb/initIndex.js | mongo -u "root" -p "abc123" --authenticationDatabase "admin"
}

function cpReplicaSetKey(){
    cd $basedir
    cp ./mongdb/replSetKey /home/
    chmod 400 /home/replSetKey
    chown -R mongod:mongod /home/replSetKey
}

function openReplicaSet(){
    sed -i 's/#keyFile/keyFile/g' /etc/mongod.conf
    sed -i 's/#replication/replication/g' /etc/mongod.conf
    sed -i 's/#replSetName/replSetName/g' /etc/mongod.conf
}

function stopMongo(){
    systemctl stop mongod
}
function startMongo(){
    systemctl start mongod
}

function main(){
  initenv
  inst
  initdb
  stopMongo
  cpReplicaSetKey
  openReplicaSet
  startMongo
}

main | tee -a ./log/inst_mongodb.log

View Code

4、安裝mongodb arbiter節點（在rs.addArb()之前其實還不能說是arbiter節點，只是起了個名字叫arbiter節點）

#!/bin/bash
basedir=$(cd "$(dirname "$0")";pwd)
function initenv(){
  setenforce 0
  sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
}
########inst mongdb 4.0#############
function inst(){
  cp -f ./mongdb/mongod-arbiter.conf /etc/
  cp -f ./mongdb/mongod-arbiter.service /usr/lib/systemd/system/
  
  mkdir -p /home/mongodb-arbiter/log
  mkdir -p /home/mongodb-arbiter/db
  chown -R mongod:mongod /home/mongodb-arbiter

  systemctl enable mongod-arbiter
  systemctl start mongod-arbiter
}
function initdb(){
  cd $basedir
  cat ./mongdb/initUser.js | mongo --port 27018 --shell
  cat ./mongdb/initIndex.js | mongo --port 27018 -u "root" -p "abc123" --authenticationDatabase "admin"
}

function openReplicaSet(){
    sed -i 's/#keyFile/keyFile/g' /etc/mongod-arbiter.conf
    sed -i 's/#replication/replication/g' /etc/mongod-arbiter.conf
    sed -i 's/#replSetName/replSetName/g' /etc/mongod-arbiter.conf
}

function stopMongo(){
    systemctl stop mongod
}
function startMongo(){
    systemctl start mongod
}

function main(){
  initenv
  inst
  initdb
  stopMongo
  openReplicaSet
  startMongo
}

main | tee -a ./log/inst_mongodb_arbiter.log

View Code

5、初始replica set

#!/bin/bash
basedir=$(cd "$(dirname "$0")";pwd)

source ./app.properties

nowtime=`date --date='0 days ago' "+%Y%m%d%H%M%S"`
daytime=`date --date='0 days ago' "+%Y%m%d"`

logPath=./log/init_repliaset_$daytime.log
function log(){
    echo $1 | tee -a $logPath
}

function logfile(){
    echo $1 >> $logPath
}

function testConfig(){
    log "check relate config."
    if [ "$masterServerIp" == "masterIp" ];then
        log "ERROR: masterServerIp is not correct in the file app.properties"
        exit 1
    fi

    if [ "$slaveServerIp" == "slaveIp" ];then
        log "ERROR: slaveServerIp is not correct in the file app.properties"
        exit 1
    fi

    local grepIp=`ifconfig | grep $masterServerIp`

    ## it is must execute in the master server which you input in the app.properties file when first initiate ths replica set
    if [ "$grepIp" == "" ];then
        log "the current server is not the master server,  please login on to $masterServerIp and execute this script."
        exit 1
    fi

    ## test the connection
    echo "use admin" > test.js
    echo "rs.slaveOk()" >> test.js
    echo "show dbs" >> test.js

    cat test.js | mongo --host $slaveServerIp -uroot -pabc123 --authenticationDatabase "admin" >> $logPath

    if [ $? != 0 ];then
            log "mongodb instance which port is 27017 in $slaveServerIp server can not connect."
            exit 1;
    fi

    cat test.js | mongo --host $slaveServerIp --port 27018 -uroot -pabc123 --authenticationDatabase "admin" >> $logPath

    if [ $? != 0 ];then
            log "mongodb instance which port is 27018 in $slaveServerIp server can not connect."
            exit 1;
    fi

    cat test.js | mongo --host $masterServerIp -uroot -pabc123 --authenticationDatabase "admin" >> $logPath

    if [ $? != 0 ];then
            log "mongodb instance which port is 27017 in $slaveServerIp server can not connect."
            exit 1;
    fi

    rm -rf test.js
    log "check relate config end."
}

function initTwoMemberReplicaSet(){
    log "init two member replica set."
    echo "use admin" > init.js
    echo "config = {_id:\"mongoSet\", members:[{_id:0,host:\"$masterServerIp:27017\",priority:2},{_id:1,host:\"$slaveServerIp:27017\",priority:1}]}" >> init.js
    echo "rs.initiate(config, {force:true})" >> init.js
    cat init.js | mongo -uroot -pabc123 --authenticationDatabase "admin" >> $logPath
    ## sleep 3 second for primary election.
    sleep 3s
    log "init two member replica set end."
}

function isElectPrimary(){
    log "check is the master node selected."
    local masterStr=`mongo  -uroot -pabc123 --authenticationDatabase "admin" --eval "rs.status()" | grep "stateStr" -B 4|grep "PRIMARY" -B 4|grep "name" | grep $masterServerIp`
    local count=0
    while [[ "$masterStr" == "" ]];do
            sleep 1s
            count=$((count+1))
            masterStr=`mongo  -uroot -pabc123 --authenticationDatabase "admin" --eval "rs.status()" | grep "stateStr" -B 4|grep "PRIMARY" -B 4|grep "name" | grep $masterServerIp`
            if [ "$masterStr" != "" ];then
                log "the master node is selected."
                break
            fi
            if [ "$count" == "30" ];then
                log "ERROR: arbiter configure timeout, please configure manually with 'rs.addArb(\"$slaveServerIp:27018\")' in $masterServerIp:27017"
                exit 1
            fi
    done
    log "check is the master node selected end."
}

function addArbiter(){
    log "add arbiter ..."
    mongo  -uroot -pabc123 --authenticationDatabase "admin" --eval "rs.addArb(\"$slaveServerIp:27018\")"
    log "add arbiter end."
}

function chekArbiter(){
    log "check arbiter."
    local arbiterStr=`mongo  -uroot -pabc123 --authenticationDatabase "admin" --eval "rs.status()" | grep "stateStr" -B 4|grep "$slaveServerIp:27018" -A 3| grep "stateStr"`
    if [ "$arbiterStr" != "ARBITER" ];then
        log "ERROR: abriter $slaveServerIp:27018: $arbiterStr, may be you need restart mongodb with 'systemctl restart mongod-arbiter' command in $slaveServerIp."
    fi
    log "check arbiter end."
}

logfile "**************************$nowtime**********************************"
testConfig
initTwoMemberReplicaSet
isElectPrimary
addArbiter
chekArbiter
log "init end..."

View Code

6、重新配置replica set

#!/bin/bash
basedir=$(cd "$(dirname "$0")";pwd)

source ./app.properties

nowtime=`date --date='0 days ago' "+%Y%m%d%H%M%S"`
daytime=`date --date='0 days ago' "+%Y%m%d"`

logPath=./log/init_repliaset_$daytime.log
function log(){
    echo $1 | tee -a $logPath
}

function logfile(){
    echo $1 >> $logPath
}

function testConfig(){
    log "check relate config."
    if [ "$masterServerIp" == "masterIp" ];then
        log "ERROR: masterServerIp is not correct in the file app.properties"
        exit 1
    fi

    if [ "$slaveServerIp" == "slaveIp" ];then
        log "ERROR: slaveServerIp is not correct in the file app.properties"
        exit 1
    fi

    ## get current replica set master node
    local currMasterNode=`mongo  -uroot -pabc123 --authenticationDatabase "admin" --eval "rs.status()" | grep "stateStr" -B 4|grep "PRIMARY" -B 4|grep "name" | cut -d ":" -f 2 | sed 's/\"//g'`
    
    if [ "`ifconfig | grep $currMasterNode`" == "" ];then
        log "the current server has no mongodb master node, please login on to $currMasterNode execute this script."
        exit 1
    fi
    ## test the connection

    echo "use admin" > test.js
    echo "rs.slaveOk()" >> test.js
    echo "show dbs" >> test.js

    cat test.js | mongo --host $slaveServerIp -uroot -pabc123 --authenticationDatabase "admin" >> $logPath

    if [ $? != 0 ];then
            log "mongodb instance which port is 27017 in $slaveServerIp server can not connect."
            exit 1;
    fi

    cat test.js | mongo --host $masterServerIp -uroot -pabc123 --authenticationDatabase "admin" >> $logPath

    if [ $? != 0 ];then
            log "mongodb instance which port is 27017 in $slaveServerIp server can not connect."
            exit 1;
    fi

    cat test.js | mongo --host $slaveServerIp --port 27018 -uroot -pabc123 --authenticationDatabase "admin" >> $logPath

    if [ $? != 0 ];then
            log "mongodb instance which port is 27018 in $slaveServerIp server can not connect."
            ## 有時，當主從服務器都重啟之后，有一個仲裁節點由於不在replica set中，但配置文件中卻存在replica的配置，造成不能夠正常啟動，但reconfig replica set后，需要將該節點reconfig到replica set中，故此處不能因為不能連接就執行exit，但輸出錯誤信息給出提示，即使服務未啟動，
            ## 執行rs.addArb()也可以添加仲裁節點，但必須確定添加的仲裁節點會啟動，否則即使通過rs.remove刪除掉了該仲裁節點，但主節點日志中也會一直打印錯誤信息，
    fi
    
    rm -rf test.js
    log "check relate config end."
}

function initTwoMemberReplicaSet(){
    log "reconfig two member replica set."
    echo "use admin" > reconfig.js
    ## the _id must be the same as before
    ## find the master server id in previous replica set
    local masterId=`mongo  -uroot -pabc123 --authenticationDatabase "admin" --eval "rs.status()" | grep "stateStr" -B 4|grep "$masterServerIp:27017" -B 1|grep "_id" | cut -d ":" -f 2|sed 's/,//g'`
    ## find the slave server id in previous replica set
    local slaveId=`mongo  -uroot -pabc123 --authenticationDatabase "admin" --eval "rs.status()" | grep "stateStr" -B 4|grep "$slaveServerIp:27017" -B 1|grep "_id" | cut -d ":" -f 2|sed 's/,//g'`
    echo "config = {_id:\"mongoSet\", "protocolVersion" : 1, members:[{_id:$masterId,host:\"$masterServerIp:27017\",priority:2},{_id:$slaveId,host:\"$slaveServerIp:27017\",priority:1}]}" >> reconfig.js
    echo "rs.reconfig(config, {force:true})" >> reconfig.js
    cat reconfig.js | mongo -uroot -pabc123 --authenticationDatabase "admin" >> $logPath
    ## sleep 3 second for primary election.
    sleep 3s
    log "reconfig two member replica set end."
}

function isElectPrimary(){
    log "check is the master node selected."
    local masterStr=`mongo  -uroot -pabc123 --authenticationDatabase "admin" --eval "rs.status()" | grep "stateStr" -B 4|grep "PRIMARY" -B 4|grep "name" | grep $masterServerIp`
    local count=0
    while [[ "$masterStr" == "" ]];do
            sleep 1s
            count=$((count+1))
            masterStr=`mongo  -uroot -pabc123 --authenticationDatabase "admin" --eval "rs.status()" | grep "stateStr" -B 4|grep "PRIMARY" -B 4|grep "name" | grep $masterServerIp`
            if [ "$masterStr" != "" ];then
                log "the master node is selected."
                break
            fi
            if [ "$count" == "30" ];then
                log "ERROR: arbiter configure timeout, please configure manually with 'rs.addArb(\"$slaveServerIp:27018\")' in $masterServerIp:27017"
                exit 1
            fi
    done
    log "check is the master node selected end."
}

function addArbiter(){
    log "remove arbiter in the master server."
    ## master server do not deploy arbiter, when reconfig, after the master node reselected you must specify the --post for delete or add
    local beforeArbiterIp=`mongo  -uroot -pabc123 --authenticationDatabase "admin" --eval "rs.status()" | grep "stateStr" -B 4|grep "ARBITER" -B 4|grep "name" | cut -d ":" -f 2 | sed 's/\"//g'`
    if [ "`echo $masterServerIp | grep $beforeArbiterIp`" != "" ];then
        log "remove $masterServerIp:27018 in replica set."
        mongo -uroot -pabc123 --authenticationDatabase "admin" --host $masterServerIp --eval "rs.remove(\"$masterServerIp:27018\")"
    fi
    log "remove arbiter in the master server end."
    log "add arbiter ..."
    mongo  -uroot -pabc123 --authenticationDatabase "admin" --host $masterServerIp --eval "rs.addArb(\"$slaveServerIp:27018\")"
    log "add arbiter end."
}

function chekArbiter(){
    log "check arbiter."
    local arbiterStr=`mongo  -uroot -pabc123 --authenticationDatabase "admin" --eval "rs.status()" | grep "stateStr" -B 4|grep "$slaveServerIp:27018" -A 3 | grep "ARBITER"`
    ## arbiter connect need time， set to 15 seconds
    local count=0
    while [[ "$arbiterStr" == "" ]];do
        sleep 1s
        count=$((count+1))
        arbiterStr=`mongo  -uroot -pabc123 --authenticationDatabase "admin" --eval "rs.status()" | grep "stateStr" -B 4|grep "$slaveServerIp:27018" -A 3 | grep "ARBITER"`
        if [ "$arbiterStr" != "" ];then
            log "the arbiter node is selected."
            break
        fi
        ## wait for 15 seconds
        if [ "$count" == "15" ];then
            log "ERROR: abriter $slaveServerIp:27018: $arbiterStr, may be you need restart mongodb with 'systemctl restart mongod-arbiter' command in $slaveServerIp."
            break
        fi
    done
    log "check arbiter end."
}

logfile "**************************$nowtime**********************************"
testConfig
initTwoMemberReplicaSet
isElectPrimary
addArbiter
chekArbiter


log "init end..."

View Code

腳本中通過rs.addArb()添加未啟動或者不存在的arbiter節點，可以添加成功，但主節點日志中會一直打印該arbiter節點相關的錯誤信息，即使通過rs.remove()刪除掉該arbiter節點，錯誤信息也會打印，該問題，重啟主節點可以解決

7、配置文件app.properties

masterServerIp=masterIp
slaveServerIp=slaveIp

View Code

執行初始化和重新配置replica set之前需要確認app.properties中配置是正確的

8、無論是初始化replica set或者重新配置replica set，前幾次都會出現not reachable/healthy，此時根據shell頁面打印的提示，在對應的服務器上執行systemctl restart mongod-arbiter即可

免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。

猜您在找 [MongoDB] 安裝MongoDB配置Replica Set MongoDB的Replica Set以及Auth的配置 MongoDB 維護Replica Set MongoDB Replica Set 集群 MongoDB 主從和Replica Set mongodb replica set介紹 MongoDB的集群模式--Replica Set MongoDB Replica Set（docker方式） MongoDB（五）-- 副本集（replica Set） mongodb replica set 和 nodejs中使用mongoose連接replica