MongoDBV3.0.7版本(shard+replica)集群的搭建及驗證

• 集群的模塊介紹：

從MongoDB官方給的集群架構了解，整個集群主要有4個模塊：Config Server、mongs、 shard、replica set：

Config Server：用來存放集群的元數據，也就是存放所有分片的的配置數據，mongos第一次啟動就需要連接configServer讀取相關數據，當configServer有數據進行更新時，也會主動推送消息到所有的mongos上，在3.0.7版本中，官方是建議配置3份的Config Server，以便掛掉兩台時，業務還能夠正常運轉。

 

mongs：Mongodb集群的的請求入口，能否自動實現數據的分布式分發，生產環境中建議部署在應用服務器上。

 

shard：分片就比如是將一張大表分散在幾個不同的shard中，實現數據分布式存儲。

replica set：主要是對每個分片進行冗余，生產環境中，一般將將副本集配置在三個節點上，兩份副本、一份仲裁。

 

• 環境規划：

• 端口及安裝路徑規划：

用途	IP	端口	備注	安裝路徑
ConfigeServer	172.16.16.120	30001		/db/configS
	172.16.16.121	30001		/db/configS
	172.16.16.122	30001		/db/configS
share1	172.16.16.124	40001	Shard1主節點	/db/shard1
	172.16.16.125	40001	Shard1副本節點	/db/shard1
	172.16.16.126	40001	Shard1仲裁節點	/db/shard1
share2	172.16.16.125	40002	Shard2主節點	/db/shard2
	172.16.16.126	40002	Shard2副本節點	/db/shard2
	172.16.16.131	40002	Shard2仲裁節點	/db/shard2
share3	172.16.16.126	40003	Shard3主節點	/db/shard3
	172.16.16.131	40003	Shard3副本節點	/db/shard3
	172.16.16.124	40003	Shard3仲裁節點	/db/shard3
share4	172.16.16.121	40004	Shard4主節點	/db/shard4
	172.16.16.124	40004	Shard4副本節點	/db/shard4
	172.16.16.125	40004	Shard4仲裁節點	/db/shard4
mongos	172.16.16.124	50001	生產環境中一般直接部署在應用端	/db/mongos
	172.16.16.125	50001		/db/mongos
	172.16.16.126	50001		/db/mongos
	172.16.16.131	50001		/db/mongos

 

• 搭建步驟：

• 下載mongodb(https://www.mongodb.org/downloads ) ，目前的最新版本是 3.0.7

opt]# tar zxvf mongodb-linux-x86_64-rhel55-3.0.7.gz
opt]# mv mongodb-linux-x86_64-rhel55-3.0.7 /usr/local/mongodb
opt]# useradd mongo
opt]# passwd mongo
Changing password for user mongo.
New UNIX password:
BAD PASSWORD: it is too simplistic/systematic
Retype new UNIX password:
passwd: all authentication tokens updated successfully.
opt]# chown -R mongo:mongo /usr/local/mongodb/
opt]# chown -R  mongo:mongo /db

• 創建相關模塊存放路徑：

• 創建configeServer目錄(172.16.16.120/121/122)：

#mkdir -p  /db/configS/data & mkdir -p /db/configS/log (存放ConfigServer的數據、日志)

• 創建shard1目錄(172.16.16.124/125/126)：

#mkdir -p  /db/shard1/data & mkdir -p /db/shard1/log (存放shard1的數據、日志)

• 創建shard2目錄(172.16.16.125/126/131 ):

#mkdir -p  /db/shard2/data & mkdir -p /db/shard2/log (存放shard2的數據、日志)

• 創建shard3目錄(172.16.16.126/131/124 ):

#mkdir -p  /db/shard3/data & mkdir -p /db/shard3/log (存放shard3的數據、日志)

• 創建shard4目錄(172.16.16.131/124/125 ):

#mkdir -p  /db/shard4/data & mkdir -p /db/shard4/log (存放shard4的數據、日志)

• 創建mongos目錄(172.16.16.124/125/126/131)

#mkdir -p  /db/mongos/log (由於mongos只做路由使用，不存數據，所以只需要建立log目錄)

•  模塊配置及啟動：

• configServer(172.16.16.120/121/122)配置及服務啟動:

編寫 /usr/local/mongodb/conf/configServer.conf，將參數都放在該文件中：

#vim /usr/local/mongodb/conf/configServer.conf
#!/bin/bash
systemLog:
   destination: file
   path: "/db/configS/log/configServer.log"                                #日志存儲位置
   logAppend: true
storage:
   journal:                                                                #journal配置
      enabled: true
   dbPath: "/db/configS/data"                                              #數據文件存儲位置
   directoryPerDB: true                                                    #是否一個庫一個文件夾    
   engine: wiredTiger                                                      #數據引擎   
   wiredTiger:                                                             #WT引擎配置
      engineConfig:
         cacheSizeGB: 6                                                    #設置為6G,默認為物理內存的一半
         directoryForIndexes: true                                         #是否將索引也按數據庫名單獨存儲
         journalCompressor: zlib
      collectionConfig:                                                    #表壓縮配置
         blockCompressor: zlib
      indexConfig:                                                         #索引配置
         prefixCompression: true
net:                                                                       #端口配置
   port: 30001                                                             #另外兩台需要分別修改為30002、30003
processManagement:                                                         #配置啟動管理方式
   fork: true
sharding:                                                                  #分片配置
   clusterRole: configsvr                                                  #分片角色

啟動configServer：

conf]$ /usr/local/mongodb/bin/mongod -f /usr/local/mongodb/conf/configServer.conf

• mongos(172.16.16.124/125/126/131)配置及服務啟動:

編寫mongos.conf，將參數都放在該文件中(4台配置文件都一樣)：

#vim /usr/local/mongodb/conf/mongos.conf
#!/bin/bash
systemLog:
    destination: file
    path: "/db/mongos/log/mongos.log"
    logAppend: true
net:
    port: 50001
sharding:
    configDB: 172.16.16.120:30001,172.16.16.121:30001,172.16.16.122:30001
processManagement:
    fork: true

啟動mongos：應保證集群中設備的時間都是一樣的，否則啟動mongos會報錯，若不相同，可先搭建一套NTP服務器

conf]$ /usr/local/mongodb/bin/mongos -f /usr/local/mongodb/conf/mongos.conf

• shard1分片+副本集配置及服務啟動(172.16.16.124/125/126 )：

#vim /usr/local/mongodb/conf/shard1.conf
#!/bin/bash
systemLog:
   destination: file                                                                     
   path: "/db/shard1/log/shard1.log"                                #日志存儲位置
   logAppend: true
storage:                                                                                   
   journal:                                                         #journal配置
      enabled: true
   dbPath: "/db/shard1/data"                                        #數據文件存儲位置
   directoryPerDB: true                                             #是否一個庫一個文件夾    
   engine: wiredTiger                                               #數據引擎   
   wiredTiger:                                                      #WT引擎配置
      engineConfig:
         cacheSizeGB: 6                                             #設置為6G,默認為物理內存的一半
         directoryForIndexes: true                                  #是否將索引也按數據庫名單獨存儲
         journalCompressor: zlib
      collectionConfig:                                             #表壓縮配置
         blockCompressor: zlib
      indexConfig:                                                  #索引配置
         prefixCompression: true
net:                                                                #端口配置
   port: 40001
processManagement:                                                  #配置啟動管理方式
   fork: true
sharding:                                                           #分片配置
   clusterRole: shardsvr
replication:                                                        
   replSetName: shard1                                              #配置副本集名稱

啟動shard1 mongod：

conf]$ /usr/local/mongodb/bin/mongod -f /usr/local/mongodb/conf/shard1.conf

•  shard2分片+副本集配置及服務啟動(172.16.16.125/126/131 )：

#vim /usr/local/mongodb/conf/shard2.conf
#!/bin/bash
systemLog:
   destination: file                                                                     
   path: "/db/shard2/log/shard2.log"                                #日志存儲位置
   logAppend: true
storage:                                                                                   
   journal:                                                         #journal配置
      enabled: true
   dbPath: "/db/shard2/data"                                        #數據文件存儲位置
   directoryPerDB: true                                             #是否一個庫一個文件夾    
   engine: wiredTiger                                               #數據引擎   
   wiredTiger:                                                      #WT引擎配置
      engineConfig:
         cacheSizeGB: 6                                             #設置為6G,默認為物理內存的一半
         directoryForIndexes: true                                  #是否將索引也按數據庫名單獨存儲
         journalCompressor: zlib
      collectionConfig:                                             #表壓縮配置
         blockCompressor: zlib
      indexConfig:                                                  #索引配置
         prefixCompression: true
net:                                                                #端口配置
   port: 40002
processManagement:                                                  #配置啟動管理方式
   fork: true
sharding:                                                           #分片配置
   clusterRole: shardsvr
replication:                                                       
   #oplogSizeMB: 
   replSetName: shard2                                              #配置副本集名稱

啟動shard2 mongod：

conf]$ /usr/local/mongodb/bin/mongod -f /usr/local/mongodb/conf/shard2.conf

• shard3分片+副本集配置及服務啟動(172.16.16.126/131/124 )：

#vim /usr/local/mongodb/conf/shard3.conf
#!/bin/bash
systemLog:
   destination: file                                                                     
   path: "/db/shard3/log/shard3.log"                                #日志存儲位置
   logAppend: true
storage:                                                                                   
   journal:                                                         #journal配置
      enabled: true
   dbPath: "/db/shard3/data"                                        #數據文件存儲位置
   directoryPerDB: true                                             #是否一個庫一個文件夾    
   engine: wiredTiger                                               #數據引擎   
   wiredTiger:                                                      #WT引擎配置
      engineConfig:
         cacheSizeGB: 6                                             #設置為6G,默認為物理內存的一半
         directoryForIndexes: true                                  #是否將索引也按數據庫名單獨存儲
         journalCompressor: zlib
      collectionConfig:                                             #表壓縮配置
         blockCompressor: zlib
      indexConfig:                                                  #索引配置
         prefixCompression: true
net:                                                                #端口配置
   port: 40003
processManagement:                                                  #配置啟動管理方式
   fork: true
sharding:                                                           #分片配置
   clusterRole: shardsvr
replication:                                                              
   #oplogSizeMB: 
   replSetName: shard3                                              #配置副本集名稱

啟動shara3 mongod：

conf]$ /usr/local/mongodb/bin/mongod -f /usr/local/mongodb/conf/shard3.conf

• shard4分片+副本集配置及服務啟動(172.16.16.131/124/125 )：

#vim /usr/local/mongodb/conf/shard4.conf
#!/bin/bash
systemLog:
   destination: file                                                                     
   path: "/db/shard4/log/shard4.log"                                #日志存儲位置
   logAppend: true
storage:                                                                                   
   journal:                                                         #journal配置
      enabled: true
   dbPath: "/db/shard4/data"                                        #數據文件存儲位置
   directoryPerDB: true                                             #是否一個庫一個文件夾    
   engine: wiredTiger                                               #數據引擎   
   wiredTiger:                                                      #WT引擎配置
      engineConfig:
         cacheSizeGB: 6                                             #設置為6G,默認為物理內存的一半
         directoryForIndexes: true                                  #是否將索引也按數據庫名單獨存儲
         journalCompressor: zlib
      collectionConfig:                                             #表壓縮配置
         blockCompressor: zlib
      indexConfig:                                                  #索引配置
         prefixCompression: true
net:                                                                #端口配置
   port: 40004
processManagement:                                                  #配置啟動管理方式
   fork: true
sharding:                                                           #分片配置
   clusterRole: shardsvr
replication:                                                          
   #oplogSizeMB: 
   replSetName: shard4                                              #復制集名

啟動shara4 mongod：

conf]$ /usr/local/mongodb/bin/mongod -f /usr/local/mongodb/conf/shard4.conf

•  集群配置：

• 副本集配置(在每個 shard的主節點上進行配置及初始化，否則會初始化失敗 )：

• shard1的副本集配置(主節點、副本節點、仲裁節點)：

bin]$ ./mongo 172.16.16.124:40001
MongoDB shell version: 3.0.7
connecting to: 172.16.16.124:40001/test
> use admin
switched to db admin
> config = { _id:"shard1", members:[
                 {_id:0,host:"172.16.16.124:40001"},
                 {_id:1,host:"172.16.16.125:40001"},
                 {_id:2,host:"172.16.16.126:40001",arbiterOnly:true}]
        }

#以下為輸出
{
        "_id" : "shard1",
        "members" : [
                {
                        "_id" : 0,
                        "host" : "172.16.16.124:40001"
                },
                {
                        "_id" : 1,
                        "host" : "172.16.16.125:40001"
                },
                {
                        "_id" : 2,
                        "host" : "172.16.16.126:40001",
                        "arbiterOnly" : true
                }
        ]
}
> rs.initiate(config);  #初始化配置
{ "ok" : 1 } 
 

• shard2的副本集配置(主節點、副本節點、仲裁節點)：

bin]$ ./mongo 172.16.16.125:40002
MongoDB shell version: 3.0.7
connecting to: 172.16.16.125:40002/test
> use admin
switched to db admin
> config = { _id:"shard2", members:[
                     {_id:0,host:"172.16.16.125:40002"},
                    {_id:1,host:"172.16.16.126:40002"},
                   {_id:2,host:"172.16.16.131:40002",arbiterOnly:true}] }
 
#以下為輸出
{
        "_id" : "shard2",
        "members" : [
                {
                        "_id" : 0,
                        "host" : "172.16.16.125:40002"
                },
                {
                        "_id" : 1,
                        "host" : "172.16.16.126:40002"
                },
                {
                        "_id" : 2,
                        "host" : "172.16.16.131:40002",
                        "arbiterOnly" : true
                }
        ]
}
> rs.initiate(config);  #初始化配置  
{ "ok" : 1 }

• shard3的副本集配置(主節點、副本節點、仲裁節點)：

bin]$ ./mongo 172.16.16.126:40003
MongoDB shell version: 3.0.7
connecting to: 172.16.16.126:40003/test
> use admin
switched to db admin
> config = { _id:"shard3", members:[
                    {_id:0,host:"172.16.16.126:40003"},
                     {_id:1,host:"172.16.16.131:40003"},
                    {_id:2,host:"172.16.16.124:40003",arbiterOnly:true}] }  
#以下為輸出
{
        "_id" : "shard3",
        "members" : [
                {
                        "_id" : 0,
                        "host" : "172.16.16.126:40003"
                },
                {
                        "_id" : 1,
                        "host" : "172.16.16.131:40003"
                },
                {
                        "_id" : 2,
                        "host" : "172.16.16.124:40003",
                        "arbiterOnly" : true
                }
        ]
}
> rs.initiate(config);  #初始化配置
{ "ok" : 1 }
 

•  shard4的副本集配置(主節點、副本節點、仲裁節點)：

bin]$ ./mongo 172.16.16.131:40004
MongoDB shell version: 3.0.7
connecting to: 172.16.16.131:40004/test
> use admin
switched to db admin
> config = { _id:"shard4", members:[
                    {_id:0,host:"172.16.16.131:40004"},
                     {_id:1,host:"172.16.16.124:40004"},
                    {_id:2,host:"172.16.16.125:40004",arbiterOnly:true}] }
 
#以下為輸出
{
        "_id" : "shard4",
        "members" : [
                {
                        "_id" : 0,
                        "host" : "172.16.16.131:40004"
                },
                {
                        "_id" : 1,
                        "host" : "172.16.16.124:40004"
                },
                {
                        "_id" : 2,
                        "host" : "172.16.16.125:40004",
                        "arbiterOnly" : true
                }
        ]
}
> rs.initiate(config);  #初始化配置
{ "ok" : 1 }
 

• 分片配置

bin]$ ./mongo 172.16.16.124:50001
mongos> use admin
switched to db admin
mongos> db.runCommand({addshard:"shard1/172.16.16.124:40001,172.16.16.125:40001,172.16.16.126:40001"});
{ "shardAdded" : "shard1", "ok" : 1 }
 
mongos>db.runCommand({addshard:"shard2/172.16.16.125:40002,172.16.16.126:40002,172.16.16.131:40002"});
{ "shardAdded" : "shard2", "ok" : 1 }
 
mongos>db.runCommand({addshard:"shard3/172.16.16.126:40003,172.16.16.131:40003,172.16.16.124:40003"});
{ "shardAdded" : "shard3", "ok" : 1 }
 
mongos>db.runCommand({addshard:"shard4/172.16.16.131:40004,172.16.16.124:40004,172.16.16.125:40004"});
{ "shardAdded" : "shard4", "ok" : 1 }

• 查看配置是否生效(仲裁不被列出 )：

mongos> db.runCommand( { listshards : 1 } );
{
        "shards" : [
                {
                        "_id" : "shard1",
                        "host" : "shard1/172.16.16.124:40001,172.16.16.125:40001"
                },
                {
                        "_id" : "shard2",
                        "host" : "shard2/172.16.16.125:40002,172.16.16.126:40002"
                },
                {
                        "_id" : "shard3",
                        "host" : "shard3/172.16.16.126:40003,172.16.16.131:40003"
                },
                {
                        "_id" : "shard4",
                        "host" : "shard4/172.16.16.124:40004,172.16.16.131:40004"
                }
        ],
        "ok" : 1
}

以上就完成了MongoDB shard+replica模式的集群搭建，接下來做業務測試。

 

• 集群測試

默認情況下，庫和集合沒有自動分片的，若有數據寫入，只會往一個shard中存儲，做個測試驗證：

bin]$ ./mongo 172.16.16.131:50001
MongoDB shell version: 3.0.7
connecting to: 172.16.16.131:50001/test
mongos> use ljaidb
switched to db ljaidb
mongos> for (var i=1;i<=10000;i++) db.ljaitable.save({"name":"ljai","age":27,"addr":"fuzhou"})
WriteResult({ "nInserted" : 1 })
mongos> db.ljaitable.stats()
{
        "sharded" : false,
        "primary" : "shard1",
        "ns" : "ljaidb.ljaitable",
        "count" : 10000,
        "size" : 670000,
        "avgObjSize" : 67,
        "storageSize" : 49152,
        "capped" : false,
        "wiredTiger" : {
                "metadata" : {
                        "formatVersion" : 1
                }
mongos> db.printShardingStatus()
--- Sharding Status ---
  sharding version: {
        "_id" : 1,
        "minCompatibleVersion" : 5,
        "currentVersion" : 6,
        "clusterId" : ObjectId("5625fc29e3c17fdff8517b73")
}
  shards:
        {  "_id" : "shard1",  "host" : "shard1/172.16.16.124:40001,172.16.16.125:40001" }
        {  "_id" : "shard2",  "host" : "shard2/172.16.16.125:40002,172.16.16.126:40002" }
        {  "_id" : "shard3",  "host" : "shard3/172.16.16.126:40003,172.16.16.131:40003" }
        {  "_id" : "shard4",  "host" : "shard4/172.16.16.124:40004,172.16.16.131:40004" }
  balancer:
        Currently enabled:  yes
        Currently running:  yes
                Balancer lock taken at Tue Oct 20 2015 21:01:26 GMT+0800 (CST) by DataServer-04:50001:1445330413:1804289383:Balancer:846930886
        Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours:
                No recent migrations
  databases:
        {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
        {  "_id" : "test",  "partitioned" : false,  "primary" : "shard1" }
        {  "_id" : "ljaidb",  "partitioned" : false,  "primary" : "shard1" }

可以看出ljaidb庫並沒有分片，且數據都在shard1上，登錄其他shard1上查看：

bin]$ ./mongo 172.16.16.124:40001
MongoDB shell version: 3.0.7
connecting to: 172.16.16.124:40001/test
shard1:PRIMARY> show dbs
ljaidb  0.000GB
local   0.000GB
shard1:PRIMARY> use ljaidb
switched to db ljaidb
shard1:PRIMARY> show tables
ljaitable
shard1:PRIMARY> db.ljaitable.find().count()
10000

驗證shard2、shard3、shard4上都沒有ljaidb這個庫：

bin]$ ./mongo 172.16.16.125:40002
MongoDB shell version: 3.0.7
connecting to: 172.16.16.125:40002/test
shard2:PRIMARY> show dbs
local  0.000GB

•  指定數據庫和集合進行分片：

為了讓某個數據庫與集合自動分片生效，對 數據庫(lymdb)及數據庫(lymtable)中的表進行分片配置：

bin]$ ./mongo 172.16.16.124:50001
MongoDB shell version: 3.0.7
connecting to: 172.16.16.124:50001/test
mongos> use admin
switched to db admin
mongos> db.runCommand( { enablesharding :"lymdb"});
{ "ok" : 1 }
mongos> db.runCommand( { shardcollection : "lymdb.lymtable",key : {_id: 1} } )
{ "collectionsharded" : "lymdb.lymtable", "ok" : 1 }
 

• 通過java或者python驅動，連接mongo集群測試：

java連接代碼：

import java.util.ArrayList;
import java.util.List;

import com.mongodb.BasicDBObject;
import com.mongodb.DB;
import com.mongodb.DBCollection;
import com.mongodb.DBObject;
import com.mongodb.MongoClient;
import com.mongodb.ServerAddress;

public class TestMongoDBShards {

    public static void main(String[] args) {
         try {
             List<ServerAddress> addresses = new ArrayList<ServerAddress>();
             ServerAddress address1 = new ServerAddress("172.16.16.124" , 50001);
             ServerAddress address2 = new ServerAddress("172.16.16.125" , 50001);
             ServerAddress address3 = new ServerAddress("172.16.16.126" , 50001);
             ServerAddress address4 = new ServerAddress("172.16.16.131" , 50001);
             addresses.add(address1);
            addresses.add(address2);
            addresses.add(address3);

             MongoClient client = new MongoClient(addresses);
             DB db = client.getDB( "lymdb" );
             DBCollection coll = db.getCollection( "lymtable" );

           //  BasicDBObject object = new BasicDBObject();
          //   object.append( "id" , 1);

          //   DBObject dbObject = coll.findOne(object);
             
             for(int i=1;i<=1000000;i++) {
                 DBObject saveData=new BasicDBObject();  
                 saveData.put("id", i);
                 saveData.put("userName", "baiwan" + i);  
                 saveData.put("age", "26");  
                 saveData.put("gender", "m");  
     
     
                 coll.save(saveData); 
             }
             
 
             

            // System. out .println(dbObject);

       } catch (Exception e) {
             e.printStackTrace();
       }
        // TODO Auto-generated method stub

    }

}

python連接代碼：

#encoding=UTF-8
import datetime

ISOTIMEFORMAT = '%Y-%m-%d %X'

from pymongo import MongoClient
conn = MongoClient("172.16.16.124",50001)
db = conn.funodb
def dateDiffInSeconds(date1,date2):
    timedelta = date2 - date1
    return timedelta.days*24*3600 +timedelta.seconds
db.funotable.drop()
date1 = datetime.datetime.now()
for i  in range(0,1000000): db.funotable.insert({"name":"ljai","age":i,"addr":"fuzhou"})
c = db.funotable.find().count()
print("count is ",c)
date2 = datetime.datetime.now()
print(date1)
print(date2)
print("消耗：",dateDiffInSeconds(date1,date2),"seconds")
conn.close()

• 測試是否自動分片：

 

mongos> db.lymtable.getShardDistribution()

Shard shard1 at shard1/172.16.16.124:40001,172.16.16.125:40001
data : 96.46MiB docs : 1216064 chunks : 4
estimated data per chunk : 24.11MiB
estimated docs per chunk : 304016

Shard shard2 at shard2/172.16.16.125:40002,172.16.16.126:40002
data : 44.9MiB docs : 565289 chunks : 4
estimated data per chunk : 11.22MiB
estimated docs per chunk : 141322

Shard shard3 at shard3/172.16.16.126:40003,172.16.16.131:40003
data : 99.39MiB docs : 1259979 chunks : 4
estimated data per chunk : 24.84MiB
estimated docs per chunk : 314994

Shard shard4 at shard4/172.16.16.124:40004,172.16.16.131:40004
data : 76.46MiB docs : 958668 chunks : 4
estimated data per chunk : 19.11MiB
estimated docs per chunk : 239667

Totals
data : 317.22MiB docs : 4000000 chunks : 16
Shard shard1 contains 30.4% data, 30.4% docs in cluster, avg obj size on shard : 83B
Shard shard2 contains 14.15% data, 14.13% docs in cluster, avg obj size on shard : 83B
Shard shard3 contains 31.33% data, 31.49% docs in cluster, avg obj size on shard : 82B
Shard shard4 contains 24.1% data, 23.96% docs in cluster, avg obj size on shard : 83B

可以看出，插入400萬條數據，都有分布各個shard上，但是不夠均勻，需要進一步研究分片的配置。